Category Archives: Linux

Limited time for Django?

Having decided that Rails development had run its course and that the future was all things Python and Django, it looks like I’ll be reining back on the web development front and sticking to simple stuff with Flask and uwsgi.

I started rewriting an application with Django following the standard tutorials and make some progress, even getting as far as a form that can subnit new records.

But, staying true to my original goal of ensuring that testing is a fundamental part of the project, I tried the simplest possible scenario,

from django.utils import timezone
from .models import Commission, Enquiry

# Create your tests here.
class CommissionMethodTests(TestCase):

    def basic_test(self):
        a = 1
        assert a == 1

Nothing could be simpler,

$ python manage.py test commissions
Creating test database for alias 'default'...
System check identified no issues (0 silenced).

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
Destroying test database for alias 'default'...

When something this simple doesn’t work and gives no error or any kind of output, there’s not a great deal of point trying to continue.

I did find another Django tutorial that suggested using ‘self.asert’, but that just give an error; I’m not a fan of failing at the first fence.

So, while I’m still like writing Python and will continue with Flask and keeping things simple, it’s time to abandon Django and look to see if WordPress plugins are up to the task.

Pythonic balancer control

In my day job I work with a lot of services that sit behind a load balancer providing high availability across multiple backend hosts.

Now, inevitably there are times when these services or their hosting servers require maintenance. And sometimes we want to do some investigation and troubleshooting against a running service, but without it taking any live traffic; it simply isn’t practical to try and involve the network team’s assistance with disabling interfaces gracefully. The usual approach is to consider using a server-side firewall to block inbound port access but this can be clumsy and can actually impact live traffic, albeit briefly.

One solution I like is to use an intermediate monitor (or watchdog) service that provides a healthcheck URL for the load balancer, say, http://192.168.1.100/my-service/healthcheck, where the returned status is derived from the application ports being monitored.

Now, we want this to be as lightweight as possible, so we can choose something like Python’s Flask and uwsgi (or Ruby Sinatra) to provide a simple service listener like,

#!/usr/bin/env python
#
from flask import Flask, abort, request, Response, redirect
import os
import requests

app = Flask(__name__)

@app.route('/my-service/healthcheck', methods=["GET"])
def heartbeat():
 resp = Response(response = "OK", status = 200, content_type = "text/plain")

# Now check for the node statuses
 nodes = ( "inbound", "outbound", "stats" )
 for node in nodes:
 req = requests.get("http://localhost:7070/" + node + "/isalive")
 if(req.status_code != 200):
 resp.status = "FAILED" 
 resp.status_code = req.status_code

return(resp)

if __name__ == "__main__":
 app.run()

And obviously I have skipped the setup with pip, virtualenv and the like, but that’s routine enough.

The beauty with this kind of approach is that with a few extra lines before the service port polling we can spoof an outage and allow the load balancer to complete any existing client connections (that the firewall approach will prevent) while marking the node out of action,

 # Check for the maintenance file and signal graceful failure
 if(os.path.isfile("maintenance")):
 resp.status = "Under maintenance. Remove maintenance file when complete"
 resp.status_code = 503

Now, by simple touching a file called maintenance in the directory where the application is run from, the next poll from the load balancer will register the failure, and we can test this with cURL,

$ curl -v http://localhost:7070/my-service/healthcheck
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 7070 (#0)
> GET /my-service/healthcheck HTTP/1.1
> Host: localhost:7070
> User-Agent: curl/7.52.1
> Accept: */*
> 
< HTTP/1.1 503 SERVICE UNAVAILABLE
< Content-Type: text/plain
< Content-Length: 2

Remove the file and the traffic will flow again. Remote control of the load balancer without stopping any services, reboot persistent and allowing us time and space to investigate as we please.

I really wanted OSMC to work

With  a couple of new 16GB SD cards for use with my Raspberry Pi collection, I thought I’d give Xbmc another try for a friendly media center over the festive break with some old vinyl I had ripped to disk.

But then I came cross OSMC as a distribution available on the Pi that fixed many of the xbmc shortcomings so I thought I’d give it a go.

One of the most frustrating things regarding Xbmc was the download of a 16Gb image only for there to be most of the space unallocated on the filesystem and no space to install any media files. OSMC certainly is a major improvement: download a small image and flash to the SD card, boot the Pi with and from there complete the installation of the rest of the system over the network.

Alas, on the Pi, it’s downhill from this point.

  • the default confluence skin is too big for the HDMI screen, left and right, top and bottom with text in the tool tips hidden,
  • the default interface is very clunky and hard to navigate with a mouse, frequently losing focus,
  • incoherent menus: trying to remember the difference between system and settings is no end of pain,
  • the OSMC skin is all but unusable: the scrolling text does all it can to avoid being selected under the pointer; the pointer itself at times getting replaced by a vertical bar of mis-render on the screen;
  • I had problems with no audio until a switch to the OSMC skin showed a ‘audio muted’ message onscreen; there was no equivalent on the confluence skin and I have no idea how the audio became muted,
  • the audio was often paused during playback even when nothing else was going on.
  • No obvious way to get music files on to the device: I could define an NFS mount for the library but no means of copying the files down: it’s a Pi so there’s no CD or SD card available.

While, as always, I appreciate the hard work that goes into putting these things together, I’m afraid that this is no way to showcase a Linux media center and I decided to bin it before letting any of my family see it.

Flask over https and local root CA

I recently had a week’s PTO and decided to spend the time getting to grips with setting up a private root CA so that I could try getting a python flask application to use client certificates (signed by the server) for authorization to request upload resources.

It was quite a struggle, more because of the root CA than anything else but since the aim was to gain an understanding of how this kind of set up worked, how to structure the requests and do the work in python (with a ruby version to follow no doubt) I am happy with the progress so far.

I will post a couple of pages describing the work I did with references and how I got past some of the major sticking points.

Fedora 24 LDAP setup for Rails applications

To support the devise authentication in my application, I need to configure a local LDAP directory. The setup details of the Fedora aren’t very good, but I cam across https://www.server-world.info/en/note?os=Fedora_23&p=openldap which worked a treat on my Fedora 24 install.

Used the following files for the build.

# Install using: ldapmodify -Y EXTERNAL -H ldapi:/// -f mydomain.ldif
#
# use slappasswd to generate SSHA passwords
#
dn: olcDatabase={1}monitor,cn=config
changetype: modify
replace: olcAccess
olcAccess: {0}to * by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth"
 read by dn.base="cn=Manager,dc=my-domain,dc=com" read by * none

dn: olcDatabase={2}mdb,cn=config
changetype: modify
replace: olcSuffix
olcSuffix: dc=my-domain,dc=com

dn: olcDatabase={2}mdb,cn=config
changetype: modify
replace: olcRootDN
olcRootDN: cn=Manager,dc=my-domain,dc=com

dn: olcDatabase={2}mdb,cn=config
changetype: modify
replace: olcRootPW
olcRootPW: {SSHA}lFyoikpFOrg....kIZ4lo85qK

dn: olcDatabase={2}mdb,cn=config
changetype: modify
add: olcAccess
olcAccess: {0}to attrs=userPassword,shadowLastChange by
 dn="cn=Manager,dc=my-domain,dc=com" write by anonymous auth by self write by * none
olcAccess: {1}to dn.base="dc=my-domain,dc=com" by * read
olcAccess: {2}to * by dn="cn=Manager,dc=my-domain,dc=com" write by * read

And,

# Install using: ldapadd -x -D cn=Manager,dc=my-domain,dc=com -W -f domain.ldif
dn: dc=my-domain,dc=com
objectClass: top
objectClass: dcObject
objectclass: organization
o: Server World
dc: my-domain

dn: cn=Manager,dc=my-domain,dc=com
objectClass: organizationalRole
cn: Manager
description: Directory Manager

dn: ou=People,dc=my-domain,dc=com
objectClass: organizationalUnit
ou: People

dn: ou=Group,dc=my-domain,dc=com
objectClass: organizationalUnit
ou: Group

An LDAP browser can then bind to the service using ‘cn=Manager’ and create users and other objects.

A Rails application can use the directory for authentication with the following in config/ldap.yml (assuming use of the devise_ldap_authenticatable gem),

development:
  # <<: *AUTHORIZATIONS
  host: 127.0.0.1
  port: 389
  attribute: cn
  base: dc=my-domain,dc=com
  admin_user: cn=Manager,dc=my-domain,dc=com
  admin_password: the_password
  ssl: false

Not really expecting this to be useful to anyone else, but it should be useful the next time I have to rebuld the laptop environment.

Fedora 23: gem install mysql2

I’m having another blast at preparing PDF documents from a database repository of infrastructure assets which requires the use of the mysql2 gem on Fedora 23.

Now, I have had many years experience building this stuff from source and even though Ruby has a reputation for being difficult to work with, this time it’s definitely Fedora  that’s bearing unberable.

Install the mysql2 gen should be a simple matter of,

gem install mysql2

But not when you get this error:

checking for ruby/thread.h... *** extconf.rb failed ***

With extra advice about probably missing developer tools or libraries. I have the compiler, mariadb-devel and ruby-devel packages installed, everything that’s required to build the gem, but still no good.

I eventually found the mkmf.log record mentioned in the error output which contained something I’d not seen before:

error: /usr/lib/rpm/redhat/redhat-hardened-cc1: No such file or directory

Searching online for this came across http://stackoverflow.com/questions/34624428/g-error-usr-lib-rpm-redhat-redhat-hardened-cc1-no-that-file-and-directory and the simple solution is to run,

dnf install redhat-rpm-config
gem install mysql2 -v '0.3.16'

And we’re done.

Whod’ve thought that an rpm-config package would be a pre-requisite for installing ruby gems? And yet another example of having to spend an hour fixing  numerous tedious problems and sub-problems introduced by system developers rather than being able to get on with the task in hand.

Initial upload of local code to repo on github

Another quick note to self ‘cos this is something that I can never remember.

After creating a local repo with the project being developed, we need to create the repo at https://github.com/slugbucket and then run the following commands on teh local workstation.

$ git push --set-upstream https://github.com/slugbucket/automation-demo-packages.git master
$ git remote add origin https://github.com/slugbucket/automation-demo-packages.git
$ git push -u origin master

Enter the github username and password when prompted.

If the repository on github was created with an initial README.md file, it will first be necessary to do a merge with the following command,

$ git pull https://github.com/slugbucket/automation-demo-packages.git

Then the push commands above will wok fine.