Category Archives: Linux

Git: stop tracking a tracked file

There are times when a project needs to include the default version of a file in the git repository that will subsequently change to support development (e.g., secret application key or sqlite3 database).

After committing the safe  default copy, any subsequent changes to the files will appear as a modified file in ‘git status’ reports and will also prevent any git flow feature finish operations if the files are not staged for commit.

Adding the files to .gitignore makes no difference.

There is a way, however, to tell git to stop tracking the file,

$ git rm --cached file1 file2

Taken from a StackOverflow posting.

Advertisements

Arch Linuxdeveloper signature errors

Every now and then when applying updates to my Antergos (Arch) Linux desktop I get an error like

$ sudo pacman -Syyu
...
error: libvirt: signature from "Christian Rebischke (Arch Linux Security Team-Member) <Chris.Rebischke@archlinux.org>" is unknown trust
:: File /var/cache/pacman/pkg/libvirt-4.6.0-3-x86_64.pkg.tar.xz is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n]

Accepting the default option means the update fails. If using the GUI client there’s just a message saying the update failed with no further explanation

It is fixed by

$ sudo pacman -S archlinux-keyring

This kind of error is one reason why I’d never consider using Arch in a work environment.

Reference: https://www.reddit.com/r/archlinux/comments/900cxa/upgrading/

 

Limited time for Django?

Having decided that Rails development had run its course and that the future was all things Python and Django, it looks like I’ll be reining back on the web development front and sticking to simple stuff with Flask and uwsgi.

I started rewriting an application with Django following the standard tutorials and make some progress, even getting as far as a form that can subnit new records.

But, staying true to my original goal of ensuring that testing is a fundamental part of the project, I tried the simplest possible scenario,

from django.utils import timezone
from .models import Commission, Enquiry

# Create your tests here.
class CommissionMethodTests(TestCase):

    def basic_test(self):
        a = 1
        assert a == 1

Nothing could be simpler,

$ python manage.py test commissions
Creating test database for alias 'default'...
System check identified no issues (0 silenced).

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
Destroying test database for alias 'default'...

When something this simple doesn’t work and gives no error or any kind of output, there’s not a great deal of point trying to continue.

I did find another Django tutorial that suggested using ‘self.asert’, but that just give an error; I’m not a fan of failing at the first fence.

So, while I’m still like writing Python and will continue with Flask and keeping things simple, it’s time to abandon Django and look to see if WordPress plugins are up to the task.

Pythonic balancer control

In my day job I work with a lot of services that sit behind a load balancer providing high availability across multiple backend hosts.

Now, inevitably there are times when these services or their hosting servers require maintenance. And sometimes we want to do some investigation and troubleshooting against a running service, but without it taking any live traffic; it simply isn’t practical to try and involve the network team’s assistance with disabling interfaces gracefully. The usual approach is to consider using a server-side firewall to block inbound port access but this can be clumsy and can actually impact live traffic, albeit briefly.

One solution I like is to use an intermediate monitor (or watchdog) service that provides a healthcheck URL for the load balancer, say, http://192.168.1.100/my-service/healthcheck, where the returned status is derived from the application ports being monitored.

Now, we want this to be as lightweight as possible, so we can choose something like Python’s Flask and uwsgi (or Ruby Sinatra) to provide a simple service listener like,

#!/usr/bin/env python
#
from flask import Flask, abort, request, Response, redirect
import os
import requests

app = Flask(__name__)

@app.route('/my-service/healthcheck', methods=["GET"])
def heartbeat():
 resp = Response(response = "OK", status = 200, content_type = "text/plain")

# Now check for the node statuses
 nodes = ( "inbound", "outbound", "stats" )
 for node in nodes:
 req = requests.get("http://localhost:7070/" + node + "/isalive")
 if(req.status_code != 200):
 resp.status = "FAILED" 
 resp.status_code = req.status_code

return(resp)

if __name__ == "__main__":
 app.run()

And obviously I have skipped the setup with pip, virtualenv and the like, but that’s routine enough.

The beauty with this kind of approach is that with a few extra lines before the service port polling we can spoof an outage and allow the load balancer to complete any existing client connections (that the firewall approach will prevent) while marking the node out of action,

 # Check for the maintenance file and signal graceful failure
 if(os.path.isfile("maintenance")):
 resp.status = "Under maintenance. Remove maintenance file when complete"
 resp.status_code = 503

Now, by simple touching a file called maintenance in the directory where the application is run from, the next poll from the load balancer will register the failure, and we can test this with cURL,

$ curl -v http://localhost:7070/my-service/healthcheck
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 7070 (#0)
> GET /my-service/healthcheck HTTP/1.1
> Host: localhost:7070
> User-Agent: curl/7.52.1
> Accept: */*
> 
< HTTP/1.1 503 SERVICE UNAVAILABLE
< Content-Type: text/plain
< Content-Length: 2

Remove the file and the traffic will flow again. Remote control of the load balancer without stopping any services, reboot persistent and allowing us time and space to investigate as we please.

I really wanted OSMC to work

With  a couple of new 16GB SD cards for use with my Raspberry Pi collection, I thought I’d give Xbmc another try for a friendly media center over the festive break with some old vinyl I had ripped to disk.

But then I came cross OSMC as a distribution available on the Pi that fixed many of the xbmc shortcomings so I thought I’d give it a go.

One of the most frustrating things regarding Xbmc was the download of a 16Gb image only for there to be most of the space unallocated on the filesystem and no space to install any media files. OSMC certainly is a major improvement: download a small image and flash to the SD card, boot the Pi with and from there complete the installation of the rest of the system over the network.

Alas, on the Pi, it’s downhill from this point.

  • the default confluence skin is too big for the HDMI screen, left and right, top and bottom with text in the tool tips hidden,
  • the default interface is very clunky and hard to navigate with a mouse, frequently losing focus,
  • incoherent menus: trying to remember the difference between system and settings is no end of pain,
  • the OSMC skin is all but unusable: the scrolling text does all it can to avoid being selected under the pointer; the pointer itself at times getting replaced by a vertical bar of mis-render on the screen;
  • I had problems with no audio until a switch to the OSMC skin showed a ‘audio muted’ message onscreen; there was no equivalent on the confluence skin and I have no idea how the audio became muted,
  • the audio was often paused during playback even when nothing else was going on.
  • No obvious way to get music files on to the device: I could define an NFS mount for the library but no means of copying the files down: it’s a Pi so there’s no CD or SD card available.

While, as always, I appreciate the hard work that goes into putting these things together, I’m afraid that this is no way to showcase a Linux media center and I decided to bin it before letting any of my family see it.

Flask over https and local root CA

I recently had a week’s PTO and decided to spend the time getting to grips with setting up a private root CA so that I could try getting a python flask application to use client certificates (signed by the server) for authorization to request upload resources.

It was quite a struggle, more because of the root CA than anything else but since the aim was to gain an understanding of how this kind of set up worked, how to structure the requests and do the work in python (with a ruby version to follow no doubt) I am happy with the progress so far.

I will post a couple of pages describing the work I did with references and how I got past some of the major sticking points.

Fedora 24 LDAP setup for Rails applications

To support the devise authentication in my application, I need to configure a local LDAP directory. The setup details of the Fedora aren’t very good, but I cam across https://www.server-world.info/en/note?os=Fedora_23&p=openldap which worked a treat on my Fedora 24 install.

Used the following files for the build.

# Install using: ldapmodify -Y EXTERNAL -H ldapi:/// -f mydomain.ldif
#
# use slappasswd to generate SSHA passwords
#
dn: olcDatabase={1}monitor,cn=config
changetype: modify
replace: olcAccess
olcAccess: {0}to * by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth"
 read by dn.base="cn=Manager,dc=my-domain,dc=com" read by * none

dn: olcDatabase={2}mdb,cn=config
changetype: modify
replace: olcSuffix
olcSuffix: dc=my-domain,dc=com

dn: olcDatabase={2}mdb,cn=config
changetype: modify
replace: olcRootDN
olcRootDN: cn=Manager,dc=my-domain,dc=com

dn: olcDatabase={2}mdb,cn=config
changetype: modify
replace: olcRootPW
olcRootPW: {SSHA}lFyoikpFOrg....kIZ4lo85qK

dn: olcDatabase={2}mdb,cn=config
changetype: modify
add: olcAccess
olcAccess: {0}to attrs=userPassword,shadowLastChange by
 dn="cn=Manager,dc=my-domain,dc=com" write by anonymous auth by self write by * none
olcAccess: {1}to dn.base="dc=my-domain,dc=com" by * read
olcAccess: {2}to * by dn="cn=Manager,dc=my-domain,dc=com" write by * read

And,

# Install using: ldapadd -x -D cn=Manager,dc=my-domain,dc=com -W -f domain.ldif
dn: dc=my-domain,dc=com
objectClass: top
objectClass: dcObject
objectclass: organization
o: Server World
dc: my-domain

dn: cn=Manager,dc=my-domain,dc=com
objectClass: organizationalRole
cn: Manager
description: Directory Manager

dn: ou=People,dc=my-domain,dc=com
objectClass: organizationalUnit
ou: People

dn: ou=Group,dc=my-domain,dc=com
objectClass: organizationalUnit
ou: Group

An LDAP browser can then bind to the service using ‘cn=Manager’ and create users and other objects.

A Rails application can use the directory for authentication with the following in config/ldap.yml (assuming use of the devise_ldap_authenticatable gem),

development:
  # <<: *AUTHORIZATIONS
  host: 127.0.0.1
  port: 389
  attribute: cn
  base: dc=my-domain,dc=com
  admin_user: cn=Manager,dc=my-domain,dc=com
  admin_password: the_password
  ssl: false

Not really expecting this to be useful to anyone else, but it should be useful the next time I have to rebuld the laptop environment.