Author Archives: julianrawcliffe

Arch Linuxdeveloper signature errors

Every now and then when applying updates to my Antergos (Arch) Linux desktop I get an error like

$ sudo pacman -Syyu
error: libvirt: signature from "Christian Rebischke (Arch Linux Security Team-Member) <>" is unknown trust
:: File /var/cache/pacman/pkg/libvirt-4.6.0-3-x86_64.pkg.tar.xz is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n]

Accepting the default option means the update fails. If using the GUI client there’s just a message saying the update failed with no further explanation

It is fixed by

$ sudo pacman -S archlinux-keyring

This kind of error is one reason why I’d never consider using Arch in a work environment.




World Cup Preview- France

Jumpers for wickets

In terms of squad depth and world class players, there are very few teams competing at this years World Cup that can match up with France. The last 10 years have been crucial for French football in bringing through and developing their youngsters into top class talent, after some disappointing tournament results. They come into this tournament having finished runner’s up in Euro 2016 in a final that everybody expected them to win.

In goal, France have their captain and most experienced player in the squad Hugo Lloris, who is coming off a decent season with Tottenham. In years gone by, Lloris has been seen as one of the most reliable goalkeepers in the world, but this season, he has shown that he is well capable of mistakes and has started to look slightly error prone. He is still a world class goalkeeper, but France will need their captain to…

View original post 1,077 more words

Ensure list only consists of strings

A nice little to remember for future use is when a list must consist of only strings:

def arrayOfStrings(l):
    nl = [s if isinstance(s, str) else str(s) for s in l]

It looks like it could be quite Pythonic and isn’t necessarily how I would have written it if left to my own devices; I might be braver and try that sort of structure in other situations.

Alas, I don’t have a reference for where I grabbed this from but it’s more than likely from StackOverflow.

Dynamic (in-memory) zip file creation

With a recent project I wanted to offer a download of a bunch of files as a ZIP download to make redistribution. I started by thinking that this would involve creating a zip file on disk and then serving it to the client.

It doesn’t.

This method creates an in-memory zip image of a temporary directory in which the required file are created. The directory is deleted before the zip file is returned to the client. Note the use of send_file to return the content and how it allows us to specify the name of the downloaded file.

If running the app under uwsgi there are some precautions to take:

  • import the send_file module from flask (the default wsgi version is broken)
  • Include ‘wsgi-disable-file-wrapper’ in the uwsgi settings
import os
import io
import tempfile
import zipfile
from peewee import *

# A simple function to write a file in a given directory
def makeFile(dir, filename, data):
    fname = str(dir) + '/' + str(filename)
    fh = open(fname, "w")
    return True
Function to create a zip file containing the application configuration files
To create the zip file, we need to create a memory file (BytesIO))
 row: Database row object containing columns for filename and
 content to be written in the form
 { {"file1": "Content for file 1}, {"file2": "Different content for file"} }
 data: BytesIO object containing the zipped files
def mkZipFile(row):
    zipdir = tempfile.mkdtemp(prefix='/tmp/')
    oldpath = os.getcwd()

    jdata = json.loads(row)
    for conf in jdata:
        for f in conf:
            makeFile(zipdir, f, conf[f])

    # Create the in-memory zip image
    data = io.BytesIO()
    with zipfile.ZipFile(data, mode='w') as z:
        for fname in os.listdir("."):

    return data

The route takes the form

Route to create a zip file containing all the required data files
 id: the numeric id of the record in the database
 ZIP: file in ZIP format containing the download files
@app.route('/download/<int:id>.zip', methods=['GET'])
def showZip(id):
    rs = == id).get()

    zfile = mkZipFile(rs)
    if not zfile:
        return Response(response="Invalid zip file", status="400")

    return send_file(


Form input validation

A technique for handling form input validation for running an arbitrary number of tests against the submitted data.

Probably doesn’t work with multiple select options

A database table to store all the possible input fields

CREATE TABLE form_items (
 name CHAR(64) NOT NULL,
 type CHAR(6) NOT NULL DEFAULT 'string',
 updated_by CHAR(32),
 updated_at DATETIME

There might be additional columns to identify the page or section that an input value is found on.

Then we have a table to identify the validation functions available

CREATE TABLE column_validations (
 updated_by CHAR(32),
 updated_at DATETIME

There might be other columns to identify the severity and whether to continue or not. in the event of failure

Then we have a many-to-many map of validators to columns

CREATE TABLE column_validation_form_items(
 column_validation_id INT REFERENCES column_validations,
 form_item_id INT REFERENCES form_items

A PeeWee model might look like

class BaseModel(Model):
  with app.app_context():
    class Meta:
    database = SqliteDatabase(app.config['DATABASE'])

class form_items(BaseModel):
  id = IntegerField(primary_key=True)
  name = CharField(unique=False)
  type = CharField(default='string')
  updated_by = CharField()
  updated_at = DateTimeField(

class column_validations(BaseModel):
  id = IntegerField(primary_key=True)
  name = CharField(unique=True)
  updated_by = CharField()
  updated_at = DateTimeField(

class column_validation_config_columns(BaseModel):
  column_validation = ForeignKeyField(column_validations)
  form_item = ForeignKeyField(form_items)
  class Meta:
    primary_key = CompositeKey('column_validation', 'form_item')

Sample function to run through the form value validations:

# Take the name of the form field and the submitted value 
 def cleanInput(name, value):
  # Array to map the validator names from the database to the function
  # Not sure if it's possible in Python to do the equivalent of a callback
  # without this type of referencing

  validators = {
    "valid_ip_address": valid_ip_address,
    "valid_url": valid_url,
    "true_or_false": true_or_false,
    "yes_or_no": yes_or_no,
    "is_integer": is_integer}

  rs = (
    .join(column_validation_form_items, JOIN.INNER)
    .join(form_items, JOIN.INNER)
    .where( == name))
  # Provide a failsafe for when no validator is found
  # Try each of the validators when the input value in non-blank
  if len(value) > 0:
    status = True
    msg = ""
    for proc in rs:
      func =
      if func in validators:
        (status, msg) = validators[func](value)
        if not status:
          return (False, "Error: %s validation failure: %s" % (name, msg))
  else: # Empty value, just return True
    return(True, value)
  return (True, msg)

Some sample validation functions. Note that these return a tuple to indicate the status along with an error message. If the status is false, dsiplay the error message and redisplay the form; otherwise continue.

Simple function to validate an IPv4 address
This needs to import socket but note how it avoids regex patterns
 addr: string containing IP address
 (True|False, addr|errmsg)
def valid_ip_address(addr):
    socket.inet_pton(socket.AF_INET, addr)
    return (True, addr)
  except socket.error:
    return (False, ("Invalid IP address, %s." % addr))

Simple function to validate an IPv4 address range in CIDR format
Pattern match taken from
 ip_range: string containing IP address range
 (True|False, ip_range|errmsg)
def valid_ip_range(ip_range):
  patt = "^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/([0-9]|[1-2][0-9]|3[0-2]))$"
  if, ip_range):
    return (True, ip_range)
  return(False, ("Invalid IPv4 range: %s" % ip_range))

Function to validate a submitted website address
 addr: String containing a URL
 (True|False, url|errmsg)
def valid_url(url):
  if len(url) > 2083:
    return(False, ("Invalid url, %s: too long." % url))
  patt = "(http|https):\/\/(\w+:{0,1}\w*@)?(\S+)(:[0-9]+)?(\/|\/([\w#!:.?+=&%@!\-\/]))?"
  if, url, re.IGNORECASE):
    return (True, url)
  return (False, ("Invalid url, %s." % url))

def true_or_false(value):
  for patt in ('true', 'false', '1', '0'):
    if, value, re.IGNORECASE):
      return (True, value)
  return(False, ("%s must be either true or false, 0 or 1" % value))

def yes_or_no(value):
  for patt in ('yes', 'no', '1', '0'):
    if, value, re.IGNORECASE):
      return (True, value)
  return(False, ("%s must be either true or false, 0 or 1" % value))

Simple function to return whether a value is an Integer
 value: String representation of an integer
 (True|False, value|errmsg)
def is_integer(value):
  if isinstance(int(value), int):
    return (True, value)
  return (False, ("%s is not an integer" % value))