Category Archives: Uncategorized

World Cup Preview- France

Jumpers for wickets

In terms of squad depth and world class players, there are very few teams competing at this years World Cup that can match up with France. The last 10 years have been crucial for French football in bringing through and developing their youngsters into top class talent, after some disappointing tournament results. They come into this tournament having finished runner’s up in Euro 2016 in a final that everybody expected them to win.

In goal, France have their captain and most experienced player in the squad Hugo Lloris, who is coming off a decent season with Tottenham. In years gone by, Lloris has been seen as one of the most reliable goalkeepers in the world, but this season, he has shown that he is well capable of mistakes and has started to look slightly error prone. He is still a world class goalkeeper, but France will need their captain to…

View original post 1,077 more words

Advertisements

Twitterbotterglory

I’m too lazy and disinterested to do a lot of tweeting but since my job is about automating processes, I’m minded to improve my “social media” presence programmatically with a simple bot to do my posting for me.

Ensure list only consists of strings

A nice little to remember for future use is when a list must consist of only strings:

def arrayOfStrings(l):
    nl = [s if isinstance(s, str) else str(s) for s in l]
    return(nl)

It looks like it could be quite Pythonic and isn’t necessarily how I would have written it if left to my own devices; I might be braver and try that sort of structure in other situations.

Alas, I don’t have a reference for where I grabbed this from but it’s more than likely from StackOverflow.

Dynamic (in-memory) zip file creation

With a recent project I wanted to offer a download of a bunch of files as a ZIP download to make redistribution. I started by thinking that this would involve creating a zip file on disk and then serving it to the client.

It doesn’t.

This method creates an in-memory zip image of a temporary directory in which the required file are created. The directory is deleted before the zip file is returned to the client. Note the use of send_file to return the content and how it allows us to specify the name of the downloaded file.

If running the app under uwsgi there are some precautions to take:

  • import the send_file module from flask (the default wsgi version is broken)
  • Include ‘wsgi-disable-file-wrapper’ in the uwsgi settings
import os
import io
import tempfile
import zipfile
from peewee import *

# A simple function to write a file in a given directory
def makeFile(dir, filename, data):
    fname = str(dir) + '/' + str(filename)
    fh = open(fname, "w")
    fh.write(data)
    fh.close()
    return True
"""
Function to create a zip file containing the application configuration files
To create the zip file, we need to create a memory file (BytesIO))
Params:
 row: Database row object containing columns for filename and
 content to be written in the form
 { {"file1": "Content for file 1}, {"file2": "Different content for file"} }
Returns:
 data: BytesIO object containing the zipped files
"""
def mkZipFile(row):
    zipdir = tempfile.mkdtemp(prefix='/tmp/')
    oldpath = os.getcwd()
    os.chdir(zipdir)

    jdata = json.loads(row)
    for conf in jdata:
        for f in conf:
            makeFile(zipdir, f, conf[f])

    # Create the in-memory zip image
    data = io.BytesIO()
    with zipfile.ZipFile(data, mode='w') as z:
        for fname in os.listdir("."):
            z.write(fname)
            os.unlink(fname)
    data.seek(0)

    os.chdir(oldpath)
    os.rmdir(zipdir)
    return data

The route takes the form

"""
Route to create a zip file containing all the required data files
Params:
 id: the numeric id of the record in the database
Returns:
 ZIP: file in ZIP format containing the download files
"""
@app.route('/download/<int:id>.zip', methods=['GET'])
def showZip(id):
    rs = db_table.select(db_table.data_column).where(db_table.id == id).get()

    zfile = mkZipFile(rs)
    if not zfile:
        return Response(response="Invalid zip file", status="400")

    return send_file(
        zfile,
        mimetype='application/zip',
        as_attachment=True,
        attachment_filename='file_bundle.zip')

References

Form input validation

A technique for handling form input validation for running an arbitrary number of tests against the submitted data.

Probably doesn’t work with multiple select options

A database table to store all the possible input fields

CREATE TABLE form_items (
 id INT PRIMARY KEY NOT NULL,
 name CHAR(64) NOT NULL,
 type CHAR(6) NOT NULL DEFAULT 'string',
 updated_by CHAR(32),
 updated_at DATETIME
);

There might be additional columns to identify the page or section that an input value is found on.

Then we have a table to identify the validation functions available

CREATE TABLE column_validations (
 id INT PRIMARY KEY NOT NULL,
 name CHAR(32) UNIQUE NOT NULL,
 updated_by CHAR(32),
 updated_at DATETIME
);

There might be other columns to identify the severity and whether to continue or not. in the event of failure

Then we have a many-to-many map of validators to columns

CREATE TABLE column_validation_form_items(
 column_validation_id INT REFERENCES column_validations,
 form_item_id INT REFERENCES form_items
);

A PeeWee model might look like

class BaseModel(Model):
  with app.app_context():
    class Meta:
    database = SqliteDatabase(app.config['DATABASE'])

class form_items(BaseModel):
  id = IntegerField(primary_key=True)
  name = CharField(unique=False)
  type = CharField(default='string')
  updated_by = CharField()
  updated_at = DateTimeField(default=datetime.datetime.now())

class column_validations(BaseModel):
  id = IntegerField(primary_key=True)
  name = CharField(unique=True)
  updated_by = CharField()
  updated_at = DateTimeField(default=datetime.datetime.now())

class column_validation_config_columns(BaseModel):
  column_validation = ForeignKeyField(column_validations)
  form_item = ForeignKeyField(form_items)
  class Meta:
    primary_key = CompositeKey('column_validation', 'form_item')

Sample function to run through the form value validations:

# Take the name of the form field and the submitted value 
 def cleanInput(name, value):
  # Array to map the validator names from the database to the function
  # Not sure if it's possible in Python to do the equivalent of a callback
  # without this type of referencing

  validators = {
    "valid_ip_address": valid_ip_address,
    "valid_ip_range":valid_ip_range,
    "valid_url": valid_url,
    "true_or_false": true_or_false,
    "yes_or_no": yes_or_no,
    "is_integer": is_integer}

  rs = (column_validations.select(column_validations.name)
    .join(column_validation_form_items, JOIN.INNER)
    .join(form_items, JOIN.INNER)
    .where(form_items.name == name))
  # Provide a failsafe for when no validator is found
  # Try each of the validators when the input value in non-blank
  if len(value) > 0:
    status = True
    msg = ""
    for proc in rs:
      func = proc.name
      if func in validators:
        (status, msg) = validators[func](value)
        if not status:
          return (False, "Error: %s validation failure: %s" % (name, msg))
  else: # Empty value, just return True
    return(True, value)
  return (True, msg)

Some sample validation functions. Note that these return a tuple to indicate the status along with an error message. If the status is false, dsiplay the error message and redisplay the form; otherwise continue.

"""
Simple function to validate an IPv4 address
This needs to import socket but note how it avoids regex patterns
Params:
 addr: string containing IP address
Returns:
 (True|False, addr|errmsg)
"""
def valid_ip_address(addr):
  try:
    socket.inet_pton(socket.AF_INET, addr)
    return (True, addr)
  except socket.error:
    return (False, ("Invalid IP address, %s." % addr))

"""
Simple function to validate an IPv4 address range in CIDR format
Pattern match taken from http://blog.markhatton.co.uk/2011/03/15/regular-expressions-for-ip-addresses-cidr-ranges-and-hostnames/
Params:
 ip_range: string containing IP address range
Returns:
 (True|False, ip_range|errmsg)
"""
def valid_ip_range(ip_range):
  patt = "^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/([0-9]|[1-2][0-9]|3[0-2]))$"
  if re.search(patt, ip_range):
    return (True, ip_range)
  return(False, ("Invalid IPv4 range: %s" % ip_range))

"""
Function to validate a submitted website address
Params:
 addr: String containing a URL
Returns:
 (True|False, url|errmsg)
"""
def valid_url(url):
  if len(url) > 2083:
    return(False, ("Invalid url, %s: too long." % url))
  patt = "(http|https):\/\/(\w+:{0,1}\w*@)?(\S+)(:[0-9]+)?(\/|\/([\w#!:.?+=&%@!\-\/]))?"
  if re.search(patt, url, re.IGNORECASE):
    return (True, url)
  return (False, ("Invalid url, %s." % url))

def true_or_false(value):
  for patt in ('true', 'false', '1', '0'):
    if re.search(patt, value, re.IGNORECASE):
      return (True, value)
  return(False, ("%s must be either true or false, 0 or 1" % value))

def yes_or_no(value):
  for patt in ('yes', 'no', '1', '0'):
    if re.search(patt, value, re.IGNORECASE):
      return (True, value)
  return(False, ("%s must be either true or false, 0 or 1" % value))

"""
Simple function to return whether a value is an Integer
Params:
 value: String representation of an integer
Returns:
 (True|False, value|errmsg)
"""
def is_integer(value):
  if isinstance(int(value), int):
    return (True, value)
  return (False, ("%s is not an integer" % value))

References

HTTP-based DB backup

If working with  Python Flask application in a closed environment it can often be handy to be able to grab a copy of the database behind the application.

Here are a few simple functions that can be used to provide a URL which will yield a (very simple) SQL export of the database schema and table contents.

"""
Simple route that captures a request to backup the database. It will
create a file containing
* the SQL commands to delete existing tables
* SQL schema commands for the database
* SQL insert operations for all the records in all the tables
params:
 None
Returns:
 File: contains database schema and rows from all the tables
"""
@app.route('/myapp/db_backup', methods=['GET'])
def myAppDbBackup():
    db_bkp = ""
    for tbl in database.get_tables():
        db_bkp = db_bkp + backupDbTable(tbl)
        if db_bkp == "":
            return Response(status="409 Empty backup")
     h = Headers()
     h.add('Content-Type', 'text/plain')
     h.add('Content-Disposition', 'attachment', filename='my_lovely_db_backup.sql')
     return Response(db_bkp, mimetype='application/x-sql', headers=h)

"""
Table backup function
Params:
 table: The name of the table to db_backup
Returns:
 String: All the table records as insert statements
"""
def backupDbTable(table):
    bkpstr = ""
    values = ""
    inscol = ""
    for col in database.get_columns(table):
        inscol = inscol + (col.name if not inscol else (", %s" % col.name))
        insert = ("INSERT INTO %s (%s) VALUES" % (table, inscol))

    bkpstr = ("%s;\n" % getTableSchema(table))
    valstr = ""
    qry = ("SELECT %s FROM %s" % (inscol, table))
    rs = database.execute_sql(qry)
    for row in rs:
        valstr = "', '".join(flattenArray(row))
        values = values + ("('" + valstr + "')" if not values else ",\n('" + valstr + "')")
    bkpstr = bkpstr + insert + values + ";\n"
    return bkpstr

"""
Function to retrieve the statement to create a table in the database
Params:
 table: The name of the table being backed up
Returns:
 String: The CREATE TABLE statement for table
"""
def getTableSchema(table):
    schema_qry = ("SELECT sql FROM sqlite_master WHERE type='table' AND name = ?")
    rs = database.execute_sql(schema_qry, [table])
    for schema in rs:
        return schema
def flattenArray(ary):
    flat = []
    for el in ary:
    if isinstance(el, list):
        flat += flattenArray(el)
    else:
        flat.append(el)
    return(flat)

It is always worth remembering that this is an intrinsically insecure operation but a small measure of protection could come from putting the app behind an Apache httpd or nginx reverse proxy with an SSL cert and user authentication. But best to limit to a closed environment.