Category Archives: Uncategorized

Twitterbotterglory

I’m too lazy and disinterested to do a lot of tweeting but since my job is about automating processes, I’m minded to improve my “social media” presence programmatically with a simple bot to do my posting for me.

Advertisements

Nice buns!

I’ve been looking for a recipe for Belgian buns and I can across this recipe

Now, I prefer to use bread flour and dried yeast and take a bit more time with the rising, and I’m not a big fan of the icing so these are just rather nice fruit buns but that’s all right by me.

Ensure list only consists of strings

A nice little to remember for future use is when a list must consist of only strings:

def arrayOfStrings(l):
    nl = [s if isinstance(s, str) else str(s) for s in l]
    return(nl)

It looks like it could be quite Pythonic and isn’t necessarily how I would have written it if left to my own devices; I might be braver and try that sort of structure in other situations.

Alas, I don’t have a reference for where I grabbed this from but it’s more than likely from StackOverflow.

Dynamic (in-memory) zip file creation

With a recent project I wanted to offer a download of a bunch of files as a ZIP download to make redistribution. I started by thinking that this would involve creating a zip file on disk and then serving it to the client.

It doesn’t.

This method creates an in-memory zip image of a temporary directory in which the required file are created. The directory is deleted before the zip file is returned to the client. Note the use of send_file to return the content and how it allows us to specify the name of the downloaded file.

If running the app under uwsgi there are some precautions to take:

  • import the send_file module from flask (the default wsgi version is broken)
  • Include ‘wsgi-disable-file-wrapper’ in the uwsgi settings
import os
import io
import tempfile
import zipfile
from peewee import *

# A simple function to write a file in a given directory
def makeFile(dir, filename, data):
    fname = str(dir) + '/' + str(filename)
    fh = open(fname, "w")
    fh.write(data)
    fh.close()
    return True
"""
Function to create a zip file containing the application configuration files
To create the zip file, we need to create a memory file (BytesIO))
Params:
 row: Database row object containing columns for filename and
 content to be written in the form
 { {"file1": "Content for file 1}, {"file2": "Different content for file"} }
Returns:
 data: BytesIO object containing the zipped files
"""
def mkZipFile(row):
    zipdir = tempfile.mkdtemp(prefix='/tmp/')
    oldpath = os.getcwd()
    os.chdir(zipdir)

    jdata = json.loads(row)
    for conf in jdata:
        for f in conf:
            makeFile(zipdir, f, conf[f])

    # Create the in-memory zip image
    data = io.BytesIO()
    with zipfile.ZipFile(data, mode='w') as z:
        for fname in os.listdir("."):
            z.write(fname)
            os.unlink(fname)
    data.seek(0)

    os.chdir(oldpath)
    os.rmdir(zipdir)
    return data

The route takes the form

"""
Route to create a zip file containing all the required data files
Params:
 id: the numeric id of the record in the database
Returns:
 ZIP: file in ZIP format containing the download files
"""
@app.route('/download/<int:id>.zip', methods=['GET'])
def showZip(id):
    rs = db_table.select(db_table.data_column).where(db_table.id == id).get()

    zfile = mkZipFile(rs)
    if not zfile:
        return Response(response="Invalid zip file", status="400")

    return send_file(
        zfile,
        mimetype='application/zip',
        as_attachment=True,
        attachment_filename='file_bundle.zip')

References

Form input validation

A technique for handling form input validation for running an arbitrary number of tests against the submitted data.

Probably doesn’t work with multiple select options

A database table to store all the possible input fields

CREATE TABLE form_items (
 id INT PRIMARY KEY NOT NULL,
 name CHAR(64) NOT NULL,
 type CHAR(6) NOT NULL DEFAULT 'string',
 updated_by CHAR(32),
 updated_at DATETIME
);

There might be additional columns to identify the page or section that an input value is found on.

Then we have a table to identify the validation functions available

CREATE TABLE column_validations (
 id INT PRIMARY KEY NOT NULL,
 name CHAR(32) UNIQUE NOT NULL,
 updated_by CHAR(32),
 updated_at DATETIME
);

There might be other columns to identify the severity and whether to continue or not. in the event of failure

Then we have a many-to-many map of validators to columns

CREATE TABLE column_validation_form_items(
 column_validation_id INT REFERENCES column_validations,
 form_item_id INT REFERENCES form_items
);

A PeeWee model might look like

class BaseModel(Model):
  with app.app_context():
    class Meta:
    database = SqliteDatabase(app.config['DATABASE'])

class form_items(BaseModel):
  id = IntegerField(primary_key=True)
  name = CharField(unique=False)
  type = CharField(default='string')
  updated_by = CharField()
  updated_at = DateTimeField(default=datetime.datetime.now())

class column_validations(BaseModel):
  id = IntegerField(primary_key=True)
  name = CharField(unique=True)
  updated_by = CharField()
  updated_at = DateTimeField(default=datetime.datetime.now())

class column_validation_config_columns(BaseModel):
  column_validation = ForeignKeyField(column_validations)
  form_item = ForeignKeyField(form_items)
  class Meta:
    primary_key = CompositeKey('column_validation', 'form_item')

Sample function to run through the form value validations:

# Take the name of the form field and the submitted value 
 def cleanInput(name, value):
  # Array to map the validator names from the database to the function
  # Not sure if it's possible in Python to do the equivalent of a callback
  # without this type of referencing

  validators = {
    "valid_ip_address": valid_ip_address,
    "valid_ip_range":valid_ip_range,
    "valid_url": valid_url,
    "true_or_false": true_or_false,
    "yes_or_no": yes_or_no,
    "is_integer": is_integer}

  rs = (column_validations.select(column_validations.name)
    .join(column_validation_form_items, JOIN.INNER)
    .join(form_items, JOIN.INNER)
    .where(form_items.name == name))
  # Provide a failsafe for when no validator is found
  # Try each of the validators when the input value in non-blank
  if len(value) > 0:
    status = True
    msg = ""
    for proc in rs:
      func = proc.name
      if func in validators:
        (status, msg) = validators[func](value)
        if not status:
          return (False, "Error: %s validation failure: %s" % (name, msg))
  else: # Empty value, just return True
    return(True, value)
  return (True, msg)

Some sample validation functions. Note that these return a tuple to indicate the status along with an error message. If the status is false, dsiplay the error message and redisplay the form; otherwise continue.

"""
Simple function to validate an IPv4 address
This needs to import socket but note how it avoids regex patterns
Params:
 addr: string containing IP address
Returns:
 (True|False, addr|errmsg)
"""
def valid_ip_address(addr):
  try:
    socket.inet_pton(socket.AF_INET, addr)
    return (True, addr)
  except socket.error:
    return (False, ("Invalid IP address, %s." % addr))

"""
Simple function to validate an IPv4 address range in CIDR format
Pattern match taken from http://blog.markhatton.co.uk/2011/03/15/regular-expressions-for-ip-addresses-cidr-ranges-and-hostnames/
Params:
 ip_range: string containing IP address range
Returns:
 (True|False, ip_range|errmsg)
"""
def valid_ip_range(ip_range):
  patt = "^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/([0-9]|[1-2][0-9]|3[0-2]))$"
  if re.search(patt, ip_range):
    return (True, ip_range)
  return(False, ("Invalid IPv4 range: %s" % ip_range))

"""
Function to validate a submitted website address
Params:
 addr: String containing a URL
Returns:
 (True|False, url|errmsg)
"""
def valid_url(url):
  if len(url) > 2083:
    return(False, ("Invalid url, %s: too long." % url))
  patt = "(http|https):\/\/(\w+:{0,1}\w*@)?(\S+)(:[0-9]+)?(\/|\/([\w#!:.?+=&%@!\-\/]))?"
  if re.search(patt, url, re.IGNORECASE):
    return (True, url)
  return (False, ("Invalid url, %s." % url))

def true_or_false(value):
  for patt in ('true', 'false', '1', '0'):
    if re.search(patt, value, re.IGNORECASE):
      return (True, value)
  return(False, ("%s must be either true or false, 0 or 1" % value))

def yes_or_no(value):
  for patt in ('yes', 'no', '1', '0'):
    if re.search(patt, value, re.IGNORECASE):
      return (True, value)
  return(False, ("%s must be either true or false, 0 or 1" % value))

"""
Simple function to return whether a value is an Integer
Params:
 value: String representation of an integer
Returns:
 (True|False, value|errmsg)
"""
def is_integer(value):
  if isinstance(int(value), int):
    return (True, value)
  return (False, ("%s is not an integer" % value))

References

HTTP-based DB backup

If working with  Python Flask application in a closed environment it can often be handy to be able to grab a copy of the database behind the application.

Here are a few simple functions that can be used to provide a URL which will yield a (very simple) SQL export of the database schema and table contents.

"""
Simple route that captures a request to backup the database. It will
create a file containing
* the SQL commands to delete existing tables
* SQL schema commands for the database
* SQL insert operations for all the records in all the tables
params:
 None
Returns:
 File: contains database schema and rows from all the tables
"""
@app.route('/myapp/db_backup', methods=['GET'])
def myAppDbBackup():
    db_bkp = ""
    for tbl in database.get_tables():
        db_bkp = db_bkp + backupDbTable(tbl)
        if db_bkp == "":
            return Response(status="409 Empty backup")
     h = Headers()
     h.add('Content-Type', 'text/plain')
     h.add('Content-Disposition', 'attachment', filename='my_lovely_db_backup.sql')
     return Response(db_bkp, mimetype='application/x-sql', headers=h)

"""
Table backup function
Params:
 table: The name of the table to db_backup
Returns:
 String: All the table records as insert statements
"""
def backupDbTable(table):
    bkpstr = ""
    values = ""
    inscol = ""
    for col in database.get_columns(table):
        inscol = inscol + (col.name if not inscol else (", %s" % col.name))
        insert = ("INSERT INTO %s (%s) VALUES" % (table, inscol))

    bkpstr = ("%s;\n" % getTableSchema(table))
    valstr = ""
    qry = ("SELECT %s FROM %s" % (inscol, table))
    rs = database.execute_sql(qry)
    for row in rs:
        valstr = "', '".join(flattenArray(row))
        values = values + ("('" + valstr + "')" if not values else ",\n('" + valstr + "')")
    bkpstr = bkpstr + insert + values + ";\n"
    return bkpstr

"""
Function to retrieve the statement to create a table in the database
Params:
 table: The name of the table being backed up
Returns:
 String: The CREATE TABLE statement for table
"""
def getTableSchema(table):
    schema_qry = ("SELECT sql FROM sqlite_master WHERE type='table' AND name = ?")
    rs = database.execute_sql(schema_qry, [table])
    for schema in rs:
        return schema
def flattenArray(ary):
    flat = []
    for el in ary:
    if isinstance(el, list):
        flat += flattenArray(el)
    else:
        flat.append(el)
    return(flat)

It is always worth remembering that this is an intrinsically insecure operation but a small measure of protection could come from putting the app behind an Apache httpd or nginx reverse proxy with an SSL cert and user authentication. But best to limit to a closed environment.

Python Flask error handlers

Writing general error handlers for Python Flask applications is very straightdorward

@app.errorhandler(sqlite3.OperationalError)
def sqlite3_op_error(err):
 return render_template('errors/500.html', errmsg=('Database error: %s' % err))
@app.errorhandler(OperationalError)
def peewee_op_error(err):
 return render_template('errors/500.html', errmsg=('ORM error: %s' % err))
@app.errorhandler(NameError)
def name_error(err):
 return render_template('errors/500.html', errmsg=('Application error: %s' % err))
@app.errorhandler(404)
def page_not_found(err):
 return render_template('errors/404.html', errmsg=(err))
@app.errorhandler(DoesNotExist)
def query_does_not_exist(err):
 return render_template('errors/409.html', errmsg=('Query error: %s' % err))
@app.errorhandler(IntegrityError)
def query_integrity_error(err):
 return render_template('errors/409.html', errmsg=('Integrify error: %s' % err))

A template might take the form

{%- extends "error_layout.html" -%}
{%- block fourzeronine -%}
{{ super() }}

<p>
A database query error has occurred: {{ errmsg }}
</p>
<p>Back to <a href="/">index</a>.
{%- endblock -%}