Category Archives: Flask

Restricted access to EB instance

Almost embarrassed to admit that I spent much of the day trying to figure why my attempts at applying a custom nginx configuration scheme to block access to the editable content on my test site at http://xword-hints.eu-west-1.elasticbeanstalk.com/ were failing because the default Python instance actually runs Apache httpd!

There was certainy enough evidence in the logs but as soon as I figured out how to SSH to the instance, it didn’t take long.

For reference, SSHing to the instance requires that the EC2 key pair be applied to the environment through the Security settings; it’s likely that this can also be done via the CLI.

Checking the EC2 control panel for the instances will give the hostname to use for SSH login; just change the path to the SSH key that has been uploaded to AWS.

$ ssh -i ~/.ssh/private-key ec2-user@ec2-pub-ip-addr-ess.eu-west-1.compute.amazonaws.com

There are  couple of ways of applying the custom configuration needed to restrict access to the editable resources but the method I settled on was by adding the content to the .ebextensions/options.config with an entry in a section called files:

option_settings:
  aws:elasticbeanstalk:application:environment:
    SECRET_KEY: ChangeMe
  aws:elasticbeanstalk:container:python:
    WSGIPath: crossword_hints.py

files:
  /etc/httpd/conf.d/xword-hints-deny.conf:
    mode: 0644
    content: |
      <LocationMatch "/(crossword-solutions|crossword-setters|setter-types|solution-types)/[0-9]+/(edit|delete)">
        Require all denied
      </LocationMatch>
      <LocationMatch "/(crossword-solutions|crossword-setters|setter-types|solution-types)/new">
        Require all denied
      </LocationMatch>
      ErrorDocument 403 /static/403-xword-hints.html

It’s important to ensure that the indentation is correct for the file definition and content; the following deployment error will be thrown if not:

Service:AmazonCloudFormation, Message:[/Resources/AWSEBAutoScalingGroup/Metadata/AWS::CloudFormation::Init/prebuild_0_crossword_hints/files//etc/httpd/conf.d/xword-hints-deny.conf] 'null' values are not allowed in templates

The application needs to includes the 403 document, 403-xword-hints.html, because the web server will pass the request for the custom error page to it as a normal HTTP request.

With all this in place, the application is reasonably safe to leave running on the internet with any attempt to create, edit or delete content yielding a permissions error.

And the updates are still be applied by a Jenkins job pulling branch code from GitHub.

Advertisements

AWS ElasticBeanstalk custom environment variables

As a holiday project I’ve been looking into using Jenkins to deploy code updates from GitHub into an Amazon AWS ElasticBeanstalk instance[1] as an early attempt at some sort of continuous delivery.

One of the features of the Flask application is that it tries to get the SECRET_KEY from an environment variable (although the code for a failsafe value doesn’t work: FIXME). The intention is that the web server environment provides the key at runtime so that different values can be used in each environment.

Now, this AWS page describes the format of the options to apply custom environment settings to an application (the name of the actual file doesn’t matter so long as it is called .config and is found in the .ebextensions directory in the uploaded code):

option_settings:
  aws:elasticbeanstalk:application:environment:
    SECRET_KEY: ChangeMe
  aws:elasticbeanstalk:container:python:
    WSGIPath: crossword_hints.py

Setting the WSGIPath variable means that I can continue to use the original application source file rather than change to the default application.py.

This file can safely be kept in the GitHub repo and setup as a simple shell build step in Jenkins prior to the code upload, thus:

SECRET_KEY=`openssl rand -base64 12`; sed -ie "s/ChangeMe/${SECRET_KEY}/" .ebextensions/options.config

Jenkins has a great AWS EB deploy plugin that uses stored credentials to mange the source bundling, upload and deployment of the application; it’s kinda strange seeing the AWS console page spring into life in response to the Jenkins job running. To save having to include the build shell step, I’m thinking of creating my own version of the plugin that allows the inclusion of custom variables.

[1] – As a development instance the application will be mostly terminated (and offline) because AWS is a very expensive way of running a bit of demo code.

References

Dynamic (in-memory) zip file creation

With a recent project I wanted to offer a download of a bunch of files as a ZIP download to make redistribution. I started by thinking that this would involve creating a zip file on disk and then serving it to the client.

It doesn’t.

This method creates an in-memory zip image of a temporary directory in which the required file are created. The directory is deleted before the zip file is returned to the client. Note the use of send_file to return the content and how it allows us to specify the name of the downloaded file.

If running the app under uwsgi there are some precautions to take:

  • import the send_file module from flask (the default wsgi version is broken)
  • Include ‘wsgi-disable-file-wrapper’ in the uwsgi settings
import os
import io
import tempfile
import zipfile
from peewee import *

# A simple function to write a file in a given directory
def makeFile(dir, filename, data):
    fname = str(dir) + '/' + str(filename)
    fh = open(fname, "w")
    fh.write(data)
    fh.close()
    return True
"""
Function to create a zip file containing the application configuration files
To create the zip file, we need to create a memory file (BytesIO))
Params:
 row: Database row object containing columns for filename and
 content to be written in the form
 { {"file1": "Content for file 1}, {"file2": "Different content for file"} }
Returns:
 data: BytesIO object containing the zipped files
"""
def mkZipFile(row):
    zipdir = tempfile.mkdtemp(prefix='/tmp/')
    oldpath = os.getcwd()
    os.chdir(zipdir)

    jdata = json.loads(row)
    for conf in jdata:
        for f in conf:
            makeFile(zipdir, f, conf[f])

    # Create the in-memory zip image
    data = io.BytesIO()
    with zipfile.ZipFile(data, mode='w') as z:
        for fname in os.listdir("."):
            z.write(fname)
            os.unlink(fname)
    data.seek(0)

    os.chdir(oldpath)
    os.rmdir(zipdir)
    return data

The route takes the form

"""
Route to create a zip file containing all the required data files
Params:
 id: the numeric id of the record in the database
Returns:
 ZIP: file in ZIP format containing the download files
"""
@app.route('/download/<int:id>.zip', methods=['GET'])
def showZip(id):
    rs = db_table.select(db_table.data_column).where(db_table.id == id).get()

    zfile = mkZipFile(rs)
    if not zfile:
        return Response(response="Invalid zip file", status="400")

    return send_file(
        zfile,
        mimetype='application/zip',
        as_attachment=True,
        attachment_filename='file_bundle.zip')

References

The new submission form

This post will detail the processing of a new submission form that contains a multiple select form element.

A widget is described as a set of versioned components where a component version could be used in many widgets:
components:
id: INT
name: STRING
description: TEXT

component_versions:
id: INT
name: STRING
component_id: INT

widgets:
id: INT
name: STRING
description: TEXT

component_version_widgets:
component_id: IN
widget_id: INT

A simple route to handle a request to create  new item could be something like:

@app.route('/widgets/new', methods=['GET', 'POST'])
def newWidget():
    # Wasn't able to find any real documentation or examples of how to use the concat
    # function but sort of figured it out by looking at the PeeWee source code. This
    # uses a local function to create a list of list of lists of the items to be
    # displayed in the multiple select element in the form
    # [ [1, <component_name>-<component_version> ], [2, <component_name>-<component_version> ], ... ]
    # that is hopefully easier to process in the template 
    cvids = dict2lol(component_versions
            .select(component_versions.id.alias("cmpntid"),
             components.name.concat("-").concat(component_versions.name).alias("cmpntname"))
             .join(components, JOIN.INNER,
             on=component_versions.component_id == components.id)
             .dicts(), 'cmpntid', 'cmpntname')
    if request.method == 'POST':
        (status, msg) = validateWidget(request.form)
         # Use .getlist() to retrieve the multiple select items
         slctd = request.form.getlist('component_version_ids')
         if status:
             # Although auto-increment works with SQLite3 it didn't seem to work
             # when creating records with PeeWee
             nextId = widgets.select(fn.Max(widgets.id)+1).scalar()
             widgets.create(id=nextId,
                                         name=request.form['name'], \
                                         description=request.form['description'], \
                                         updated_by="config.admin", \
                                          updated_at=datetime.datetime.now())
             for cvid in slctd:
                  component_version_widget.create(widget_id=nextId, component_version_id=cvid)
                  logActivity('create', 'component_version_widgets', nextId, ("component_version_id: %s" % pvid))

             logActivity('create', 'widgets', nextId, request.form['name'])
              flash(("Saved new widget, update for %s." % request.form['name']))
              return redirect('/widgets')
        else:
             flash("Widget creation failed: %s." % msg)
              slctd = []
              for pv in request.form.getlist('component_version_ids'):
                 slctd.append(int(pv))
                 wdgt = {"description": request.form['description'],
                           "name": request.form['name'],
                           "component_version_ids": slctd}
    else:
        wdgt = {'name': 'Widget name', 'description': 'Brief description', "component_version_ids": []}
    return render_template('views/widgets/new.html', wdgt=wdgt, cvids=cvids, req=request)

Notes:

  • cvids: a list of lists of select options in the form [id, name]. Possibly being
    lazy but this is a bit simpler than trying to process what might come out of a PeeWee
    result set
  • slctd: a list of selected options from the select form; either from the database or the
    submitted form data. When creating this list from the submitted form we need to cast the id values as int’s or they won’t be recognised by the template inline conditional.
  • wdgt: dict containing the values for the template form items
  • formatting python code in the edit window is quite tricky so no guarantees that it compiles cleanly

The template to display this might look a bit like,

<form name="widgets" method="POST" action="{{ req.path }}">

<p>
Widget Widget name Description {{ rel['description'] }}
Component versions:
{%- for opt in cvids-%} {{ opt[1] }} {%- endfor -%}
</p>
<input type="submit" name="submit" value="Save widget"> </form>

The only real point to note in here is the method by which the ‘selected’ attribute is added to the option in the select list: the inline if.

References

The edit form submission

This post will detail the processing of an edit submission form that contains a multiple select form element.

A widget is described as a set of versioned components where a component version could be used in many widgets:

components:
id: INT
name: STRING
description: TEXT

component_versions:
id: INT
name: STRING
component_id: INT

widgets:
id: INT
name: STRING
description: TEXT

component_version_widgets:
component_id: INT
widget_id: INT

@app.route('/widgets/<int:id>/edit', methods=['GET', 'POST'])
def editWidget(id):
    # Get the component names and versions and format them for easy display
    # in the template
    cvids = dict2lol(component_versions
                .select(component_versions.id.alias("cmpntverid"),
                 components.name.concat("-").concat(component_versions.name).alias("cmpntname"))
                 .join(components, JOIN.INNER,
                 on=component_versions.component_id == components.id)
                 .dicts(), 'cmpntverid', 'cmpntname')

    if request.method == 'POST':
         (status, msg) = validateWidget(request.form)
         slctd = request.form.getlist('component_version_ids')
         if status:
             savewdgt = widgets(id=id,
                       name=request.form['name'], \
                       description=request.form['description'], \
                       updated_by="config.admin", \
                       updated_at=datetime.datetime.now())
             savewdgt.save()
            # Delete the current component versions for this widget and save
             # the values submitted in the form
             qry = component_version_widgets.delete().where(component_version_widgets.widget_id == id)
             qry.execute()
             for cvid in slctd:
                 component_version_widgets.create(widget_id=id, component_version_id=cvid)
 logActivity('update', 'component_version_widgets', id, ("component_version_id: %s" % cvid))

                logActivity('update', 'widgets', id, request.form['name'])
            flash(("Saved update for widget, %s." % request.form['name']))
            return redirect('/widgets')
        else:
             flash(("widget update failed: %s." % msg))
             slctd = []
             for cv in request.form.getlist('component_version_ids'):
                slctd.append(int(cv))
                wdgt = {"description": request.form['description'], "name": request.form['name'],
 "component_version_ids": slctd}
    else:
        # Display the initial edit form with details from the database
         # Get the details of the item to be edited 
         try:
             rs = widgets.select().where(widgets.id == id).get()
         except DoesNotExist:
             flash(("Cannot locate widget record with id = %s" % id))
         return redirect('/widgets')


    slctd = []
    for cvid in component_version_widgets \
           .select(component_version_widgets.component_version_id) \
           .where(component_version_widgets.widget_id == id):
        slctd.append(cvid.component_version_id)
        wdgt = {'name': rs.name, 'description': rs.description, 'component_version_ids': slctd}

    return render_template('views/widgets/edit.html', wdgt=wdgt, cvids=cvids, req=request)

Notes:

  • cvids: a list of lists of select options in the form [id, name]. Possibly being lazy but this is a bit simpler than trying to process what might come out of a PeeWee result set
  • slctd: a list of selected options from the select form; either from the database or the
    submitted form data. When creating this list from the submitted form we need to cast the id values as int’s or they won’t be recognised by the template inline conditional.
  • wdgt: dict containing the values for the template form items

The dict2lol function is simply:

def dict2lol(rs, val, text):
    lol = []
    for row in rs:
        lol.append([row[val], row[text]])
    return(lol)

The template to display this might look a bit like,

<form name="widgets" method="POST" action="{{ req.path }}">
<p>
Widget Widget name Description {{ rel['description'] }}
Component versions:
{%- for opt in cvids-%} {{ opt[1] }} {%- endfor -%}
</p>
<input type="submit" name="submit" value="Save widget"> </form>

The only real point to note in here is the method by which the ‘selected’
attribute is added to the option in the select list: the inline if.

References

POST form handling with Python Flask

A series of posts describing how to handle POST form submission with Python Flask.

Flask is a great framework for building simple HTTP-based applications but without the overhead and baggage of something like Django; Sinatra provides a similar situation when compared to Ruby on Rails.

One of the great things about Rails is how easy it is to work with many-to-many relationships and multiple select form elements.

But having abandoned working with Rails (and with no urge to go back) but managing to use some of the structure in Python Flask apps I do have to devise a method for handling new and edit routes using PeeWee as an ORM.

Most of this is probably self-evident but figuring out how to work with the multiple select form elements was quite tricky.

Simple POST form processing

The general form for a simple (single form) new page could be routes that accept GET and POST requests:

@app.route('/widgets/new', methods=['GET', 'POST'])
def newWidget():
  • If the request.method is ‘POST’ this is a form submission:
    • validate the form data and return a tuple containing the status and any error message
    • If the status id good:
      • Create a new record in the database
      • If processing a multiple select element:
      • for each item in the multiple select create a new many-to-many table entry
      • Record the activity in the log table
    • generate a flash message for display on the next page
    • redirect to the index page
    • If the status is bad:
      • Create a flash message with the error from the validation check
      • create a dict containing the submitted form elements to be used in the template
  • If not a POST request
    • prepare a dict containing default values for the form template

In all cases:

  • Generate objects to include database content to be passed to the temaplate
  • render the template

The general form for a simple (single form) edit page could be

  • Retrieve the edited object’s details from the database
  • If the request.method is ‘POST’ this is a form submission:
    • validate the form data and return a tuple containing the status and any error message
    • If the status id good:
      • Create a new record in the database
      • Record the activity in the log table
      • If processing a multiple select element:
        • delete the current many-to-many records the match the id of the edited object
        • for each item in the multiple select create a new many-to-many table entry
        • Record the activity in the log table
      • generate a flash message for display on the next page
      • redirect to the index page
    • If the status is bad:
      • Create a flash message with the error from the validation check
      • create a dict containing the submitted form elements to be used in the template
  • If not a POST request
    • prepare a dict containing edited item’s values for the form template

In all cases:

  • Generate objects to include database content to be passed to the template
  • render the template

    So far, so good, and I’m sure there’s nothing revelatory in any of this. The following posts will include some simple example code to demonstrate these types of form processing.

SQLite3 autoincrement primary and foreign keys

Just a wee nugget I picked up from the PeeWee documentation that needs to be saved for later reference ‘cos I’m bound to forget this with later projects and I’m including unnecessary code in my applications.

I had cursed PeeWee for not appearing to support incrementing primary keys when inserting new records and having to include a line to calculate the next insert id via a query (not a safe operation, I know), but it turns out that there is an alternate way to specify a primary key in the PeeWee model:

id = PrimryKeyField()

rather than

id = IntegerField(primary_key=True)

(though like most ORMs it doesn’t think you need the id column; I prefer to see it). This comes, however, from a Playhouse extension which I want to avoid.

It also seems that foreign keys are not honored by default and that they have to be explicity requested with,

database = SqliteDatabase(app.config['DATABASE'], pragmas=((‘foreign_keys’, ‘on’),))

when making the database connection. It doesn’t break my run of unit tests which is also a bonus.

Finally, a compound primary is better specified in the model as,

class Meta:
    primary_key = CompositeKey('column1', 'column2')

rather than  unique index for the unit tests to still work; the column names don’t include the ‘_id’ part of what might actually be in the database.

References