Category Archives: Uncategorized

Machine reading of crossword grid

Having spent some time ove the weekend entering clues, solutions and breakdowns of some sample crosswords for the hints database web application that I’m occasionally working on, and really struggling with it taking 30 – 45 minutes to add a complete grid I hit upon the idea of maybe trying to write a program that could scan a photo of the grid, determine the grid layout (i.e, identify the positions of the across and down clues) and then use OCR to detect the solution.

Just as a diversion, you understand. Probably won’t get very far. And I’ve still got a bunch of more relevant feature requests to be working on.

But it does mean that I can do a few blog posts with some cool sounding transforms and other mathematical wizardry without really knowing what I’m talking about (perhaps, I know people that have done this as a career, perhaps not).

OpenCV has python bindings and some good tutorials so it might be worth exploring for a while. And probably not as daft as trying to use image recognition in the garden for a smart scarecrowmagpie.

Advertisements

Disabling gnome-keyring

One of my pet peeves with cinnamon on the Antergos desktop is the incessant prompts from the gnome keyring when starting Chrome or Firefox with Gmail or other service loaded. I’ve never requested that Gnome store my passwords nor has it ever prompted for details of what it’s after and how to remove it.

In the past I have even deleted the gnome-keyring binary just to stop the really annoying behaviour, but there is a better way as described at https://www.linuxsecrets.com/archlinux-wiki/wiki.archlinux.org/index.php/GNOME/Keyring.html.

Just copy the gnome-keyring files from /etc/xdg/autostart into ~/.config/autostart and add settings to disable the daemon. According to http://ask.xmodulo.com/disable-gnome-keyring-linux-desktop.html, the implementation in gnome-keyring isn’t that great in any case so the service is probably not that useful anyway.

I’ll choose if I want the desktop to squirrel away my authentication data.

Python Flask activity logging

This post describes a simple pproach to recording login, logout, insert, update and delete operations that are carried out by a Python Flask application.

This is not the same as a table auditing function similar to Ruby’s Paperclip module.

https://github.com/slugbucket/crossword-hints/issues/9 is an example feature branch for a personal project implementing this.

The activity log will be stored in a database table with the following structure:
* rowid – auto-assigned unique id for the activity record
* actor – the name of the user performing the action
* action – one of login, insert, update, delete or logout,
* item_type -the table on which the operation has been performed.
* item_id – the numeric id of the item under operation
* act_action – details of the content that has been changed

This can be modeled in Flask using PeeWee

...
from peewee import *
from datetime import date, timedelta, datetime

...

class BaseModel(Model):
    with application.app_context():
    class Meta:
        database = database

    class activity_logs(BaseModel):
        rowid = AutoField()
        actor = CharField(max_length=32)
        action = CharField(max_length=32)
        item_type = CharField(max_length=32)
        item_id = IntegerField()
        act_action = TextField()
        created_at = CharField(max_length=32)
        updated_at = DateTimeField(default=datetime.now())

The application context is called application to work better when deployed to AWS.

Having used Flask-login to control the login process. we have a variable. current_user, that contains a numeric id of the user from their record in teh database.

The users model includes a method to return the name of the user and this can be used for the ‘actor’ value.

A simple function can create the activity log reord:

def add_log(actor, action, item_type, item_id, activity):
    log = activity_logs(actor=actor,
          action=action,
          item_type=item_type,
          item_id=item_id,
          act_action=activity,
          created_at=datetime.now(),
          updated_at=datetime.now())
    log.save()

And activities to be logged can be logged with something like,

log = ("name: %s\ndescription: %s" % (name, description))
add_log(users.get_name(current_user), 'update', 'solution_types', id, log)

Where,

  • name and description are variables containing form-submitted data
  • activity_logs is the PeeWee model of the databse table
  • users is a PeeWee ORM model of an application user that can be authenticated.

When used the (sqlite3) database records will look like,

sqlite> select * from activity_logs;
crossy|login|user|1|Successful login for crossy|2019-02-15 18:44:38.257050|2019-02-15 18:44:38.257061
crossy|update|solution_types|12|name: Homophone
description: Sounds like a word that has a different spelling|2019-02-15 18:45:03.227188|2019-02-15 18:45:03.227193
crossy|logout|user|1|Successful logout user for crossy|2019-02-15 18:48:11.043243|2019-02-15 18:48:11.043249

With these records it should be possible to display an activity report and perhaps even recover previous versions of particular database records.

Improvements

  • Storing the full record on each update is inefficient; only differences should be recorded.
  • The logging function should be able to determine what to include in the log message based on the

Configure Python Flask application for LDAP authentication

Using a crossword-hints Flask application to include authentication for adding, modifying and deleting database content.
We will use ad a users table to the database and hand off the authentication to the directory.

Install the python-ldp pip (3.1.0 as of February 2019)

$ pip install python-ldap --user

Install the Flask-login pip (0.4.1 s of February 2019)

$ pip install flask-login --user

Include the modules in the Flask application

from flask_login import LoginManager, UserMixin, login_required, login_user, logout_user
import ldap

Configure Flask-login in the application

# flask-login
login_manager = LoginManager()
login_manager.init_app(application)
login_manager.login_view = "crossword_login"

The login_view indicated the function to be called to handle the user login.

Flask-login also requires a user_loader function that returns an integer id value for the user,

@login_manager.user_loader
def load_user(id):
    return users.get(users.rowid == int(id))

Add a function to get a connection to the directory (but no authentication at this point)

def get_ldap_connection():
    conn = ldap.initialize('ldap://localhost:389')
    return conn

Add a route to the login form. This accepts GET and POST requests to display the form and then
process the submitted form. Notice that after successful login there is a redirect to a page passed as the next parameter in the query string when an application route requires login: this could be a securiity concern if steps are not taken to prevent the application accepting a spoofed resource as input rather than a permitted resource in the intended application.

@application.route("/login", methods=['GET', 'POST'])
def crossword_login():
    try:
        if current_user.is_authenticated:
        flash('You are already logged in.')
        return redirect(request.path)
    except:
        pass

    if request.method == 'POST':
        username = request.form.get('username')
        password = request.form.get('password')

        try:
            users.try_login(username, password)
        except ldap.INVALID_CREDENTIALS:
            flash(
                'Invalid username or password. Please try again.', 'danger')
            return(render_template('views/login/login.html', u=username, r=request))

        user = users.get(users.username == username)
        login_user(user)
        return redirect(request.args.get("next"))
    else:
        username = 'username'

    return(render_template('views/login/login.html', u=username, r=request))

A template for the login form could be

{%- import "macros.html" as f -%}
Use this form to login to the site to maintain restricted content.
</p>
<form name="crossword_login" action="{{ r.full_path }}" method="POST">
<fieldset id="crossword_login">
<legend>Login</legend>
{{ f.label('username', 'Username') }}
{{ f.input_tag('username', value=(u|escape()|default('')), class="xwordhints") }}

{{ f.label('password', 'Password') }}
{{ f.input_tag('password', type="password", class="xwordhints") }}
</fieldset>

<p>
<input type="submit" name="login" value="Login" />
<input type="reset" value="Reset" />
<br /><br />
<a href="/crossword-hints">Back to index</a>
</p>
</form>

macros.html (there’s a good GitHub gist with some macros but I can’t find a link to it) contains helpers to create the form input elements. The form action
uses the request’s full path which includes the query string containing the next parameter.

Provide a logout route. Note that this requires a valid login. Going direct to /logout will show a prompt to login!

@application.route("/logout")
@login_required
def logout():
    logout_user()
    flash("Logout successful. Please close browser for best security.")
    return(redirect("/"))

A PeeWee user model

class users(BaseModel):
    rowid = AutoField()
    username = CharField(null=False, max_length=32, unique=True)
    created_at = DateTimeField(default=datetime.now())
    updated_at = DateTimeField(default=datetime.now())

    @staticmethod
    def try_login(username, password):
        conn = get_ldap_connection()
        conn.simple_bind_s(
            'uid=%s,ou=People,dc=my-domain,dc=com' % username, password
        )

    def is_authenticated(self):
        return True

    def is_active(self):
        return True

    def is_anonymous(self):
        return False

    def get_id(self):
        conn = get_ldap_connection()
        return(self.rowid)

    def get_name(self):
       return(self.username)

The get_name method is used by HTML templates to display the logged in username. For authentication the try_login method just attempts to bind to the directory as the user.

{% if current_user.get_id() is not none %}Logged in {{ current_user.get_name() }}&nbsp; <a href="/logout">Logout</a>{% endif %}

Routes that need authentication (and authoristion) before access simply need to state ‘login_required):

@application.route("/crossword-setters/<int:id>/edit", methods=["GET", "POST"])
@login_required
def crossword_setters_edit(id):

Accessing this route will redirect to the login form and successful login will redirect back to the edit page.

Unit tests

When requiring authentication to enable certain operations in the application, consideration needs to be given to the impact on running unit tests.

The protecting views section of the Flask-login site indicates that this is simply a matter of setting the following option:

LOGIN_DISABLED=True

which disables the login_required decoration applied to routes. Adding this to a settings file used when running the tests allows everything to complete as expected.

APP_SETTINGS='test-settings.py' python crossword-hints-tests.py

 

Improvements

  • Don’t use the SQL database to store the users
  • Control authorisation according to group membership.
  • Prevent authentication leakage via spoofed ‘next’ resources.
  • Don’t request a login when going direct to the logout route.

References

ArchLinux OpenLDAP setup for webapp authentication

Setting up a simple OpenLDAP directory to allow authentication from a web (or other) application.

Based on https://wiki.archlinux.org/index.php/OpenLDAP

Configure LDAP service

Install openldap packages

Create a salted hashed password for the directory manager:

$ slappasswd -h {SSHA}

Enter and confirm a password and receive output such as

{SSHA}c920AmsQ9Evay0YaCU/r0GAdnMroyL4O

Configure /etc/openldap/slapd.conf to use the following:

database mdb
suffix "dc=my-domain,dc=com"
rootdn "cn=Manager,dc=my-domain,dc=com"
...
rootpw {SSHA}c920AmsQ9Evay0YaCU/r0GAdnMroyL4O

As the superuser, setup the directory database and start the service,

# cp /var/lib/openldap/openldap-data/DB_CONFIG.example /var/lib/openldap/openldap-data/DB_CONFIG
# slaptest -f /etc/openldap/slapd.conf -F /etc/openldap/slapd.d
# slapindex
# chown -R ldap:ldap /etc/openldap/slapd.d /var/lib/openldap/openldap-data
# systemctl start slapd

Test connection,

$ ldapsearch -D "cn=Manager,dc=my-domain,dc=com" -W -x '(objectclass=*)'
Enter LDAP Password:

Create base directory, users and groups

Create an LDIF file, say, base.ldif, for the directory manager and base groups (whether or not the groups are used by the application is another matter):

dn: dc=my-domain,dc=com
objectClass: dcObject
objectClass: organization
dc: my-domain
o: My-domain
description: My-domain directory

dn: cn=Manager,dc=my-domain,dc=com
objectClass: organizationalRole
cn: Manager
description: Directory Manager

# People, my-domain.com
dn: ou=People,dc=my-domain,dc=com
ou: People
objectClass: top
objectClass: organizationalUnit

# Groups, my-domain.com
dn: ou=Groups,dc=my-domain,dc=com
ou: Groups
objectClass: top
objectClass: organizationalUnit

# xword-admins group
dn: cn=xword-admins,ou=Groups,dc=my-domain,dc=com
objectClass: top
objectClass: posixGroup
gidNumber: 10000

# xword-users group
dn: cn=xword-users,ou=Groups,dc=my-domain,dc=com
objectClass: top
objectClass: posixGroup
gidNumber: 10001
$ ldapadd -D "cn=Manager,dc=my-domain,dc=com" -W -x -c -f /path/to/base.ldif

(-c allows for repeated runs of the same LDIF as typos and errors are wrinkled out)
Enter the password when prompted and check that the OUs and groups are created with:

$ ldapsearch -D "cn=Manager,dc=my-domain,dc=com" -W -x '(objectclass=*)'

Add groups and users

Generate a hashed password for each user, to be used in the subsequent LDIF file,

$ slappasswd -h {SSHA}

Create another ldap, say, users.ldif. contining something like,

dn: uid=xword,ou=People,dc=my-domain,dc=com
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: shadowAccount
uid: xword
cn: Crossword Hints
sn: Hints
givenName: Crossword
userPassword: {SSHA}V7i9IYaW4Z4Z2mVjbtOk0PcinzgJof9y
labeledURI: http://xword-hints.my-domain.com/
loginShell: /bin/bash
uidNumber: 9999
gidNumber: 9999
homeDirectory: /home/xword
description: Crossword hints website user

# Add xword to the LDAP xword-admins group
# group
dn: cn=xword-admins,ou=Groups,dc=my-domain,dc=com
changetype: modify
add: memberuid
memberuid: xword

# Add xword to the LDAP xword-admins group
dn: cn=xword-users,ou=Groups,dc=my-domain,dc=com
changetype: modify
add: memberuid
memberuid: xword

Then add the users to the directory

$ ldapadd -D "cn=Manager,dc=my-domain,dc=com" -W -x -c -f /path/to/users.ldif

Check the users can authenticate through the directory,

$ ldapsearch -D "uid=xword,ou=People,dc=my-domain,dc=com" -W -x '(objectclass=*)'

This will probably show the entire directory contents; this can be tightened up as needed.

The directory is now ready for user authentication.

Troubleshooting

Reset the password described at https://www.digitalocean.com/community/tutorials/how-to-change-account-passwords-on-an-openldap-server although the Arch LDAP server does not have the EXTERNAL mechanism enabled by default; it’s probably easier to rebuild the directory.

AWS ElasticBeanstalk custom environment variables

As a holiday project I’ve been looking into using Jenkins to deploy code updates from GitHub into an Amazon AWS ElasticBeanstalk instance[1] as an early attempt at some sort of continuous delivery.

One of the features of the Flask application is that it tries to get the SECRET_KEY from an environment variable (although the code for a failsafe value doesn’t work: FIXME). The intention is that the web server environment provides the key at runtime so that different values can be used in each environment.

Now, this AWS page describes the format of the options to apply custom environment settings to an application (the name of the actual file doesn’t matter so long as it is called .config and is found in the .ebextensions directory in the uploaded code):

option_settings:
  aws:elasticbeanstalk:application:environment:
    SECRET_KEY: ChangeMe
  aws:elasticbeanstalk:container:python:
    WSGIPath: crossword_hints.py

Setting the WSGIPath variable means that I can continue to use the original application source file rather than change to the default application.py.

This file can safely be kept in the GitHub repo and setup as a simple shell build step in Jenkins prior to the code upload, thus:

SECRET_KEY=`openssl rand -base64 12`; sed -ie "s/ChangeMe/${SECRET_KEY}/" .ebextensions/options.config

Jenkins has a great AWS EB deploy plugin that uses stored credentials to mange the source bundling, upload and deployment of the application; it’s kinda strange seeing the AWS console page spring into life in response to the Jenkins job running. To save having to include the build shell step, I’m thinking of creating my own version of the plugin that allows the inclusion of custom variables.

[1] – As a development instance the application will be mostly terminated (and offline) because AWS is a very expensive way of running a bit of demo code.

References

Pastry – handle with care

Finally, I’m getting to understand what it means to handle shortcrust pastry as little as possible.

So there has been plenty of practice making Bakewell tarts and some with homemade lemon curd. The next batch will include some of the homemade jam.

But the best creation so far has been a lemon meringue pie for the first time ever.

IMG_20181216_170017097_HDR

It’s the first time I’ve been able to make that size of pastry in one piece and the first go ever at making the meringue. Delicious.