Python Flask pagination

Flask pagination

The [crossword hints]() Flask application I have been writing has been expanding and I have taken some time to add more clues and solutions such that there is now some need to break up the index list into more manageable chunks using pagination.

I’ve been considering a couple of options:

  • the flask-paginate pip which is modelled on Rails’ will_paginate (which I have used in the past.
  • a Flask snippet from Armin Ronacher, which on the face of it looks like it will be harder to implement;
  • a hand-rolled solution, but why re-invent the wheel?

Flask-paginate

This should be the preferred solution because it reduces the amount of coding required in the application, but I had a cserious problems when it came to rendering the pagination: it’s missing much of the necessary CSS to display the page listing.

At first, the page listing just appeared as a bulleted list. It turns out that Flask-paginate uses a CSS framework called bootstrap but the documentation
doesn’t include any real mention of it, let alone describing it as essential.

StackOverflow responses to Flask=paginate styling problems just said to install
the bootstrap CSS files, but when I tried this it just completely mangled much of
the other navigation styling I already had. I hate messing around with CSS at the best of times so there’s no way I’m wasting time fiddling around with minimised 3rd-party dependent CSS. Time to ‘git checkout — ‘ the files I’d been working on.

Snippet 44

I was wary of attempting to implementing the suggestion in the snippet, not
because I thought it wouldn’t be any good (total regard for Armin’s code), but because of my lack opf confidence working with what looked like a bit more advanced Python and Flask and whether my ropey application could support it being added without a complete rewrite.
I need not have worried.

Anyway, the task breaks down into the following stages:

  • a pagination class
  • a view helper,
  • routing and rendering: URLs and views,
  • CSS to support a reasonable page layout

Pagination class

First off we need a class to contain properties and methods needed to describe the pagination:

  • total number of records,
  • items per page
  • number of pages
  • to determine whether the current page needs next and/or previous links
  • an iterator over the pages; I don’t think I would have been able to write this
    so concisely, so definitely good value here,
class Pagination(object):
    def __init__(self, page, per_page, total_count):
        self.page = page
        self.per_page = per_page
        self.total_count = total_count

    @property
    def pages(self):
        return int(ceil(self.total_count / float(self.per_page)))

    @property
    def has_prev(self):
        return self.page > 1

    @property
    def has_next(self):
        return self.page < self.pages

    def iter_pages(self, left_edge=2, left_current=2,
        right_current=5, right_edge=2):
        last = 0
        for num in range(1, self.pages + 1):
            if num <= left_edge or \
              (num > self.page - left_current - 1 and \
            num < self.page + right_current) or \
            num > self.pages - right_edge:
            if last + 1 != num:
                yield None
            yield num
            last = num

I had to make a tiny change to what appeared in the snippet, replacing xrange with just range.

View helper

There needs to be a view helper that can be used in a template macro to generate the link to the other pages.

def url_for_other_page(page):
    args = request.view_args.copy()
    args['page'] = page
    return url_for(request.endpoint, **args)
application.jinja_env.globals['url_for_other_page'] = url_for_other_page

Although this has been added to my main application file it is something that can be relocated somewhere more sensible when refactoring work starts; registering the helper with Jinja this way is certainly something I can try with future projects so this snippet is teaching more than just pagination. Win-win.

Routing and rendering

The trickiest part of the process is probably changing the index route to support displaying a particular page of content.

Default and paged indexes

The easiest way to drive the pagination routing is to associate a page number with the route

@application.route("/crossword-solutions/", defaults={'page': 1})
@application.route('/crossword-solutions/page/<int:page>')
def crossword_solution_index(page):

The index routing needs to include:

  • a count of the number of items
  • a query that only selects the necessary records for display,
  • a check to capture out of range page requests,
  • create a pagination instance and pass it to the template to render

This gives an index router like:

    count = crossword_solutions.select(fn.COUNT(crossword_solutions.rowid)).scalar()
    PER_PAGE=25
    offset = ((int(page)-1) * PER_PAGE)
    solutions = crossword_solutions.raw("""
        SELECT cs1.name as setter, cs2.solution AS solution, cs2.rowid AS csid, st1.name AS soltype
        FROM crossword_setters cs1
        INNER JOIN crossword_solutions cs2
        ON cs1.rowid = cs2.crossword_setter_id
        INNER JOIN solution_types st1
        ON cs2.solution_type_id = st1.rowid
        ORDER BY cs2.solution
        LIMIT %s, %s""" % (offset, PER_PAGE))
    # Display a 409 not found page for an out of bounds request
    if not solutions and page != 1:
        return(render_template('errors/409.html', errmsg="Requested page out of bounds"), 409 )
    pagination = Pagination(page, PER_PAGE, count)
    return(render_template('views/crossword-solutions/index.html',
                            r=request,
                            solns=solutions,
                            pagination=pagination))

Although I’m using PeeWee as the ORM, I’ve found that queries like this one work better as plain SQL particularly with the offset and limit. What we notice here is that the pagination has no connection at all with the database result set like, say, will_paginate, but still doesn’t lose any functionality.

Pagination macro

I’m still not very experienced with Jinja2 macros so really appreciated the sample provided by the snippet. I modified it slightly to include a previous page link and also display next and previous as grayed-out boxes and the last and first pages respectively.

{% macro render_pagination(pagination) %}

<br />
{% endmacro %}

This is added to an existing macros,html template file.

Template rendering

The macros file is imported into a high-level layout template as

{%- import "macros.html" as f -%}

and child templates apply the pagination with,

{{ f.render_pagination(pagination) }}

Pagination stylesheet

I dislike HTML layout in general and loathe CSS in particular so I will settle for whatever gives a reasonable look despite any ‘obvious’ inefficiencies and any prolonged CSS development will inevitably provoke much cussing and cursing. I have based the styling for my current project around ‘digg_pagination’ styles I came across for old Rails projects with as few tweaks as possible; I really don’t why some settings get included from some sections and not other.

The styling boils down to the following sections (I include it only as an example
and not a recommendation; no-one in their right mind would trust me to design CSS)

.pagination {
margin: 10px 0;
margin-top: 0;
margin-bottom: 0;
font-size: 0.7em;
}
.pagination ul li {
border-color: #105cb6;
border-width: 0 0 1px;
border-radius: 0;
float: left;
list-style-type: none;
}
.pagination ul li span.nolink {
padding: 2px 10px 2px 10px;
display: block;
/* float: left; */
margin-right: 1px;
font-size: 8pt;
font-weight: bold;
border: 1px solid #9aafe5;
color: #D5D5D5;
}
.pagination a, .pagination ul li span {
padding: 2px 10px 2px 10px;
display: block;
/* float: left; */
margin-right: 1px;
font-weight: bold;
color: #105cb6;
border: 1px solid #9aafe5;
}
.pagination ul li span.ellipsis {
/* font-size: 10pt; */
font-weight: normal;
padding: 2px;
margin: 1px;
border-color: #fff;
}

Which I think gives a reasonable look to the pagination with greyed out boxes for the prev and next links on the first and last pages and clearly indicates the active page.

References

I highly recommend http://flask.pocoo.org/snippets/44/ as a great place to start if looking at pagination for a Flask application; in general, Armin’s posts here are top-notch and very informative.

Advertisements

Python Flask activity logging

This post describes a simple pproach to recording login, logout, insert, update and delete operations that are carried out by a Python Flask application.

This is not the same as a table auditing function similar to Ruby’s Paperclip module.

https://github.com/slugbucket/crossword-hints/issues/9 is an example feature branch for a personal project implementing this.

The activity log will be stored in a database table with the following structure:
* rowid – auto-assigned unique id for the activity record
* actor – the name of the user performing the action
* action – one of login, insert, update, delete or logout,
* item_type -the table on which the operation has been performed.
* item_id – the numeric id of the item under operation
* act_action – details of the content that has been changed

This can be modeled in Flask using PeeWee

...
from peewee import *
from datetime import date, timedelta, datetime

...

class BaseModel(Model):
    with application.app_context():
    class Meta:
        database = database

    class activity_logs(BaseModel):
        rowid = AutoField()
        actor = CharField(max_length=32)
        action = CharField(max_length=32)
        item_type = CharField(max_length=32)
        item_id = IntegerField()
        act_action = TextField()
        created_at = CharField(max_length=32)
        updated_at = DateTimeField(default=datetime.now())

The application context is called application to work better when deployed to AWS.

Having used Flask-login to control the login process. we have a variable. current_user, that contains a numeric id of the user from their record in teh database.

The users model includes a method to return the name of the user and this can be used for the ‘actor’ value.

A simple function can create the activity log reord:

def add_log(actor, action, item_type, item_id, activity):
    log = activity_logs(actor=actor,
          action=action,
          item_type=item_type,
          item_id=item_id,
          act_action=activity,
          created_at=datetime.now(),
          updated_at=datetime.now())
    log.save()

And activities to be logged can be logged with something like,

log = ("name: %s\ndescription: %s" % (name, description))
add_log(users.get_name(current_user), 'update', 'solution_types', id, log)

Where,

  • name and description are variables containing form-submitted data
  • activity_logs is the PeeWee model of the databse table
  • users is a PeeWee ORM model of an application user that can be authenticated.

When used the (sqlite3) database records will look like,

sqlite> select * from activity_logs;
crossy|login|user|1|Successful login for crossy|2019-02-15 18:44:38.257050|2019-02-15 18:44:38.257061
crossy|update|solution_types|12|name: Homophone
description: Sounds like a word that has a different spelling|2019-02-15 18:45:03.227188|2019-02-15 18:45:03.227193
crossy|logout|user|1|Successful logout user for crossy|2019-02-15 18:48:11.043243|2019-02-15 18:48:11.043249

With these records it should be possible to display an activity report and perhaps even recover previous versions of particular database records.

Improvements

  • Storing the full record on each update is inefficient; only differences should be recorded.
  • The logging function should be able to determine what to include in the log message based on the

Configure Python Flask application for LDAP authentication

Using a crossword-hints Flask application to include authentication for adding, modifying and deleting database content.
We will use ad a users table to the database and hand off the authentication to the directory.

Install the python-ldp pip (3.1.0 as of February 2019)

$ pip install python-ldap --user

Install the Flask-login pip (0.4.1 s of February 2019)

$ pip install flask-login --user

Include the modules in the Flask application

from flask_login import LoginManager, UserMixin, login_required, login_user, logout_user
import ldap

Configure Flask-login in the application

# flask-login
login_manager = LoginManager()
login_manager.init_app(application)
login_manager.login_view = "crossword_login"

The login_view indicated the function to be called to handle the user login.

Flask-login also requires a user_loader function that returns an integer id value for the user,

@login_manager.user_loader
def load_user(id):
    return users.get(users.rowid == int(id))

Add a function to get a connection to the directory (but no authentication at this point)

def get_ldap_connection():
    conn = ldap.initialize('ldap://localhost:389')
    return conn

Add a route to the login form. This accepts GET and POST requests to display the form and then
process the submitted form. Notice that after successful login there is a redirect to a page passed as the next parameter in the query string when an application route requires login: this could be a securiity concern if steps are not taken to prevent the application accepting a spoofed resource as input rather than a permitted resource in the intended application.

@application.route("/login", methods=['GET', 'POST'])
def crossword_login():
    try:
        if current_user.is_authenticated:
        flash('You are already logged in.')
        return redirect(request.path)
    except:
        pass

    if request.method == 'POST':
        username = request.form.get('username')
        password = request.form.get('password')

        try:
            users.try_login(username, password)
        except ldap.INVALID_CREDENTIALS:
            flash(
                'Invalid username or password. Please try again.', 'danger')
            return(render_template('views/login/login.html', u=username, r=request))

        user = users.get(users.username == username)
        login_user(user)
        return redirect(request.args.get("next"))
    else:
        username = 'username'

    return(render_template('views/login/login.html', u=username, r=request))

A template for the login form could be

{%- import "macros.html" as f -%}
Use this form to login to the site to maintain restricted content.
</p>
<form name="crossword_login" action="{{ r.full_path }}" method="POST">
<fieldset id="crossword_login">
<legend>Login</legend>
{{ f.label('username', 'Username') }}
{{ f.input_tag('username', value=(u|escape()|default('')), class="xwordhints") }}

{{ f.label('password', 'Password') }}
{{ f.input_tag('password', type="password", class="xwordhints") }}
</fieldset>

<p>
<input type="submit" name="login" value="Login" />
<input type="reset" value="Reset" />
<br /><br />
<a href="/crossword-hints">Back to index</a>
</p>
</form>

macros.html (there’s a good GitHub gist with some macros but I can’t find a link to it) contains helpers to create the form input elements. The form action
uses the request’s full path which includes the query string containing the next parameter.

Provide a logout route. Note that this requires a valid login. Going direct to /logout will show a prompt to login!

@application.route("/logout")
@login_required
def logout():
    logout_user()
    flash("Logout successful. Please close browser for best security.")
    return(redirect("/"))

A PeeWee user model

class users(BaseModel):
    rowid = AutoField()
    username = CharField(null=False, max_length=32, unique=True)
    created_at = DateTimeField(default=datetime.now())
    updated_at = DateTimeField(default=datetime.now())

    @staticmethod
    def try_login(username, password):
        conn = get_ldap_connection()
        conn.simple_bind_s(
            'uid=%s,ou=People,dc=my-domain,dc=com' % username, password
        )

    def is_authenticated(self):
        return True

    def is_active(self):
        return True

    def is_anonymous(self):
        return False

    def get_id(self):
        conn = get_ldap_connection()
        return(self.rowid)

    def get_name(self):
       return(self.username)

The get_name method is used by HTML templates to display the logged in username. For authentication the try_login method just attempts to bind to the directory as the user.

{% if current_user.get_id() is not none %}Logged in {{ current_user.get_name() }}&nbsp; <a href="/logout">Logout</a>{% endif %}

Routes that need authentication (and authoristion) before access simply need to state ‘login_required):

@application.route("/crossword-setters/<int:id>/edit", methods=["GET", "POST"])
@login_required
def crossword_setters_edit(id):

Accessing this route will redirect to the login form and successful login will redirect back to the edit page.

Unit tests

When requiring authentication to enable certain operations in the application, consideration needs to be given to the impact on running unit tests.

The protecting views section of the Flask-login site indicates that this is simply a matter of setting the following option:

LOGIN_DISABLED=True

which disables the login_required decoration applied to routes. Adding this to a settings file used when running the tests allows everything to complete as expected.

APP_SETTINGS='test-settings.py' python crossword-hints-tests.py

 

Improvements

  • Don’t use the SQL database to store the users
  • Control authorisation according to group membership.
  • Prevent authentication leakage via spoofed ‘next’ resources.
  • Don’t request a login when going direct to the logout route.

References

ArchLinux OpenLDAP setup for webapp authentication

Setting up a simple OpenLDAP directory to allow authentication from a web (or other) application.

Based on https://wiki.archlinux.org/index.php/OpenLDAP

Configure LDAP service

Install openldap packages

Create a salted hashed password for the directory manager:

$ slappasswd -h {SSHA}

Enter and confirm a password and receive output such as

{SSHA}c920AmsQ9Evay0YaCU/r0GAdnMroyL4O

Configure /etc/openldap/slapd.conf to use the following:

database mdb
suffix "dc=my-domain,dc=com"
rootdn "cn=Manager,dc=my-domain,dc=com"
...
rootpw {SSHA}c920AmsQ9Evay0YaCU/r0GAdnMroyL4O

As the superuser, setup the directory database and start the service,

# cp /var/lib/openldap/openldap-data/DB_CONFIG.example /var/lib/openldap/openldap-data/DB_CONFIG
# slaptest -f /etc/openldap/slapd.conf -F /etc/openldap/slapd.d
# slapindex
# chown -R ldap:ldap /etc/openldap/slapd.d /var/lib/openldap/openldap-data
# systemctl start slapd

Test connection,

$ ldapsearch -D "cn=Manager,dc=my-domain,dc=com" -W -x '(objectclass=*)'
Enter LDAP Password:

Create base directory, users and groups

Create an LDIF file, say, base.ldif, for the directory manager and base groups (whether or not the groups are used by the application is another matter):

dn: dc=my-domain,dc=com
objectClass: dcObject
objectClass: organization
dc: my-domain
o: My-domain
description: My-domain directory

dn: cn=Manager,dc=my-domain,dc=com
objectClass: organizationalRole
cn: Manager
description: Directory Manager

# People, my-domain.com
dn: ou=People,dc=my-domain,dc=com
ou: People
objectClass: top
objectClass: organizationalUnit

# Groups, my-domain.com
dn: ou=Groups,dc=my-domain,dc=com
ou: Groups
objectClass: top
objectClass: organizationalUnit

# xword-admins group
dn: cn=xword-admins,ou=Groups,dc=my-domain,dc=com
objectClass: top
objectClass: posixGroup
gidNumber: 10000

# xword-users group
dn: cn=xword-users,ou=Groups,dc=my-domain,dc=com
objectClass: top
objectClass: posixGroup
gidNumber: 10001
$ ldapadd -D "cn=Manager,dc=my-domain,dc=com" -W -x -c -f /path/to/base.ldif

(-c allows for repeated runs of the same LDIF as typos and errors are wrinkled out)
Enter the password when prompted and check that the OUs and groups are created with:

$ ldapsearch -D "cn=Manager,dc=my-domain,dc=com" -W -x '(objectclass=*)'

Add groups and users

Generate a hashed password for each user, to be used in the subsequent LDIF file,

$ slappasswd -h {SSHA}

Create another ldap, say, users.ldif. contining something like,

dn: uid=xword,ou=People,dc=my-domain,dc=com
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: shadowAccount
uid: xword
cn: Crossword Hints
sn: Hints
givenName: Crossword
userPassword: {SSHA}V7i9IYaW4Z4Z2mVjbtOk0PcinzgJof9y
labeledURI: http://xword-hints.my-domain.com/
loginShell: /bin/bash
uidNumber: 9999
gidNumber: 9999
homeDirectory: /home/xword
description: Crossword hints website user

# Add xword to the LDAP xword-admins group
# group
dn: cn=xword-admins,ou=Groups,dc=my-domain,dc=com
changetype: modify
add: memberuid
memberuid: xword

# Add xword to the LDAP xword-admins group
dn: cn=xword-users,ou=Groups,dc=my-domain,dc=com
changetype: modify
add: memberuid
memberuid: xword

Then add the users to the directory

$ ldapadd -D "cn=Manager,dc=my-domain,dc=com" -W -x -c -f /path/to/users.ldif

Check the users can authenticate through the directory,

$ ldapsearch -D "uid=xword,ou=People,dc=my-domain,dc=com" -W -x '(objectclass=*)'

This will probably show the entire directory contents; this can be tightened up as needed.

The directory is now ready for user authentication.

Troubleshooting

Reset the password described at https://www.digitalocean.com/community/tutorials/how-to-change-account-passwords-on-an-openldap-server although the Arch LDAP server does not have the EXTERNAL mechanism enabled by default; it’s probably easier to rebuild the directory.

The pain of Nagios and Arch Linux

Or, there’s more to systemd than just service files.

I recently had cause to investigate writing a simple Nagios http monitor and although I’ve worked in and around Nagios I’ve not really had cause to try writing the monitor scripts.

There was an aspect of the task I wasn’t too sure about so I thought the best way to get to grips with fully understanding  the environment would be to try and get Nagios running on my desktop system.

Moral of the story: there’s a reason why Arch will never be used in an enterprise environment.

First, I should have been warned off by seeing that nagios for Arch only comes via the AUR.

But, there’s a reasonably detailed page describing it. The real problems came with nginx, php-fpm and fastcgi.

The php-fpm service isn’t enabled or started by default: systemctl soon fixes that.

For some bizarre reason, it turned out that although nginx had been configured to use /usr/share/nagios/share and the root directory for PHP scripts, it was trying to access PHP scripts in /etc/nginx/html/nagios. I had to cheat here and just created a symbolic link between the directories which got the basic page displaying.

But then to get the CGI scripts running I had to look more closely at the Arch nginx configuration for fcgiwrap and having seen that the fcgiwrap service wouldn’t run, I finally twigged that nginx was referring to a .sock file and that there was a systemd socket file.

So I now have Nagios running on Arch. Mostly.

Tomorrow I need to figure out the cause of these errors:

Error: Could not read object configuration data!

But at least we’re now dealing with Nagios problems rather than Arch.

References

Places of interest

 

htpasswd without Apache

If wanting to restrict access to website content under nginx but don’t want to install Apache use the following to generate an htpasswd file

printf "USER:$(openssl passwd -apr1 P@55w0rd)\n" > /etc/nginx/auth/htpasswd

Then use the following nginx rules to

location /api {
    satisfy all;    

    allow 127.0.0.1;
    deny  all;

    auth_basic           "Administrator’s Area";
    auth_basic_user_file /etc/nginx/auth/htpasswd;
}

 

References

Restricted access to EB instance

Almost embarrassed to admit that I spent much of the day trying to figure why my attempts at applying a custom nginx configuration scheme to block access to the editable content on my test site at http://xword-hints.eu-west-1.elasticbeanstalk.com/ were failing because the default Python instance actually runs Apache httpd!

There was certainy enough evidence in the logs but as soon as I figured out how to SSH to the instance, it didn’t take long.

For reference, SSHing to the instance requires that the EC2 key pair be applied to the environment through the Security settings; it’s likely that this can also be done via the CLI.

Checking the EC2 control panel for the instances will give the hostname to use for SSH login; just change the path to the SSH key that has been uploaded to AWS.

$ ssh -i ~/.ssh/private-key ec2-user@ec2-pub-ip-addr-ess.eu-west-1.compute.amazonaws.com

There are  couple of ways of applying the custom configuration needed to restrict access to the editable resources but the method I settled on was by adding the content to the .ebextensions/options.config with an entry in a section called files:

option_settings:
  aws:elasticbeanstalk:application:environment:
    SECRET_KEY: ChangeMe
  aws:elasticbeanstalk:container:python:
    WSGIPath: crossword_hints.py

files:
  /etc/httpd/conf.d/xword-hints-deny.conf:
    mode: 0644
    content: |
      <LocationMatch "/(crossword-solutions|crossword-setters|setter-types|solution-types)/[0-9]+/(edit|delete)">
        Require all denied
      </LocationMatch>
      <LocationMatch "/(crossword-solutions|crossword-setters|setter-types|solution-types)/new">
        Require all denied
      </LocationMatch>
      ErrorDocument 403 /static/403-xword-hints.html

It’s important to ensure that the indentation is correct for the file definition and content; the following deployment error will be thrown if not:

Service:AmazonCloudFormation, Message:[/Resources/AWSEBAutoScalingGroup/Metadata/AWS::CloudFormation::Init/prebuild_0_crossword_hints/files//etc/httpd/conf.d/xword-hints-deny.conf] 'null' values are not allowed in templates

The application needs to includes the 403 document, 403-xword-hints.html, because the web server will pass the request for the custom error page to it as a normal HTTP request.

With all this in place, the application is reasonably safe to leave running on the internet with any attempt to create, edit or delete content yielding a permissions error.

And the updates are still be applied by a Jenkins job pulling branch code from GitHub.