Category Archives: Linux

Python Flask activity logging

This post describes a simple pproach to recording login, logout, insert, update and delete operations that are carried out by a Python Flask application.

This is not the same as a table auditing function similar to Ruby’s Paperclip module.

https://github.com/slugbucket/crossword-hints/issues/9 is an example feature branch for a personal project implementing this.

The activity log will be stored in a database table with the following structure:
* rowid – auto-assigned unique id for the activity record
* actor – the name of the user performing the action
* action – one of login, insert, update, delete or logout,
* item_type -the table on which the operation has been performed.
* item_id – the numeric id of the item under operation
* act_action – details of the content that has been changed

This can be modeled in Flask using PeeWee

...
from peewee import *
from datetime import date, timedelta, datetime

...

class BaseModel(Model):
    with application.app_context():
    class Meta:
        database = database

    class activity_logs(BaseModel):
        rowid = AutoField()
        actor = CharField(max_length=32)
        action = CharField(max_length=32)
        item_type = CharField(max_length=32)
        item_id = IntegerField()
        act_action = TextField()
        created_at = CharField(max_length=32)
        updated_at = DateTimeField(default=datetime.now())

The application context is called application to work better when deployed to AWS.

Having used Flask-login to control the login process. we have a variable. current_user, that contains a numeric id of the user from their record in teh database.

The users model includes a method to return the name of the user and this can be used for the ‘actor’ value.

A simple function can create the activity log reord:

def add_log(actor, action, item_type, item_id, activity):
    log = activity_logs(actor=actor,
          action=action,
          item_type=item_type,
          item_id=item_id,
          act_action=activity,
          created_at=datetime.now(),
          updated_at=datetime.now())
    log.save()

And activities to be logged can be logged with something like,

log = ("name: %s\ndescription: %s" % (name, description))
add_log(users.get_name(current_user), 'update', 'solution_types', id, log)

Where,

  • name and description are variables containing form-submitted data
  • activity_logs is the PeeWee model of the databse table
  • users is a PeeWee ORM model of an application user that can be authenticated.

When used the (sqlite3) database records will look like,

sqlite> select * from activity_logs;
crossy|login|user|1|Successful login for crossy|2019-02-15 18:44:38.257050|2019-02-15 18:44:38.257061
crossy|update|solution_types|12|name: Homophone
description: Sounds like a word that has a different spelling|2019-02-15 18:45:03.227188|2019-02-15 18:45:03.227193
crossy|logout|user|1|Successful logout user for crossy|2019-02-15 18:48:11.043243|2019-02-15 18:48:11.043249

With these records it should be possible to display an activity report and perhaps even recover previous versions of particular database records.

Improvements

  • Storing the full record on each update is inefficient; only differences should be recorded.
  • The logging function should be able to determine what to include in the log message based on the
Advertisements

Configure Python Flask application for LDAP authentication

Using a crossword-hints Flask application to include authentication for adding, modifying and deleting database content.
We will use ad a users table to the database and hand off the authentication to the directory.

Install the python-ldp pip (3.1.0 as of February 2019)

$ pip install python-ldap --user

Install the Flask-login pip (0.4.1 s of February 2019)

$ pip install flask-login --user

Include the modules in the Flask application

from flask_login import LoginManager, UserMixin, login_required, login_user, logout_user
import ldap

Configure Flask-login in the application

# flask-login
login_manager = LoginManager()
login_manager.init_app(application)
login_manager.login_view = "crossword_login"

The login_view indicated the function to be called to handle the user login.

Flask-login also requires a user_loader function that returns an integer id value for the user,

@login_manager.user_loader
def load_user(id):
    return users.get(users.rowid == int(id))

Add a function to get a connection to the directory (but no authentication at this point)

def get_ldap_connection():
    conn = ldap.initialize('ldap://localhost:389')
    return conn

Add a route to the login form. This accepts GET and POST requests to display the form and then
process the submitted form. Notice that after successful login there is a redirect to a page passed as the next parameter in the query string when an application route requires login: this could be a securiity concern if steps are not taken to prevent the application accepting a spoofed resource as input rather than a permitted resource in the intended application.

@application.route("/login", methods=['GET', 'POST'])
def crossword_login():
    try:
        if current_user.is_authenticated:
        flash('You are already logged in.')
        return redirect(request.path)
    except:
        pass

    if request.method == 'POST':
        username = request.form.get('username')
        password = request.form.get('password')

        try:
            users.try_login(username, password)
        except ldap.INVALID_CREDENTIALS:
            flash(
                'Invalid username or password. Please try again.', 'danger')
            return(render_template('views/login/login.html', u=username, r=request))

        user = users.get(users.username == username)
        login_user(user)
        return redirect(request.args.get("next"))
    else:
        username = 'username'

    return(render_template('views/login/login.html', u=username, r=request))

A template for the login form could be

{%- import "macros.html" as f -%}
Use this form to login to the site to maintain restricted content.
</p>
<form name="crossword_login" action="{{ r.full_path }}" method="POST">
<fieldset id="crossword_login">
<legend>Login</legend>
{{ f.label('username', 'Username') }}
{{ f.input_tag('username', value=(u|escape()|default('')), class="xwordhints") }}

{{ f.label('password', 'Password') }}
{{ f.input_tag('password', type="password", class="xwordhints") }}
</fieldset>

<p>
<input type="submit" name="login" value="Login" />
<input type="reset" value="Reset" />
<br /><br />
<a href="/crossword-hints">Back to index</a>
</p>
</form>

macros.html (there’s a good GitHub gist with some macros but I can’t find a link to it) contains helpers to create the form input elements. The form action
uses the request’s full path which includes the query string containing the next parameter.

Provide a logout route. Note that this requires a valid login. Going direct to /logout will show a prompt to login!

@application.route("/logout")
@login_required
def logout():
    logout_user()
    flash("Logout successful. Please close browser for best security.")
    return(redirect("/"))

A PeeWee user model

class users(BaseModel):
    rowid = AutoField()
    username = CharField(null=False, max_length=32, unique=True)
    created_at = DateTimeField(default=datetime.now())
    updated_at = DateTimeField(default=datetime.now())

    @staticmethod
    def try_login(username, password):
        conn = get_ldap_connection()
        conn.simple_bind_s(
            'uid=%s,ou=People,dc=my-domain,dc=com' % username, password
        )

    def is_authenticated(self):
        return True

    def is_active(self):
        return True

    def is_anonymous(self):
        return False

    def get_id(self):
        conn = get_ldap_connection()
        return(self.rowid)

    def get_name(self):
       return(self.username)

The get_name method is used by HTML templates to display the logged in username. For authentication the try_login method just attempts to bind to the directory as the user.

{% if current_user.get_id() is not none %}Logged in {{ current_user.get_name() }}&nbsp; <a href="/logout">Logout</a>{% endif %}

Routes that need authentication (and authoristion) before access simply need to state ‘login_required):

@application.route("/crossword-setters/<int:id>/edit", methods=["GET", "POST"])
@login_required
def crossword_setters_edit(id):

Accessing this route will redirect to the login form and successful login will redirect back to the edit page.

Unit tests

When requiring authentication to enable certain operations in the application, consideration needs to be given to the impact on running unit tests.

The protecting views section of the Flask-login site indicates that this is simply a matter of setting the following option:

LOGIN_DISABLED=True

which disables the login_required decoration applied to routes. Adding this to a settings file used when running the tests allows everything to complete as expected.

APP_SETTINGS='test-settings.py' python crossword-hints-tests.py

 

Improvements

  • Don’t use the SQL database to store the users
  • Control authorisation according to group membership.
  • Prevent authentication leakage via spoofed ‘next’ resources.
  • Don’t request a login when going direct to the logout route.

References

ArchLinux OpenLDAP setup for webapp authentication

Setting up a simple OpenLDAP directory to allow authentication from a web (or other) application.

Based on https://wiki.archlinux.org/index.php/OpenLDAP

Configure LDAP service

Install openldap packages

Create a salted hashed password for the directory manager:

$ slappasswd -h {SSHA}

Enter and confirm a password and receive output such as

{SSHA}c920AmsQ9Evay0YaCU/r0GAdnMroyL4O

Configure /etc/openldap/slapd.conf to use the following:

database mdb
suffix "dc=my-domain,dc=com"
rootdn "cn=Manager,dc=my-domain,dc=com"
...
rootpw {SSHA}c920AmsQ9Evay0YaCU/r0GAdnMroyL4O

As the superuser, setup the directory database and start the service,

# cp /var/lib/openldap/openldap-data/DB_CONFIG.example /var/lib/openldap/openldap-data/DB_CONFIG
# slaptest -f /etc/openldap/slapd.conf -F /etc/openldap/slapd.d
# slapindex
# chown -R ldap:ldap /etc/openldap/slapd.d /var/lib/openldap/openldap-data
# systemctl start slapd

Test connection,

$ ldapsearch -D "cn=Manager,dc=my-domain,dc=com" -W -x '(objectclass=*)'
Enter LDAP Password:

Create base directory, users and groups

Create an LDIF file, say, base.ldif, for the directory manager and base groups (whether or not the groups are used by the application is another matter):

dn: dc=my-domain,dc=com
objectClass: dcObject
objectClass: organization
dc: my-domain
o: My-domain
description: My-domain directory

dn: cn=Manager,dc=my-domain,dc=com
objectClass: organizationalRole
cn: Manager
description: Directory Manager

# People, my-domain.com
dn: ou=People,dc=my-domain,dc=com
ou: People
objectClass: top
objectClass: organizationalUnit

# Groups, my-domain.com
dn: ou=Groups,dc=my-domain,dc=com
ou: Groups
objectClass: top
objectClass: organizationalUnit

# xword-admins group
dn: cn=xword-admins,ou=Groups,dc=my-domain,dc=com
objectClass: top
objectClass: posixGroup
gidNumber: 10000

# xword-users group
dn: cn=xword-users,ou=Groups,dc=my-domain,dc=com
objectClass: top
objectClass: posixGroup
gidNumber: 10001
$ ldapadd -D "cn=Manager,dc=my-domain,dc=com" -W -x -c -f /path/to/base.ldif

(-c allows for repeated runs of the same LDIF as typos and errors are wrinkled out)
Enter the password when prompted and check that the OUs and groups are created with:

$ ldapsearch -D "cn=Manager,dc=my-domain,dc=com" -W -x '(objectclass=*)'

Add groups and users

Generate a hashed password for each user, to be used in the subsequent LDIF file,

$ slappasswd -h {SSHA}

Create another ldap, say, users.ldif. contining something like,

dn: uid=xword,ou=People,dc=my-domain,dc=com
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: shadowAccount
uid: xword
cn: Crossword Hints
sn: Hints
givenName: Crossword
userPassword: {SSHA}V7i9IYaW4Z4Z2mVjbtOk0PcinzgJof9y
labeledURI: http://xword-hints.my-domain.com/
loginShell: /bin/bash
uidNumber: 9999
gidNumber: 9999
homeDirectory: /home/xword
description: Crossword hints website user

# Add xword to the LDAP xword-admins group
# group
dn: cn=xword-admins,ou=Groups,dc=my-domain,dc=com
changetype: modify
add: memberuid
memberuid: xword

# Add xword to the LDAP xword-admins group
dn: cn=xword-users,ou=Groups,dc=my-domain,dc=com
changetype: modify
add: memberuid
memberuid: xword

Then add the users to the directory

$ ldapadd -D "cn=Manager,dc=my-domain,dc=com" -W -x -c -f /path/to/users.ldif

Check the users can authenticate through the directory,

$ ldapsearch -D "uid=xword,ou=People,dc=my-domain,dc=com" -W -x '(objectclass=*)'

This will probably show the entire directory contents; this can be tightened up as needed.

The directory is now ready for user authentication.

Troubleshooting

Reset the password described at https://www.digitalocean.com/community/tutorials/how-to-change-account-passwords-on-an-openldap-server although the Arch LDAP server does not have the EXTERNAL mechanism enabled by default; it’s probably easier to rebuild the directory.

The pain of Nagios and Arch Linux

Or, there’s more to systemd than just service files.

I recently had cause to investigate writing a simple Nagios http monitor and although I’ve worked in and around Nagios I’ve not really had cause to try writing the monitor scripts.

There was an aspect of the task I wasn’t too sure about so I thought the best way to get to grips with fully understanding  the environment would be to try and get Nagios running on my desktop system.

Moral of the story: there’s a reason why Arch will never be used in an enterprise environment.

First, I should have been warned off by seeing that nagios for Arch only comes via the AUR.

But, there’s a reasonably detailed page describing it. The real problems came with nginx, php-fpm and fastcgi.

The php-fpm service isn’t enabled or started by default: systemctl soon fixes that.

For some bizarre reason, it turned out that although nginx had been configured to use /usr/share/nagios/share and the root directory for PHP scripts, it was trying to access PHP scripts in /etc/nginx/html/nagios. I had to cheat here and just created a symbolic link between the directories which got the basic page displaying.

But then to get the CGI scripts running I had to look more closely at the Arch nginx configuration for fcgiwrap and having seen that the fcgiwrap service wouldn’t run, I finally twigged that nginx was referring to a .sock file and that there was a systemd socket file.

So I now have Nagios running on Arch. Mostly.

Tomorrow I need to figure out the cause of these errors:

Error: Could not read object configuration data!

But at least we’re now dealing with Nagios problems rather than Arch.

References

Places of interest

 

htpasswd without Apache

If wanting to restrict access to website content under nginx but don’t want to install Apache use the following to generate an htpasswd file

printf "USER:$(openssl passwd -apr1 P@55w0rd)\n" > /etc/nginx/auth/htpasswd

Then use the following nginx rules to

location /api {
    satisfy all;    

    allow 127.0.0.1;
    deny  all;

    auth_basic           "Administrator’s Area";
    auth_basic_user_file /etc/nginx/auth/htpasswd;
}

 

References

Git: stop tracking a tracked file

There are times when a project needs to include the default version of a file in the git repository that will subsequently change to support development (e.g., secret application key or sqlite3 database).

After committing the safe  default copy, any subsequent changes to the files will appear as a modified file in ‘git status’ reports and will also prevent any git flow feature finish operations if the files are not staged for commit.

Adding the files to .gitignore makes no difference.

There is a way, however, to tell git to stop tracking the file,

$ git rm --cached file1 file2

Taken from a StackOverflow posting.

Arch Linuxdeveloper signature errors

Every now and then when applying updates to my Antergos (Arch) Linux desktop I get an error like

$ sudo pacman -Syyu
...
error: libvirt: signature from "Christian Rebischke (Arch Linux Security Team-Member) <Chris.Rebischke@archlinux.org>" is unknown trust
:: File /var/cache/pacman/pkg/libvirt-4.6.0-3-x86_64.pkg.tar.xz is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n]

Accepting the default option means the update fails. If using the GUI client there’s just a message saying the update failed with no further explanation

It is fixed by

$ sudo pacman -S archlinux-keyring

This kind of error is one reason why I’d never consider using Arch in a work environment.

Reference: https://www.reddit.com/r/archlinux/comments/900cxa/upgrading/