Category Archives: DevOps

Restricted access to EB instance

Almost embarrassed to admit that I spent much of the day trying to figure why my attempts at applying a custom nginx configuration scheme to block access to the editable content on my test site at were failing because the default Python instance actually runs Apache httpd!

There was certainy enough evidence in the logs but as soon as I figured out how to SSH to the instance, it didn’t take long.

For reference, SSHing to the instance requires that the EC2 key pair be applied to the environment through the Security settings; it’s likely that this can also be done via the CLI.

Checking the EC2 control panel for the instances will give the hostname to use for SSH login; just change the path to the SSH key that has been uploaded to AWS.

$ ssh -i ~/.ssh/private-key

There are  couple of ways of applying the custom configuration needed to restrict access to the editable resources but the method I settled on was by adding the content to the .ebextensions/options.config with an entry in a section called files:

    SECRET_KEY: ChangeMe

    mode: 0644
    content: |
      <LocationMatch "/(crossword-solutions|crossword-setters|setter-types|solution-types)/[0-9]+/(edit|delete)">
        Require all denied
      <LocationMatch "/(crossword-solutions|crossword-setters|setter-types|solution-types)/new">
        Require all denied
      ErrorDocument 403 /static/403-xword-hints.html

It’s important to ensure that the indentation is correct for the file definition and content; the following deployment error will be thrown if not:

Service:AmazonCloudFormation, Message:[/Resources/AWSEBAutoScalingGroup/Metadata/AWS::CloudFormation::Init/prebuild_0_crossword_hints/files//etc/httpd/conf.d/xword-hints-deny.conf] 'null' values are not allowed in templates

The application needs to includes the 403 document, 403-xword-hints.html, because the web server will pass the request for the custom error page to it as a normal HTTP request.

With all this in place, the application is reasonably safe to leave running on the internet with any attempt to create, edit or delete content yielding a permissions error.

And the updates are still be applied by a Jenkins job pulling branch code from GitHub.

AWS ElasticBeanstalk custom environment variables

As a holiday project I’ve been looking into using Jenkins to deploy code updates from GitHub into an Amazon AWS ElasticBeanstalk instance[1] as an early attempt at some sort of continuous delivery.

One of the features of the Flask application is that it tries to get the SECRET_KEY from an environment variable (although the code for a failsafe value doesn’t work: FIXME). The intention is that the web server environment provides the key at runtime so that different values can be used in each environment.

Now, this AWS page describes the format of the options to apply custom environment settings to an application (the name of the actual file doesn’t matter so long as it is called .config and is found in the .ebextensions directory in the uploaded code):

    SECRET_KEY: ChangeMe

Setting the WSGIPath variable means that I can continue to use the original application source file rather than change to the default

This file can safely be kept in the GitHub repo and setup as a simple shell build step in Jenkins prior to the code upload, thus:

SECRET_KEY=`openssl rand -base64 12`; sed -ie "s/ChangeMe/${SECRET_KEY}/" .ebextensions/options.config

Jenkins has a great AWS EB deploy plugin that uses stored credentials to mange the source bundling, upload and deployment of the application; it’s kinda strange seeing the AWS console page spring into life in response to the Jenkins job running. To save having to include the build shell step, I’m thinking of creating my own version of the plugin that allows the inclusion of custom variables.

[1] – As a development instance the application will be mostly terminated (and offline) because AWS is a very expensive way of running a bit of demo code.


Express yourself

I am still committed to the idea of the workplace as somewhere for creative expression.

Not artistic or musical or anything like that, and perhaps intensely corporate, but where individuals are afforded the opportunity to express and develop their ideas. And through this comes true personal development.

Over the years I have seen that whenever members of  team are given the freedom and autonomy to develop solutions to problems, the end results are of a higher quality.

And any time I hear a (micro) manager say that you can’t have everyone going off doing their own thing all the time, I know that that is someone who seeks control and actively tries to inhibit realisation of potential.

Using process to support the team

One of the early joys of my new job is the use of Jira to manage the change control process.

This matters because Jira is quite unobtrusive and fits in naturally with the development process.

Compare this against a previous employer with as bloated a process as could possibly be imagined (there was a category for a P3 emergency) and where low-impact had to wait at least 2 days and with the expectation that teams change how they go about their work to support the process: the process is more important than getting stuff done.

And the managers wonder why the teams hated it.


Let yourself be impressed

Quite often when we’re with junior or less-experienced members of a team, we’ll ask them to carry out an apparently simple task and because we consider it trivial or inconsequential we don’t feel it necessary to say ‘well done’ or ‘thank you’.

But bear in mind that the junior team member doesn’t think of the task as of no consequence and will consider that completion as an achievement and appreciation and recognition indicates that they are making a genuinely valuable contribution to the team effort.

Any junior team member will want to impress so any opportunity that arises where they notice or say something novel must be embraced, no matter how trivial it may appear to be because it helps reinforce the collaborative nature of team work and will mean everyone continues to look for ways to move forward and improve on what they are doing.

Explicit exclusions

When deciding what tasks need to added to the delivery of a particular iteration of whatever is being worked on it’s often easy to overlook what won’t be included and it becomes tempting to add that nice simple little feature.

But this is scope-creep and it needs to be avoided at all costs. The best way to avoid this risk is to have a list of items that are definitely not going to be included. They might be for inclusion in the next iteration but any time that there’s the temptation to include that feature there’s a reminder (and preferably with annotations) to explain why it is not to be worked on right now.

This will help to maintain focus on the most important project goals and ensuring that we’re sticking to the design. document, deliver principle.

Design. Document. Deliver.

There is only one way to have a fully maintainable and supportable system: document what has been designed and implement what has been documented.

Anything other than should be considered as having its long-term supportability and maintainability compromised.

Now, it is sometimes inevitable and necessary that systems be put in place that have not been properly documented before build time. That’s okay, but what is required in these circumstances is acknowledgement and agreement up front that ideal practice is not being followed and  compromised system is the result.

I worked on a high-profile (and high-stress) project where comments from the managers were often like: ‘don’t worry, junior admin that’s working way beyond experience and expectations will have documented all the components’. That upsets me. Not only are the (supposedly) experienced managers absolving their own responsibility they are heaping pressure where it least deserves to be.

By agreeing up front that the expectation is for a system that cannot (and will not) be fully documented we are making an effort to reduce the stress on the implementer who already has the unpleasant task of trying to create complex applications on the fly (and inevitably feeling that they’re making the best of a bad job) by saying that there’s a shared responsibility when it comes to the later questioning of why nothing matches up.

Sweating the small stuff

In a previous role, in preparation for a major datacenter migration project, I figured that there were a ton of really small mini- or micro-projects that we needed to get sorted before we’d be in a position to undertake the big stuff: migration of filestore from Netware to Windows, internal app upgrades, non-disruptive re-platforming, the kind of things that were often talked about in the office for half an hour and then lost to the ether, or, jobs that were important to a particular team member, that would have been started but lose momentum when the bigger stuff takes over.

Realising that these small-scale projects are a genuine way for the team to deliver incremental improvements, rewarding in their own right, I figured that why they usually stopped or lost was because they were no actual targets, that the individual knew what was required and didn’t need to write any of it down.

Also, knowing that techies are lousy are keeping track of their ideas, I asked the powers that be for some low-key project management assistance, not a formal project, but keeping tabs on the wee items that need to be done, most of which wouldn’t appear on the radar of a proper project.

Because the PM wasn’t really interested in tracking it ‘properly’ (because it was a proper project, had no budgetary implications, no vendor managements, or contracts to negotiate) he put as little time and effort into as possible, just checking up on the progress the devs and infrastructure were making on their areas. And it was just two or three people, agreeing on a set of tasks to be done whenever they got done.

But after a while, I noticed that all he was doing was getting agreement on: what needed to be done, who was going to do it and when it should be done by; he was an independent (and uninterested arbiter).

All the tasks were simple and achievable, but having an agreed list where each part needed to be ticked off at regular brief meetings was key. I already appreciated the need for actively tracking tasks (I tried using a now-defunct tool called todoyu) but trying to get the team to manage their own work like this just doesn’t work; that external guide was the key.

And it delivered. Not everything, of course, but enough to make a difference; the stuff that didn’t get done that way turned out to be much more complex and needed to be a project in its own right, but this becomes obvious.

To me, there are two real benefits of an approach like this,

  • stuff gets done,
  • there is a record of stuff getting done

When a team is working toward a strategic objective there are many small items that get in the way or get forgotten about in the flood of important matters, but in the end it all counts.

By the end of next week

Just a recap of a one-to-one meeting I had a little while back with a colleague and an on-the-spot thought experiment that came to mind.

I am well known for arranging even apparently small pieces of work as a project and how it is essential to break them down into bite-sized, well-scoped tasks.

As part of the planning exercise I asked my colleague how long she thought it would take to complete task X.

By the end of next week was the reply (Aside: is it just me or do most people think the end of next week is a reasonable length of time to complete a discrete piece of work?).

Now, my colleague is a working mum and only (I say ‘only’) works for 25 hours a week. Having heard ‘the end of next week’ so many times triggered a quasi-mathematical thought in me and it goes like this:

  • You have 25 hours per week for work, so by the end of next week you have a total of 50 hours available,
  • assuming a 20% utilisation rate for this particular task gives a completion allowance of 10 hours,
  • so, if you started it now and worked on nothing else (chance would be a fine thing)  would you have it completed by the end of tomorrow – the next 10 hours?
  • No.
  • Then the end of next week isn’t long enough. Allow another two weeks.

The consequence of this type of thinking is that this simple seeming little task that you’ll be squeezing in with everything else is actually going to take a month to complete.

Now, this idea makes people really uncomfortable but the reality is that it actually does that long to complete these pieces of work – I’ll discuss soon of my ideas around working inefficiencies that lead to this later – and that accepting this position we (can hope to) remove some of the pressures we experience when we don’t seem to be making much progress toward the goal and enter the panic spiral as we get closer to deadline.

This idea is intended to be a way of self-analysing expected effort and estimating/guessing whether it might be achievable by the expected deadline. This is also about allowing people to brave with their assessments and not afraid to suggest something that might ordinarily appear to be unreasonable.

As a follow-on from this, there will be situations where the PHB says this isn’t acceptable and the item needs to be delivered sooner. Well, in this case it can then be seen that something else will have yo make way to free up time for the task. This is agreed up front so everyone is at the same level of expectation. This should help contain the stress on the task implementer believing they are expected to complete a task in addition to normal activities. A too-short delivery window can increase the pressure in a way that worrying about completing the task on time detracts from the task itself and leads to a vicious circle.

I include this under the DevOps category because it is entirely cultural  and experience suggests many organisations don’t work this way and will need to change their approach to people in allowing them the space to challenge existing ideas.