JW.PE's RSS Feed

Establishing A Career Framework

Tue, 04 Aug 2026 19:05:00 GMT

One of the projects that I've now led twice - once at TravelLocal, and once, more recently, for a client - is the establishment of a software engineering career framework. Nowadays, career frameworks for engineering organisations are ubiquitous, especially in larger teams. However, when I started my software engineering journey in the early 2010s, they had not yet been popularised. I distinctly remember the first time I encountered one, reading this blog post by Camille Fournier about her experience introducing an engineering career framework at Rent The Runway. More than anything, I was impressed by the intent behind the framework - providing clarity to team members wishing to progress in their roles.

In the 10+ years since then, examples of career frameworks have proliferated, with plenty of high-profile examples in the public domain - GitLab's and CircleCI's are two that I've referenced myself in the past. There's even a whole website, https://progression.fyi/, dedicated to sharing open examples. In that time, I've also come to believe that a well-designed career framework is a key tool in successfully growing an engineering org.

What is a career framework?
Why do we need a career framework?
How do we get started?
Embedding the career framework

What is a career framework?

To put it simply, a career framework is a document or tool that outlines the expectations for each role in your engineering organisation. These expectations are usually defined as a set of behaviours or competencies that are expected to be fulfilled by a person working in that specific role. Unlike a job description (more on that later), a career framework is a living document that grows and evolves with your organisation.

Why do we need a career framework?

It can be tempting to think that establishing a career framework is unnecessary work when your team is small - and if you're a handful of engineers, you're probably right. However, a timely introduction of a career framework can make your life as an engineering leader significantly easier.

To me, a career framework solves three core challenges for your team. These are the challenges of clarity, consistency, and ambition.

Clarity

In many jobs, your job description is the only written record of your expectations in the role. However, job descriptions are designed for a specific purpose - hiring - and so are not optimised for people already in the role. They become outdated as soon as the role is filled. What this means is that starting from day one, a team member has a growing gap around the question "What is expected of me?"

Good managers will regularly fill this gap in 1:1 conversations, performance reviews, and by defining clear goals for their team members. However, in all of these cases, the context for expectations is largely in the heads of manager and team member. If, as an engineer, I want to clearly establish what I need to do to succeed, I have no quick point of reference. If, as a manager, I want to evaluate the progress and contributions of my team, I have no easy and consistent method of comparison.

A career framework solves this issue by taking the majority of this context, and making it explicit. A document can't - and shouldn't try to - replace the relationship between team member and manager. But by establishing an agreed-upon set of behaviours for each role, it provides a shared understanding and point of reference to answer the key question of "What is expected of me?" This gives agency to team members to own their own performance and progression, and provides a fair and equal baseline for managers to evaluate the success of their team.

Consistency

Fairness leads directly to the second key challenge answered by a career framework. Establishing a shared understanding of what is expected of team members in each role means that everyone is being evaluated against the same, known behaviours. This can go a long way towards preventing a situation in which success in the organisation is predicated purely on the whims of your manager. Nobody wants to work in an organisation where people succeed or fail thanks to some opaque system of assessment that involves pleasing certain key figures - and yet, many orgs operate this way. By putting the expectations out in the open - and using them in processes such as hiring, performance management, and promotions - team members can feel confident that they are aiming for the same benchmark as their peers.

"But I judge the performance of my team fairly and objectively!" Sorry, but you probably don't. Unconscious bias is a real phenomenon, and no matter your intentions, it's all too easy to see one person's confident decision-making as another's reckless risk-taking. Establishing clear behaviours beforehand goes some way to eliminating these inconsistent perceptions of the same actions across your team. This is particularly relevant for team members from minority backgrounds, who are disproportionately negatively impacted by opaque hiring or promotion processes.

Ambition

People enjoy challenge and progression in their work. By clearly documenting expectations at each level of your engineering organisation, a career framework gives team members a signpost for what they could achieve within the org. This allows them to own their self-development in a structured way. For example, perhaps a key behaviour for Senior Engineers in your organisation is technical leadership of projects. An aspiring engineer might choose to focus on developing this behaviour by shadowing an existing tech lead, or proactively requesting to take on the tech lead role for a smaller project.

Without clarity around the roles that exist in the org and what they entail, ambitious team members may feel that their opportunities to grow within the team are limited. This can lead to challenges with retention as high performers seek more well-defined opportunities by taking another job.

With a career framework in place, conversations with high-performing team members can focus on what behaviours they should develop, not whether opportunities for growth in the team exist.

How do we get started?

Hopefully I've convinced you of the value of establishing a career framework for your team. So how do we go about doing that?

Wait until you need it

It can be tempting to think that before you start hiring for your fledgling engineering team, you need to have a career framework in place. In my experience, making a career framework too early is almost as bad as not making one at all. For one, it may be wasted work - in an early-stage organisation, nothing is guaranteed, and you could almost certainly be focussing on a higher-leverage activity that would increase the org's chances of success. Secondly, you simply won't have enough information to clearly define the potential roles in your team. While some behaviours are fairly consistent across organisations, the culture and best practices of your team, and the specific goals of your org will significantly affect what 'good' looks like. Trying to anticipate this before you've even hired your first team member is unlikely to produce anything that has meaning and relevance once your team has grown and established its ways of working.

A better time to make a career framework is when you are starting to feel the friction of not having one. Once you have a few engineers who are working effectively, and the core of the team is established, your focus may turn to developing your existing team or expanding the team with new skills. At this point, questions like "How can I grow in my role?" or "How do we fairly evaluate these candidates for a new hire?" may come to the fore. This is a great time to make a first pass of your framework, knowing that it will have immediate applications.

Keep it simple

A good career framework provides enough information to give clear direction to the team, but not so much that it's overwhelming. What that looks like will be different depending on your team size and seniority mix - a team of five engineers with around the same experience needs way less detail than a team of 100 engineers with a wide variety of experience levels. But in all cases, try to keep things a simple as possible. A framework with tens of behaviours at each level simply won't be read and used by the majority of your team.

To be concrete about this, when you're starting out, make a short list of key behaviours for your most common role. These should cover the core activities and responsibilities of the job. If most people are Senior Engineers, define that role first, as you'll have the most actual experience to draw on in terms of what is working and what isn't in your team. Structure the behaviours as clear, one-or-two sentence statements, for example:

I deliver high-quality, readable code that follows the team’s conventions. I seek feedback through pairing and reviews to improve the quality of my contributions.

Behaviours should be things that people do! The framework should consist of actions that it is possible to clearly demonstrate, not intangibles like 'has good architecture sense'.

Once you've established a baseline of key behaviours for your most common role, sense check them with those people actually doing that job. Do the behaviours reflect their experience of their day-to-day work? If not, why not? Iterate until you've got something that feels true of your org as it is now. Only then should you start to deepen and expand.

Copy and customise

As mentioned above, you'll need to have an understanding of your team as a living, working entity to be able to make a useful career framework. However, that doesn't mean that you have to start completely from scratch. Unless you're working on something truly unique, or working in a radically different way from every other engineering team out there (and you probably aren't), the behaviours needed to be an effective software engineer at your organisation are likely to overlap significantly with those at other orgs.

Take some time to read through the existing state of the art, looking at both industry-leading orgs for inspiration, but also for orgs that are more reflective of your scale and maturity. If you see behaviours that reflect the best practices you feel should exist in your team, feel free to incorporate them. Think about the categories of behaviour that are covered by existing frameworks - does your framework broadly cover the same main aspects of everyone's job?

While you may need to customise behaviours to fit into the context of your team, drawing on existing open frameworks will give you a good sense of whether your framework is covering the correct bases.

Aim for good enough, then evolve

As mentioned above, a career framework should be a living document. The documented behaviours should reflect the expectations of the team you have now, not the team you want to have when your org has grown to 1000 engineers. With this in mind, challenge yourself to get something simple and meaningful in front of your engineers as soon as possible. By starting with your most common role, you can validate the approach early and see if it resonates with your existing team members.

Once you've got the framework out in the open in the team, actively solicit feedback while building it out to cover all of the existing roles in your team. For small teams, this might only be two or three roles: Engineer, Senior Engineer, and Staff Engineer, for example. Only once you've got clear expectations for each existing role should you start thinking about extending the framework to cover aspects of your hypothetical org chart. Try to link behaviours from one level to the next - behaviours at Senior level should be a natural progression from those at Engineer, and so on. Some behaviours will be the same in multiple roles, and that's fine!

You can evolve your framework in several ways as the situation demands. Maybe you've hired your first engineering manager - this could be a good time to split your framework into distinct Individual Contributor and Management tracks. Or you might find your most senior engineer is going above and beyond what's defined in their role; this could be a signal to introduce a Principal Engineer role for them to aim for. Perhaps you find yourself in a situation where you have lots of team members at the Engineer level, and some clearly have greater responsibilities than others. Maybe you need to start considering if you need multiple intra-role levels (e.g. E1, E2).

I personally find it helpful to write a short accompanying document alongside the career framework itself, explaining where and how it will be used. This explainer will also evolve as you embed the framework in more aspects of your process.

Finally, as time goes on, what behaviours are expected of each role will likely evolve as the team does. It's good to build in a process to review the framework on e.g. an annual basis, to ensure it's still relevant for the team. The best way for a framework to become obsolete is to let it diverge too far from the reality of your team.

A note about salaries

I am strongly of the opinion that you should align salary bands within your org with your career framework. It simplifies a whole class of difficult management problems around fair compensation if you can confidently state that everyone in a certain role is within a given salary band. Whether you choose to transparently surface these bands alongside the roles, how wide the bands are, and whether the bands between different roles overlap - all of these will depend on your own situation, constraints and preferences. But whether or not the bands are visible to the team, being able to comfortably justify to yourself why someone's salary is at a certain level relative to their peers will set you up for productive conversations with team members and prospective hires on the topic of remuneration.

Embedding the career framework

Well done - you've done the hard work of designing and rolling out your career framework. Now you have to use it! In future posts, I'll cover three areas where you can make the career framework a core part of your process: hiring, performance management, and professional development.

Until then, if you have any thoughts about the post, or questions about establishing your own career framework, drop me an email at work at jw.pe.

October 2025 Good Things

Mon, 01 Dec 2025 12:58:00 GMT

Better late than never on my October good things!

Getting Married!

The main reason that I didn't write this post at the end of last month was that my (now) wife and I were busy getting married! It was absolutely amazing. I felt incredibly lucky: not only to be marrying someone as wonderful and well-suited to me as my partner, but also to be able to do so in the presence of so many people we love.

There's so much I could say about the day, and so many incredible memories I will cherish for years to come, but for the moment I am just feeling grateful. The love that we felt from friends and family alike was in some ways overwhelming - it's not often that you're the at the centre of attention of everyone you care about - but ultimately very validating. It really reminded me how much I value my relationships, and how investing in them is one of the most worthwhile things I can do.

Sparks Co-Working

While getting married was very much the priority in October, I did make another positive change during the month - I started working out of a co-working space! I've been wanting to do this for some time - working from home every day for more than three years has been quite isolating, and terrible for my focus. I will always advocate for flexible and remote working, as I think people should have the option to work from wherever makes the most sense for them. With that said, losing the ability to work from the office during COVID deprived me of a healthy routine of getting out of the house, getting some fresh air, and getting some free exercise on the way to work.

I've finally rectified this by signing up to work at Sparks Bristol. For anyone who isn't familiar - Sparks is a re-imagining of a department store in the old Marks & Spencer building in Broadmead. Instead of being run by a single company, the building is managed by two charities, and the ground floor is occupied by a whole bunch of small retail businesses, charity installations, and the like. It's a great place to come and shop for art, jewellery, clothing, or homeware from local independent businesses. However, until this month, I wasn't aware that Sparks also offered desk spaces and artist studio spaces on the second floor of the building. Fortunately, my friend and fellow consultant Sim was also looking for a co-working space around the same time as me, and invited me to tag along on his visit to the space.

To cut a long story short: we both really liked Sparks, and have signed up for desks. The small team who manage the building are really friendly and helpful, the desks in the White Studio (where I work) are extremely affordable at £120 / month, and the space is charming, quirky, and functional.

It's hard to imagine what working in a cavernous former department store is like until you try it, but it has something of the energy of being in school after everyone has gone home that I remember from occasions in my childhood. It feels somewhat like you shouldn't be allowed to be there, like you're peeking behind the curtain of normal everyday life. It adds an element of fun to coming to work, and because Sparks is very DIY, it feels like there is a lot of flexibility and possibility with how the space can be used.

For the more practically-minded, here are the facilities that Sparks has:

Adjustable-height desks (not quite adjustable enough to be standing desks) and office chairs.
Kitchen with kettle, microwaves, fridges, etc.
Functional heating (important in a relatively old building with single-glazed windows).
Good Wi-Fi - I have done plenty of video calls and had no issues whatsoever with bandwidth.
A secure bike rack indoors, although it does require carrying your bike up stairs, so not great for accessibility considerations.

And things I'm not aware it has, which other, more expensive co-working spaces might have:

Showers / changing facilities.
Dedicated meeting rooms - although the space is big enough that you could easily find a space to have a meeting without disturbing anyone.

Obviously I am not affiliated with Sparks, and these facilities and prices might change, but I am currently really enjoying working here. I would encourage anyone in Bristol looking for co-working with a twist to come join us! You can enquire here: https://sparksbristol.co.uk/hire-a-space/desks/

Running

I've been running twice a week pretty consistently since the middle of last year. As exercise goes, running is something that works well for me: low-cost, gets me outdoors, and I can fit a short run into my work day quite easily. In October I passed my annual goal of 60 runs for the year, which felt satisfying.

Something I really value about my running routine is that it demonstrated my ability to form new, healthy habits. The science of habits is something I'm very interested in, but last year I was in a headspace where positive change felt really difficult, if not impossible. Starting my running habit provided me with concrete evidence that I could make beneficial changes in my life that would stick.

Other Stuff

We saw Australian comedy trio Aunty Donna at the Hippodrome - completely unhinged and hilarious.
I read the final part of Philip Pullman's Book of Dust trilogy, The Rose Field. I devoured it in two sittings, both thanks to the incredible prose, and wanting to discover the ending of a journey starting 30 years ago with Northern Lights. I am still mulling over my opinions on the narrative, but I can say that I found the book compelling and beautiful.

September 2025 Good Things

Tue, 30 Sep 2025 22:46:00 GMT

In my ongoing quest to write more, here are some things I've been enjoying this month. If I can, I'll write one of these each month.

Hollow Knight: Silksong

This game has taken over my free time since I bought it earlier this month. I loved the original Hollow Knight, with its melancholy, brooding atmosphere, beautiful art, and tough but satisfying gameplay. In fact, I replayed it (including beating The Radiance, which I hadn't done before) before I allowed myself to buy Silksong, just to get in the right frame of mind.

Having spent many hours with the game, I can confidently say that Silksong is a worthy successor to the original Hollow Knight. The incredible music and art direction are back, with stunning environments that feel very alive and at times very hostile. The story, focussing on Hornet as the protagonist, ties in satisfyingly with the original game without being limited by it. I won't go into detail to avoid spoilers, but there are also some incredible twists later in the game that will leave you amazed at the sheer ambition of Team Cherry.

The biggest change is the gameplay. While the game follows the tried-and-true Metroidvania formula, the world is significantly bigger than Hollow Knight, and the challenges noticeably harder. Having just replayed the original, I found the first Act of Silksong manageable in comparison, but from Act II the difficulty spikes significantly. While bosses are a bit tougher than Hollow Knight, the real difference in challenge for me came from the environment and regular enemies. Benches (save points) are few and far-between, and some of the compulsory platforming sections are extremely tough, interspersed with high-damage enemies for that extra level of risk.

While the increased difficulty is at times frustrating, the feeling of satisfaction at defeating a particularly powerful boss - or navigating an impossible-feeling platforming sequence - is commensurately higher. Without wishing to spoil anything, I can say that beating the game's final boss left me suffused with adrenaline and feeling triumphant for hours afterwards.

Hollow Knight is one of my favourite games ever, and Silksong is as good a sequel as I could have hoped for, so I'd give it a 9.5/10.

Tame Impala

Everyone's favourite one-man psych rock polymath Kevin Parker is back at it again with a new album, Deadbeat, due out in October. So far there are three singles out, End of Summer, Loser, and Dracula. While the Tame Impala subreddit's reaction has been decidedly mixed - people are mad that the music isn't as exploratory and emotionally resonant as his old stuff - I have personally enjoyed all three tracks so far. Dracula is definitely my favourite, an upbeat dance-pop number about a never-ending party with an extremely catchy chorus.

Parker also did a wide-ranging and interesting interview with GQ in which he talks about his childhood, his creative process, but also his - as far as it appears to me - intense self-criticism and dissatisfaction with his own achievements. For example:

“I know that everyone that sees me and my life and my career thinks that I am the furthest thing from being a deadbeat, because I’m successful,” he says in a rush. “But the more success that I have, the more I feel like I’m living a lie. It’s a sham.”

It's hard to believe that someone who has singlehandedly created a musical vehicle of the scope and level of success of Tame Impala would feel this way, but perhaps this inner critic has been a part of what has driven his constant reaching for musical growth. Nonetheless, it makes me a little sad that Parker isn't able to appreciate his achievements for what they are. As someone who struggles a lot with my own inner critic, it is quite humanising to see that even someone as conventionally successful as Kevin Parker faces a similar challenge. For me, it just reinforces the idea that satisfaction can't come from external validation - it has to come from self-acceptance and making your own meaning.

Lost and Grounded Brewery

A couple of my friends and I went to a fun Oktoberfest event at Lost and Grounded the other day. They had some great live music, and an excellent Festbier on tap. Although they have a somewhat limited range, Lost and Grounded are definitely one of my favourite Bristol breweries - Keller Pils is a fantastic flagship beer.

Greens Organise

Zack Polanski's recent victory in the Green Party's internal leadership elections seems to have supercharged the movement somewhat, which is really exciting. We recently surpassed 80,000 members in the party, and Zack is doing a fantastic job of staying in the public eye with a series of high-profile news articles and interviews.

Meanwhile, Greens Organise - a membership group within the Green Party with the aim of organising activists to achieve the party's eco-socialist and left-wing goals - had its first AGM yesterday. It was really energising to see lots of people coming together to discuss how to help the Green Party reach new heights, but also strengthen its ties with the wider Left.

With Labour currently trying to pander to right-wing scaremongering about immigration, and showing no signs of the boldness needed to improve the lives of people in the UK, the Green Party needs to continue to grow and provide a viable left-wing alternative to stand up for the disenfranchised in society. If you're already a Green Party member, I'd recommend signing up to Greens Organise. If you're not - what are you waiting for? https://join.greenparty.org.uk/

Other Stuff

This rug - we bought it to put under our bed and it's very cosy.
Morbid but quite gripping - https://illwill.com/fascism-and-the-spectacle-of-death - an essay on the increasing presence of death in everyone's awareness due to capitalism, and how an indifference to death can shade into the glorification of death and violence inherent in fascism.
An essay called Nine Things I Learned in Ninety Years, written by the creator of the Choose Your Own Adventure book format.

Website Refresh 2025

Fri, 21 Feb 2025 22:18:00 GMT

I've been wanting to write again recently, but I also find myself procrastinating a lot instead of writing. This month, my main source of procrastination has been "fixing" my website - by which I mean breaking it on purpose by upgrading things, and then fixing it again. This is somewhat of a cliché for anyone who maintains their own site. Why actually write when you could just tinker with things that aren't important to anyone except you? In an attempt to get something valuable out of the process, here's a ~~short~~ description of the changes I made, if only for my own benefit when I inevitably forget what I did.

Step 1: pnpm

The pulling of the long string began when I opened VS Code, intending to write something for my blog, and noticed that the copyright date was outdated. It was hard-coded to 2024. Since the site is built in Gatsby, there's no reason why I couldn't alter the footer code to get the current year instead. So I did that, and it was of course very easy. But did I stop there? No, I did not. I thought "Hmm, it's 2025 now, maybe I should stop using an odd combination of yarn and npm to install things and get with the times." So I switched my package manager to pnpm.

I should note that I've used pnpm in other projects, and I like it, it's great. The concept of not actually having to have a hundred versions of the same JS library stored in various node_modules is a good one. However, switching to it from yarn without stopping to think about how that might go is not a good idea. The crucial, and in hindsight, obvious step that I missed was ignoring my yarn.lock and just reinstalling the project dependencies from package.json with pnpm. Literally 1 minute of searching after the fact showed me that I could have simply imported the contents of the lockfile into pnpm using pnpm import.

Having not done this, and not having aggressively pinned the minor versions of my dependencies in the project definition, when I reinstalled the dependencies, things upgraded. Things upgraded and broke.

Step 2: Gatsby nightmares

When we were working on a recent project together, and I suggested Gatsby, my friend Sim alerted me to the fact that it was the second-most-disliked tool in the JavaScript ecosystem according to the State of JavaScript 2024 survey. I was surprised by this, as I'd not really had any issues with it in the past. However, the next few hours made me truly understand where this sentiment comes from.

I won't bore you and my future self with too much detail here, but post-reinstall my site would not build. I tracked this down to an issue with an outdated Gatsby plugin I use: gatsby-remark-relative-images (an unmaintained plugin for... something to do with relative image paths). This plugin requires a very specific version of cheerio, but doesn't specify that version in its own package.json. What is cheerio? I'm not sure, but it involves HTML parsing. It turns out that it had changed how it was exported in one of its releases in a way that broke the plugin.

The upside of this problem was that I learned how to use pnpm's overrides feature to pin the version of cheerio to 0.22.0 for that dependency only. It looks like this:

// package.json
 "pnpm": {
    "overrides": {
      "gatsby-remark-relative-images>cheerio": "0.22.0"
    }
  }

The downside was I spent about 2 hours debugging the issue, time that I was intending to spend on writing.

Step 3: More Gatsby nightmares feat. hydration

However, this wasn't the end of my woes. After fixing the first issue, my site would build, but on load would flash up with the content and then show an empty screen. Even better, I only discovered this issue when I deployed it to Netlify, since the build worked OK locally. But why?

As is the fashion, the way Gatsby works by default when it receives a request is to load the static HTML page it has generated in its build step, followed by hydrating a React app to allow interactivity on the page. This is billed as a "static site generator enhanced with React Hydration." Never mind that my website is completely static and doesn't need to hydrate a React app.

In its infinite wisdom, by default Gatsby doesn't use the same workflow in development as it does in production. The gatsby develop command used in dev does not do the static HTML generation step, instead using a webpack server (I think?) to serve the pages. This means you get hot reloads and all that good stuff, but it crucially also means that any hydration errors are not seen in development.

Hydration errors occur when the static HTML loaded doesn't match the HTML generated by React. This makes React upset. Sometimes it can recover from a hydration error by fully re-rendering the page, but sometimes it can't.

To debug these issues, I discovered that setting Gatsby's DEV_SSR flag forces pages to server render on dev, leading to the hydration workflow and at least allowing the errors to be seen locally. In my mind, it's not very intuitive that you'd need to turn DEV_SSR on to see a problem that is occurring between a pre-built HTML page and the React hydration step, but here we are.

My hydration errors were caused by two sets of problems. The one that was completely bricking the site on client load was fixed by following a random person's suggestion on a GitHub thread somewhere: installing the latest version of Gatsby's modified reach-router. It's still unclear to me why this wasn't already installed as a dependency of Gatsby itself, but perhaps it was a version mismatch. Once this was installed, the site would actually load correctly on build, but it was still doing a full client re-render to recover from a different hydration error. This somewhat defeats the point of a static site generator, in my opinion.

The second issue was caused by an image component loading with the alt-text set on the server-side HTML, but empty in the client render. According to React, the alt prop for this component was empty, even though I could happily log out the property in console.

At this point, I gave up on trying to fix the hydration issue and instead just upgraded everything (again). This was partly inspired by the fact that Gatsby v5 has a mode called partial hydration, which basically allows you to turn hydration off for all components that don't need it. Imagine that, a static site generator that just generates static pages! Anyway, I still haven't tried this setting, but I'm keen to.

Step 4: Upgrades

It turns out that just upgrading everything more fixed my problems. Replacing the defunct gatsby-image plugin with gatsby-plugin-image (great naming convention) made the hydration errors disappear. It also allowed me to delete a bunch of image code hanging around from the original starter repo, so it was a win-win.

Feeling buoyed by my success, I upgraded Gatsby and its accompanying plugins to v5. Amazingly, this time nothing broke. So now the site is using a version of Gatsby that works in the Year of our Lord 2025. This should tide me over until the inevitable Astro rewrite 🙃.

Step 5: Bulma and styles

The last thing to upgrade was Bulma, the CSS framework I use. This was reasonably painless, and moved me away from using node-sass (bad, deprecated) to dart-sass (good, modern). Honestly I couldn't care less what compiles my CSS, but I just want things to keep working. The curse of JavaScript web development is that everything goes out of date immediately and has to be replaced with whatever is new, cool, and popular.

Anyway, this upgrade also allowed me to update my single scss file, the aptly-named all.scss, to remove a bunch of custom styles I no longer needed. The new version of Bulma is modular, and allows customisation of its various components by overriding Sass variables when you import them. Here's an example for the title element:

// all.scss
$site-accent: #004225cc;
$site-title-family: "Playfair Display", serif;

@use "bulma/sass/elements/title" with (
  $title-family: $site-title-family,
  $subtitle-family: $site-title-family,
  $subtitle-color: $site-accent,

);

This took me a while to figure out, but led to a very slimmed-down stylesheet, which I like. The only frustration is that some customisation options that seem like they should have corresponding Bulma variables just don't (looking at you, default font size). So unfortunately I couldn't delete my own styles entirely.

I took the opportunity to make the site look a little nicer and tidier. I'm no designer - check out Sim's site if you want to see what happens when someone who is a designer makes a personal website - but hopefully it's at least pretty readable and consistent.

Step 6: Actually write

Today we reached the final step of actually writing something, to salvage some meaning from my self-inflicted labours. I have to admit that while the first part of the journey was largely frustrating, I am happy with the end product, so I suppose I shouldn't begrudge myself going down the rabbit hole.

If you encounter any similar problems with your Gatsby site, I hope maybe you'll find a useful solution here. Otherwise, maybe you can take some inspiration to break things and then put them back together again better.

Book Review: In Ascension

Fri, 31 May 2024 16:34:00 GMT

I read In Ascension in March 2024, and at the time I wrote down quite a few thoughts about the book because I enjoyed it so much. I'm now revisiting my notes to attempt to turn them into a coherent review of the book. Note: I've tried to keep the review section fairly spoiler-free, but there is some discussion of the pacing and characterisation. The subsequent two sections - questions and theories about the book - are full of spoilers!

Review

First and foremost, I found In Ascension very compelling. I read most of the book in a couple of sittings, and found it difficult to put it down. For me, this is an increasingly rare experience as I've grown older, and the internet has rotted my attention span, so I always value it very highly.

A way that In Ascension stood out for me from other sci-fi that I've read recently was the level of mystery. Science fiction often loves an explanation, and I think a mark of good sci-fi for me is getting the right balance of what's explained and what's implied. When a book is written well, concepts can be introduced tangentially and explored in their interactions with the characters and plot. In Ascension took this to an extreme, where some of the ideas remain a mystery (at least to me) throughout the novel. This could have been frustrating in a less well-constructed book, but it worked very effectively with the pacing and plot - it felt like the mysteries and level of understanding I encountered as a reader were mirrored by the protagonist. It wasn't that the book was failing to explain things that the characters knew, but rather that there were things that couldn't be known to the characters or the reader, because they were so strange.

To return to the pacing - one of the most effective aspects of In Ascension for me was the gradual escalation of risk and strangeness throughout the story. For me, there was a sense of connectedness and seeming inevitability that built both within the plot, and the writing itself. By the denouement, it felt like everything in the book was rushing and growing to the point of conclusion. This was partly offset by a change in perspective late in the novel, but I'm sure this was deliberate - the pulling of the punch, so to speak, just added to the idea that there are some things that can't be known.

My one gripe with the book - and it's a small one -was that I found the protagonist somewhat difficult to empathise with. Leigh, while being an interesting and well-developed character, felt strangely flat. Throughout the story, the only real emotions we see her engage with are those of curiosity and guilt. Her hunger for knowledge of the unknown and the unknowable pulls her forward, and repeatedly wins out over her guilt at the things she has to sacrifice. However, it's hard to find much depth in these sacrifices, since ultimately Leigh doesn't actually seem to want the connections and relationships she's giving up - instead, the source of the guilt is feeling that she should care more about them.

It'd be interesting to know if this characterisation was deliberate on the part of the author, or rather just not an area of focus in an otherwise extremely tightly-written novel. Unfortunately, I missed a Q&A with Martin MacInnes at my local bookshop, so I may never know.

I hope that it's fairly evident that I'd wholeheartedly recommend In Ascension. I think it appeals both to regular sci-fi readers, and to those who enjoy being drawn into a story where the veil of understanding is never fully lifted.

Questions

Having finished the book, I was left with loads of questions. I've collected some of these, stream-of-consciousness style, below. Spoilers for the whole novel follow.

What is the significance of the vent that the characters visit on the ship? Is it really deeper than anywhere else in the ocean? Why do the divers experience euphoria and a subsequent illness on entering the water? If it's the deepest part of the ocean yet discovered, why is it not further explored by the end of the book?
What is the alien artefact? Where does it appear from - I think this may be mentioned but I'd need to reread to check. Is the artefact related to the vent? Did it come from the vent? What do the concentric oval patterns mean? Perhaps the weirdest thing - how does it communicate with the dreams of scientists / engineers to pass on knowledge of the propulsion system?
Archaea and algae are big themes throughout. Why does the algae grow wildly after the acceleration of the spacecraft? What is the gene from the vent that is spliced into the algae?
Is it the acceleration of the spacecraft that messes with the bodies of Leigh and her other crew members? Is it some side-effect of the propulsion system? Or is it to do with leaving the heliosphere?
There's some ominous talk about the Great Filter being either space travel itself - travelling in interstellar space is impossible / destructive to a species - or what you find as a result of space travel. No conclusions here other than that humans' first attempt was unsuccessful, but it doesn't appear to have destroyed the species immediately.
It's heavily implied that the crew travel backwards in time before / while returning to Earth. Is this a consequence of leaving the heliosphere? Or is it something to do with the alien artefact?

Theories

While I really enjoyed the experience of reading In Ascension, I also think it's fun to speculate and discuss what everything meant - to attempt to make sense of some of the mysteries that linger at the end of the novel. With this in mind, I looked at some reviews to see if anyone had any good ideas for what was happening. Once again, spoilers.

The best clue I got for building my own understanding was from someone who referenced the description of salmon in the book. There's a focus on salmon returning to their spawning grounds, and their bodies disintegrating after they've laid their eggs to provide nutrients for their unborn children. This is a real phenomenon, although I'm not sure about the nutrient bit. This, along with the strong emphasis on cycles throughout the book, provides a possible explanation for events.

Leigh and her crewmates are participating in the 'life cycle' of life on Earth, although this life cycle takes place in a non-linear way through time. A lot of focus, early in the book, is put on the significance of the moment that archaea and bacteria combine to make eukaryotic life. The suggestion could be that the crew's return to Earth, with the rapidly flourishing algae taking over their ship and bodies, is the event that precipitates the creation of eukaryotes. In this cyclic version of events, the eventual evolution of humans and their travel to space is both the cause and the effect, with the journey acting as the completion of a loop. This lines up strongly with the scant evidence we have of what happens to the crew post 'The Event'. Their estimated "when", from looking at the stars, is 2 billion years ago, fitting with our knowledge of when eukaryotes first appear. The deterioration of the crew's bodies as the algae become more vital mirrors the salmon, reinforcing the idea of them as 'parents' bringing new life to Earth, their jobs done once the journey has taken place.

While this is a satisfying theory to me, it by no means explains everything in the story. The significance of the vent is still unclear, as is the role of the alien artefact. It serves as the catalyst for the journey by providing the propulsion system to humans, but is it also the mediator for the physical transformation that the crew undergo, or their travel through time on leaving the heliosphere?

One implication of the theory, at least for me, is that life on Earth is coming to a conclusion in the current time. This fits well with the backdrop of environmental degradation that the story is set against, as well as the increasing ill-effects that we see Helena and her children experiencing in the final chapters. In this reading, the mission can be seen as the final significant event of the planet in linear time, sowing the seeds for its own 'renewal'. Anything afterwards is a sort of post-script. It's not the case that humans immediately cease to exist, but more the sense that a tipping point has been reached from which there is no way back.

Do you have any interesting theories about the meaning of the book? If so, I'd love to hear them.

Consulting Update - March 2023

Wed, 22 Mar 2023 19:16:00 GMT

Somehow, it's already March 2023, and past time for another update on the progress of my consulting business. Excitingly, it's going pretty well so far!

After my launch in November, my focus for the last two months of 2022 was on building a basic version of the business, and promoting myself in order to generate inbound work. In 2023, I've shifted gears to focus on longer-term client work. So far the project I'm working on has been great fun, and has accelerated my timeline for achieving my initial business goals. Naturally, however, this focus on client work has impacted my available time for writing content and promoting my brand - hence the quarterly, rather than monthly, update.

Goals

Taking stock of my initial business goals, progress has been promising. This gives me a good footing for continuing with the consulting experiment, and increases my confidence that moving into the world of self-employment was the right choice.

♻️ Create sustainable revenue

Metrics:

✅ Reach an average revenue of £5000 / month - With the client work I've taken on so far in 2023, I've managed to hit my initial revenue goal from February onwards. However, in order to maintain a steady revenue stream, I need to continue to broaden and diversify my client base.
✅ Maintain a 4-day work schedule - I've kept my client commitments within my 4-day work schedule. Having now maintained this schedule since the launch of my consultancy, I feel very validated to find that it works effectively for me and for clients. The shorter working week means that I'm focussed and energised each day, and I feel that I'm delivering a lot of value as a result. The three days away from work give me plenty of time to rest, sort out non-work chores and responsibilities, and enjoy time with my friends and family.

🌱 Help organisations that align with my values

Metrics:

✅ 100% of clients pass the 'credible argument for good' test - The long-term client project I've been working on this quarter is in the renewable energy sector, and so comfortably meets this metric. I'm continuing to use this metric as a barometer for taking on new work.

✨ Build a reputation for excellence

Metrics:

100% of clients would like to work with me again - I don't have enough data here because I haven't completed work with many clients yet. I'm doing my best to make sure that my current clients would answer yes to this question, and make this metric 100% from the start.
X% of new work comes from client recommendations - The long-term project that I've been working on recently came to me via my network from previous full-time roles. While I wouldn't count this as a client recommendation, it's always nice when previous colleagues are excited to work with you again. I imagine it will take a while for me to settle on a reasonable target for this metric, but I'll keep it in mind when finishing work with clients.

Projects and initiatives

As mentioned above, I've been focussed on client work recently. The long-term project that I'm working on has a heavy focus on software engineering and architecture design, which has been refreshing and fun after several years of more leadership- and management-oriented work. It's gratifying to come back to coding and realise that you still know a few things.

There's some good writing out there on the value of moving back and forth between engineering and leadership roles - see The Engineer/Manager Pendulum as an example. I definitely feel the benefit of having been in an engineering leadership role when working on code - I find it much easier to have some emotional distance from the work I'm creating, and I'm much more implementation-agnostic than I used to be, because I care more about my clients' business outcomes than the code itself. This isn't to say that I no longer care about code quality or good architecture; I'm just more aware that it's possible to trade off, for example, an elegant implementation in order to gain operational simplicity. I also feel a lot more empathy for non-engineering stakeholders than perhaps I did earlier in my career, and my skills for engaging across different business functions - for diplomacy, if you will - have been helped by my time in leadership.

Outside of client work, I've been slowly improving the mechanics of my business. My instinct is to over-engineer processes, so I've been trying to deliberately keep things simple. I'm relying a lot on Notion to organise myself, from keeping daily notes to generating and tracking invoices.

Challenges

My content writing has fallen by the wayside due to my focus on client work. I'm keen to write the (hopefully shorter) counterpart to my article on self-development as an early-stage engineering leader, as well as turning another half-dozen ideas I have floating around in my brain into articles. Finding a way to set aside consistent time to develop my content and brand, alongside client work, is a challenge I'm keen to solve in the next few months.

Next steps

With the end of Q1 fast approaching, and Q2 looking fairly busy with existing clients, it's likely that project work will continue to be front and centre for the next few months. Alongside that, I'll aim to find a sustainable way to keep raising the profile of the consultancy. If you'd like to work together in the second half of the year, now is a great time to get in touch with me. You can reach me by email at work at jw.pe, or on LinkedIn, Twitter or Mastodon.

Consulting Update - December 2022

Tue, 06 Dec 2022 14:27:00 GMT

It’s been one month since I announced my consulting launch, so I thought I’d check in with an update on my progress so far. I’m hoping to do this on a monthly basis, in order to stay true to my aim of building in the open.

Goals

It’s still very early days for my experiment, so I haven’t got much to report on the goal front, but as a reminder, here are the goals I set myself:

♻️ Create sustainable revenue

Metrics:

Reach an average revenue of £5000 / month - I’ve been spending the last month trying to build my brand, and I have yet to make a dent in my revenue goals, so no update here. One interesting side effect of posting this revenue goal is that I’ve had some people make the mistake of trying to use it to reverse-engineer my day rate. Please don’t do this, as you’ll only be disappointed when it comes to the contract process. I see this as an initial revenue milestone to prove the validity of the business, nothing more or less.
Maintain a 4-day work schedule - I’ve found the lack of structure at this early stage of the business quite challenging, as I’ll discuss below. Nonetheless, I’ve been pretty good at keeping my work within these limits - the flexibility of choosing your own work time is nice, when working on ‘internal’ work such as brand building.

🌱 Help organisations that align with my values

Metrics:

100% of clients pass the 'credible argument for good' test - This has been a useful way to screen initial approaches for work, so I’ll continue to use this metric.

✨ Build a reputation for excellence

Metrics:

100% of clients would like to work with me again - While I wouldn’t call them clients, it was really gratifying to see some of the responses and endorsements from my network when I initially announced this experiment. I’m very grateful to those people I’ve worked with in the past who boosted my announcement, or left positive feedback.
X% of new work comes from client recommendations - It’s too early days to have a number for this, but some of the work I am exploring is as a direct result of previous roles, so this is starting out well.

Initiatives

Over the last month, my main focus has been on promoting the existence of my consulting experiment, as well as trying to build my reputation through writing relevant content.

Promotion

The soft launch of the consultancy centred on blog content promoted through social media channels. I put out my launch article on LinkedIn and Twitter. Of those channels, LinkedIn was by far the most effective, resulting in around 70 reactions, comments and reposts, and approximately 3000 impressions. Twitter, by contrast, fell very flat, with single-digit engagement and only a couple of hundred impressions.

I think that the issue here is that Twitter requires a very different posting strategy to LinkedIn. Instead of long-form content, effective use of Twitter seems to rely on drip-feeding interesting content via regular threads and tweets. Most of these might not land, but the occasional one will blow up and drive a lot of engagement. I’ve always been a fairly low-posting Twitter user, so I struggle with this pattern. However, given that Twitter has been undergoing a fairly rapid transformation into a dreadful tyre-fire thanks to the Musk acquisition, I’m leaning towards not using it as a platform for work promotion in future.

The other avenue for promotion that I’ve relied on is word-of-mouth discussions with friends and ex-colleagues. This is a smaller but much more engaged group of people, and I’m hoping one that will continue to yield useful results and contacts.

Content

Aside from these update posts, I’ve been working on some more long-form content around engineering management and team growth. A couple of weeks ago, I published an article about self-development as an early-stage engineering leader. It ended up growing to a slightly unwieldy size, and took longer to finish than I’d hoped, but I was happy with the content. Disappointingly, it didn’t seem to resonate with my (admittedly small) audience. I have a couple of theories why this might have been the case:

Too long - The article wasn’t short enough to digest in five minutes. Even though I provided a TL;DR summary at the end, it’s possible people were put off by the volume of content.
Too generic - While I hope the advice in the article was useful, perhaps it wasn’t specific or engaging enough for my audience to apply easily.
Not enough reach - It could be that the article just didn’t find the right people to read it, because I have a small audience.

I’m going to continue to experiment with this kind of content, but I’m really interested in any feedback - please, if you like or dislike any of my content, or think it could be improved in some way, I’d love to hear about it. Critical feedback is particularly welcome and valuable.

Challenges

The biggest challenge I’ve experienced with my consulting experiment is consistently motivating myself to develop the business. I know from experience that I work effectively when I have extrinsic motivations - for example, accountability to an employer, a colleague, or a client. However, finding that inner drive to consistently do the work needed to get clients in the door - when the only person I’m accountable to is myself - doesn’t come easily to me. I’m very much of the mindset that consistency and intrinsic motivation are learnable skills, but I’m going to need to keep practicing them to make consulting a success.

Next Steps

I’m going to continue building the brand and looking for potential clients for the next month. With the end of the year approaching, I’m conscious that many businesses will be winding down work and preparing for a big start in 2023. That said, for organisations with budget windows following the calendar year, some teams will be in a ‘use it or lose it’ situation with remaining budget - if that’s you, and you want to get a few days of useful consulting booked in before the holidays, let me know.

I’ll keep on writing valuable content around my areas of expertise - I’m approaching this through the lens of “What would I have found useful to read when I was getting started in my engineering management career?” I’m also thinking about local organisations that I can work with to increase my profile within the Bristol tech community, as well as how I can make better use of in-person meetups and events to find potential clients.

As a reminder, if you’d like for us to work together, you can contact me at work at jw.pe, or send me a message through another medium if we're already in touch. I’ll look forward to hearing from you!

Self-Development As An Engineering Leader: Gaining

Mon, 14 Nov 2022 13:36:00 GMT

A key factor in your ability to successfully grow an engineering team is your capacity for self-development. Over the course of scaling from zero to a team of double figures, your responsibilities and priorities will change dramatically. This may feel uncomfortable, as the skills that make you effective in your role as the sole engineer on the team will only be marginally helpful in managing a team of ten people. In fact, holding on too tightly to your original role and responsibilities will likely stifle the growth of your team and hold you back from being an effective leader.

With this in mind, perhaps the most enduring skills that will help you on the journey are humility and a desire to learn. You'll often find yourself facing new situations where you won't immediately know the right way to proceed, and being able to admit this to yourself - and others - is very important. However, as a leader, your team will be relying on you to provide a path forward, so you'll need to be able to educate yourself quickly in order to make (or facilitate) a decision. The good news is that if you're reading about engineering leadership, you've already demonstrated that you're keen to learn and able to recognise that you may not have all the answers.

I like to consider self-growth in the context of engineering leadership through two lenses. These can be summed up as "What do I need to gain?" and "What do I need to let go of?". Firstly, what are the areas I need to build on to lead my team and help the organisation succeed? This typically means developing new skills, gaining new knowledge and nurturing relationships. Secondly, and just as importantly, what are the things that I should let go of in order to maximise effectiveness and develop my team? I believe this question is often overlooked, but letting go of things is a powerful and necessary part of self-growth. This could be delegating responsibility to someone on your team in order to provide them with a learning opportunity, deprioritising an aspect of your role that isn't a valuable use of your time, or even letting go of a way of thinking or mental model that has become counterproductive. In this article, we’ll focus on the first of those two questions: “What do I need to gain to continue to be a good leader as my organisation grows?”

Develop relevant skills

A critical challenge for budding software leaders that feels so obvious that it shouldn’t need stating, but is frequently overlooked by organisations, is this: being a good software engineer does not automatically make you a good manager or leader. Management and leadership are their own disciplines with their own sets of skills, which are broadly orthogonal to engineering skills. For new organisations with little experience of running a technology team, it’s common for engineers to be hired or promoted largely on the basis of their technical skill. This can lead to situations where an effective engineer is turned into an ineffective manager, at a net loss to the individual and the organisation.

As the leader of a new or growing engineering team, you can set yourself up for success by approaching the running of a team as a new problem domain with new skills to be learnt. With leadership skills like effective communication and empathy, there’s often a (largely mistaken) view that they are innate, rather than learnable. Certainly, if you’re someone to whom skills like listening or reading others’ emotions comes easily, you’ll have a head start. But even if you find these areas challenging, practicing and soliciting feedback will allow you to improve, just like any other skill.

It may be helpful to identify a peer or mentor who can help you practice and give you honest feedback - for example, through conducting mock 1:1s, or roleplaying challenging situations. Look around at your organisation or network for someone who you identify with being a ‘good’ leader, and see if they’re willing to share some time to help you grow. Most likely, they’ll find this experience beneficial too, and so will be happy to help.

Another way to develop these skills is through reading, watching and listening to experts. As the technology industry continues to grow and mature, we’re being blessed with a variety of great books about engineering management. There’s also a conference organisation, LeadDev, which focusses exclusively on the skills and challenges of engineering leadership. There’s so much great content out there - it’s just a matter of going out and finding it.

The skills that make a good technology leader are many and varied, but some areas that I’d really recommend focussing on as your team grows are:

Listening effectively
Providing positive and critical feedback
Managing conflict
Speaking and writing engagingly

Depending on your previous experience, these may not be skills that you’ve approached as ‘things I can learn to be better at’. In your role as a leader, now is a great opportunity to start (or continue) cultivating these skills to make your team happier and more successful.

Broaden your knowledge

When you’re the sole engineer on a project - especially a greenfield project, as is common in start-ups - you’re responsible for every aspect of the technology. As a result, you probably know all of the ins and outs of your codebase, all of the implementation details behind key features, and so on. You’re coding day in, day out, so you’re constantly exposed to the evolution of languages, frameworks, and libraries that power your product. As you work, your understanding of these tools grows. You revisit and evolve your assumptions and mental models, improving your design patterns over time.

This deep technical knowledge and constant learning is part of the joy of being a subject matter expert. However, as your team grows and your responsibilities evolve, the importance of technical knowledge at this level of detail will decrease. In its place, knowledge about systems, organisations and best practices will become more valuable and relevant.

Zooming out

In practical terms, this may mean shifting your learning focus away from keeping up with every aspect of your stack, and instead exploring more abstract concepts. Systems thinking is a useful way of modelling both technical and organisational problems. Thinking about process development - how to intentionally design the way your team works to achieve the results you want - becomes increasingly important as your team scales beyond a few people.

One way to think of this shift in focus is as a zooming-out process, from the detail level to the outcome level. While you may still be the subject matter expert in some areas, in others, you’ll be thinking less about the implementation and more about the inputs and outputs. One pleasant benefit I’ve experienced from this process is being able to let go of feelings about the ‘right’ way to do things, rooted in your own biases. When you’re looking at a more outcome-focussed view of technical areas, you’ll likely be much more agnostic to how the result is achieved. This leads to increased trust in your team members, and empowers them to take ownership over the details of their work.

That said, even in areas where you’re no longer the technical authority, it’s wise to keep engaged with broad trends. This will allow you to have meaningful discussions with the people who have taken on the responsibility for this area. Even being able to ask the right questions will help the team to come to conclusions on big decisions, such as what technology to explore, or when to build vs buy. Showing humility and a willingness to learn from your team is one of the most effective ways to stay engaged with areas that you’re no longer working on directly. This approach has the added benefit of developing team members’ communication and teaching skills.

Looking beyond engineering

As you shift time away from technical detail, it’s important to broaden your knowledge outside your fledgling team. When you’re the sole coder and your entire organisation is a handful of people, it’s easy to stay current with other areas of work simply through osmosis. As growth occurs across the organisation, you’ll need to be more intentional about maintaining an awareness of the work and goals of other teams. Part of your responsibility as a leader is to act as an effective conduit between your team and the rest of the organisation, broadcasting information both internally and externally (while avoiding becoming a communication bottleneck!) If you don’t spend time learning about other aspects of the organisation’s work, you won’t be able to accurately represent needs and interests that impact on your team. This learning will also allow you to identify opportunities where your team can contribute more broadly to the success of the organisation.

Related to, but distinct from, understanding other teams’ work, is understanding the growth levers of your organisation. For revenue-driven organisations, this means understanding what aspects of the organisation can be changed to increase revenue. For example, an online sales business will have levers around conversion rate, customer re-purchase rate, customer volume, order value and so forth. Having clear knowledge of these levers will help to inform your decision-making as a leader, and also make it much easier to understand the motivations behind other decisions in the organisation. They can be helpful to identify areas for opportunity, and also to predict likely challenges that will arise as the organisation grows. Your conversion rate may be high when targeting highly-engaged early adopters, but will that conversion rate be sustainable as you address a larger market? It may seem obvious to know the basics of how your organisation fulfils its goals, but it can be surprisingly easy when you’ve been in the weeds of software development to lose sight of these levers. Staying in touch with them as the organisation grows is a key part of your own leadership growth.

Invest in your relationships

A critical aspect of self-growth in a growing organisation is making sure that you invest in your relationships. If you haven't built trust and good communication with your organisation's leadership, your peers and your team, then you won't be able to take advantage of your own skills and knowledge to effect change.

The first step in this process is simply to make sure that avenues of communication exist. Hopefully you work in an organisation where the concept of 1:1s is already established, but if not, start establishing it. Ask your manager (if you have one in your role - if not, it may be useful to find a mentor) for a regular catch-up, and schedule catch-ups with your direct reports. You may also find it valuable to schedule 1:1 time with other leaders in the organisation.

Team relationships

From my experience, 1:1s with your team are one of the most valuable ways you can spend your time as a manager and leader. There’s a plethora of great advice out there about how to run effective 1:1s, and there are many valid approaches. My philosophy around 1:1s has always been that they’re a time that belongs to your team, and as such each 1:1 should be driven by what works for that team member. Maybe they prefer to bring an agenda of topics they’d like to discuss, or maybe they prefer you to drive the conversation. Sometimes they might want to dive deep into an aspect of their work and get your advice, and other times they might just want to vent and get something off their chest. There’s definitely not enough space here to cover the details of a good 1:1, but some key outcomes that I look for from these conversations are:

Team members feel listened to and valued - this is priority number one. The strength of your relationship with your team rests on mutual trust and respect, and listening and understanding their priorities will build this trust.
Team members have a chance to give and receive feedback - whether this is feedback from them on the direction of a project, or feedback from you on their development, having space for feedback is very important.
You and your team member have a shared understanding of goals - if there’s important information you need to communicate to your team, 1:1s are a great place to check understanding of that info. Likewise, if a team member has their own professional goals, they can use this space to keep you informed of them.

I’ll revisit the topic of 1:1s at another time, but remember - 1:1s shouldn’t be the only source of your relationship with your team members. Building authentic relationships with your team, through both formal and informal means, is a big part of effective leadership.

Manager relationships

Alongside your relationship with your direct reports, cultivating your relationship with your manager will help you to grow alongside your organisation. As an engineering leader in a small organisation, it’s likely that you’ll be reporting directly to the CEO or founder. You may even be a founder, in which case you’ll have a slightly different dynamic to navigate.

In the first case, your manager is the person ultimately responsible for the success of the organisation. They’ll have a lot on their plate, and as a result will likely expect you to operate with a lot of autonomy. In the context of this dynamic, having a good understanding of their priorities and concerns will make your life much easier. You’ll be able to anticipate work that will come your way, and align your goals with theirs. Listening and asking the right questions will help you to get this information, as well as increasing your manager’s trust that you have a clear understanding of their needs.

Non-technical founders or CEOs will also want to be able to hand off technical problems and challenges to you, with the understanding that these will be dealt with. This means you also need to get comfortable with telling your manager what you need to succeed. While it might feel easier to just agree to a project, then go away and figure it out, being clear and up-front about the constraints and required resources will benefit your manager much more. They’ll be able to make an informed decision based on your expertise. This doesn’t mean being able to instantly estimate work, or accurately predict the exact cost of a project. Rather, it means taking the time to work out what the implications of a piece of work are, and bringing those clearly to the table. If a poorly-scoped project is going to take 6 months of your team’s time, then your manager would almost certainly know that information before starting, rather than waiting until you’re 3 months in to find out.

Don’t forget to ask for feedback from your manager. While some leaders my find it easy to give feedback, others will err on the side of politeness and conflict aversion, robbing you of an opportunity to learn and improve. You can help to avoid this process by articulating when and how you like to receive feedback. This will make it easier for your manager, and ensure you get the guidance you need to grow.

A final point on your relationship with your manager - as a founder of a small organisation, there’s no guarantee that they themselves have management training or experience. They could be learning as they go, and experiencing organisational growth for the first time. You can help to get the best out of your relationship by clearly articulating your needs, and providing them with advice and resources on how best to manage you. Chances are, they’ll really appreciate it.

Peer relationships

As mentioned above, as organisations grow, your knowledge of the work and challenges of other teams becomes less readily available. One way to combat this is to invest in your relationships with other leaders in your organisation. Understanding what they’re trying to achieve, and making sure that they understand your goals, will allow you to work in alignment. A lack of relationships between different areas of an organisation is often the cause of wasted resources, working at cross purposes, or internal politicking.

While it may not be necessary or feasible for you to have 1:1s with all other leaders in the organisation, a regular catch-up with anyone whose team’s work frequently overlaps with yours can be valuable. This will allow you to find shared solutions to common challenges, identify areas where you can support each other’s work, and build a shared narrative to present to the rest of the organisation.

Not only do peer relationships offer important organisational benefits, they can also benefit you professionally. Having an opportunity to learn from people who are going through a similar growth trajectory to you is really helpful. If you’re able to put in place simple ways to share this learning, such as a leadership lunch-and-learn or a book club, you’ll be able to pool your shared knowledge to the benefit of all.

Treating your peers as your team may feel unnatural at first, especially if you are deeply invested in the growth and success of your own team. However, it’s definitely worth it in terms of helping you, and the organisation, to grow together.

Summary

We’ve covered a lot of ground here, so let’s recap. In order to grow as a leader alongside our fledgling organisation, you’ll need to:

Develop relevant skills by treating management and leadership as distinct, learnable disciplines.
Broaden your knowledge of systems, processes and trends, as well as your knowledge of the work of other teams and the key levers of your organisation.
Invest in your relationships with your team, manager and peers.

However, growth isn’t all about gaining. Next time, we’ll cover some of the things you’ll need to let go of as you grow in your role as a leader.

This article is part of a series I’m writing on scaling an engineering team from zero - look out for further posts on the same topic.

Want support and advice on growing as an engineering leader? I’m currently available for consulting on the areas discussed above - see my consulting launch post for details. If you have thoughts or questions about the above, feel free to get in touch with me on Twitter, Mastadon, or by email.

I'm Launching A Consultancy

Tue, 01 Nov 2022 14:17:00 GMT

Today is 1st November, 2022, which feels like as good a day as any to begin my experiment with consulting. I'm framing this next step in my career as an experiment because I want to approach it with openness and curiosity, as well as applying an evidence-driven approach to my goals.

My career in the world of software engineering started with consulting and contracting. After making the switch to full-time employment a decade ago, returning to the world of consulting feels like coming full-circle. In that time, I've worked for a variety of great organisations, and learned an enormous amount. I'm excited to share some of that knowledge and experience with organisations whose values are in line with my own.

TL;DR

I'm launching a software engineering management consultancy, focussing on three areas:

🌅 Engineering team management, leadership, and coaching
✅ Preparing for investment
⚙️ Hands-on software engineering

If you want to work with me, please drop me an email at work at jw.pe.

What I offer

When planning my consulting offering, I spent some time reflecting on the unique value that I can bring. I've chosen three key areas where I believe my experience and skills can make a real difference.

🌅 Engineering team management, leadership, and coaching

Having led two successive technology teams over the last five years, engineering leadership is where I believe I can offer the most value, and where I get the most enjoyment from my work. I can help your organisation to build and retain an effective engineering team, improve your engineering processes, and coach or mentor developing engineering leaders. My approach focusses on:

Leadership centred on empathy and consensus-building.
Creating systems and processess to empower the team to drive its own development and growth.
Building team structures that are appropriate for the level of maturity of the organisation, while laying the groundwork for future expansion.
Establishing a working environment where team members understand their responsibilities and the value of their work, without unnecessary stress or pressure.
Making sure that communication between key stakeholders and the engineering team is transparent and effective, with clear shared goals and metrics.
Agreeing shared values, behaviours and standards to clarify expectations for all team members.

My experience is primarily working with engineering teams from early-stage start-ups to Series B scale-ups, with an engineering / product headcount between 1-20 people. I am happy to advise on processes for larger teams, but my experience will be less relevant.

✅ Preparing for investment

I've been through several investment rounds in previous roles, and have helped to prepare organisations to successfully secure those investments.

I've worked extensively on pitch decks, providing a trustworthy technical voice to investors. I would be happy to review investor materials and provide unbiased feedback, or work with you to craft the best story for the technology team in your organisation.

As deals move forward, I've prepared due diligence reports that have provided investors with confidence in the technical strength of the organisation. There's no substitute for having an empowered and well-run engineering team, but communicating these strengths effectively is very important. Given the current challenging investment environment for technology businesses, being adequately prepared for due diligence is crucial. I can help your organisation to get ready for the due diligence process by evaluating your technical infrastructure, team, and processes, then producing an assessment report with actionable recommendations. Alternatively, I can assist your team with an ongoing due diligence process by providing technical insight and hands-on support.

⚙️ Hands-on software engineering

I've spent the majority of the last 10 years working hands-on with code. My areas of expertise are back-end web development and infrastructure.

Rather than presenting an exhaustive list of technologies, two examples of web stacks I've worked with extensively are:

Stack 1:

Front-end: React, Redux, TypeScript
Back-end: NodeJS, Express, TypeScript
Data stores: PostgreSQL, S3
Infrastructure: AWS, CircleCI, Terraform, Docker

Stack 2:

Front-end: Angular, JavaScript
Back-end: Python, Django, Celery
Data stores: PostgreSQL, S3, Redis, Kafka
Infrastructure: AWS, CircleCI, Docker

Many of my skills are cross-functional and I'm always happy to learn new technologies, although this may depend on the duration of our work together.

I've worked to scale systems from pre-launch to heavy production usage, and will always try to choose an implementation to fit the level of maturity of your product, balancing the need for extensibility with avoiding overengineering.

I pride myself on communicating effectively with both technical and non-technical team members, as well as producing high-quality, robust and well-documented code.

Goals

With my experiment-driven approach in mind, I've set myself some goals for this new chapter, as well as considering metrics that I can use to measure my success.

♻️ Create sustainable revenue

A key goal for the coming months is to prove that consulting is a viable way for me to earn an income. It's critical to me that I can do this in a sustainable way. With this in mind, I'm aiming to work a 4-day week, giving myself time to balance delivering on my work goals with enjoying life outside of work.

Metrics:

Reach an average revenue of £5000 / month - As an initial income goal, this will prove the success of my experiment by allowing me to comfortably sustain my lifestyle. I'm confident that the value I bring will allow me to reach this goal in a fraction of my available schedule. Once I've proven the sustainability of the model I will set a new revenue goal, but for me the long-term aim is to reach a threshold that allows me to enjoy my life, not to earn as much as possible. Note: Please don't misinterpret this metric by trying to use it to calculate my day rate! This will only lead to disappointment.
Maintain a 4-day work schedule - Experience has shown me that I can deliver timely, high-quality work in a 4-day schedule. I want to have a healthy balance between my work and other important aspects of my life.

🌱 Help organisations that align with my values

It's important to me that I work with organisations that I believe are doing something positive in the world, so I want to achieve my revenue goal without compromising on my principles.

This is a bit of a tough goal to measure, since my assessment of an organisation as 'good' is completely subjective, but I'd want to be able to make a credible argument to friends and family for how each organisation I work with is benefitting the world and fits with my morals.

Metrics:

100% of clients pass the 'credible argument for good' test - See above.

✨ Build a reputation for excellence

I want people and organisations to want to work with me. I believe that I've left a good impression on my previous colleagues, and I'd like to continue to do so with my clients.

Metrics:

100% of clients would like to work with me again - I want to build positive relationships with anyone I'm helping, and leave them satisfied that my work brought them value.
X% of new work comes from client recommendations - Not sure what a good number to put against this is, but ideally I want to get to a place where word-of-mouth recommendations are driving a good chunk of my inbound work.

I aim to be transparent about my progress towards these goals, both to hold myself accountable and because I think it's interesting. I really enjoy reading about other people developing their businesses in the open, so it seems reasonable to try to take the same approach.

Why now?

My career in software has taken me in some interesting directions so far - from travel to politics, from coding to managing to leading. I've had the privilege to work in a variety of roles that have allowed me to better understand the landscape of technology organisations, and to discover my own strengths and topics that I'm passionate about. I'm sure that there's some nostalgia talking, but it's always felt to me like the next step on my journey was clear - when one chapter finished, the next one quickly began. Now, for the first time I can remember, that next step isn't so obvious, so I have to make my own way.

It feels scary to put yourself out there and trade solely on your own merit, which has held me back from launching this experiment over the last month that I've been seriously thinking about it. However, I've always liked the idea of working for myself, and with the expertise I've developed, I feel confident enough in my abilities to give it a try.

Work with me

If you like the sound of what I'm setting out to do, and you think I could add value to your organisation, please don't hesitate to get in touch to outline how we could work together. You can drop me an email at work at jw.pe, or send me a message through another medium if we're already in touch.

I'll likely be spending at least some of this month on setup and promotion, so my short-term availability may be limited until December. That said, if you have a small project where I can help you immediately, I'd love to hear about it. I'm currently discussing some larger projects in the new year, so if you have something sizeable in mind, the sooner you can let me know about it, the better.

I look forward to hearing from you!

Lockdown Walks Beyond Bristol

Wed, 13 Jan 2021 11:02:00 GMT

One of the few highlights of my 2020 was walking. Since I've lived in Bristol, I've wanted to make more of an effort to explore the beautiful local countryside. This was always something that fell by the wayside in favour of easier, lazier activities. With the onset of the pandemic, though, options for things to do outside the house dwindled.

Fortunately, during the first lockdown my girlfriend bought an excellent book - Beyond Bristol by Robin Tetlow. This little book has made a huge difference to our lockdown lifestyle. Full to the brim with carefully-researched walks through the fields, woods and villages of Gloucestershire and Somerset, it's a great primer for walkers who want to explore the area around Bristol. In my experience, instructions in walking guides can be frustratingly vague or require lots of prior knowledge. This isn't the case at all with Robin's directions. Each step in each walk we've been on so far has been clear, concise and unambiguous. I can't recommend it enough. It's clear from the creative routes and little details in the guide just how well the author knows the land around Bristol, and how much of his time and passion he's put into exploring and cataloguing it for the benefit of other walkers.

Most weekends, when the rules allow it, we've driven out of Bristol - the walks generally start around half an hour's drive away from the city¹ - parked in a little village, and set off for a few hours of leisurely walking. Given the sheer amount of time we're all spending at home these days, being out in the fresh air and hills of the Mendips or the Cotswolds is a very revitalising experience. No screens, no video calls; just chatting, climbing over stiles and engaging with the world outside.

There are still plenty of walks in the book that we've yet to try, but some we've done multiple times. Some of my favourites are:

A walk from Dursley (of Cam and Dursley fame) over some impressive hilltops following the Cotswold Way, and back along the edge of the forest.
A short (by the book's standard) walk from Hawkesbury Upton, passing through one valley in particular that seemed like something out of a Miyazaki film.
A climb up Crook Peak on the south edge of the Mendips, with expansive views over the Bristol Channel and down to north Devon.

I'm looking forward to more of these walks, and if by some miracle we manage to do them all, there's a second book!

I think this period of uncertainty and crisis has really highlighted to me the importance of doing things that are good for the soul, and walking definitely fulfils that for me. I might write in a bit more detail about some of the individual walks I've done over the last year, but I wanted to publish this book recommendation to encourage anyone who's looking for a place to start. I'd be interested to hear about any other good walks in the Bristol area.

1. I haven't tried to get to any of the walks using public transport yet - I'm sure that some of them, especially those starting in larger towns, should be accessible by bus or train. Smaller villages might be a bit more of a challenge. If you were feeling particularly dedicated, you could cycle to the start of some of the walks, but this would make for a long day!

Day 4: OAuth Flows

Mon, 08 Aug 2016 15:24:36 GMT

On Friday I finished off the initial implementation of obtaining an OAuth token from GitHub. Their documentation makes it pretty straightforward, but for anyone unfamiliar with the OAuth2 flow, it goes a little something like this:

The application generates an authentication URL for GitHub based on several parameters including the CLIENT_ID (obtained from the OAuth application on GitHub's side) and a redirect_url that users will be sent back to after authorizing.
The user follows this URL which takes them to an authorization page on GitHub, where they can choose to authorize the application to use the scopes (essentially permissions) that it has requested. More on scopes in a moment.
If the user authorizes the application, they are redirected back to the previously supplied redirect_url. GitHub passes a temporary code with this redirect call, which can temporarily be used to request a full access token for the user.
Behind the scenes, the application makes a POST request to GitHub, with more client information and the temporary code. In return it receives the OAuth access token.
The application can now use the access token to make authenticated calls on behalf of the user.

Steps 1 to 4 can be implemented in two fairly simple Django views. Let's go through the code for the initial generation of the authorization URL first. Note: the application code is now on GitHub so you can follow along there!

Generating an authorization URL

# models.py
from django.db import models
from django.contrib.postgres.fields import ArrayField
from django.utils import timezone

import uuid

class OauthToken(models.Model):
    """
    Stores Ouath Credentials for contacting github.
    """
    token = models.CharField(max_length=255, blank=True, null=True)
    scopes = ArrayField(base_field=models.CharField(max_length=255))
    state = models.CharField(max_length=255)
    timestamp = models.DateTimeField(default=timezone.now)

    def save(self, *args, **kwargs):
        self.state = self.state or uuid.uuid4()
        super(OauthToken, self).save(*args, **kwargs)

This is our database model model to hold data about the OAuth token. The token itself will live in the token field, and the scopes that the token has access to will live in a fancy Postgres ArrayField. The state field allows us to preserve state between generating the authorization URL and receiving the callback - after this it is unused. Finally, we have a timestamp because if there's one thing I've learned from working with Django for years, it's that everything is easier when your models are timestamped.

# views.py
from django.conf import settings
from django.shortcuts import get_object_or_404, reverse

import json
import requests
from rest_framework.views import APIView
from rest_framework.response import Response

from .models import OauthToken, User

OAUTH_URL = "https://github.com/login/oauth/"
API_URL = "https://api.github.com/"

class GitHubAuthorize(APIView):

    def get(self, request):

        # build an empty OauthToken
        desired_scopes = [
            'user:email',
            'write:repo_hook',
            'public_repo'
        ]

        token = OauthToken.objects.create(scopes=desired_scopes)

        # construct oauth URL for GitHub
        oauth_url = self._get_oauth_url(request, token)

        data = {'oauth_url': oauth_url}

        return Response({'data': data})

    def _get_oauth_url(self, request, token):

        redirect_uri = request.build_absolute_uri(
            reverse('oauth:callback'))

        params = {
            'state': token.state,
            'client_id': settings.GITHUB_CLIENT_ID,
            'scope': ' '.join(token.scopes),
            'redirect_uri': redirect_uri,
        }

        uri = OAUTH_URL + "authorize"

        oauth_url = self._add_params(uri, params)
        return oauth_url

    def _add_params(self, uri, params):
        uri_with_params = uri + '?' + '&'.join(['{0}={1}'.format(k, v) for k, v in params.items()])
        return uri_with_params

Our view is a subclass of DRF's APIView - this allows us to easily generate JSON responses and do a bunch of other awesome things like have pluggable authentication and permissions. In the main get method, we build a list of desired scopes to request from GitHub. We then make an empty OauthToken, with a randomly generated state field, to make sure that we can check that any callbacks we receive were actually initiated on our site. When we receive a callback, we will check for the existence of an OauthToken object with the same state passed in the callback. If one exists, we can be confident that it's a real authorization response, not a spoofed call from another site.

Our _get_oauth_url method just adds this state, our application's client ID, the desired scopes and our redirect URI as URL parameters to the base GitHub OAuth URL, https://github.com/login/oauth/.

Since the application will eventually have a separate client-side app, rather than redirecting the user directly to the authorization URL, we're returning it in a JSON response. The client application will then use this to redirect the user, but for the purposes of testing we can just get the resulting URL and plug it into our browser to authorize our application.

Success!

Recieving the OAuth callback and obtaining an access token

Once the user has authorized our GitHub app, we need to listen for a callback from GitHub and exchange the temporary code we receive for a full access token.

#models.py
class User(models.Model):
    """
    A basic user model that is used for authentication.
    """

    email = models.EmailField(unique=True)
    github_id = models.IntegerField()
    token = models.ForeignKey(OauthToken)

This model will hold basic user information. It would be totally fine to keep the user and OAuth token in a single model - I believe TravisCI does this - but I chose this implementation in order not to have to create empty user models in order to verify the legitimacy of a callback based on the state field as described above. A more mature implementation would probably handle this state flow separately, using expiring keys in Redis or something similar, and only create a unified token-user model once the callback had been confirmed valid. This is always something we can build later if we so desire.

# views.py
class GitHubCallback(APIView):

    def get(self, request):

        state = request.query_params.get('state')
        code = request.query_params.get('code')

        # Check that this is a legitimate callback for a request made from the site
        token = get_object_or_404(OauthToken, state=state)

        try:
            data = self._get_access_token(code)
        except requests.HTTPError as e:
            return self._generate_error_response(e)

        if set(token.scopes) == set(data['scope'].split(',')):
            token.token = data.get('access_token')
            token.save()

            user = self._get_or_create_user(token=token)

            return Response({'data': {
                'user_email': user.email,
                'access_token': token.token
            }})

        else:
            return Response({'errors': [
                {'status': 403, 'detail': "User has not authorized the correct scopes"}
            ]})

From our callback, we get the state and the code out of the URL parameters, and look for the OauthToken instance corresponding to the state to make sure it's the real deal. Once we find it, we exchange the code for the access_token. Let's look at that in more detail:

def _get_access_token(self, code):

     uri = OAUTH_URL + "access_token"
     params = {
         'code': code,
         'client_id': settings.GITHUB_CLIENT_ID,
         'client_secret': settings.GITHUB_CLIENT_SECRET,
     }
     headers = {'Accept': 'application/json'}

     response = requests.post(uri, headers=headers, params=params)
     response.raise_for_status()

     return response.json()

This function is very simple: we send the code to GitHub along with our application's client ID and secret, and get back a response including the access token and the scopes that the user has authorized.

It's probably a good idea to check at this point that the user-authorized scopes are the ones we requested - without those we can't do the things the service needs to do.

if set(token.scopes) == set(data['scope'].split(',')):
    token.token = data.get('access_token')
    token.save()

Currently, we'll just fail if the scopes are incorrect. In future, we could warn the user or redirect them back to the authorization page. If the scopes are right, we save the token object with the access token we got back. Finally, since we successfully have a token, we create a user to store GitHub-specific data like email and ID.

def _get_or_create_user(self, token):

    user_url = API_URL + 'user'

    headers = {'Authorization': 'token {}'.format(token.token)}

    response = requests.get(user_url, headers=headers)
    data = response.json()

    try:

        user = User.objects.get(email=data['email'], github_id=data['id'])
        old_token = user.token
        user.token = token
        user.save()
        old_token.delete()

    except User.DoesNotExist:

        user = User.objects.create(
            email=data['email'], github_id=data['id'], token=token)

    return user

Here we're calling GitHub's user API to get details about the authorized user. If the user has been through the authorization flow before - e.g. if our application's authentication has expired because they haven't visited the site for a long time - we don't want to recreate a new user. In this case we'll just delete the old token and replace it with the new one. If they're new to the flow, we'll make a user object for them and attach the OAuth token.

Later on we'll use this user for our own authentication purposes - we want to remember that the user has authorized with GitHub so they don't have to go through the full flow every time they visit the site.

What's next?

In the current implementation, we're just spitting out the user's email and access token at the end of the callback function, to prove that authorization has been successful. In reality, though, we want to generate our own authentication token for the user that they can use to take actions on our service, and probably redirect them to some client application with this token as a header. We'll use JWTs for our authentication, so tune in next time for a discussion of generating JWTs and making authenticated calls with them!

As always, if you have any questions or criticism, hit up the comments!

Day 2-3: Back to Django

Fri, 05 Aug 2016 12:09:49 GMT

The last couple of days have been relatively productive. Thanks to GitHub's excellent API documentation, I've got to the stage where I can reliably authenticate a user with my application via Oauth, and use their key to pull basic user info such as their private email address and GitHub ID. In this post I'll talk briefly about the setup of the application and some of the design choices I've made so far.

I've written the server element of the project using Django 1.10. I went with Django mainly because of my familiarity with the ecosystem and the ease of setting up REST endpoints with Django REST Framework (DRF), which is definitely something I'll want to do further down the line.

Layout

The project layout is slightly different from the standard created by Django's startproject command. For simplicity's sake, I prefer to put settings.py, wsgi.py etc in the root directory of the Django application, rather than having them in a separate, named folder. As a result, my tree looks like this:

.
├── alexandria_server
│   ├── permissions
│   │   ├── migrations
│   │   │   └── 0001_initial.py
│   │   ├── models.py
│   │   ├── urls.py
│   │   └── views.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── docker-compose.yml
├── Dockerfile
├── manage.py
└── requirements.txt

All of the configuration and deployment code lives at the project root, along with manage.py. The application root (alexandria_server) contains settings, base URL configuration and WSGI info, as well as any app directories.

Configuration

It was an interesting experience trying to see how few of Django's built-in modules were necessary for it to work. I've built Django projects before without many of the contrib apps, but I think this was the first time that I'd completely removed the auth app. Since the only login will be through GitHub initially, the user model in the application can be very basic and thus doesn't require the password hashing and other helpful features that Django auth provides.

Here's my extremely minimal settings.py file currently. Initially, DRF was pretty unhappy with my removal of the auth app, as it tries to assign an AnonymousUser object to the request on an unauthenticated call. However, the UNAUTHENTICATED_USER setting can be changed to another object, or None, which circumvents this issue.

"""
Django settings for alexandria_server project.

Generated by 'django-admin startproject' using Django 1.10.
"""

import os
import dj_database_url

# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.abspath(__file__))

SECRET_KEY = os.environ.get('SECRET_KEY')

if os.environ.get('DEBUG'):
    DEBUG = True
else:
    DEBUG = False

GITHUB_CLIENT_ID = os.environ.get('GITHUB_CLIENT_ID')
GITHUB_CLIENT_SECRET = os.environ.get('GITHUB_CLIENT_SECRET')

ALLOWED_HOSTS = []


# Application definition

INSTALLED_APPS = [
    # Core
    'django.contrib.contenttypes',
    'django.contrib.messages',
    # Third-party
    'rest_framework',
    # Internal
    'alexandria_server.permissions',
]

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

ROOT_URLCONF = 'alexandria_server.urls'

WSGI_APPLICATION = 'alexandria_server.wsgi.application'


# Database
DATABASES = {
    'default': dj_database_url.parse(os.environ.get('DATABASE_URL')),
}

# Internationalization
# https://docs.djangoproject.com/en/1.10/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_L10N = True

USE_TZ = True

# DRF
REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': [],
    'DEFAULT_PERMISSION_CLASSES': [],
    'DEFAULT_RENDERER_CLASSES': (
        'rest_framework.renderers.JSONRenderer',
    ),
    'UNAUTHENTICATED_USER': None
}

Only a couple of interesting things worth pointing out here. Firstly, as you can see, most of the default Django apps are gone. I'm not entirely sure if I need contenttypes or messages either, so those may disappear.

I'm using Kenneth Reitz's dj-database-url to easily populate the configuration for the default database. I like this approach because it makes it trivially easy to set your DB locally, or in the cloud, simply by changing the DATABASE_URL environment variable.

Finally, I'm going for the classic 12 factor app approach to settings, so all variables and secrets are pulled in from environment variables. I prefer this tactic to the alternative of having a settings.py file per environment, perhaps not checked in to source control. I feel like it gives more flexibility but also more consistency between environments.

Database

My go-to database choice for Django has always been Postgres. They play together very nicely, and Postgres has a host of awesome features which are fully supported in Django that no other database has. It's also extremely easy, thanks to Docker, to run a Postgres database locally in a container. Here's the setup in my docker-compose.yml:

db:
    image: postgres:9.5
    environment:
        - POSTGRES_PASSWORD=some_password
        - POSTGRES_USER=some_user

It's amazing to be able to spin up a local database with only 5 lines of configuration code.

In a separate post, I'll talk about the Oauth flow in more detail. If you have any questions or comments about project layout, or configuring a basic Django setup, let me know in the comments.

Open Source Adventures, Day 1: Ideas and Skeletons

Tue, 02 Aug 2016 22:43:38 GMT

Today I began work on my project in earnest. It always helps me to visualize my goals if I can draw them out, and I still find pen and paper to be the best medium for rapidly exploring ideas. Since I'm at a very early stage, the notes that I made were pretty high level, but helped me to get an idea of the main moving parts I'd want for the app.

Some basic requirements for the app to work, as shown in the list on the right:

Authenticate with GitHub. The goal for the project is to automatically build documentation whenever a relevant repository is pushed to, much like continuous integration tools like Travis or Circle. This means that being linked to GH is pretty much a no-brainer.
Parse Markdown and convert it to HTML. Ideally in the long term I'd like the tool to support multiple documentation formats but Markdown is ubiquitous and easy to parse.
Build a tree of static files and push them to a CDN (probably S3). Ultimately the docs you view should just be static files for simplicity and speed of loading. At some point they will probably be accessed through a web app with authentication so that only registered users can view them, but this can come later.

There will have to be some boilerplate around this core logic, especially on the authentication/permissions front for interacting with the GitHub API. However, those three bullet points encompass the main purpose of the project. My aim is to build an MVP which covers these bases, and then expand as and when I've ascertained the usefulness of the tool.

My current idea is to separate the 'worker' unit - the part of the application that listens for webhooks, retrieves raw documentation, parses the Markdown and spits out HTML - from the 'authentication' unit, which will handle storage of user tokens, repo references, serving a REST API etc. I'm hoping with this approach I'll be able to make the worker element stateless and easily horizontally scalable, since I imagine building and uploading the compiled documentation will be the time bottleneck in the app flow.

With my ideas slightly more in order, I started building an application skeleton that I could iterate on. I often find that the hardest part of a project is writing the first line of code, and breaking that barrier by laying simple groundwork is very helpful. It makes it easy to start building experimental views, run unit tests, and all the good stuff that allows rapid development. From a psychological perspective, the hurdle of starting is overcome and you can start focussing on the more interesting and challenging parts of the problem at hand.

I'm using Docker to run the application at the moment, since I'm familiar with it and it's portable. Here's my bare-bones Dockerfile for Python-based projects:

FROM python:3.5

ADD requirements.txt /code/
WORKDIR /code/

RUN pip install -r requirements.txt
ADD . /code/

CMD python run.py

I generally use docker-compose to run locally as it makes everything so easy. Whether or not this will stay in through the final product, I don't know, but it makes running locally and in production fairly seamless, so there's a good chance.

Tomorrow I plan to focus on getting the basics of authenticating and GitHub token storage down, so I can start playing around with the webhook API using my own credentials. I'll hopefully post an update showing some actual code or results at that point, but no promises!

Let me know in the comments or via email if you have any initial suggestions or questions about the project!

Open Source Adventures, Day 0

Mon, 01 Aug 2016 00:00:00 GMT

Today is August 1st. Now that I have been back in Bristol for a little while, it's time to knuckle down on a project that I've been thinking about in some form or another for the last few months.

I don't have many details yet because the ideas are still all in my head. However, I have a pretty good idea of the problem that I want to solve, so here's the summary:

As a software engineer working on a team, you want to write documentation to accompany your code. If your project involves multiple moving parts (for example a microservice architecture), it can be hard to keep documentation consistent and accessible across the different repositories that comprise the overall application. On one hand, you want to write your documentation alongside your code so that it stays up to date with changes to APIs. On the other hand, you don't want to have to trawl through 50 services' GitHub repositories to find the documentation on the one endpoint you care about.

I want to build a tool that allows developers to write documentation alongside code, but read documentation in a single, presentable, easily accessible way. In other words, a documentation tool that allows distributed writing and centralised reading.

Open source is amazing, and I have built my career on the shoulders of great developers who make the open source packages I use every day. For this reason, among others, I want to build this project in the open. If teams find it useful, and want to self-host the tool, I will be delighted. If I can provide a paid service for those people who don't want to run it themselves, in the style of Sentry or Nginx, all the better.

I want to make sure that I stay motivated and hold myself accountable while I'm working on this project, hence the announcement. I intend to post regular updates on my progress for anyone who is interested, and to make sure that I work consistently towards a goal.

If you're interested in more details, please let me know in the comments or via email. As soon as I have something tangible to show, I'll add links to a GitHub repository for anyone who wants to contribute!

Quantifying Sufjan Stevens with the Genius API and NLTK

Fri, 22 Jul 2016 18:45:06 GMT

I am an avid Sufjan Stevens fan. I really enjoy his peaceful, somewhat melancholy style of music. As a person who is not religious myself, one of the things that I find interesting about his songs is that religious - specifically Christian - themes seem to appear quite broadly across his oeuvre. A song will often have a superficial, literal interpretation, and potentially another allegorical interpretation with religious connotations. Stevens has discussed his faith, and how it influences his music, in several interviews. Nonetheless, discussions rage online as to whether his songs should be viewed through a lens of religion or simply taken at face value.

I was thinking about this yesterday while listening to his song about the elusive, possibly extinct ivory-billed woodpecker, 'Lord God Bird':

It made me wonder how often Sufjan Stevens actually references God in his work. Is he talking more, or less, about religion as time goes on? Since I have been itching to do some lighthearted Python coding, I decided I'd try to find out using www.genius.com as a lyrics resource. Fortunately, Genius has a decent REST API which can be accessed using an OAuth token. For experiments like mine with no users, it's possible to generate a client token after having registered an application.

Firstly, I needed to get a complete list of Sufjan songs to work with. Using requests, it was fairly easy to throw together a minimal API client that accessed the artist songs endpoint at api.genius.com/artist/<id>/songs. This endpoint only returns 20 results by default, so it was necessary to page through until I'd made a list of all songs by the artist.

import requests

CLIENT_ACCESS_TOKEN = "some_token"

BASE_URI = "https://api.genius.com"

def _get(path, params=None, headers=None):

    url = '/'.join([BASE_URI, path])

    token = "Bearer {}".format(CLIENT_ACCESS_TOKEN)

    if headers:
        headers['Authorization'] = token
    else:
        headers = {"Authorization": token}

    response = requests.get(url=url, params=params, headers=headers)
    response.raise_for_status()

    return response.json()

def get_artist_songs(artist_id):

    current_page = 1
    next_page = True
    songs = []

    while next_page:

        path = "artists/{}/songs/".format(artist_id)
        params = {'page': current_page}
        data = _get(path=path, params=params)

        page_songs = data['response']['songs']

        if page_songs:
            songs += page_songs
            current_page += 1
        else:
            next_page = False

    return songs

This is a fairly naive implementation - it'll keep paging through until it gets to the first empty page of results - but it worked fine for my purposes. I chose to separate out the request logic because I wanted to implement some other API calls using the same approach.

Running the get_artist_songs function with artist_id=958 got me a list of songs by Sufjan Stevens. I filtered out any that didn't list him as the primary artist:

songs = get_artist_songs(958)
songs = [song for song in songs if song['primary_artist']['id'] == 958]

This left me with 284 Sufjan songs. I didn't do any filtering beyond this, so it's possible that the data set contains duplicates - I don't know how strict Genius is about removing multiple instances of the same song.

Now that I had my song list, I needed to find the lyrics for each song. Unfortunately, this wasn't a field that was present in the song objects I'd got back from the API. However, I had the URL of the song itself on Genius. I used BeautifulSoup, another of my favourite Python libraries, to grab the song lyrics directly from the HTML of the page... or that's what I would have done if it were permitted by the Genius Terms of Service. To be honest, since Genius are shamelessly annotating other people's web content I'm sure they'll be cool with this. Here's what that code might potentially look like:

import requests
from bs4 import BeautifulSoup

def scrape_lyrics(url):

    response = requests.get(url)
    html = response.text

    soup = BeautifulSoup(html, 'html.parser')

    lyrics = soup.find(name="lyrics")
    lyrics.script.decompose()

    lyrics_text = " / ".join([lyric for lyric in lyrics.stripped_strings])
    return lyrics_text

BeautifulSoup makes things so simple. It's easy to find the lyrics tag in the HTML, and then get all child strings within the tag using lyrics.stripped_strings. stripped_strings just removes extraneous whitespace, but otherwise behaves like the more commonly-used strings. I joined the strings together with / for aesthetic purposes, but it's really not necessary.

Using this function, I was able to retrieve the lyrics for all 284 songs on my list.

for song in songs:
    url = song['url']
    lyrics = scrape_lyrics(url)
    song['lyrics'] = lyrics

Armed with this trove of data, I could start doing some rudimentary analysis. My initial idea was to use the release dates of songs to investigate whether Sufjan had become more, or less, godly over time. However, only 34 of the 284 songs had release date information available through the Genius API, so that idea was quickly nipped in the bud. If I'm feeling motivated at some point, I may revisit this and combine the data with another information source to get a better picture of God references per year.

What I can tell you, however, is which of Sufjan's songs feature the most references to God. I chose the words 'God', 'Lord', 'Jesus' and 'Christ' as explicit references - if I missed any, please let me know.

First, let's look at the lyrics in aggregate:

import nltk

stop_words = set(nltk.corpus.stopwords.words('english'))
stop_words.update(['.', ',', '"', "'", '?', '!', ':', ';', '(', ')', '[', ']', '{', '}', '/']) 

all_lyrics = " ".join([song['lyrics'] for song in songs])

all_tokens = nltk.word_tokenize(all_lyrics)
all_tokens = [token.lower() for token in all_tokens if token.lower() not in stop_words]
fdist = nltk.FreqDist(all_tokens)

Here I am bringing NLTK to bear, admittedly at a very rudimentary level. I tokenized the lyrics, splitting them up into individual words, and then built a frequency distribution from the tokens. I removed stop words and punctuation to make the results less noisy.

The fdist object has a couple of fun options. Looking directly at the words we chose above, we can see how often Sufjan talks about God in all of his songs:

god_words = ['christ', 'jesus', 'lord', 'god']
god_count = sum([fdist[word] for word in god_words])

For the whole body of lyrics I was analyzing, direct references to God appeared a whopping 187 times in 69 distinct songs. In the absence of a comparison with a non-Christian artist, I don't want to draw any direct conclusions from this, but I think it's fair to say that Sufjan revisits this theme a lot.

We can also see what other things Stevens talks about a lot in his songs by plotting the most common 50 words in the frequency distribution.

fdist.plot(50)

Unsurprisingly, 'love' is the most common noun across all songs. Perhaps more surprisingly, 'Christmas' is extremely common too - that is, until you know that Sufjan Stevens has released two Christmas albums.

We can go into a bit more detail, and look at the songs that feature God the most frequently:

god_list = []
for song in songs:
    tokens = nltk.word_tokenize(song['lyrics'])
    tokens = [token.lower() for token in tokens if token.lower() not in stop_words]
    fdist = nltk.FreqDist(tokens)
    god_count = sum([fdist[word] for word in god_words])
    god_list.append((song['title'], god_count))

# filter out songs with no reference to god
god_list = [song for song in god_list if song[1]]

# sort by number of references
god_list = sorted(god_list, key=lambda tup:tup[1], reverse=True)

Here then, are the top 10 Sufjan Stevens songs by Godliness:

Come On! Let's Boogey to the Elf Dance
Away In A Manger
Holy Holy, Etc.
Get Real Get Right
Oh God, Where Are You Now? (In Pickerel Lake? Pigeon? Marquette? Mackinaw?)
Seven Swans
Hark! The Herald Angels Sing!
Ding-a-ling-a-ring-a-ling
Once in Royal David's City
The Transfiguration

Aside from Christmas songs, the album that appears the most is Seven Swans, with two tracks ('Seven Swans' and 'The Transfiguration') featuring in the top 10. This makes sense, as the album is acknowledged as one of the more overtly religious of his works.

I hope you've enjoyed this little adventure into data collection and analysis using Python, with a distinct Sufjan Stevens theme. Thanks go to the singer himself, as well as Genius for providing the data for this post.

Back in Bristol

Mon, 11 Jul 2016 15:18:00 GMT

It's been a long time since I've written anything on this blog, and I have a lot of ground to cover. Since I last updated, I've spent 18 months in America, worked on some amazing projects and learned an enormous amount. I'm hoping to talk a bit more about some of the technical concepts I've had the chance to explore in subsequent posts. However, for the moment, I just want to address my short-term goals and current situation.

I've moved back to Bristol - perhaps my favourite city in the UK - and will be working on my own software projects. I am also interested in doing more technical writing; I'll aim to put out content on this blog, as well as on Twitter, on a more regular schedule.

It would be great to become more involved again with the Bristol tech scene. I am particularly interested in discussing the following topics:

systems architecture
Docker, and container orchestration/devops
web development using Django/Flask and Python
continuous integration and deployment
JavaScript and ES2015

If you run a meetup group, are working on a project that covers one of these areas, or just want to get coffee and chat, please let me know!

I am also considering taking on some short-term consulting/contract projects, so I'm happy to discuss potential opportunities if they are relevant to my experience and interests.

Making a Mockery of Python

Wed, 10 Dec 2014 15:11:53 GMT

This post was originally published on the TrackMaven engineering blog.

Today we will be talking about mocking. No, not the joke at someone else's expense kind. Mocking is a technique to temporarily replace real parts of our code with fake, simpler parts, so that we can check that the rest of the parts are working as we intend. Here we'll consider some simple use cases for mocking in Python tests, and try to show how this extremely useful technique can make us better at testing.

Why do we mock?

When we carry out unit testing, our goal is to test a small section of code - for example a function or class method - in isolation. In other words, we should only be testing the code that is contained in said function. If we rely on calls to other pieces of code in our test, then we find ourselves in an unfortunate situation where changes to the nested code can break the test, despite the code of the function being tested remaining the same. This is best illustrated by example:

# function.py
def add_and_multiply(x, y):

    addition = x + y
    multiple = multiply(x, y)

    return (addition, multiple)


def multiply(x, y):

    return x * y

# test.py
import unittest


class MyTestCase(unittest.TestCase):
    def test_add_and_multiply(self):

        x = 3
        y = 5

        addition, multiple = add_and_multiply(x, y)

        self.assertEqual(8, addition)
        self.assertEqual(15, multiple)

if __name__ == "__main__":
    unittest.main()

$ python test.py
.
----------------------------------------------------------------------
Ran 1 test in 0.001s

OK

In the simple case above, we have a function that adds and multiplies two numbers, and returns both the sum and the multiple. The add_and_multiply function calls a second function, multiply to perform the multiplication operation.

Suppose we decided that we wanted to dispense with 'traditional' mathematics, and redefine our multiply function to always add three to the numbers being multiplied.

Our new 'multiplication' function looks like this:

def multiply(x, y):

    return x * y + 3

Now we encounter a problem. Our test code hasn't changed. The function we are supposedly testing hasn't changed. However, the test_add_and_multiply test will now fail:

$ python test.py
F
======================================================================
FAIL: test_add_and_multiply (__main__.MyTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 13, in test_add_and_multiply
    self.assertEqual(15, multiple)
AssertionError: 15 != 18

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)

The issue here is that our original test was not a true unit test. Despite intending only to test our outer function, we are implicitly testing the inner function as well, since our desired result depends on its behaviour. This may seem like a pointless distinction in the simple case above, but in a situation where we are testing a complex piece of logic - for example, a Django view function that calls various different inner functions based on certain conditionals - it becomes more important to separate the testing of the view logic from the results of the function calls.

There are two ways to solve this problem. We either ignore it, call our unit test an integration test and move on, or we can turn to mocking. The disadvantage of the first course of action is that an integration test only tells us something is broken somewhere along the line of function calls - it makes it much harder to identify where the issue lies. This is not to say that integration tests aren't useful, because they are. However, unit tests and integration tests solve different problems, and should be used in tandem. So if we want to be good testers, we choose the alternative: the mock library.

What is mock?

mock is a Python package so awesome, it was added to the standard library in Python 3. For those of us peasants still toiling in the UnicodeError-strewn fields of Python 2.x, you can install it through pip:

pip install mock==1.0.1

There are many different ways to use mock. We can use it to monkey-patch functions, create fake objects, or even as a context manager. All of these implementations serve one overall purpose - replacing parts of our code with replicas that we can use to a) gather information and b) return contrived responses.

mock's documentation can be quite dense, and finding information on a particular use-case can be tricky. Here, we'll take a look at a common scenario - replacing a nested function to check its inputs and outputs.

We will mock you

Let's rewrite our unit test, using the power of mock. Then we'll discuss what's happening, and why it is useful from the perspective of testing.

# test.py
import mock
import unittest


class MyTestCase(unittest.TestCase):

    @mock.patch('multiply')
    def test_add_and_multiply(mock_multiply):

        x = 3
        y = 5

        mock_multiply.return_value = 15

        addition, multiple = add_and_multiply(x, y)

        mock_multiply.assert_called_once_with(3, 5)

        self.assertEqual(8, addition)
        self.assertEqual(15, multiple)

if __name__ == "__main__":
    unittest.main()

At this point, we can change the multiply function to do whatever we want - it could return the multiple plus three, return None, or return your favourite line from Monty Python and the Holy Grail - and our test above will still pass. This is because we are mocking the multiply function. In true unit test fashion, we don't care about what happens inside the multiply function; from the perspective of our add_and_multiply test, we only care that multiply was called with the right arguments. We assume (hopefully, correctly) that what is going on inside multiply is itself being tested by another unit test.

What just happened?

The syntax used above may look confusing at first. Let's consider the relevant lines more closely:

@mock.patch('multiply')
def test_add_and_multiply(mock_multiply):

We've used the mock.patch decorator to replace multiply with a mock object. We then insert this into our test by passing it as an argument, which we've called mock_multiply. Within the context of the test, any call to multiply will be redirected to our mock_multiply object.

Cries of terror - "How can we be replacing a function with an object!?" Don't worry! This is Python, so functions are objects. Normally, when we call multiply(), we are using the __call__ method of the multiply function object. With our mock in place, however, our multiply() call instead calls the __call__ method of our mock object.

mock_multiply.return_value = 15

In order to get our mock function to return anything, we need to specify the return_value attribute. This tells our mock object what to give back when it is called.

addition, multiple = add_and_multiply(x, y)

mock_multiply.assert_called_once_with(3, 5)

In the test, we then called our outer function, add_and_multiply. This will call our nested multiply function, and if we've mocked it correctly, the call will be received by our mock object instead. To check that this has happened, we can rely on a smart feature of mock objects - they store any arguments that they were called with. The assert_called_once_with method of the mock object is a nice shortcut to check, as the name suggests, if the object was called once with a specific set of arguments. If it was, we are happy and the test passes. If it wasn't, assert_called_once_with will let us know by raising an AssertionError.

What have we achieved?

Well, quite a lot actually. Firstly, we have isolated the functionality of add_and_multiply from the functionality of multiply by mocking the nested function. This means that our unit test is only testing logic specifically inside add_and_multiply. Only changes to the code of add_and_multiply will affect the success or failure of the test.

Secondly, we can now control the outputs of our nested function to make sure our outer function handles different cases. For example, our add_and_multiply function might have conditional logic based on the result of multiply: say, we only want to return a value if the multiple is greater than 10. We could easily test that our logic works by generating contrived outputs from multiply to mimic the case where the multiple is less than 10, and the case where the multiple is greater. This feature of mock objects is great for testing control flow.

Finally, we can now make sure that our mocked function is being called the correct number of times, with the correct arguments. Since our mock object is sitting where our multiply function normally sits, we know that any calls made to it would normally go to multiply. When testing a complex function, it is extremely reassuring to make sure that each step is being called correctly.

The example given above only scratches the surface of what mock can do. In an upcoming post, we'll look at some more in-depth examples of using the library, as well as identifying some pitfalls to avoid. Meanwhile, questions are welcome in the comments, or on Twitter!

Want support and advice on growing as an engineering leader? I’m currently available for consulting - see my consulting launch post for details. If you have thoughts or questions about the above, feel free to get in touch with me on Twitter, Mastadon, or by email.

Using Markdown with Django 1.7

Fri, 09 May 2014 03:26:17 GMT

Markdown is great. As a formatting language, I find it to be pretty much unparalleled for speed and ease of syntax. Not having to think overly hard about style allows you to focus purely on the quality of the content.

At work, we use GitHub extensively, which means that Markdown is a standard. Issues, pull requests and comments all accept the syntax, so it feels very familiar. Given a choice, I now prefer to do all of my writing in Markdown. Using Draft makes this very convenient - I can write in Markdown and output my writing to HTML. However, this has always felt like a bit of an extra step to me. What I really wanted to be able to do was just dump my Markdown into my blog and have it render as HTML.

One way to do this would be to use a static site generator like Pelican or Jekyll. However, my site is a Django app, and I want to keep it that way. Unfortunately, Django deprecated their built-in support for Markdown rendering in 1.5. Fortunately, there is an easy way around this!

Using Markdown with Django 1.5+

The Python markdown package is the secret to adding Markdown support back into our Django templates. In order to make it as easy as possible to render Markdown, we want to create a custom template filter. It will look a little something like this:

In order to render content written in Markdown on a page, we can just add the filter to the relevant context variable. Don't forget to load the file containing the template tag!

Et voila - now we can write our content as Markdown, throw it into a text field on a Django form, and see it rendered as HTML on our display page. Easy! So easy, in fact, that this post itself was written in Markdown and put straight into my blog.

Note: For the security-minded, it's also possible to change the escape behaviour of the markdown function to disallow raw HTML from being passed through. This way, we can expose a Markdown form to our users without worrying that they will inject malicious code into our pages! In the example above, I've used safe_mode = 'escape' which automatically escapes raw HTML to a safe equivalent.