Kodiak is an open source GitHub App that automates updating and merging pull requests. It eliminates the tedious process of updating and merging branches and only takes a couple minutes to setup.

This post covers the history of Kodiak’s development from inception until now.

Why we built Kodiak

At work our code repository’s main branch must always be in a deployable state. Failing tests on main is a serious issue that prevents us deploying new changes.

We test all pull requests that are merged into main, but with default GitHub Branch Protection settings, it’s easy to break main.

Breaking main in two easy steps

Two pull requests can conflict with each other without triggering a merge conflict.

One pull request (A) can delete a function, while another pull request (B) in parallel creates a new call site for that function.

Pull request A:

-# emails.py
-def send_email(address, message):
-    ...

Pull request B:

 # main.py
 from app import emails
 def signup(name, address):
     user = create_user(name, address)
+    emails.send_message(user.address, "Welcome to our app")

Both pull requests A & B are open at the same time and pass tests. Pull request A is then merged, followed by pull request B.

main
 |
 B (fail)
 |
 A (pass)
 |
 .

Now our main branch is failing tests because the new call site added by B calls a missing function.

Keeping main green

We can prevent breaking our main branch by always testing our pull requests against the latest version of main.

When pull request A is merged into main, we also update pull request B with the latest changes on main. Now pull request B’s tests will fail because of the changes pull request A introduced, which prevents us from merging a broken pull request into main.

GitHub can enforce this testing flow via the the “Require branches to be up to date before merging” branch protection rule. GitHub describes this rule as:

Require branches to be up to date before merging

This ensures pull requests targeting a matching branch have been tested with the latest code.

Updating pull requests is annoying

At work, the “Require branches to be up to date before merging” branch protection rule ensures our main branch is green, but this slows down merging pull requests.

Whenever a pull request is merged, everyone else needs to update their pull request with the latest changes on main before merging. This requires running CI again and can take upwards of 10 minutes.

If multiple developers are trying to merge pull requests at the same time, there’s a race for a developer to update their pull request, wait for CI to finish, and merge the pull request before the other developer. If you’re too slow, you have to start waiting all over again.

This waiting and toil is annoying for developers. This became a regular complaint during our meetings, so I decided to automate it.

I looked at open source projects like Bulldozer, and even made a pull request, but none of the existing projects I found solved the problem of efficiently updating and merging pull requests.

This was the impetus for building Kodiak. I ended up building the first version during finals in May 2019.

Automating pull request updates and merges

The initial version of Kodiak in May 2019 was basic, but included the main features of Kodiak that exist today. The app would update and merge pull requests in order of when the pull requests were added to the queue.

There wasn’t a persistent queue so any restart would clear the merge queues. Also, no GitHub Check Run messages meant you couldn’t see Kodiak working in the GitHub UI.

Even without this, the initial version solved the immediate problem.

Polishing the app

Future updates added a configuration file, error handling, queue persistence via Redis, and automated tests.

Using Kodiak at work allowed us to quickly find ways to improve the app. Using GitHub Check Run messages to show Kodiak working on pull requests made the app feel alive. With a relatively small change, a developer could see Kodiak updating and merging a pull request within the GitHub UI.

kodiak waiting for checks

Finding our first users

Our first Kodiak users were my coworkers. While the project was open source and anyone on GitHub could install the app, I wasn’t doing any advertising. So for the first few months, we were the sole users.

In July, Vercel discovered Kodiak while searching for “auto update PRs bot”.

After helping them get setup on Kodiak and making some changes to better support their use cases, Kodiak got it’s largest boost when the Vercel CEO tweeted about Kodiak.

This tweet launched Kodiak.

Growing Kodiak

We didn’t actively promote Kodiak because marketing is hard and isn’t as fun as coding. Word of mouth and open source projects like Vercel’s next.js using Kodiak were the key drivers in user growth.

Developers will notice Kodiak on a popular open source project start using it themselves.

Using Kodiak every day at work allowed us to quickly add and refine new features. As external users grew, we eventually started building most new features to support external users and even received occasional external contributions.

Eventually some feature requests felt like too much work for me to complete. So I would push them off. However, when someone offered to sponsor the project in exchange, I gladly accepted their feature request.

Monetization

Kodiak’s first source of revenue was GitHub Sponsors. We had less than five sponsors, but it was enough to cover our hosting costs, especially since GitHub was matching all sponsorship money at the time.

In January of 2020, we started work on monetizing Kodiak, so all private GitHUb organizations would require payment to use Kodiak. We didn’t need money to run Kodiak, but we were curious to see what it would be like to monetize a project. We wanted experience and if we could get some money out of it, that would be a nice plus.

Timeline

Here’s the timeline of monetization within Kodiak:

  • Fall 2019 — Thinking about pricing for Kodiak
  • January, 2020 — Build kodiakhq.com landing page and documentation website
  • January 28th, 2020 — Start building Kodiak Dashboard (app.kodiakhq.com)
  • March 5th, 2020 — Release Kodiak Dashboard with read only information
  • April 22nd, 2020 — Update Kodiak to support subscriptions
  • April 23rd, 2020 — Roll out paywall to three users
  • May 1st, 2020 — Roll out paywall to remaining users

Building a Homepage

We first built a landing page and documentation website for our new domain, kodiakhq.com. Our documentation was originally all in our GitHub Readme, so we reorganized our docs and made them searchable with Algolia’s DocSearch.

I felt that having a professional looking homepage was important if users were going to trust Kodiak enough to pay for a subscription.

Designing the dashboard

Charging money for a product requires a lot of infrastructure.

We need a website to accept credit card payments and we need to track usage to know how much to charge an organization.

We started out by designing the Kodiak dashboard in Figma. We copied designs from CircleCI for their payment flows and usage information, because we had a similar usage model.

Initially I tried using GitHub’s Marketplace payments for Kodiak. I made a verification request in mid-January 2020, which was the first step, but didn’t hear back. This pushed me to integrate with a third party payments provider, which was great anyway because GitHub wanted an outrageous 25% fee at the time.

We initially had this idea that we needed to support PayPal as a payment method since GitHub supported it and European users liked PayPal more than credit cards. But after struggling to get PayPal or Braintree payments setup, we settled on Stripe.

Once we finalized a Figma design, we built out a skeleton React app.

Coding the dashboard

Our dashboard was built out in React for the UI and Django for the backend with Postgres. I wanted to use FastAPI for the backend, but configuring SQLAlchemy and testing was too difficult, so I quickly initialized a Django app.

Coding the UI and API was similar to my tech stack at work, which made it quick to get moving.

Tracking usage required sending data from the GitHub App to the web API’s Postgres database, so we used Redis as a queue between these two systems.

The dashboard allowed us to show active Kodiak users and accept payments, but it also allowed us to display graphs on Kodiak’s activity, to show how useful Kodiak was at updating and merging pull requests.

We first shipped the Dashboard as read-only, without payments. This way we could ensure our usage tracking worked correctly and our charts looked correct before adding billing.

Finally we shipped our paywall with a 30 day free trial. We only require an email, so we can communicate with users, and we don’t even verify it.

After 30 days, we manually paywall users.

Introducing the paywall

Introducing the paywall caused some users to leave, which was sad, but every time a new user signed up felt great. I manually paywalled individual accounts, so the paywall was introduced gradually. I’m still glad I haven’t automated this step because it’s useful to be selective in who to paywall.

It’s better to wait for an account to grow to have more users, than paywall them when they only have a couple users and don’t find much value from Kodiak. Kodiak’s value grows as the users grow.

A couple users complained the price, $4.99 per active Kodiak user per month, was too much compared to GitHub’s per user costs. But we couldn’t compete on price with GitHub. Thankfully most users didn’t find the price objectionable.

Some users asked for discounts or special deals, which we gave without question. I’d rather have a customer than no customer.

Maintaining Kodiak

Kodiak was initially a passion project and I spent a lot of my free time tinkering on it, but now it’s a job.

Once users started paying for Kodiak, it became a job. Any downtime still causes me anxiety, which can happen at any hour since there’s users in Europe, the US, and APAC.

Until recently, we had a longstanding bug that would leave Kodiak frozen for only a few users. After years, we finally replaced our Redis library with one that had proper network timeouts and resolved the issue.

The GitHub API has also been no friend. We’ve seen numerous bugs and regressions. Dealing with GitHub Support is never fun because it’s so difficult to get through to engineering teams. Building on another company’s platform is a challenge that I wish I could avoid.

I’m proud of Kodiak

Writing this blog post has been a great way for me to see how far the project has come over the years.

It’s satisfying that so many people have found Kodiak useful and I’m proud of all the features we’ve built out: the GitHub App, our bot testing setup, the web API, the dashboard UI, the documentation site.

Check out the GitHub repository or see the Kodiak App on the GitHub Marketplace.