Problem Matchers with GitHub Actions

GitHub Actions Problem Matchers display your build errors as Annotations in your Pull Requests.

For supported output styles, like Typescript errors, GitHub will parse your GitHub Action logs and create Annotations on your pull request.

With an error like the following, GitHub will show the error message on our pull request.

src/pages/PlacesCreateView.page.tsx(31,30): error TS2345: Argument of type 'string | undefined' is not assignable to parameter of type 'string | null'.

Annotations save us from having to dig through our GitHub Actions logs to find an error.

GitHub Annotation

Using built-in matchers with monorepos

The actions/setup-node Action has built-in problem matchers for Eslint and Typescript, so these errors will be parsed automatically.

One catch is that GitHub expects paths in errors to be either absolute or relative to the repository root.

Typescript outputs paths relative to the project root, which breaks when we run Typescript in a subdirectory of our repository.

Absolute paths in Typescript errors

In our case, we have Typescript projects in ui/ and functions/. Running the Typescript project within ui/ will output error messages like with paths like src/pages/PlacesCreateView.page.tsx, instead of ui/src/pages/PlacesCreateView.page.tsx.

This relative path causes Typescript errors to not be displayed as Annotations in the Pull Request.

A simple fix is to convert our relative error paths to absolute paths with a shell script.

tsc | awk -v prefix="$PWD" '{print prefix "/" $0}'

With absolute paths, our errors will render in our pull request.

relative

src/pages/PlacesCreateView.page.tsx
  31:9  error  'x' is assigned a value but never used  @typescript-eslint/no-unused-vars

absolute

/Users/christopherdignam/projects/foodieyak/ui/src/pages/PlacesCreateView.page.tsx
  31:9  error  'x' is assigned a value but never used  @typescript-eslint/no-unused-vars

Building a custom problem matcher

Vitest outputs errors in a format that doesn’t match actions/setup-node’s built-in Eslint or Typescript problem matchers, so we need to make our own matcher.

Here’s an example error message we want to parse.

 FAIL  src/textutils.test.ts > startCase
AssertionError: expected 'Braised 1/2 Chicken' to deeply equal 'Baised 1/2 Chicken'
 ❯ src/textutils.test.ts:8:28
      6|   ["all lower case words", "All Lower Case Words"],
      7| ])("startCase", (input: string, expected: string) => {
      8|   expect(startCase(input)).toEqual(expected)
       |                            ^
      9| })
     10|

  - Expected   "Baised 1/2 Chicken"
  + Received   "Braised 1/2 Chicken"

Reading the Problem Matcher docs and using the Eslint-stylish matcher as an example, we need:

file path
line number
column number
error message

Because this information is spread across multiple lines in the Vitest output, we need to use a multiline matcher. This is a list of regex expressions that will be applied in succession to match all of the lines we need from our error.

When the first pattern in our list of patterns matches, GitHub will apply each consecutive pattern in the list to the following lines in succession. All patterns must match for the problem matcher to apply.

Creating the Regex patterns

To anchor our problem matcher around the error, we want to match on the line containing “FAIL”.

 FAIL  src/textutils.test.ts > startCase

We don’t need to capture any information from this line, we just want to start our match sequence. The regex ^ FAIL will match this first line and look like the following problem matcher pattern:

{
  "regexp": "^ FAIL  "
}

The next line we want to capture in full.

AssertionError: expected 'Braised 1/2 Chicken' to deeply equal 'Baised 1/2 Chicken'

We can use the regex (.*) and specify that message uses the first capture group in this regex 1:

{
  "regexp": "(.*)",
  "message": 1
}

For the following line, we can capture the file name, line number, and column.

 ❯ src/textutils.test.ts:8:28

We can use a regex of ` ❯ (.*):(\d+):(\d+)`, which will look like the following problem matcher when escaped for JSON:

{
  "regexp": " ❯ (.*)\\:(\\d+)\\:(\\d+)",
  "file": 1,
  "line": 2,
  "column": 3
}

Putting these regexes together, we get a full multilne problem matcher:

{
  "problemMatcher": [
    {
      "owner": "vitest",
      "pattern": [
        {
          "regexp": "^ FAIL  "
        },
        {
          "regexp": "(.*)",
          "message": 1
        },
        {
          "regexp": " ❯ (.*)\\:(\\d+)\\:(\\d+)",
          "file": 1,
          "line": 2,
          "column": 3
        }
      ]
    }
  ]
}

A couple additional problems with Vitest

Output has color, which breaks Problems Matchers because of a bug in GitHub Actions.
Errors uses relative paths like Typescript, so we need to convert our paths to absolute paths.

The following code works around these issues:

# NO_COLOR disables colors in Vitest.
# We redirect stderr to stdout to allow sed to parse the full output and insert
# absolute paths.
NO_COLOR=1 vitest 2>&1 | sed "s#src/#$PWD/src/#"

Adding our custom matcher to GitHub Actions

To enable our custom Vitest Problem Matcher, we need to add the file to our repository (.github/problem-matcher-vitest.json) and run the following command before our tests in CI to register our matcher:

echo "::add-matcher::.github/problem-matcher-vitest.json"

In our GitHub Actions workflow file, this will look like the following yaml:

jobs:
  test_ui:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v3
      - run: yarn install --frozen-lockfile
        working-directory: "./ui"
      # register our problem matcher.
      - run: echo "::add-matcher::.github/problem-matcher-vitest.json"
      - run: ./s/test
        working-directory: "./ui"

Now GitHub will parse your GitHub Actions logs and render your Vitest errors in your pull request.

GitHub Annotation

Additional notes

The codebase behind problem matchers is written in C#, so the problem matcher Regex expressions use C# regex instead of Javascript: src/Runner.Worker/IssueMatcher.cs

Regex101 is a great tool for testing regular expressions, but be sure to disable the “global” and “multi line” flags like we have in ❯ (.*)\:(\d+)\:(\d+). Matcher patterns only work against a single line at a time.

To store an expression like ❯ (.\*)\:(\d+)\:(\d+) in a JSON string, you need to escape the \’s. Use Regex101’s Javascript code generator to generate an escaped regex string:

// Copy the regex string " ❯ (.*)\\:(\\d+)\\:(\\d+)" to insert into the problem
// matcher json.
const regex = new RegExp(" ❯ (.*)\\:(\\d+)\\:(\\d+)", "");