GitHub Actions

info

What you'll learn

• How to run Cypress tests with GitHub Actions as part of CI/CD pipeline
• How to parallelize Cypress test runs within GitHub Actions
• How to cache build artifacts between installation jobs and worker jobs

info

GitHub Actions + Cypress Screencasts

1. What is Continuous Integration?
2. Actions & Workflows
3. Example App Overview
4. Understanding how to configure a workflow
5. Running Tests in GitHub Actions CI/CD Workflow
6. Debugging Test Failures in CI
7. Running Tests in Parallel

GitHub offers developers Actions that provide a way to automate, customize, and execute your software development workflows within your GitHub repository. Detailed documentation is available in the GitHub Action Documentation.

Cypress GitHub Action

Workflows can be packaged and shared as GitHub Actions. GitHub maintains many, such as the checkout and Upload/Download Artifact Actions actions used below.

The Cypress team maintains the official Cypress GitHub Action for running Cypress tests. This action provides npm installation, custom caching, additional configuration options and simplifies setup of advanced workflows with Cypress in the GitHub Actions platform.

Basic Setup

The example below is basic CI setup and job using the Cypress GitHub Action to run Cypress tests within the Electron browser. This GitHub Action configuration is placed within .github/workflows/main.yml.

name: Cypress Tests

on: [push]

jobs:
  cypress-run:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      # Install NPM dependencies, cache them correctly
      # and run all Cypress tests
      - name: Cypress run
        uses: cypress-io/github-action@v2
        with:
          build: npm run build
          start: npm start

tip

Try it out

To try out the example above yourself, fork the Cypress Kitchen Sink example project and place the above GitHub Action configuration in .github/workflows/main.yml.

How this action works:

• On push to this repository, this job will provision and start GitHub-hosted Ubuntu Linux instance for running the outlined steps for the declared cypress-run job within the jobs section of the configuration.
• The GitHub checkout Action is used to checkout our code from our GitHub repository.
• Finally, our Cypress GitHub Action will:
  • Install npm dependencies
  • Build the project (npm run build)
  • Start the project web server (npm start)
  • Run the Cypress tests within our GitHub repository within Electron.

Testing in Chrome and Firefox with Cypress Docker Images

GitHub Actions provides the option to specify a container image for the job. Cypress offers various Docker Images for running Cypress locally and in CI.

Below we add the container attribute using a Cypress Docker Image built with Google Chrome and Firefox. For example, this allows us to run the tests in Firefox by passing the browser: firefox attribute to the Cypress GitHub Action.

name: Cypress Tests using Cypress Docker Image

on: [push]

jobs:
  cypress-run:
    runs-on: ubuntu-latest
    container: cypress/browsers:node12.18.3-chrome87-ff82
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      # Install NPM dependencies, cache them correctly
      # and run all Cypress tests
      - name: Cypress run
        uses: cypress-io/github-action@v2
        with:
          # Specify Browser since container image is compile with Firefox
          browser: firefox

Caching Dependencies and Build Artifacts

The Cypress team maintains the official Cypress GitHub Action for running Cypress tests. This action provides npm installation, custom caching, additional configuration options and simplifies setting up advanced workflows with Cypress in the GitHub Actions platform.

info

Caching of dependencies and build artifacts between installation and worker jobs can be accomplished with the Upload/Download Artifact Actions.

The install job below uses the upload-artifact action and will save the state of the build directory for the worker jobs.

name: Cypress Tests with installation job

on: [push]

jobs:
  install:
    runs-on: ubuntu-latest
    container: cypress/browsers:node12.18.3-chrome87-ff82
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Save build folder
        uses: actions/upload-artifact@v2
        with:
          name: build
          if-no-files-found: error
          path: build

      - name: Cypress install
        uses: cypress-io/github-action@v2
        with:
          # Disable running of tests within install job
          runTests: false
          build: yarn build

The download-artifact action will retrieve the build directory saved in the install job, as seen below in a worker job.

name: Cypress Tests with Dependency and Artifact Caching

on: [push]

jobs:
  # install:
  # ....
  cypress-run:
    runs-on: ubuntu-latest
    container: cypress/browsers:node12.18.3-chrome87-ff82
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Download the build folders
        uses: actions/download-artifact@v2
        with:
          name: build
          path: build

      # Install NPM dependencies, cache them correctly
      # and run all Cypress tests
      - name: Cypress run
        uses: cypress-io/github-action@v2
        with:
          # Specify Browser since container image is compile with Firefox
          browser: firefox
          build: yarn build

Parallelization

The Cypress Dashboard offers the ability to parallelize and group test runs along with additional insights and analytics for Cypress tests.

GitHub Actions offers a matrix strategy for declaring different job configurations for a single job definition. Jobs declared within a matrix strategy can run in parallel which enables us run multiples instances of Cypress at same time as we will see later in this section.

Before diving into an example of a parallelization setup, it is important to understand the two different types of GitHub Action jobs that we will declare:

• Install Job: A job that installs and caches dependencies that will be used by subsequent jobs later in the GitHub Action workflow.
• Worker Job: A job that handles execution of Cypress tests and depends on the install job.

Install Job

The separation of installation from test running is necessary when running parallel jobs. It allows for reuse of various build steps aided by caching.

First, we'll define the install step that will be used by the worker jobs defined in the matrix strategy.

For the steps, notice that we pass runTests: false to the Cypress GitHub Action to instruct it to only install and cache Cypress and npm dependencies without running the tests.

The upload-artifact action will save the state of the build directory for the worker jobs.

name: Cypress Tests with installation job

on: [push]

jobs:
  install:
    runs-on: ubuntu-latest
    container: cypress/browsers:node12.18.3-chrome87-ff82
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Save build folder
        uses: actions/upload-artifact@v2
        with:
          name: build
          if-no-files-found: error
          path: build

      - name: Cypress install
        uses: cypress-io/github-action@v2
        with:
          # Disable running of tests within install job
          runTests: false
          build: yarn build

Worker Jobs

Next, we define the worker job named ui-chrome-tests that will run Cypress tests with Chrome as part of a parallelized matrix strategy.

info

Note

Using our Cypress GitHub Action we specify install: false since our dependencies and build were cached in our install job.

The download-artifact action will retrieve the build directory saved in the install job.

name: Cypress Tests with Install Job and UI Chrome Job x 5

on: [push]

jobs:
  install:
  # ...

  ui-chrome-tests:
    runs-on: ubuntu-latest
    container: cypress/browsers:node12.18.3-chrome87-ff82
    needs: install
    strategy:
      fail-fast: false
      matrix:
        # run copies of the current job in parallel
        containers: [1, 2, 3, 4, 5]
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Download the build folders
        uses: actions/download-artifact@v2
        with:
          name: build
          path: build

      - name: 'UI Tests - Chrome'
        uses: cypress-io/github-action@v2
        with:
          # we have already installed all dependencies above
          install: false
          start: yarn start:ci
          wait-on: 'http://localhost:3000'
          wait-on-timeout: 120
          browser: chrome
          record: true
          parallel: true
          group: 'UI - Chrome'
          spec: cypress/tests/ui/*
        env:
          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
          # Recommended: pass the GitHub token lets this action correctly
          # determine the unique run id necessary to re-run the checks
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

note

The above configuration using the parallel and record options of the Cypress GitHub Action requires setting up recording to the Cypress Dashboard.

Setting up Parallelization

To setup multiple containers to run in parallel, the matrix option of the strategy configuration can be set to containers: [1, 2, 3, 4, 5], which will start 5 instances of the defined container image.

info

Note

The containers array is filled with filler (or dummy) items to provision the desired number of CI machine instances within GitHub Actions.

caution

Ensure Correct Container

The container attribute must be specified using the Cypress Docker Image in the configuration that was used in the install job.

#...
ui-chrome-tests:
  runs-on: ubuntu-latest
  container: cypress/browsers:node12.18.3-chrome87-ff82
  needs: install
  strategy:
    fail-fast: false
    matrix:
      # run copies of the current job in parallel
      containers: [1, 2, 3, 4, 5]

Using the Cypress Dashboard with GitHub Actions

In the GitHub Actions configuration we have defined in the previous section, we are leveraging three useful features of the Cypress Dashboard:

1. Recording test results with the record: true option to the Cypress Dashboard:

   • In-depth and shareable test reports.
   • Visibility into test failures via quick access to error messages, stack traces, screenshots, videos, and contextual details.
   • Integrating testing with the pull-request (PR) process via commit status check guards and convenient test report comments.
   • Detecting flaky tests and surfacing them via Slack alerts or GitHub PR status checks.

2. Parallelizing test runs and optimizing their execution via intelligent load-balancing of test specs across CI machines with the parallel: true option.

3. Organizing and consolidating multiple cypress run calls by labeled groups into a single report within the. Cypress Dashboard. In the example above we use the group: "UI - Chrome" option to organize all UI tests for the Chrome browser into a group labeled "UI - Chrome" in the Cypress Dashboard report.

Cypress Real World Example with GitHub Actions

A complete CI workflow against multiple browsers, viewports and operating systems is available in the Real World App (RWA).

Clone the Real World App (RWA) and refer to the .github/workflows/main.yml file.

Common Problems and Solutions

Re-run jobs passing with empty tests

We recommend passing the GITHUB_TOKEN secret (created by the GH Action automatically) as an environment variable. This will allow the accurate identification of each build to avoid confusion when re-running a build.

name: Cypress tests
on: [push]
jobs:
  cypress-run:
    name: Cypress run
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Cypress run
        uses: cypress-io/github-action@v2
        with:
          record: true
        env:
          # pass GitHub token to detect new build vs re-run build
          GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}

Pull requests commit message is merge SHA into SHA

You can overwrite the commit message sent to the Dashboard by setting an environment variable. See Issue #124 for more details.

name: Cypress tests
on: [push]
jobs:
  cypress-run:
    name: Cypress run
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Cypress run
        uses: cypress-io/github-action@v2
        with:
          record: true
        env:
          # overwrite commit message sent to Dashboard
          COMMIT_INFO_MESSAGE: ${{github.event.pull_request.title}}
          # re-enable PR comment bot
          COMMIT_INFO_SHA: ${{github.event.pull_request.head.sha}}

info

We also recommend adding COMMIT_INFO_SHA to re-enable Cypress bot PR comments. See this comment for more details.

See also

• Test anything that runs in the browser with Cypress and GitHub Actions