Playwright Buildkite Setup

Buildkite auto-detects a .buildkite/pipeline.yml in your repository. This guide covers a basic run and a parallel sharded setup that streams results to TestDino during the run.

Prerequisites

A TestDino account with a project created
A TestDino API key (Generate API Keys)
A Buildkite pipeline connected to your repository, with an agent in a cluster
A Playwright test project (or clone the TestDino Example Repository to get started)
@testdino/playwright installed and configured in playwright.config.ts|js|mjs:

npm install @testdino/playwright

playwright.config.ts

import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['@testdino/playwright', {
      token: process.env.TESTDINO_TOKEN,
    }],
  ],
});

Set Up Your API Key

Store your TestDino API key as a Buildkite cluster secret so it reaches the job without being written into .buildkite/pipeline.yml.

In Buildkite, go to Agents
Open Clusters and select the cluster your pipeline runs on
Open Secrets, then click New Secret
Set the Key to TESTDINO_TOKEN
Paste your TestDino API key into the value
Save the secret

The secrets plugin reads the secret at runtime and exposes it as an environment variable. Buildkite redacts secret values from build logs automatically.

Basic Pipeline Config

For a simple setup without sharding, run your tests with TESTDINO_TOKEN injected. Results stream to TestDino during the run, so no separate upload step is needed.

.buildkite/pipeline.yml: Basic pipeline

.buildkite/pipeline.yml

steps:
  - label: ":playwright: Run tests with TestDino"
    plugins:
      - secrets#v2.4.0:
          variables:
            TESTDINO_TOKEN: "TESTDINO_TOKEN"
      - docker#v5.11.0:
          image: "mcr.microsoft.com/playwright:v1.61.1-jammy"
          propagate-environment: true
          environment:
            - "TESTDINO_TOKEN"
    command:
      - npm ci
      - npx playwright install --with-deps
      - npx playwright test

TipThe secrets plugin injects TESTDINO_TOKEN and the docker plugin forwards it into the container with propagate-environment. Results stream live even when tests fail, so failures land on the dashboard without a separate upload step.

Sharded Test Runs

For large test suites, run shards in parallel. Each shard streams its own results, and TestDino groups them into one run when they share a ci-run-id.

How it works

parallelism: 4 fans the step out into 4 Buildkite jobs
Each shard runs Playwright with --shard=<n>/<total> from its parallel-job index
Each shard streams its results to TestDino during the run
Every shard sets the same TESTDINO_CI_RUN_ID so TestDino merges them into a single run
No merge or upload step runs after the shards finish

Full sharded config

.buildkite/pipeline.yml: Sharded pipeline

.buildkite/pipeline.yml

steps:
  - label: ":playwright: shard %n"
    parallelism: 4
    plugins:
      - secrets#v2.4.0:
          variables:
            TESTDINO_TOKEN: "TESTDINO_TOKEN"
      - docker#v5.11.0:
          image: "mcr.microsoft.com/playwright:v1.61.1-jammy"
          propagate-environment: true
          environment:
            - "TESTDINO_TOKEN"
    env:
      # Same run id across all shards groups them into one run in TestDino.
      # $$ defers interpolation to job runtime (Buildkite interpolates the YAML at upload time).
      TESTDINO_CI_RUN_ID: "$$BUILDKITE_BUILD_NUMBER"
    command:
      # BUILDKITE_PARALLEL_JOB is 0-based; Playwright shards are 1-based.
      - export SHARD=$$(($$BUILDKITE_PARALLEL_JOB + 1))
      - export TOTAL=$$BUILDKITE_PARALLEL_JOB_COUNT
      - npm ci
      - npx playwright install --with-deps
      - npx playwright test --shard=$$SHARD/$$TOTAL

Key details

Detail	Notes
`parallelism: 4`	Buildkite runs 4 copies of the step as separate parallel jobs.
`BUILDKITE_PARALLEL_JOB`	0-based index of the current job. Add 1 for Playwright’s 1-based `--shard`.
`BUILDKITE_PARALLEL_JOB_COUNT`	Total shard count, passed as the denominator of `--shard`.
`TESTDINO_CI_RUN_ID`	Read by `@testdino/playwright` to group all shards into one run.
`$$` prefix	Buildkite interpolates the YAML at upload time. `$$` defers runtime variables so they resolve when the job runs, not at upload.
`TESTDINO_TOKEN`	Injected by the `secrets` plugin and redacted from logs.

Set the ciRunId reporter option in playwright.config to process.env.TESTDINO_CI_RUN_ID if you prefer to pass it through the config instead of relying on the environment variable.

Pipeline execution

After the build starts, Buildkite shows each shard as a separate job under the step. Each shard streams its results, and TestDino merges them into one run as they complete.

Results in TestDino

The test run appears in your TestDino dashboard with full failure details, flaky detection, and trend data as tests complete.

Troubleshooting

Results not appearing on the dashboard

Confirm @testdino/playwright is in your playwright.config reporter array and TESTDINO_TOKEN is injected. The docker plugin must list TESTDINO_TOKEN under environment so the value reaches the container. Results stream during the run, so no separate upload step is required.

Sharded runs show up as separate runs

Confirm every shard sets the same TESTDINO_CI_RUN_ID. If the id resolves to an empty value, check the $$ prefix: without it Buildkite interpolates BUILDKITE_BUILD_NUMBER at upload time, before the value exists.

TESTDINO_TOKEN not available in the job

Open Agents → Clusters → your cluster → Secrets and confirm a secret with the exact key TESTDINO_TOKEN exists. The secrets plugin variables map is ENV_VAR: "SecretKey", and secret keys are case-sensitive.

CI Optimization

Reduce CI time with smart reruns

Environment Mapping

Route test results to Dev, Staging, or Production by branch

Integrations

Connect Slack, Jira, Linear, Asana, and more

TestDino MCP

Access test results and fix issues with AI agents

Getting Started

Installation & Setup

AI & MCP

Guides

Project

Test Management

Organization

Integrations

Data Privacy

Resources

Prerequisites

Set Up Your API Key

Basic Pipeline Config

Sharded Test Runs

How it works

Full sharded config

Key details

Pipeline execution

Results in TestDino

Troubleshooting

CI Optimization

Environment Mapping

Integrations

TestDino MCP

​Prerequisites

​Set Up Your API Key

​Basic Pipeline Config

​Sharded Test Runs

​How it works

​Full sharded config

​Key details

​Pipeline execution

​Results in TestDino

​Troubleshooting

​Related

CI Optimization

Environment Mapping

Integrations

TestDino MCP

Prerequisites

Set Up Your API Key

Basic Pipeline Config

Sharded Test Runs

How it works

Full sharded config

Key details

Pipeline execution

Results in TestDino

Troubleshooting

Related