Playwright Azure DevOps Pipeline Setup

What you’ll learn

How to upload Playwright results from Azure DevOps pipelines to TestDino
How to configure sharded test runs with merged reporting
How to set up the TESTDINO_TOKEN as a secret pipeline variable

Set up an Azure DevOps pipeline to upload Playwright test results to TestDino and view aggregated analytics, failure analysis, and flaky test detection on your dashboard. This guide covers a basic pipeline, sharded pipeline for parallel test execution across multiple shards, and merged reporting.

Prerequisites

Before setting up, ensure you have:

A TestDino account with a project created
A TestDino API key (Generate API Keys)
An Azure DevOps project with pipelines enabled
A Playwright test project (or clone the TestDino Example Repository to get started)
Playwright configured with JSON and HTML reporters in playwright.config.js:

playwright.config.js

// ...existing config

reporter: [
  ['html', { outputDir: './playwright-report' }],  // Optional
  ['json', { outputFile: './playwright-report/report.json' }],  // ✅ Required
]

Set Up Your API Key

Store your TestDino API key as a secret pipeline variable so it is available to your pipeline without exposing it in logs or config files.

Open your Azure DevOps pipeline
Click Edit on the pipeline
Click Variables
Click New variable
Set the name to TESTDINO_TOKEN
Paste your TestDino API key as the value
Check Keep this value secret
Save the pipeline

Warning Never commit your API key directly in pipeline files. Always use secret variables. Secret variables are not exposed in pipeline logs.

Basic Pipeline Config

For a simple setup without sharding, add the upload step after your Playwright tests.

azure-pipelines.yml — Basic pipeline

azure-pipelines.yml

trigger:
  branches:
    include:
      - main

pr:
  branches:
    include:
      - main

variables:
  CI: "true"

pool:
  vmImage: ubuntu-latest

steps:
  - checkout: self

  - task: UseNode@1
    inputs:
      version: "20.x"
    displayName: Install Node.js

  - script: npm ci
    displayName: Install dependencies

  - script: npx playwright install --with-deps
    displayName: Install Playwright browsers

  - script: npx playwright test
    displayName: Run Playwright tests

  - script: npx tdpw upload ./playwright-report --token="$TESTDINO_TOKEN"
    displayName: Upload results to TestDino
    condition: always()
    env:
      TESTDINO_TOKEN: $(TESTDINO_TOKEN)

Tip The condition: always() ensures the upload runs even if tests fail. The env block maps the secret variable to an environment variable accessible in the script.

Upload Options

Flag	Description	Default
`--environment <value>`	Target environment tag (staging, production, qa)	unknown
`--tag <values>`	Comma-separated run tags for categorization (max 5)	None
`--upload-images`	Upload image attachments	false
`--upload-videos`	Upload video attachments	false
`--upload-html`	Upload HTML reports	false
`--upload-traces`	Upload trace files	false
`--upload-files`	Upload file attachments (.md, .pdf, .txt, .log)	false
`--upload-full-json`	Upload all attachments	false
`--json`	Output results as JSON to stdout (for CI/CD)	false
`-v, --verbose`	Enable verbose logging	false

Sharded Test Runs

For larger test suites, Azure DevOps matrix strategy splits tests across multiple jobs. Each shard produces a blob report that gets merged before uploading to TestDino.

How it works

Azure DevOps runs Playwright across 4 shards using a matrix strategy
Each shard publishes its blob report as a pipeline artifact
A separate MergeAndUpload stage downloads all blob reports, merges them into a single report.json, and uploads to TestDino
The merge stage runs even if some shards fail (condition: always())

Full sharded config

azure-pipelines.yml — Sharded pipeline

azure-pipelines.yml

trigger:
  branches:
    include:
      - main

pr:
  branches:
    include:
      - main

variables:
  CI: "true"

stages:
  - stage: Test
    jobs:
      - job: Playwright
        pool:
          vmImage: ubuntu-latest
        strategy:
          matrix:
            shard1:
              SHARD: 1/4
            shard2:
              SHARD: 2/4
            shard3:
              SHARD: 3/4
            shard4:
              SHARD: 4/4
        steps:
          - checkout: self

          - task: UseNode@1
            inputs:
              version: "20.x"
            displayName: Install Node.js

          - script: npm ci
            displayName: Install dependencies

          - script: npx playwright install --with-deps
            displayName: Install Playwright browsers

          - script: npx playwright test --shard=$(SHARD)
            displayName: Run Playwright shard $(SHARD)

          - task: PublishTestResults@2
            condition: always()
            inputs:
              testResultsFormat: JUnit
              testResultsFiles: test-results/junit.xml
              mergeTestResults: false
              testRunTitle: Playwright shard $(SHARD)
            displayName: Publish shard test results

          - task: PublishPipelineArtifact@1
            condition: always()
            inputs:
              targetPath: blob-report
              artifact: blob-report-$(System.JobPositionInPhase)
              publishLocation: pipeline
            displayName: Publish shard blob report

  - stage: MergeAndUpload
    dependsOn: Test
    condition: always()
    jobs:
      - job: MergeAndUpload
        pool:
          vmImage: ubuntu-latest
        steps:
          - checkout: self

          - task: UseNode@1
            inputs:
              version: "20.x"
            displayName: Install Node.js

          - script: npm ci
            displayName: Install dependencies

          - task: DownloadPipelineArtifact@2
            inputs:
              patterns: "blob-report-*/*"
              path: all-blob-reports
            displayName: Download blob reports

          - script: |
              mkdir -p playwright-report all-blob-reports-flat
              find all-blob-reports -type f -exec cp {} all-blob-reports-flat/ \;
              if find all-blob-reports-flat -type f | grep -q .; then
                npx playwright merge-reports --reporter=json ./all-blob-reports-flat > playwright-report/report.json
              else
                echo "No blob report files were found to merge."
                exit 1
              fi
            displayName: Merge Playwright reports to JSON
            condition: always()

          - script: |
              if [ ! -f playwright-report/report.json ]; then
                echo "Merged report was not created, so TestDino upload is being skipped."
                exit 0
              fi
              npx tdpw upload ./playwright-report --token="$TESTDINO_TOKEN"
            displayName: Upload results to TestDino
            condition: always()
            env:
              TESTDINO_TOKEN: $(TESTDINO_TOKEN)

          - task: PublishPipelineArtifact@1
            condition: always()
            inputs:
              targetPath: playwright-report
              artifact: playwright-report
              publishLocation: pipeline
            displayName: Publish merged Playwright report

Info The MergeAndUpload stage uses dependsOn: Test with condition: always() so it runs even when some shards fail. The upload step also checks if the merged report exists before attempting the upload, avoiding unnecessary errors.

Key details in the sharded config

Config Block	What It Does
`strategy: matrix`	Defines 4 shards with `SHARD: 1/4` through `4/4`
`--shard=$(SHARD)`	Passes the shard value directly to Playwright (Azure DevOps uses 1-based values)
`PublishPipelineArtifact`	Saves each shard’s blob report as a named artifact (`blob-report-0`, `blob-report-1`, etc.)
`DownloadPipelineArtifact`	Downloads all blob report artifacts matching `blob-report-/`
`find ... -exec cp`	Flattens all blob files into a single directory for merging
`merge-reports --reporter=json`	Merges blob reports into a single `report.json`
`condition: always()`	Ensures merge and upload run regardless of shard outcomes
`env: TESTDINO_TOKEN`	Maps the secret variable to an environment variable for the script

Pipeline execution

After the pipeline runs, Azure DevOps shows all shard jobs and the merge stage in the pipeline view. Azure DevOps pipeline execution view showing 4 Playwright shard jobs and MergeAndUpload stage

Results in TestDino

Once uploaded, the test run appears in your TestDino dashboard with full failure details, flaky detection, and trend data. TestDino test run screen showing uploaded results from Azure DevOps pipeline with pass/fail counts and failure details

TestDino test run screen showing uploaded results from Azure DevOps pipeline with pass/fail counts and failure details

Rerun Failed Tests

Cache test metadata to enable selective reruns:

- script: npx tdpw cache --token="$TESTDINO_TOKEN"
  displayName: Cache rerun metadata
  condition: always()
  env:
    TESTDINO_TOKEN: $(TESTDINO_TOKEN)

Rerun only failed tests on the next run:

- script: |
    FAILED=$(npx tdpw last-failed --token="$TESTDINO_TOKEN")
    if [ -n "$FAILED" ]; then
      npx playwright test $FAILED
    else
      echo "No failed tests found."
    fi
  displayName: Rerun failed tests
  env:
    TESTDINO_TOKEN: $(TESTDINO_TOKEN)

For advanced rerun strategies, caching patterns, and CI optimization techniques, see CI Optimization.

Troubleshooting

Upload step skipped after test failure

Add condition: always() to the upload step so it runs regardless of test exit code
For sharded runs, ensure the MergeAndUpload stage has condition: always() and dependsOn: Test

No report found at ./playwright-report

Verify your playwright.config.ts outputs to playwright-report/ (default location)
For sharded runs, ensure all shards publish blob-report as a pipeline artifact and the merge step writes to playwright-report/

TESTDINO_TOKEN not available in script

Secret variables in Azure DevOps are not automatically available as environment variables. You must map them explicitly using the env block in the script step
Verify the variable name matches exactly: TESTDINO_TOKEN: $(TESTDINO_TOKEN)

Blob reports not found during merge

Ensure each shard uses PublishPipelineArtifact with condition: always() to publish even on failure
The DownloadPipelineArtifact step uses pattern blob-report-*/*. Verify artifact names match this pattern.
Check that blob-report directory exists after running Playwright (configure reporter: [['blob']] in playwright.config.ts)

Next Steps

CI Optimization

Reduce CI time with smart reruns

Branch Mapping

Map branches to environments for organized test runs

Integrations

Connect Slack, Jira, Linear, Asana, and more

Azure DevOps Extension

View test runs inside Azure DevOps

​Prerequisites

​Set Up Your API Key

​Basic Pipeline Config

​Upload Options

​Sharded Test Runs

​How it works

​Full sharded config

​Key details in the sharded config

​Pipeline execution

​Results in TestDino

​Rerun Failed Tests

​Troubleshooting

​Next Steps

CI Optimization

Branch Mapping

Integrations

Azure DevOps Extension

Prerequisites

Set Up Your API Key

Basic Pipeline Config

Upload Options

Sharded Test Runs

How it works

Full sharded config

Key details in the sharded config

Pipeline execution

Results in TestDino

Rerun Failed Tests

Troubleshooting

Next Steps