> ## Documentation Index
> Fetch the complete documentation index at: https://docs.testdino.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Playwright Azure DevOps Pipeline Setup
> Run Playwright in Azure DevOps Pipelines and upload results to TestDino, with sharded jobs.
**What you'll learn**
* How to upload Playwright results from Azure DevOps pipelines to TestDino
* How to configure sharded test runs with merged reporting
* How to set up the TESTDINO\_TOKEN as a secret pipeline variable
Set up an Azure DevOps pipeline to upload Playwright test results to TestDino and view aggregated analytics, failure analysis, and flaky test detection on your dashboard.
This guide covers a basic pipeline, sharded pipeline for parallel test execution across multiple shards, and merged reporting.
## Prerequisites
Before setting up, ensure you have:
* A TestDino account with a project created
* A TestDino API key ([Generate API Keys](/guides/generate-api-keys))
* An Azure DevOps project with pipelines enabled
* A Playwright test project (or clone the TestDino Example Repository to get started)
* Playwright configured with JSON and HTML reporters in `playwright.config.js`:
```javascript playwright.config.js theme={null}
// ...existing config
reporter: [
['html', { outputDir: './playwright-report' }], // Optional
['json', { outputFile: './playwright-report/report.json' }], // ✅ Required
]
```
## Set Up Your API Key
Store your TestDino API key as a secret pipeline variable so it is available to your pipeline without exposing it in logs or config files.
1. Open your Azure DevOps pipeline
2. Click **Edit** on the pipeline
3. Click **Variables**
4. Click **New variable**
5. Set the name to `TESTDINO_TOKEN`
6. Paste your TestDino API key as the value
7. Check **Keep this value secret**
8. Save the pipeline
**Warning**
Never commit your API key directly in pipeline files. Always use secret variables. Secret variables are not exposed in pipeline logs.
## Basic Pipeline Config
For a simple setup without sharding, add the upload step after your Playwright tests.
```yaml azure-pipelines.yml theme={null}
trigger:
branches:
include:
- main
pr:
branches:
include:
- main
variables:
CI: "true"
pool:
vmImage: ubuntu-latest
steps:
- checkout: self
- task: UseNode@1
inputs:
version: "20.x"
displayName: Install Node.js
- script: npm ci
displayName: Install dependencies
- script: npx playwright install --with-deps
displayName: Install Playwright browsers
- script: npx playwright test
displayName: Run Playwright tests
- script: npx tdpw upload ./playwright-report --token="$TESTDINO_TOKEN"
displayName: Upload results to TestDino
condition: always()
env:
TESTDINO_TOKEN: $(TESTDINO_TOKEN)
```
**Tip**
The `condition: always()` ensures the upload runs even if tests fail. The `env` block maps the secret variable to an environment variable accessible in the script.
## Upload Options
| Flag | Description | Default |
| :---------------------- | :-------------------------------------------------- | :------ |
| `--environment ` | Target environment tag (staging, production, qa) | unknown |
| `--tag ` | Comma-separated run tags for categorization (max 5) | None |
| `--upload-images` | Upload image attachments | false |
| `--upload-videos` | Upload video attachments | false |
| `--upload-html` | Upload HTML reports | false |
| `--upload-traces` | Upload trace files | false |
| `--upload-files` | Upload file attachments (.md, .pdf, .txt, .log) | false |
| `--upload-full-json` | Upload all attachments | false |
| `--json` | Output results as JSON to stdout (for CI/CD) | false |
| `-v, --verbose` | Enable verbose logging | false |
## Sharded Test Runs
For larger test suites, Azure DevOps matrix strategy splits tests across multiple jobs. Each shard produces a blob report that gets merged before uploading to TestDino.
### How it works
1. Azure DevOps runs Playwright across 4 shards using a matrix strategy
2. Each shard publishes its blob report as a pipeline artifact
3. A separate `MergeAndUpload` stage downloads all blob reports, merges them into a single `report.json`, and uploads to TestDino
4. The merge stage runs even if some shards fail (`condition: always()`)
### Full sharded config
```yaml azure-pipelines.yml theme={null}
trigger:
branches:
include:
- main
pr:
branches:
include:
- main
variables:
CI: "true"
stages:
- stage: Test
jobs:
- job: Playwright
pool:
vmImage: ubuntu-latest
strategy:
matrix:
shard1:
SHARD: 1/4
shard2:
SHARD: 2/4
shard3:
SHARD: 3/4
shard4:
SHARD: 4/4
steps:
- checkout: self
- task: UseNode@1
inputs:
version: "20.x"
displayName: Install Node.js
- script: npm ci
displayName: Install dependencies
- script: npx playwright install --with-deps
displayName: Install Playwright browsers
- script: npx playwright test --shard=$(SHARD)
displayName: Run Playwright shard $(SHARD)
- task: PublishTestResults@2
condition: always()
inputs:
testResultsFormat: JUnit
testResultsFiles: test-results/junit.xml
mergeTestResults: false
testRunTitle: Playwright shard $(SHARD)
displayName: Publish shard test results
- task: PublishPipelineArtifact@1
condition: always()
inputs:
targetPath: blob-report
artifact: blob-report-$(System.JobPositionInPhase)
publishLocation: pipeline
displayName: Publish shard blob report
- stage: MergeAndUpload
dependsOn: Test
condition: always()
jobs:
- job: MergeAndUpload
pool:
vmImage: ubuntu-latest
steps:
- checkout: self
- task: UseNode@1
inputs:
version: "20.x"
displayName: Install Node.js
- script: npm ci
displayName: Install dependencies
- task: DownloadPipelineArtifact@2
inputs:
patterns: "blob-report-*/*"
path: all-blob-reports
displayName: Download blob reports
- script: |
mkdir -p playwright-report all-blob-reports-flat
find all-blob-reports -type f -exec cp {} all-blob-reports-flat/ \;
if find all-blob-reports-flat -type f | grep -q .; then
npx playwright merge-reports --reporter=json ./all-blob-reports-flat > playwright-report/report.json
else
echo "No blob report files were found to merge."
exit 1
fi
displayName: Merge Playwright reports to JSON
condition: always()
- script: |
if [ ! -f playwright-report/report.json ]; then
echo "Merged report was not created, so TestDino upload is being skipped."
exit 0
fi
npx tdpw upload ./playwright-report --token="$TESTDINO_TOKEN"
displayName: Upload results to TestDino
condition: always()
env:
TESTDINO_TOKEN: $(TESTDINO_TOKEN)
- task: PublishPipelineArtifact@1
condition: always()
inputs:
targetPath: playwright-report
artifact: playwright-report
publishLocation: pipeline
displayName: Publish merged Playwright report
```
**Info**
The `MergeAndUpload` stage uses `dependsOn: Test` with `condition: always()` so it runs even when some shards fail. The upload step also checks if the merged report exists before attempting the upload, avoiding unnecessary errors.
### Key details in the sharded config
| Config Block | What It Does |
| :------------------------------ | :------------------------------------------------------------------------------------------ |
| `strategy: matrix` | Defines 4 shards with `SHARD: 1/4` through `4/4` |
| `--shard=$(SHARD)` | Passes the shard value directly to Playwright (Azure DevOps uses 1-based values) |
| `PublishPipelineArtifact` | Saves each shard's blob report as a named artifact (`blob-report-0`, `blob-report-1`, etc.) |
| `DownloadPipelineArtifact` | Downloads all blob report artifacts matching `blob-report-*/*` |
| `find ... -exec cp` | Flattens all blob files into a single directory for merging |
| `merge-reports --reporter=json` | Merges blob reports into a single `report.json` |
| `condition: always()` | Ensures merge and upload run regardless of shard outcomes |
| `env: TESTDINO_TOKEN` | Maps the secret variable to an environment variable for the script |
### Pipeline execution
After the pipeline runs, Azure DevOps shows all shard jobs and the merge stage in the pipeline view.
### Results in TestDino
Once uploaded, the test run appears in your TestDino dashboard with full failure details, flaky detection, and trend data.
## Rerun Failed Tests
Cache test metadata to enable selective reruns:
```yaml theme={null}
- script: npx tdpw cache --token="$TESTDINO_TOKEN"
displayName: Cache rerun metadata
condition: always()
env:
TESTDINO_TOKEN: $(TESTDINO_TOKEN)
```
Rerun only failed tests on the next run:
```yaml theme={null}
- script: |
FAILED=$(npx tdpw last-failed --token="$TESTDINO_TOKEN")
if [ -n "$FAILED" ]; then
npx playwright test $FAILED
else
echo "No failed tests found."
fi
displayName: Rerun failed tests
env:
TESTDINO_TOKEN: $(TESTDINO_TOKEN)
```
For advanced rerun strategies, caching patterns, and CI optimization techniques, see [CI Optimization](/guides/playwright-ci-optimization).
## Troubleshooting
* Add `condition: always()` to the upload step so it runs regardless of test exit code
* For sharded runs, ensure the `MergeAndUpload` stage has `condition: always()` and `dependsOn: Test`
* Verify your `playwright.config.ts` outputs to `playwright-report/` (default location)
* For sharded runs, ensure all shards publish `blob-report` as a pipeline artifact and the merge step writes to `playwright-report/`
* Secret variables in Azure DevOps are not automatically available as environment variables. You must map them explicitly using the `env` block in the script step
* Verify the variable name matches exactly: `TESTDINO_TOKEN: $(TESTDINO_TOKEN)`
* Ensure each shard uses `PublishPipelineArtifact` with `condition: always()` to publish even on failure
* The `DownloadPipelineArtifact` step uses pattern `blob-report-*/*`. Verify artifact names match this pattern.
* Check that `blob-report` directory exists after running Playwright (configure `reporter: [['blob']]` in `playwright.config.ts`)
## Next Steps
Reduce CI time with smart reruns
Map branches to environments for organized test runs
Connect Slack, Jira, Linear, Asana, and more
View test runs inside Azure DevOps