Python CLI (testdino)

The testdino CLI is a Python package that connects pytest-based Playwright tests to the TestDino platform.

It caches test metadata, retrieves the last failed tests, and uploads reports with all their attachments.

1. Package and Platform Information

• Package name: testdino (available on PyPI (opens in a new tab))
• Supported Python versions: 3.9, 3.10, 3.11, 3.12 or higher
• License: MIT

2. When to Use the Python CLI

Pick this CLI if you're running Playwright tests through pytest.

It works with the pytest-playwright plugin and integrates cleanly into Python-based CI pipelines.

3. Prerequisites

• Python 3.9, 3.10, 3.11, 3.12 or higher
• An existing pytest + Playwright test suite
• pytest-playwright-json installed
• A TestDino account with an API token

Quick Start

1. JSON Report Requirement

The upload command needs a JSON report to work. Without it, TestDino can't parse your test results. Make sure you generate this report when running tests.

2. Install Required Plugins

pip install pytest-playwright-json pytest-html testdino

3. Run Tests with JSON and HTML Reports

pytest --playwright-json=test-results/report.json --html=test-results/index.html --self-contained-html

4. Five Minute Tutorial

Here's how to get started:

1. Install Packages

Install the CLI and pytest plugins:

pip install pytest-playwright-json pytest-html

2. Run Tests

Run pytest with the --playwright-json flag (and optionally HTML reporting).

3. Upload Reports

testdino upload ./test-results --token="your-token"

4. Cache Metadata

testdino cache --working-dir test-results --token="your-token"

5. View Results

Open TestDino to view your test run.

Features

• Test metadata caching after Playwright runs
• Last failed tests for faster CI pipelines
• Zero configuration report discovery
• Smart shard detection
• CI/CD ready for GitHub Actions, GitLab CI, Jenkins, and Azure DevOps
• Token-based API authentication

Commands

1. Cache Command

Store test execution metadata after Playwright runs. This data powers the last-failed command and feeds into TestDino analytics.

1.1 Usage

Basic usage:

testdino cache --token="your-token"

With a custom working directory:

testdino cache --working-dir ./test-results --token="your-token"

With verbose logging:

testdino cache --verbose --token="your-token"

1.2 Options

2. Last Failed Command

Retrieve cached test failures for intelligent reruns. This command outputs test identifiers that you can pass directly to pytest.

2.1 Usage

Basic usage:

testdino last-failed --token="your-token"

Run only the last failed tests:

pytest $(testdino last-failed --token="your-token")

With a custom branch and commit:

testdino last-failed --branch="main" --commit="abc123" --token="your-token"

2.2 Options

3. Upload Command

Upload Playwright test reports with attachments to TestDino.

The testdino upload command requires a JSON report to function.

You must:

• Install required plugins:

  pip install pytest-playwright-json pytest-html

• Run your tests with the --playwright-json flag:

  pytest --playwright-json=test-results/report.json

• Optionally generate HTML reports:

  pytest --html=test-results/index.html --self-contained-html

3.1 Usage

Basic upload:

testdino upload ./test-results --token="your-token"

Upload with specific attachments:

testdino upload ./test-results --token="your-token" --upload-images --upload-videos

Upload all attachments:

testdino upload ./test-results --token="your-token" --upload-full-json

3.2 Options

4. Environment Variables

You can set your token as an environment variable instead of passing it with each command:

export TESTDINO_TOKEN="your-api-token"

CI/CD Integration

1. GitHub Actions

name: Playwright Tests
on: [push, pull_request]
 
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
 
      - name: Install dependencies
        run: |
          pip install pytest pytest-playwright pytest-playwright-json pytest-html testdino
          playwright install chromium --with-deps
 
      - name: Run tests
        run: |
          pytest \
            --playwright-json=test-results/report.json \
            --html=test-results/index.html \
            --self-contained-html
 
      - name: Cache rerun metadata
        if: always()
        run: testdino cache --working-dir test-results --token="${{ secrets.TESTDINO_TOKEN }}" -v
 
      - name: Upload test reports
        if: always()
        run: testdino upload ./test-results --token="${{ secrets.TESTDINO_TOKEN }}" --upload-full-json

2. GitLab CI

image: python:3.11
 
stages:
  - test
 
playwright-tests:
  stage: test
  script:
    - pip install pytest pytest-playwright pytest-playwright-json pytest-html testdino
    - playwright install chromium --with-deps
    - pytest --playwright-json=test-results/report.json --html=test-results/index.html --self-contained-html
    - testdino upload ./test-results --token="$TESTDINO_TOKEN" --upload-full-json
  when: always

3. Jenkins

pipeline {
    agent any
 
    environment {
        TESTDINO_TOKEN = credentials('testdino-token')
    }
 
    stages {
        stage('Test') {
            steps {
                sh 'pip install pytest pytest-playwright pytest-playwright-json pytest-html testdino'
                sh 'playwright install chromium --with-deps'
                sh 'pytest --playwright-json=test-results/report.json --html=test-results/index.html --self-contained-html'
                sh 'testdino upload ./test-results --token="$TESTDINO_TOKEN" --upload-full-json'
            }
        }
    }
}

Examples

1. Basic Workflow

Run tests with JSON and HTML reports:

pytest \
  --playwright-json=test-results/report.json \
  --html=test-results/index.html \
  --self-contained-html

Cache metadata (specify the directory where test results are located):

testdino cache --working-dir test-results --token="your-token"

2. Intelligent Test Reruns

Run tests with JSON and HTML reports, then cache results:

pytest \
  --playwright-json=test-results/report.json \
  --html=test-results/index.html \
  --self-contained-html

testdino cache --working-dir test-results --token="your-token"

On the next rerun, execute only previously failed tests:

pytest $(testdino last-failed --token="your-token")

3. Complete CI/CD Workflow

Run all tests with JSON and HTML reports:

pytest \
  --playwright-json=test-results/report.json \
  --html=test-results/index.html \
  --self-contained-html

Cache test metadata (specify the directory where test results are located):

testdino cache --working-dir test-results --token="$TESTDINO_TOKEN"

Rerun only failed tests if the first run had failures:

if [ $? -ne 0 ]; then
  FAILED=$(testdino last-failed --token="$TESTDINO_TOKEN")
  [ -n "$FAILED" ] && pytest $FAILED
fi

Authentication

1. Getting Your Token

• Create an account at TestDino (opens in a new tab).
• Go to Settings, then API Tokens.
• Generate a new token.
• Store it securely in your CI/CD secrets.

2. Security Best Practices

• Never commit tokens to version control
• Use environment variables or CI/CD secrets
• Rotate tokens regularly