Test Controls

Test Controls sets per-test build rules in TestDino: quarantine a flaky test so its failures stop failing the build, cover only 1 known error, or make a test critical so it must pass on the first try. The TestDino reporter applies the rules when the test run ends. A quarantined test case keeps running and reporting. Its failures stay in test run results, notifications, analytics, and flaky rates; only the build result changes.

​

Quick Reference

Viewers and billing users can open the Test Controls page but not change it.

​

Open Test Controls

The Test Controls page in the project sidebar has 4 tabs: Quarantined, Candidates (tests that were flaky in more than 5% of their test runs in the last 30 days and are not quarantined or critical), Critical, and Watching. The header shows how many quarantine slots are used, for example 3/12 quarantine slots used. Each test also has a Test Controls button on its test case page in a test run and in the Test Explorer side panel. The button reads Quarantined · ends with the end date when the test is quarantined, or Critical when it is critical. Click it to quarantine the test, mark it critical, assign it, or watch it.

​

Quarantine a test

A quarantine needs an end date and an assignee. 1 quarantine covers the test on every browser and Playwright project it runs on. A test that is already quarantined opens for editing instead, and a critical test cannot be quarantined. The dialog shows the quarantine limit and disables Quarantine when the project is at it, for example Quarantine limit reached (12 of 12). Release a test, or ask an admin to raise the limit. A test that has not run in the project recently cannot be quarantined.

​

Quarantine several tests at once

Select test cases in Test Explorer to quarantine them together, for example when an outside service is down. Filter by tag, spec file, or search first, then tick the rows (the header checkbox selects the whole page), and click Quarantine N tests. Before you save, the dialog counts the tests it would quarantine. Each test gets its own quarantine and is released on its own.

​

How the build result changes

When the test run ends, the reporter checks every failure against the project’s rules: The reporter prints TestDino Test Controls: fetching rules… after it authenticates. At the end of the test run it prints the rules it loaded (TestDino Test Controls: 3 rules (2 quarantined, 0 known errors, 1 critical)), the quarantined tests that ran (Quarantined tests in this run: 2 (2 failed, excluded from the build)), and a verdict. When no quarantined test ran and no critical test passed only on retry, only the rules line appears. The verdict is one of: Playwright uses the last status a reporter returns. If another reporter in your config sets the test run’s status, list TestDino after it.

​

What the test run and notifications show

The test run keeps its real status. A test run with a failed test case reads Failed even when quarantine passed the build. Its quarantined test cases are tagged Quarantined, and the test run notes 2 quarantined failures covered, so they didn't fail the build. To find these test runs, filter the test runs list by Test Controls: Changed by Test Controls, Had quarantined failures, or Critical test passed only on retry. In a test run’s summary, Filter → Test Controls narrows it to Quarantined or Critical: passed only on retry test cases.

​

Mark a test critical

A critical test must pass on the first try. If it fails and then passes on a retry, the build fails. Click Test Controls on the test case page, then Mark critical. Owners and admins can mark up to 50 critical tests per project. A quarantined test cannot be marked critical until its quarantine is released. When a critical test passes only on retry:

• The test run stays Passed, and the test case is tagged Critical: passed only on retry.
• Slack gets a second message right below the test run message, such as Critical test passed only on retry in run #42, with each test linked to its page.
• The PR comment opens with a caution box naming the test, and the PR description gets a Critical test passed only on retry line.
• The GitHub check title adds ❗ 1 critical passed on retry, and the quality gate fails.

A critical test that fails outright is an ordinary failure: the build fails as it would for any failure.

​

Watch or assign a test

Click Test Controls, then Watch, to get a message when the test changes state: it starts failing, passes only on retry, or passes again. Click Stop watching to end it.

• The first test run after you start watching only records the test’s state, so the first message comes with the next change.
• A test sends you at most 1 message every 6 hours. A change inside that window is sent after it, if the test is still in its new state.
• The message is a Slack direct message from the TestDino Slack app when the project’s Slack app can reach you, otherwise an email. A Slack workspace connected before Test Controls sends email until the Slack app is reconnected.

To assign a test, click Assign in the Assignee row. The assignee starts watching the test, so they get the same messages. Test run summaries, PR comments, and Slack channel messages never show the assignee’s name.

​

When a quarantine ends

Every quarantine ends. The Released view on the Quarantined tab shows why: A pass only on retry resets the count to 0, and a test run where the test was skipped does not count. The Passes in a Row column shows progress, for example 6 of 10. Expand a row to see its Details, its History, and Covered failures: the test runs in the last 90 days whose failure the quarantine kept from failing the build, each linked to its page. From a row’s menu:

• Extend or edit changes the end date, assignee, or note.
• Link ticket records 1 Jira or Linear issue. When it is marked done, the quarantine is released.
• Release ends the quarantine now. The test fails the build again when it fails.

​

Configure Test Controls settings

Click Settings on the Test Controls page. Owners and admins can change these: The limit is how many tests can be quarantined at once. At least 1 test is always allowed, and lowering it never releases a test.

​

Troubleshooting

​

Related