Test automation
What to automate and what not to, tools by layer, a maintainable test structure, flaky tests and running everything in CI.
Automation doesn't replace testers; it frees them. Machines repeat checks quickly and consistently on every build, while people focus on exploratory testing, new features and judgement. The goal is fast, trustworthy feedback, not the largest possible number of scripts.
What to automate
| Good candidates | Poor candidates |
|---|---|
| Smoke tests on every build | Features that change every week |
| Regression of stable, critical journeys | One-off checks |
| API and business-logic tests | Visual judgement and UX feel |
| Data-driven tests with many inputs | Tests that need physical interaction |
| Checks across many browsers or devices | Exploratory testing |
A practical rule: automate a check when it is stable, valuable and will be run many times. Push tests down the pyramid — an API test is faster and more stable than a UI test for the same rule.
Tools by layer
| Layer | Popular tools |
|---|---|
| Unit | Jest, Vitest, JUnit, pytest, NUnit |
| API | Postman/Newman, REST Assured, pytest + requests, Karate |
| Web UI | Playwright, Cypress, Selenium WebDriver |
| Mobile UI | Appium, Espresso, XCUITest, Maestro |
| Performance | k6, JMeter, Gatling, Locust |
| Visual regression | Playwright screenshots, Percy, Applitools |
A maintainable structure
Keep tests readable and keep selectors in one place. The Page Object pattern wraps a page's elements and actions in a class, so a changed button only needs fixing once.
// pages/login.page.ts
export class LoginPage {
constructor(private page: Page) {}
async open() { await this.page.goto('/login'); }
async login(email: string, password: string) {
await this.page.getByLabel('Email').fill(email);
await this.page.getByLabel('Password').fill(password);
await this.page.getByRole('button', { name: 'Sign in' }).click();
}
}
// tests/login.spec.ts
test('user sees dashboard after login', async ({ page }) => {
const login = new LoginPage(page);
await login.open();
await login.login(USER.email, USER.password);
await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible();
});
Flaky tests
A flaky test passes and fails without any code change. Flaky suites destroy trust, so we treat them as bugs.
| Cause | Fix |
|---|---|
Fixed waits (sleep(3)) | Wait for a condition: element visible, request finished |
| Brittle selectors (long CSS paths) | Roles, labels or dedicated data-testid attributes |
| Shared or leftover test data | Create isolated data per test, clean up after |
| Test order dependency | Each test sets up its own state |
| Unstable environment or third-party services | Mock external services, monitor the test environment |
Automation in CI/CD
Run fast suites on every pull request so a change can't merge while red. Run full regression, cross-browser and performance suites nightly or before releases. Run tests in parallel to keep feedback under ten minutes, and publish readable reports with screenshots, videos and traces for failures. See Shift-left in practice for our pipeline.
Measuring success
- Time from commit to feedback
- Share of regression covered automatically on critical journeys
- Flaky test rate (aim for close to zero)
- Bugs found by automation before release vs. bugs found by users