← QA Library
Chapter 7 / 9 · 8 Min. Lesezeit

Test automation

What to automate and what not to, tools by layer, a maintainable test structure, flaky tests and running everything in CI.

Automation doesn't replace testers; it frees them. Machines repeat checks quickly and consistently on every build, while people focus on exploratory testing, new features and judgement. The goal is fast, trustworthy feedback, not the largest possible number of scripts.

What to automate

Good candidatesPoor candidates
Smoke tests on every buildFeatures that change every week
Regression of stable, critical journeysOne-off checks
API and business-logic testsVisual judgement and UX feel
Data-driven tests with many inputsTests that need physical interaction
Checks across many browsers or devicesExploratory testing

A practical rule: automate a check when it is stable, valuable and will be run many times. Push tests down the pyramid — an API test is faster and more stable than a UI test for the same rule.

Tools by layer

LayerPopular tools
UnitJest, Vitest, JUnit, pytest, NUnit
APIPostman/Newman, REST Assured, pytest + requests, Karate
Web UIPlaywright, Cypress, Selenium WebDriver
Mobile UIAppium, Espresso, XCUITest, Maestro
Performancek6, JMeter, Gatling, Locust
Visual regressionPlaywright screenshots, Percy, Applitools

A maintainable structure

Keep tests readable and keep selectors in one place. The Page Object pattern wraps a page's elements and actions in a class, so a changed button only needs fixing once.

// pages/login.page.ts
export class LoginPage {
  constructor(private page: Page) {}
  async open() { await this.page.goto('/login'); }
  async login(email: string, password: string) {
    await this.page.getByLabel('Email').fill(email);
    await this.page.getByLabel('Password').fill(password);
    await this.page.getByRole('button', { name: 'Sign in' }).click();
  }
}

// tests/login.spec.ts
test('user sees dashboard after login', async ({ page }) => {
  const login = new LoginPage(page);
  await login.open();
  await login.login(USER.email, USER.password);
  await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible();
});

Flaky tests

A flaky test passes and fails without any code change. Flaky suites destroy trust, so we treat them as bugs.

CauseFix
Fixed waits (sleep(3))Wait for a condition: element visible, request finished
Brittle selectors (long CSS paths)Roles, labels or dedicated data-testid attributes
Shared or leftover test dataCreate isolated data per test, clean up after
Test order dependencyEach test sets up its own state
Unstable environment or third-party servicesMock external services, monitor the test environment

Automation in CI/CD

CommitLint + unitBuildAPI testsUI smokeDeploy
Fast checks run on every pull request; longer suites run nightly.

Run fast suites on every pull request so a change can't merge while red. Run full regression, cross-browser and performance suites nightly or before releases. Run tests in parallel to keep feedback under ten minutes, and publish readable reports with screenshots, videos and traces for failures. See Shift-left in practice for our pipeline.

Measuring success

  • Time from commit to feedback
  • Share of regression covered automatically on critical journeys
  • Flaky test rate (aim for close to zero)
  • Bugs found by automation before release vs. bugs found by users
If nobody trusts or reads the results, the automation isn't working, however many tests it has.