Skip to content

test: move replay coverage toward stable mobile app patterns #320

@thymikee

Description

@thymikee

Summary

The current replay coverage is useful, but too much of the core smoke/behavior validation still depends on Apple system apps and simulator-specific surfaces. That makes the suite less representative of real mobile app development patterns and more sensitive to OS churn and host-environment flakiness.

Proposed Improvements

  1. Move most smoke and interaction coverage off system apps and onto a controlled fixture app with stable accessibility identifiers.
  2. Keep system-app coverage only where it validates platform-specific behavior:
    • app launch / close
    • deep links
    • permissions / alerts
    • home / app switcher
    • screenshots / recordings
  3. Split replay coverage by intent:
    • smoke: fast, minimal confidence checks
    • behavior: selectors, fill, scroll, keyboard, lifecycle, alerts
    • benchmark/soak: longer end-to-end journeys
  4. Add flows that better match real mobile app usage:
    • cold start -> deep link -> navigation
    • text entry + keyboard show/dismiss
    • background -> foreground resume
    • modal/sheet open/close
    • permission prompt accept/dismiss/recovery
    • interrupted flow after home/app switcher
    • long-list rediscovery and scroll recovery
  5. Add one explicit host-focus regression for iOS simulator so we catch cases where Simulator steals macOS focus during automation.
  6. Prefer structural assertions over screenshot dependence in CI where possible.

Why This Matters

  • System Settings / Simulator UI changes are outside our control and make otherwise-valid tests flaky.
  • A fixture app gives us stable selectors and lets us model the patterns real mobile teams actually care about.
  • The iOS simulator focus issue is a host-OS-visible regression and should have dedicated end-to-end coverage.

Suggested Starting Point

  1. Build a small fixture/demo app with stable ids and common mobile UI patterns.
  2. Migrate smoke and behavior replays from Settings/system-app journeys to that fixture.
  3. Keep benchmarks as longer, more realistic user journeys.
  4. Add a focused iOS simulator “does not steal host focus” regression test.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions