Skip to content

Add e2e tests with ecosystem repositories to prevent regressions #12

@fengmk2

Description

@fengmk2

Background

Issue #10 revealed that our CI tests were insufficient — they only verified vp --version after installation, which doesn't exercise critical code paths like vp exec and vp run. The Cannot find module 'which' error on Windows went undetected because no test actually ran commands through vite-plus.

Goal

Design and implement an e2e testing workflow that runs real-world ecosystem repositories against the action on all platforms (ubuntu, macos, windows) to catch regressions before they reach users.

Testing Plan

1. Ecosystem Repository Matrix

Include real repositories that use setup-vp in their CI to validate the full workflow:

Repository Key Commands Purpose
voidzero-dev/vite-plus vp install, vp run build, vp run test Core toolchain — validates install + build + test
cloudflare/vinext (or similar) vp exec, vp install, vp run Real consumer — validates exec path that broke in #10

2. Test Scenarios

Each ecosystem repo should be tested with:

  • Fresh install (no cache) on all 3 platforms
  • Cached install (cache hit) on all 3 platforms
  • vp exec — the code path that triggered Windows runner failures with latest action version #10
  • vp run — script execution
  • vp install — dependency installation
  • vp env use — Node.js version switching
  • Multiple Node.js versions (e.g., 22, 24)

3. Workflow Design

  • Create a separate e2e.yml workflow (to avoid slowing down the main test.yml)
  • Trigger on push to main, pull_request, and schedule (e.g., daily) to catch upstream breakage
  • Clone ecosystem repos at pinned refs for reproducibility
  • Use fail-fast: false so all platform results are visible even if one fails

4. Example Structure

jobs:
  e2e-vite-plus:
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v6
      - uses: ./
        with:
          version: latest
          cache: false
      - uses: actions/checkout@v6
        with:
          repository: voidzero-dev/vite-plus
          path: ecosystem/vite-plus
      - name: Install and build
        working-directory: ecosystem/vite-plus
        run: |
          vp install
          vp run build
      - name: Run tests
        working-directory: ecosystem/vite-plus
        run: vp run test

Acceptance Criteria

  • e2e workflow exists and runs on all 3 platforms
  • At least 2 ecosystem repositories are included in the matrix
  • vp exec, vp run, and vp install are all exercised
  • Scheduled runs catch upstream breakage early
  • The e2e test would have caught Windows runner failures with latest action version #10 before it reached users

Related

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Priority

None yet

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions