ci: Add template integration test pipeline#23
Merged
Conversation
GitHub Actions workflow that applies setup_project.py with 5 different configurations (mono-default, mono-renamed, mono-extra-pkgs, single-package, mono-postgres) and verifies each produces a valid project (uv sync, ruff, pyright, pytest). Includes reusable shell script for local testing. Also fixes pre-existing test_hooks.py bug: adds devcontainer-policy-blocker.sh to expected hooks list and sets its executable bit in git. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Warning Rate limit exceeded
⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughAdds a GitHub Actions workflow for template integration testing, a reusable Bash test harness to validate generated templates across multiple configurations, and updates test hooks to include a devcontainer policy hook. The CI runs unit tests then matrixed integration tests invoking the script per configuration. Changes
Sequence Diagram(s)sequenceDiagram
participant GHA as GitHub Actions
participant Unit as Unit Tests Job
participant Integration as Integration Tests Job
participant Script as test_template_integration.sh
participant Template as Template Files
participant Tools as Tools (uv, ruff, pyright, pytest)
GHA->>Unit: trigger unit-tests
Unit->>Unit: run pytest tests/ -v
Unit-->>GHA: report success
GHA->>Integration: trigger integration-test (matrix)
Integration->>Script: invoke with config args
Script->>Template: copy template to work-dir
Script->>Template: run setup_project.py to apply template
Script->>Template: scan for unresolved placeholders
Script->>Template: verify directory structure (mono/single)
Script->>Tools: run uv sync (install deps)
Script->>Tools: run ruff (lint + format)
Script->>Tools: run pyright (type check)
Script->>Tools: run pytest (package tests)
Script->>Template: validate .devcontainer/docker-compose.yml (if services)
Script-->>Integration: return pass/fail
Integration-->>GHA: report matrix results
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes 🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Add comment explaining cd $WORK_DIR governs all subsequent steps, and note that --packages is ignored in single-package mode. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@scripts/test_template_integration.sh`:
- Around line 42-45: Reject unsafe work directories before running the recursive
delete: in the validation block that currently checks SOURCE_DIR and WORK_DIR
non-empty, add explicit guards in the script that verify WORK_DIR is not equal
to SOURCE_DIR, is not "/", "." or "~", is not the user's home directory, and is
not the repository root (or any shared/global path you want to protect); if any
check fails, print an error and exit. Ensure the same guards are applied before
the rm -rf "$WORK_DIR" call (referencing the WORK_DIR and SOURCE_DIR variables
and the rm -rf invocation) so a bad value is rejected early.
- Around line 137-145: The test currently only checks for top-level apps/libs
when PROJECT_TYPE != "single"; update the test to validate the exact package set
in $PACKAGES: parse $PACKAGES (comma/space-separated), build the expected
relative paths under apps/ or libs/ depending on PROJECT_TYPE, then assert each
expected directory exists and that the set of actual directories under apps/ and
libs/ exactly matches the expected set (fail if unexpected/stale default dirs
are present); reference the PROJECT_TYPE and PACKAGES variables and ensure the
new checks replace or augment the existing apps/libs existence assertions so
mono-renamed and mono-extra-pkgs matrix cases fail if the package
rename/extra-packages logic in setup_project.py regresses.
- Around line 192-199: The current check only ensures a top-level 'services:'
key exists in .devcontainer/docker-compose.yml but doesn't verify the requested
service; update the block that runs when SERVICES != "none" to also validate
that the compose file contains the specific service named by the SERVICES
variable (e.g., look for a service key matching "$SERVICES" under services), and
call step_fail with a clear message if that service is missing before calling
step_pass; use the existing SERVICES variable, the
.devcontainer/docker-compose.yml path, and the step_fail/step_pass helpers to
implement this additional check.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: c8dafaaa-f666-4146-9117-c644127fe1be
📒 Files selected for processing (4)
.claude/hooks/devcontainer-policy-blocker.sh.github/workflows/template-integration.ymlscripts/test_template_integration.shtests/test_hooks.py
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@scripts/test_template_integration.sh`:
- Around line 115-126: The grep currently only searches a fixed set of file
extensions so extensionless text files (e.g., .devcontainer/Dockerfile) can
escape detection; update the PLACEHOLDER_HITS search (the command that uses
PLACEHOLDER_PATTERN and --include/--exclude flags) to also scan extensionless
files by either adding a broad --include='*' (and keep relevant --exclude
entries) or explicitly including common extensionless filenames like Dockerfile
and CI config names so all text files are checked for TEMPLATE placeholders.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 3e79777c-4467-43e4-9ff5-1dfeb578d2a7
📒 Files selected for processing (4)
.github/workflows/template-integration.ymldocs/CHANGELOG.mddocs/DECISIONS.mdscripts/test_template_integration.sh
Comment on lines
+115
to
+126
| # Search for actual template placeholders (not GitHub Actions ${{ }} expressions). | ||
| # The pattern matches {{word}} but NOT ${{word}} (GHA syntax). | ||
| # Excludes setup_project.py (defines them) and test files (reference them in fixtures). | ||
| PLACEHOLDER_PATTERN='\{\{(project_name|namespace|description|author_name|author_email|python_version|base_branch|year)\}\}' | ||
| PLACEHOLDER_HITS=$(grep -rE "$PLACEHOLDER_PATTERN" \ | ||
| --include='*.py' --include='*.toml' --include='*.yml' --include='*.yaml' \ | ||
| --include='*.md' --include='*.json' --include='*.cfg' --include='*.ini' \ | ||
| --include='*.txt' --include='*.sh' \ | ||
| --exclude='setup_project.py' \ | ||
| --exclude='test_setup_project.py' \ | ||
| --exclude='test_template_integration.sh' \ | ||
| . 2>/dev/null || true) |
There was a problem hiding this comment.
Scan all text files for unreplaced placeholders.
This allowlist skips extensionless text files, so files like .devcontainer/Dockerfile can retain {{python_version}} and Step 3 still passes. That defeats the main “template fully applied” assertion.
Suggested fix
-PLACEHOLDER_HITS=$(grep -rE "$PLACEHOLDER_PATTERN" \
- --include='*.py' --include='*.toml' --include='*.yml' --include='*.yaml' \
- --include='*.md' --include='*.json' --include='*.cfg' --include='*.ini' \
- --include='*.txt' --include='*.sh' \
+PLACEHOLDER_HITS=$(grep -rIE "$PLACEHOLDER_PATTERN" \
--exclude='setup_project.py' \
--exclude='test_setup_project.py' \
--exclude='test_template_integration.sh' \
. 2>/dev/null || true)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| # Search for actual template placeholders (not GitHub Actions ${{ }} expressions). | |
| # The pattern matches {{word}} but NOT ${{word}} (GHA syntax). | |
| # Excludes setup_project.py (defines them) and test files (reference them in fixtures). | |
| PLACEHOLDER_PATTERN='\{\{(project_name|namespace|description|author_name|author_email|python_version|base_branch|year)\}\}' | |
| PLACEHOLDER_HITS=$(grep -rE "$PLACEHOLDER_PATTERN" \ | |
| --include='*.py' --include='*.toml' --include='*.yml' --include='*.yaml' \ | |
| --include='*.md' --include='*.json' --include='*.cfg' --include='*.ini' \ | |
| --include='*.txt' --include='*.sh' \ | |
| --exclude='setup_project.py' \ | |
| --exclude='test_setup_project.py' \ | |
| --exclude='test_template_integration.sh' \ | |
| . 2>/dev/null || true) | |
| # Search for actual template placeholders (not GitHub Actions ${{ }} expressions). | |
| # The pattern matches {{word}} but NOT ${{word}} (GHA syntax). | |
| # Excludes setup_project.py (defines them) and test files (reference them in fixtures). | |
| PLACEHOLDER_PATTERN='\{\{(project_name|namespace|description|author_name|author_email|python_version|base_branch|year)\}\}' | |
| PLACEHOLDER_HITS=$(grep -rIE "$PLACEHOLDER_PATTERN" \ | |
| --exclude='setup_project.py' \ | |
| --exclude='test_setup_project.py' \ | |
| --exclude='test_template_integration.sh' \ | |
| . 2>/dev/null || true) |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@scripts/test_template_integration.sh` around lines 115 - 126, The grep
currently only searches a fixed set of file extensions so extensionless text
files (e.g., .devcontainer/Dockerfile) can escape detection; update the
PLACEHOLDER_HITS search (the command that uses PLACEHOLDER_PATTERN and
--include/--exclude flags) to also scan extensionless files by either adding a
broad --include='*' (and keep relevant --exclude entries) or explicitly
including common extensionless filenames like Dockerfile and CI config names so
all text files are checked for TEMPLATE placeholders.
- Reject unsafe --work-dir values (/, $HOME, same as --source-dir) before the rm -rf to prevent accidental data loss - Verify each requested package exists in the correct directory (libs/ or apps/) based on the lib:/app: prefix convention, catching rename regressions that leave default names in place - Check that the specific service (db, redis) is defined in the generated docker-compose.yml, not just that the file exists Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
template-integration.yml) that appliessetup_project.pywith 5 different configurations in a matrix and verifies each produces a valid, working projectscripts/test_template_integration.sh) that can also be run locally for debuggingtest_hooks.pywheredevcontainer-policy-blocker.shwas not in the expected hooks list, and sets its missing executable bit in gitMatrix Configurations
Verification Steps (per config)
setup_project.pywith CLI args{{placeholder}}patterns remainuv syncdependenciesruff check+ruff format --checkpyrighttype checkingpytesttest suiteTest plan
🤖 Generated with Claude Code
Summary by CodeRabbit
Tests
Documentation
Chores