diff --git a/doc/changes/unreleased.md b/doc/changes/unreleased.md index 82a9043e2..82dc04fb6 100644 --- a/doc/changes/unreleased.md +++ b/doc/changes/unreleased.md @@ -19,16 +19,15 @@ details, check out the [Workflow Extensions](https://exasol.github.io/python-too ## Features -* #756: Added `dependency-update.yml` to automate resolving vulnerabilities with a generated pull request - -## Bugfix - -* #563: Fixed merge-gate to prevent auto-merges from happening when integration tests failed - -## Feature * #829: Extended removing a job from a workflow to also remove it from the `needs` of another job * #825: Created two workflows by splitting up previous ones: * Moved the periodic jobs in `ci.yml` to its own `periodic-validation.yml` * Moved the unit tests job in `checks.yml` to its own `fast-tests.yml` * #730: Added workflow extensions to `fast-tests` and `merge-gate` +* #756: Added `dependency-update.yml` to automate resolving vulnerabilities with a generated pull request +* #792: Improved `dependency-update.yml` documentation + +## Bugfix + +* #563: Fixed merge-gate to prevent auto-merges from happening when integration tests failed diff --git a/doc/user_guide/features/github_workflows/index.rst b/doc/user_guide/features/github_workflows/index.rst index abdbccf2a..c763d1925 100644 --- a/doc/user_guide/features/github_workflows/index.rst +++ b/doc/user_guide/features/github_workflows/index.rst @@ -25,8 +25,9 @@ workflows from the templates. .. attention:: - In most cases, we recommend using _all_ workflows without change to ensure - consistent interdependencies. For more details, see :ref:`ci_actions`. + In most cases, we recommend using **all** workflows without change to ensure + consistent interdependencies. For more details, see :ref:`manual_activities` + and :ref:`automated_activities`. Workflows @@ -136,79 +137,27 @@ and is maintained by the PTB and what is project-specific. tests. It's encouraged to add to this workflow extension additional approval requests, similar to ``approve-run-slow-tests``. -.. _ci_actions: -CI Actions ----------- -.. _dependency_update: - -Dependency Update -^^^^^^^^^^^^^^^^^ - -The ``dependency-update.yml`` workflow is used to resolve vulnerabilities by updating our project dependencies. - -It can be triggered manually and is also scheduled to run weekly. - -The workflow first audits dependencies for known vulnerabilities. If vulnerabilities -are detected, it updates the dependencies using Poetry. When the ``poetry.lock`` is changed, -then it creates a pull request with the update. - -.. _pr_merge_yml: +.. _manual_activities: -Merge -^^^^^ +Manual Activities +----------------- -When a pull request is merged to main, then the ``pr-merge.yml`` workflow is activated. +The following workflows are typically triggered directly by a developer during normal +project maintenance. -.. mermaid:: - :name: merge-diagram - - graph TD - %% Workflow Triggers (Solid Lines) - pr-merge[pr-merge.yml] --> publish-docs[gh-pages.yml] - -.. _periodic_validation_yml: +.. _ci_yml: -Periodic Validation -^^^^^^^^^^^^^^^^^^^ +Create a PR +^^^^^^^^^^^ -Once a week, this `periodic-validation.yml` is triggered on the default branch. Its main -purpose is to ensure that our critical checks and tests continue to run, but it also -sends the results of the linting tools and test coverage to Sonar for an overall report. +Modifying any pull request triggers the workflow ``ci.yml``. -.. literalinclude:: ../../../../exasol/toolbox/templates/github/workflows/periodic-validation.yml +.. literalinclude:: ../../../../exasol/toolbox/templates/github/workflows/ci.yml :language: yaml - :start-at: schedule: - :end-at: - cron: "0 0 * * 6" - -.. mermaid:: - - graph TD - %% Define Nodes - checks[checks.yml] - periodic_validation[periodic-validation.yml] - fast-tests[fast-tests.yml] - slow_checks[slow-checks.yml] - report[report.yml] - - %% Workflow Triggers - periodic_validation --> checks - periodic_validation --> fast-tests - periodic_validation --> slow_checks - - %% Dependencies - checks -.->|needs| report - fast-tests -.->|needs| report - slow_checks -.->|needs| report - -.. _ci_yml: - -Pull Request -^^^^^^^^^^^^ - -When any pull request is opened, synchronized, or reopened, then the ``ci.yml`` will be -triggered. + :start-at: on: + :end-at: types: [opened, synchronize, reopened] When configured as described on :ref:`github_project_configuration`, the ``run-slow-tests`` job requires a developer to manually approve executing the slower @@ -264,10 +213,24 @@ then the subsequent jobs will not be started. style approver fill:#fff,stroke:#333,stroke-dasharray: 5 5 style slow_run fill:#fff,stroke:#333,stroke-dasharray: 5 5 +.. _pr_merge_yml: + +Merge a PR +^^^^^^^^^^ + +Merging a pull request to the default branch, activates the ``pr-merge.yml`` workflow. + +.. mermaid:: + :name: merge-diagram + + graph TD + %% Workflow Triggers (Solid Lines) + pr-merge[pr-merge.yml] --> publish-docs[gh-pages.yml] + .. _cd_yml: -Release -^^^^^^^ +Create a Release +^^^^^^^^^^^^^^^^ When the nox session ``release:trigger`` is used, a new tag is created & pushed to main. This starts the release process by activating the ``cd.yml`` workflow. @@ -282,3 +245,68 @@ to main. This starts the release process by activating the ``cd.yml`` workflow. %% Dependencies / Waiting (Dotted Lines) check-release-tag -.->|needs| build-and-publish[build-and-publish.yml] build-and-publish -.->|needs| gh-pages[gh-pages.yml] + +.. _automated_activities: + +Automated Activities +-------------------- + +The following workflows are triggered automatically by schedules. + +.. _dependency_update: + +Update Dependencies +^^^^^^^^^^^^^^^^^^^ + +The ``dependency-update.yml`` workflow runs on the default branch. It +checks for known vulnerabilities and tries to fix them by updating dependencies. + +.. literalinclude:: ../../../../exasol/toolbox/templates/github/workflows/dependency-update.yml + :language: yaml + :start-at: on: + :end-at: workflow_dispatch: + +The workflow first audits dependencies for known vulnerabilities: + +* If no vulnerabilities are detected, then no update is needed. +* If vulnerabilities are detected, it updates the dependencies using ``poetry update``. + + * If the ``poetry.lock`` is unchanged, then no further action is taken. + * If the ``poetry.lock`` is changed, then it creates a branch, stages the commit, + creates a pull request, and sends a Slack notification. + +Afterwards, users need to perform some manual steps which are described in the PR description. + +.. _periodic_validation_yml: + +Validate Periodically +^^^^^^^^^^^^^^^^^^^^^ + +The ``periodic-validation.yml`` runs on the default branch. It ensures that critical +checks and tests continue to run and sends the results of the linting tools and test +coverage to Sonar for an overall report. + +.. literalinclude:: ../../../../exasol/toolbox/templates/github/workflows/periodic-validation.yml + :language: yaml + :start-at: schedule: + :end-at: - cron: "0 0 * * 6" + +.. mermaid:: + + graph TD + %% Define Nodes + checks[checks.yml] + periodic_validation[periodic-validation.yml] + fast-tests[fast-tests.yml] + slow_checks[slow-checks.yml] + report[report.yml] + + %% Workflow Triggers + periodic_validation --> checks + periodic_validation --> fast-tests + periodic_validation --> slow_checks + + %% Dependencies + checks -.->|needs| report + fast-tests -.->|needs| report + slow_checks -.->|needs| report diff --git a/exasol/toolbox/templates/github/workflows/dependency-update.yml b/exasol/toolbox/templates/github/workflows/dependency-update.yml index c6783bfd8..1419bff35 100644 --- a/exasol/toolbox/templates/github/workflows/dependency-update.yml +++ b/exasol/toolbox/templates/github/workflows/dependency-update.yml @@ -91,9 +91,13 @@ jobs: BASE_BRANCH=$(gh repo view --json defaultBranchRef -q .defaultBranchRef.name) PR_BODY="Automated dependency update for \`poetry.lock\`. - This PR was created by the dependency update workflow after running: - - \`poetry run -- nox -s dependency:audit\` - - \`poetry update\`" + This PR was created by the workflow \`dependency-update.yml\` + + Please perform the following actions on a locally checked out branch: + - [ ] Execute \`poetry run -- nox -s workflow:generate -- --all\` + - [ ] Use \`poetry run -- nox -s dependency:audit\` to check for vulnerabilities requiring manual action + - [ ] Update file \`doc/changes/unreleased.md\` + " PR_URL=$(gh pr create \ --base "$BASE_BRANCH" \