From 28d54b917012b8d904b5859b2ab15ff73a393b27 Mon Sep 17 00:00:00 2001 From: Ariel Schulz Date: Tue, 24 Feb 2026 13:53:14 +0100 Subject: [PATCH 01/11] Tiny fix ups --- doc/user_guide/configuration.rst | 1 + .../features/github_workflows/create_and_update.rst | 10 +++++----- .../github_workflows/github_project_configuration.rst | 11 ++++++----- 3 files changed, 12 insertions(+), 10 deletions(-) diff --git a/doc/user_guide/configuration.rst b/doc/user_guide/configuration.rst index 0967c43f0..e7f5538ef 100644 --- a/doc/user_guide/configuration.rst +++ b/doc/user_guide/configuration.rst @@ -11,3 +11,4 @@ especially when starting a new project. features/metrics/sonar Formatting + features/github_workflows/github_project_configuration diff --git a/doc/user_guide/features/github_workflows/create_and_update.rst b/doc/user_guide/features/github_workflows/create_and_update.rst index 9aa3b7710..a1c0932b4 100644 --- a/doc/user_guide/features/github_workflows/create_and_update.rst +++ b/doc/user_guide/features/github_workflows/create_and_update.rst @@ -3,12 +3,12 @@ Creating and Updating the GitHub Workflows in Your Project ========================================================== -PTB can initially generate the GitHub workflows in your project and also +The PTB can initially generate the GitHub workflows in your project and also update existing workflows. The workflows are based on Jinja templates with variables populated by the -PTB. The PTB reads the values from various places within your project, see -:ref:`template_variables`. +PTB. The PTB reads the values from various attributes and properties of your +project's config, see :ref:`template_variables`. Please note that the PTB only updates the values in the GitHub workflows when *updating* the workflows. So, after updating the :ref:`list of Python versions @@ -36,8 +36,8 @@ Many workflows are using a Build-matrix to iterate over multiple versions of Python and/or the Exasol Docker DB. This is to make sure your code is valid, free of bugs and working correctly for each combination of these items. -PTB has a default for these versions, but you can override it in file -``noxconfig.py``, e.g. +The PTB has a default for these versions, but you can override it in the +``noxconfig.py`` file, e.g. .. code-block:: shell diff --git a/doc/user_guide/features/github_workflows/github_project_configuration.rst b/doc/user_guide/features/github_workflows/github_project_configuration.rst index c962f2fb5..3696f26ca 100644 --- a/doc/user_guide/features/github_workflows/github_project_configuration.rst +++ b/doc/user_guide/features/github_workflows/github_project_configuration.rst @@ -7,8 +7,9 @@ Branch Protection ----------------- The best and most maintainable way to have solid branch protection -(:code:`Settings/Branches/main`) is to require the workflow :code:`CI / Allow -Merge` to pass successfully. +(:code:`Settings/Branches/main`) is to require the workflow :code:`merge-gate / Allow +Merge` to pass successfully. Additionally, if it make sense for your project, +you can further require that ``SonarCloud Code Analysis`` passes. .. note:: Setting the required status checks to pass before merging is only possible @@ -17,7 +18,7 @@ Merge` to pass successfully. Manual Approval --------------- -If your CI workflow involves slow or expensive steps you can guard these to be +If your CI workflow involves slow or expensive steps, you can guard these to be executed only after manual approval. The CI workflow will automatically create a GitHub environment named :code:`manual-approval`. You only need to add reviewers in (:code:`Settings/Environments/manual-approval`) and move the @@ -27,11 +28,11 @@ file :code:`.github/workflows/merge-gate.yml`. Secrets ------- -For accessing specific services in the Internet, your project often needs a +For accessing specific services in the internet, your project often needs a related *token* or other credentials. These credentials can be acquired by registering on the service's website. -In many cases your company or organization may manage the credentials +In many cases, your company or organization may manage the credentials centrally and enable the use in multiple projects. The credentials can be managed as Secrets in GitHub and can be made accessible to GitHub projects and used by their workflows. From 6e305804c4c81f95a33051bd9dbc477548340713 Mon Sep 17 00:00:00 2001 From: Ariel Schulz Date: Tue, 24 Feb 2026 14:17:51 +0100 Subject: [PATCH 02/11] Ensure workflow:generate works for newly created project --- test/integration/project-template/nox_test.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/test/integration/project-template/nox_test.py b/test/integration/project-template/nox_test.py index 5a0d55e66..d41d67248 100644 --- a/test/integration/project-template/nox_test.py +++ b/test/integration/project-template/nox_test.py @@ -64,3 +64,12 @@ def test_release_prepare(self, poetry_path, run_command): output = run_command(release_prepare) assert output.returncode == 0 + + def test_install_github_workflows(self, poetry_path, run_command): + install_workflows = self._command(poetry_path, task="workflow:generate", + add_ons=["all"]) + output = run_command(install_workflows) + assert output.returncode == 0 + + file_list = run_command(["ls", ".github/workflows"]).stdout.splitlines() + assert len(file_list) == 13 From 8f68807a326515a63fde55992bf68019e1d91b67 Mon Sep 17 00:00:00 2001 From: Ariel Schulz Date: Tue, 24 Feb 2026 14:34:54 +0100 Subject: [PATCH 03/11] Give basic text start --- .../github_workflows/create_and_update.rst | 43 ++++++++++++++++--- 1 file changed, 38 insertions(+), 5 deletions(-) diff --git a/doc/user_guide/features/github_workflows/create_and_update.rst b/doc/user_guide/features/github_workflows/create_and_update.rst index a1c0932b4..1f6b5d4ad 100644 --- a/doc/user_guide/features/github_workflows/create_and_update.rst +++ b/doc/user_guide/features/github_workflows/create_and_update.rst @@ -49,6 +49,24 @@ The PTB has a default for these versions, but you can override it in the Some workflows are expected to not depend on a specific Python version and will use only the lowest Python version in the list specified above. +.. _customize_workflows: + +Customize Workflows for Your Project +------------------------------------ + +The PTB allows you to customise workflows by targeting specific jobs or steps: + +* Remove a job by its job_name. +* Replace a step (referenced by step_id) with one or more new steps. +* Insert steps after a specific step_id. + +.. note:: + + These operations do not currently cascade. For example, removing a job + without accounting for its downstream dependencies may result in errors. + You must manually adjust any subsequent steps that rely on the removed + job's or step's output. + .. _update_workflows: Add all Workflows to Your Project @@ -56,11 +74,26 @@ Add all Workflows to Your Project .. code-block:: shell - tbx workflow install all + poetry run -- nox -s workflow:generate -- all .. warning:: - #. If you already have various workflows, you may want to run the - :code:`update` command instead of the :code:`install` command. + Some workflows depend on other workflows. Please ensure you have all + the required workflows if you do not install all of them. + +.. note:: + + The commands: + + * ``tbx workflow install all`` - used to install workflows + * ``tbx workflow update all`` - used to update workflows + + are considered historic variants of this command. + + **Deprecation Notice:** + These ``tbx`` endpoints are marked as **deprecated** and are scheduled for removal + by **April 22nd, 2025**. - #. Some workflows depend on other workflows. Please ensure you have all - the required workflows if you do not install all of them. + Please note that these legacy commands do not allow users to use their specified + ``.workflow-patcher.yml`` file to further customise or patch workflows. Users + should transition to the ``nox``-based command to leverage full customisation + features. From 9b43d1190dc77fd0856b593bcc671fdb292eebce Mon Sep 17 00:00:00 2001 From: Ariel Schulz Date: Tue, 24 Feb 2026 15:04:10 +0100 Subject: [PATCH 04/11] Split up API into sub-files & fix so BaseConfig can be viewed --- doc/api/base_config.rst | 13 ++++++++++ doc/api/index.rst | 10 ++++++++ doc/{api.rst => api/workflow_exceptions.rst} | 5 ---- doc/index.rst | 2 +- .../github_workflows/template_variables.rst | 1 + exasol/toolbox/config.py | 24 +++++++++++-------- 6 files changed, 39 insertions(+), 16 deletions(-) create mode 100644 doc/api/base_config.rst create mode 100644 doc/api/index.rst rename doc/{api.rst => api/workflow_exceptions.rst} (83%) diff --git a/doc/api/base_config.rst b/doc/api/base_config.rst new file mode 100644 index 000000000..d925014e8 --- /dev/null +++ b/doc/api/base_config.rst @@ -0,0 +1,13 @@ +.. _base_config: + +BaseConfig +========== + +.. currentmodule:: exasol.toolbox.config + +.. autoclass:: BaseConfig + :members: + :undoc-members: + :show-inheritance: + :member-order: bysource + :special-members: __init__ diff --git a/doc/api/index.rst b/doc/api/index.rst new file mode 100644 index 000000000..e738b4d1a --- /dev/null +++ b/doc/api/index.rst @@ -0,0 +1,10 @@ +.. _api: + +:octicon:`cpu` API Reference +============================= + +.. toctree:: + :maxdepth: 2 + + base_config + workflow_exceptions diff --git a/doc/api.rst b/doc/api/workflow_exceptions.rst similarity index 83% rename from doc/api.rst rename to doc/api/workflow_exceptions.rst index 9950235c8..de27c018a 100644 --- a/doc/api.rst +++ b/doc/api/workflow_exceptions.rst @@ -1,8 +1,3 @@ -.. _api: - -:octicon:`cpu` API Reference -============================= - .. _workflow_exceptions: Workflow Exceptions diff --git a/doc/index.rst b/doc/index.rst index bc00b76a2..48aa5aa29 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -52,5 +52,5 @@ Documentation of the Exasol-Toolbox developer_guide/developer_guide tools github_actions/github_actions - api + api/index changes/changelog diff --git a/doc/user_guide/features/github_workflows/template_variables.rst b/doc/user_guide/features/github_workflows/template_variables.rst index f8355e3fa..5ee89e403 100644 --- a/doc/user_guide/features/github_workflows/template_variables.rst +++ b/doc/user_guide/features/github_workflows/template_variables.rst @@ -12,3 +12,4 @@ necessary. .. literalinclude:: ../../../../exasol/toolbox/config.py :language: python :start-at: github_template_dict + :end-before: @computed_field diff --git a/exasol/toolbox/config.py b/exasol/toolbox/config.py index e0fdd3bfb..05ef9d9ac 100644 --- a/exasol/toolbox/config.py +++ b/exasol/toolbox/config.py @@ -44,7 +44,7 @@ def filter_not_specified_methods( ) -> tuple[str, ...]: """ Filter methods which were specified with a @hookimpl but where not specified - in `exasol.toolbox.nox.plugins.NoxTasks`. + in ``exasol.toolbox.nox.plugins.NoxTasks``. """ return tuple(name for name, _ in methods if name not in METHODS_SPECIFIED_FOR_HOOKS) @@ -52,7 +52,7 @@ def filter_not_specified_methods( def validate_plugin_hook(plugin_class: type[Any]): """ Validate methods in a class for at least one pluggy @hookimpl marker and verifies - that this method is also specified in `exasol.toolbox.nox.plugins.NoxTasks`. + that this method is also specified in ``exasol.toolbox.nox.plugins.NoxTasks``. """ methods_with_hook = get_methods_with_hook_implementation(plugin_class=plugin_class) @@ -154,7 +154,7 @@ class BaseConfig(BaseModel): This is used to extend the default excluded_python_paths. If a more general path that would be seen in other projects, like .venv, needs to be added into this argument, please instead modify the - `exasol.toolbox.config.DEFAULT_EXCLUDED_PATHS`. + ``exasol.toolbox.config.DEFAULT_EXCLUDED_PATHS``. """, ) plugins_for_nox_sessions: tuple[ValidPluginHook, ...] = Field( @@ -164,7 +164,7 @@ class BaseConfig(BaseModel): by the python-toolbox. As described on the plugins pages: - https://exasol.github.io/python-toolbox/main/user_guide/customization.html#plugins - https://exasol.github.io/python-toolbox/main/developer_guide/plugins.html, - possible plugin options are defined in `exasol.toolbox.nox.plugins.NoxTasks`. + possible plugin options are defined in ``exasol.toolbox.nox.plugins.NoxTasks``. """, ) dependency_manager: DependencyManager = Field( @@ -199,10 +199,10 @@ def documentation_path(self) -> Path: @property def minimum_python_version(self) -> str: """ - Minimum Python version declared from the `python_versions` list + Minimum Python version declared from the ``python_versions`` list This is used in specific testing scenarios where it would be either - costly to run the tests for all `python_versions` or we need a single metric. + costly to run the tests for all ``python_versions`` or we need a single metric. """ versioned = [Version.from_string(v) for v in self.python_versions] min_version = min(versioned) @@ -219,9 +219,10 @@ def excluded_python_paths(self) -> tuple[str, ...]: - lint:code - lint:security - lint:typing + where it is desired to restrict which Python files are considered within the - PROJECT_CONFIG.source_code_path path, like excluding `dist`, `.eggs`. As such, - this property is used to exclude such undesired paths. + ``PROJECT_CONFIG.source_code_path`` path, like excluding ``dist``, ``.eggs``. + As such, this property is used to exclude such undesired paths. """ return tuple( sorted(DEFAULT_EXCLUDED_PATHS.union(set(self.add_to_excluded_python_paths))) @@ -244,7 +245,7 @@ def pyupgrade_argument(self) -> tuple[str, ...]: @property def sonar_code_path(self) -> Path: """ - Relative path needed in nox session `sonar:check` to create the coverage XML + Relative path needed in Nox session ``sonar:check`` to create the coverage XML. """ return self.source_code_path.relative_to(self.root_path) @@ -269,6 +270,9 @@ def version_filepath(self) -> Path: @computed_field # type: ignore[misc] @property def github_workflow_directory(self) -> Path: + """ + Path to the GitHub workflow directory. + """ return self.root_path / ".github" / "workflows" @computed_field # type: ignore[misc] @@ -290,7 +294,7 @@ def github_template_dict(self) -> dict[str, Any]: def github_workflow_patcher_yaml(self) -> Path | None: """ For customizing the GitHub workflow templates provided by the PTB, - a project can define a `.workflow-patcher.yml` file containing instructions to + a project can define a ``.workflow-patcher.yml`` file containing instructions to delete or modify jobs in the PTB template. Modification includes replacing and inserting steps. From 28f7e40883e82b7860a38324924fd9254f5a9f9c Mon Sep 17 00:00:00 2001 From: Ariel Schulz Date: Tue, 24 Feb 2026 15:09:58 +0100 Subject: [PATCH 05/11] Add small text description for --- .../features/github_workflows/create_and_update.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/user_guide/features/github_workflows/create_and_update.rst b/doc/user_guide/features/github_workflows/create_and_update.rst index 1f6b5d4ad..f3d4dbdfb 100644 --- a/doc/user_guide/features/github_workflows/create_and_update.rst +++ b/doc/user_guide/features/github_workflows/create_and_update.rst @@ -67,6 +67,10 @@ The PTB allows you to customise workflows by targeting specific jobs or steps: You must manually adjust any subsequent steps that rely on the removed job's or step's output. +To utilize this feature, create a ``.workflow-patcher.yml`` file in your project's +root directory. This will be automatically detected, validated by a pydantic model, and +used when you :ref:`install or update your workflows `. + .. _update_workflows: Add all Workflows to Your Project From 9afecf8a3498bfaa7023868bd6274d403afb8212 Mon Sep 17 00:00:00 2001 From: Ariel Schulz Date: Tue, 24 Feb 2026 15:17:08 +0100 Subject: [PATCH 06/11] Add workflow_patcher page --- .../features/github_workflows/index.rst | 1 + .../github_workflows/template_variables.rst | 4 ++-- .../github_workflows/workflow_patcher.rst | 15 +++++++++++++++ exasol/toolbox/config.py | 9 +++------ 4 files changed, 21 insertions(+), 8 deletions(-) create mode 100644 doc/user_guide/features/github_workflows/workflow_patcher.rst diff --git a/doc/user_guide/features/github_workflows/index.rst b/doc/user_guide/features/github_workflows/index.rst index 13a8b3d7c..63aa4af03 100644 --- a/doc/user_guide/features/github_workflows/index.rst +++ b/doc/user_guide/features/github_workflows/index.rst @@ -10,6 +10,7 @@ GitHub Workflow Templates github_project_configuration create_and_update template_variables + workflow_patcher The PTB ships with configurable GitHub workflow templates covering the most common CI/CD setup variants for Python projects. The templates are defined in: diff --git a/doc/user_guide/features/github_workflows/template_variables.rst b/doc/user_guide/features/github_workflows/template_variables.rst index 5ee89e403..f6ff6078b 100644 --- a/doc/user_guide/features/github_workflows/template_variables.rst +++ b/doc/user_guide/features/github_workflows/template_variables.rst @@ -6,8 +6,8 @@ Template Variables Underlying the CLI, the PTB uses Jinja to dynamically generate project-specific workflows. The rendering process is supported by the ``github_template_dict`` found in your ``noxconfig.py::PROJECT_CONFIG``. This dictionary is inherited by default from -``BaseConfig.py``, ensuring a standardized baseline that can be easily overridden, if -necessary. +:py:attr:`exasol.toolbox.config.BaseConfig.github_template_dict`, ensuring a +standardized baseline that can be easily overridden, if necessary. .. literalinclude:: ../../../../exasol/toolbox/config.py :language: python diff --git a/doc/user_guide/features/github_workflows/workflow_patcher.rst b/doc/user_guide/features/github_workflows/workflow_patcher.rst new file mode 100644 index 000000000..0c606cec5 --- /dev/null +++ b/doc/user_guide/features/github_workflows/workflow_patcher.rst @@ -0,0 +1,15 @@ +.. _workflow_patcher: + +Workflow Patcher +================ + +Underlying the CLI, the PTB uses, if defined, a ``.workflow_patcher.yml`` file to +customize generated project-specific workflows. The rendering process is supported by +the ``github_workflow_patcher_yaml`` found in your ``noxconfig.py::PROJECT_CONFIG``. +This Path is inherited by default from +:py:attr:`exasol.toolbox.config.BaseConfig.github_workflow_patcher_yaml` +ensuring a standardized baseline that can be easily overridden, if necessary. + +.. literalinclude:: ../../../../exasol/toolbox/config.py + :language: python + :start-at: github_workflow_patcher_yaml diff --git a/exasol/toolbox/config.py b/exasol/toolbox/config.py index 05ef9d9ac..061d2b1e8 100644 --- a/exasol/toolbox/config.py +++ b/exasol/toolbox/config.py @@ -294,12 +294,9 @@ def github_template_dict(self) -> dict[str, Any]: def github_workflow_patcher_yaml(self) -> Path | None: """ For customizing the GitHub workflow templates provided by the PTB, - a project can define a ``.workflow-patcher.yml`` file containing instructions to - delete or modify jobs in the PTB template. Modification includes replacing and - inserting steps. - - This feature is a work-in-progress that will be completed with: - https://github.com/exasol/python-toolbox/issues/690 + a project can define a ``.workflow-patcher.yml`` file containing + instructions to delete or modify jobs in the PTB template. + Modification includes replacing and inserting steps. """ workflow_patcher_yaml = self.root_path / ".workflow-patcher.yml" if workflow_patcher_yaml.exists(): From 68a25b90f67a4a7e27f7fbe8a3a8afc7fcea7fd8 Mon Sep 17 00:00:00 2001 From: Ariel Schulz Date: Tue, 24 Feb 2026 15:31:20 +0100 Subject: [PATCH 07/11] Add autodoc_pydantic --- doc/conf.py | 1 + poetry.lock | 67 ++++++++++++++++++++++++++++++++++++++++++++++++-- pyproject.toml | 1 + 3 files changed, 67 insertions(+), 2 deletions(-) diff --git a/doc/conf.py b/doc/conf.py index afc9576f4..95375d240 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -29,6 +29,7 @@ extensions = [ "sphinx.ext.todo", "sphinx.ext.autodoc", + "sphinxcontrib.autodoc_pydantic", "sphinx.ext.viewcode", "sphinx.ext.napoleon", "sphinx.ext.intersphinx", diff --git a/poetry.lock b/poetry.lock index 634301c0c..7b3bf768e 100644 --- a/poetry.lock +++ b/poetry.lock @@ -143,6 +143,30 @@ files = [ {file = "attrs-25.4.0.tar.gz", hash = "sha256:16d5969b87f0859ef33a48b35d55ac1be6e42ae49d5e853b597db70c35c57e11"}, ] +[[package]] +name = "autodoc-pydantic" +version = "2.2.0" +description = "Seamlessly integrate pydantic models in your Sphinx documentation." +optional = false +python-versions = "<4.0.0,>=3.8.1" +groups = ["main"] +files = [ + {file = "autodoc_pydantic-2.2.0-py3-none-any.whl", hash = "sha256:8c6a36fbf6ed2700ea9c6d21ea76ad541b621fbdf16b5a80ee04673548af4d95"}, +] + +[package.dependencies] +pydantic = ">=2.0,<3.0.0" +pydantic-settings = ">=2.0,<3.0.0" +Sphinx = ">=4.0" + +[package.extras] +docs = ["myst-parser (>=3.0.0,<4.0.0)", "sphinx-copybutton (>=0.5.0,<0.6.0)", "sphinx-rtd-theme (>=2.0.0,<3.0.0)", "sphinx-tabs (>=3,<4)", "sphinxcontrib-mermaid (>=0.9.0,<0.10.0)"] +erdantic = ["erdantic (<2.0)"] +linting = ["ruff (>=0.4.0,<0.5.0)"] +security = ["pip-audit (>=2.7.2,<3.0.0)"] +test = ["coverage (>=7,<8)", "defusedxml (>=0.7.1)", "pytest (>=8.0.0,<9.0.0)", "pytest-sugar (>=1.0.0,<2.0.0)"] +type-checking = ["mypy (>=1.9,<2.0)", "types-docutils (>=0.20,<0.21)", "typing-extensions (>=4.11,<5.0) ; python_version <= \"3.9\""] + [[package]] name = "autodocsumm" version = "0.2.14" @@ -666,7 +690,7 @@ files = [ {file = "colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6"}, {file = "colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44"}, ] -markers = {main = "platform_system == \"Windows\" or sys_platform == \"win32\"", dev = "platform_system == \"Windows\""} +markers = {main = "sys_platform == \"win32\" or platform_system == \"Windows\"", dev = "platform_system == \"Windows\""} [[package]] name = "colorlog" @@ -2507,6 +2531,30 @@ files = [ [package.dependencies] typing-extensions = ">=4.14.1" +[[package]] +name = "pydantic-settings" +version = "2.13.1" +description = "Settings management using Pydantic" +optional = false +python-versions = ">=3.10" +groups = ["main"] +files = [ + {file = "pydantic_settings-2.13.1-py3-none-any.whl", hash = "sha256:d56fd801823dbeae7f0975e1f8c8e25c258eb75d278ea7abb5d9cebb01b56237"}, + {file = "pydantic_settings-2.13.1.tar.gz", hash = "sha256:b4c11847b15237fb0171e1462bf540e294affb9b86db4d9aa5c01730bdbe4025"}, +] + +[package.dependencies] +pydantic = ">=2.7.0" +python-dotenv = ">=0.21.0" +typing-inspection = ">=0.4.0" + +[package.extras] +aws-secrets-manager = ["boto3 (>=1.35.0)", "boto3-stubs[secretsmanager]"] +azure-key-vault = ["azure-identity (>=1.16.0)", "azure-keyvault-secrets (>=4.8.0)"] +gcp-secret-manager = ["google-cloud-secret-manager (>=2.23.1)"] +toml = ["tomli (>=2.0.1)"] +yaml = ["pyyaml (>=6.0.1)"] + [[package]] name = "pyfakefs" version = "5.9.3" @@ -2679,6 +2727,21 @@ files = [ [package.dependencies] six = ">=1.5" +[[package]] +name = "python-dotenv" +version = "1.2.1" +description = "Read key-value pairs from a .env file and set them as environment variables" +optional = false +python-versions = ">=3.9" +groups = ["main"] +files = [ + {file = "python_dotenv-1.2.1-py3-none-any.whl", hash = "sha256:b81ee9561e9ca4004139c6cbba3a238c32b03e4894671e181b671e8cb8425d61"}, + {file = "python_dotenv-1.2.1.tar.gz", hash = "sha256:42667e897e16ab0d66954af0e60a9caa94f0fd4ecf3aaf6d2d260eec1aa36ad6"}, +] + +[package.extras] +cli = ["click (>=5.0)"] + [[package]] name = "python-slugify" version = "8.0.4" @@ -3953,4 +4016,4 @@ type = ["pytest-mypy"] [metadata] lock-version = "2.1" python-versions = ">=3.10,<4.0" -content-hash = "231df9e065279a52f02698bee3a9eab2706b8be77ec1b72d8ef61ff4a9e6af75" +content-hash = "a37eb8ffe43f83a51a784ed9deb85ca6f82f5f5779e0ae2377006cbc4fef6ec8" diff --git a/pyproject.toml b/pyproject.toml index d01a5f05b..886453e24 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -27,6 +27,7 @@ classifiers = [ "Programming Language :: Python :: 3.14", ] dependencies = [ + "autodoc-pydantic (>=2.2.0,<3.0.0)", "bandit[toml]>=1.7.9,<2", "black>=24.1.0,<26", "coverage>=6.4.4,<8.0.0", From 2d99932bb61e23063f659503bdb3cf4d3ad74522 Mon Sep 17 00:00:00 2001 From: Ariel Schulz Date: Tue, 24 Feb 2026 15:48:07 +0100 Subject: [PATCH 08/11] Add model to API --- doc/api/index.rst | 1 + doc/api/workflow_patcher_config.rst | 36 +++++++++++++++ doc/changes/unreleased.md | 3 +- .../github_workflows/create_and_update.rst | 9 +++- .../features/github_workflows/index.rst | 4 +- .../github_workflows/workflow_patcher.rst | 44 ++++++++++++++++++- exasol/toolbox/config.py | 11 ++--- .../toolbox/util/workflows/patch_workflow.py | 37 +++++++++------- 8 files changed, 121 insertions(+), 24 deletions(-) create mode 100644 doc/api/workflow_patcher_config.rst diff --git a/doc/api/index.rst b/doc/api/index.rst index e738b4d1a..8ad43c961 100644 --- a/doc/api/index.rst +++ b/doc/api/index.rst @@ -8,3 +8,4 @@ base_config workflow_exceptions + workflow_patcher_config diff --git a/doc/api/workflow_patcher_config.rst b/doc/api/workflow_patcher_config.rst new file mode 100644 index 000000000..5935dd729 --- /dev/null +++ b/doc/api/workflow_patcher_config.rst @@ -0,0 +1,36 @@ +.. _workflow_patcher_config: + +WorkflowPatcherConfig +===================== + +.. currentmodule:: exasol.toolbox.util.workflows.patch_workflow + +.. autoclass:: WorkflowPatcherConfig + :members: + :undoc-members: + :show-inheritance: + :member-order: bysource + :special-members: __init__ + +.. autoclass:: Workflow + :members: + :undoc-members: + :show-inheritance: + :member-order: bysource + :special-members: __init__ + +.. autoclass:: StepCustomization + :members: + :undoc-members: + :show-inheritance: + :member-order: bysource + :special-members: __init__ + +.. autoclass:: StepContent + :members: + :undoc-members: + :show-inheritance: + :member-order: bysource + :special-members: __init__ + +.. autofunction:: validate_workflow_name diff --git a/doc/changes/unreleased.md b/doc/changes/unreleased.md index 22d3fc7c7..c41100a9a 100644 --- a/doc/changes/unreleased.md +++ b/doc/changes/unreleased.md @@ -8,12 +8,13 @@ * #712: Added basic logging to workflow processing * #714: Added logic to modify a workflow using the .workflow-patcher.yml * #717: Restricted workflow names in .workflow-patcher.yml to template workflow names -* #719: Added nox session `workflow:generate` to generate/update workflows using the `.workflow-patcher.yml` (if desired) +* #719: Added Nox session `workflow:generate` to generate/update workflows using the `.workflow-patcher.yml` (if desired) ## Documentation * #705: Described how the versions of poetry and python are retrieved * #706: Added description how to ignore findings to the User Guide +* #720: Added documentation for Nox session `workflow:generate` ## Refactoring diff --git a/doc/user_guide/features/github_workflows/create_and_update.rst b/doc/user_guide/features/github_workflows/create_and_update.rst index f3d4dbdfb..3a7f401e6 100644 --- a/doc/user_guide/features/github_workflows/create_and_update.rst +++ b/doc/user_guide/features/github_workflows/create_and_update.rst @@ -68,9 +68,16 @@ The PTB allows you to customise workflows by targeting specific jobs or steps: job's or step's output. To utilize this feature, create a ``.workflow-patcher.yml`` file in your project's -root directory. This will be automatically detected, validated by a pydantic model, and +root directory, as specified further in :ref:`workflow_patcher`. This file will be +automatically detected, validated by a pydantic model, and used when you :ref:`install or update your workflows `. +.. note:: + The pydantic validation checks that the yml file is in the expected format + and that the specified workflow names exist. However, when a workflow is being + generated, each removed job or modified step_id is checked to see if it exists. + If it does not an exist, an exception will be raised (:ref:`workflow_exceptions`). + .. _update_workflows: Add all Workflows to Your Project diff --git a/doc/user_guide/features/github_workflows/index.rst b/doc/user_guide/features/github_workflows/index.rst index 63aa4af03..87e37228f 100644 --- a/doc/user_guide/features/github_workflows/index.rst +++ b/doc/user_guide/features/github_workflows/index.rst @@ -21,13 +21,15 @@ workflows from the templates. .. code-block:: bash - poetry run -- tbx workflow --help + poetry run -- nox -s workflow:generate --help .. attention:: In most cases, we recommend using _all_ workflows without change to ensure consistent interdependencies. For more details, see :ref:`ci_actions`. + The deprecated alternate is to use the CLI provided by + ``poetry run -- tbx workflow --help``. This will be removed by April 22nd, 2025. Workflows --------- diff --git a/doc/user_guide/features/github_workflows/workflow_patcher.rst b/doc/user_guide/features/github_workflows/workflow_patcher.rst index 0c606cec5..d73adeca4 100644 --- a/doc/user_guide/features/github_workflows/workflow_patcher.rst +++ b/doc/user_guide/features/github_workflows/workflow_patcher.rst @@ -6,10 +6,52 @@ Workflow Patcher Underlying the CLI, the PTB uses, if defined, a ``.workflow_patcher.yml`` file to customize generated project-specific workflows. The rendering process is supported by the ``github_workflow_patcher_yaml`` found in your ``noxconfig.py::PROJECT_CONFIG``. -This Path is inherited by default from +This filepath is inherited by default from :py:attr:`exasol.toolbox.config.BaseConfig.github_workflow_patcher_yaml` ensuring a standardized baseline that can be easily overridden, if necessary. .. literalinclude:: ../../../../exasol/toolbox/config.py :language: python :start-at: github_workflow_patcher_yaml + +Model +------- + +.. code-block:: yaml + + workflows: + - name: pr-merge + remove_jobs: + - publish-docs + step_customizations: + - action: REPLACE | INSERT_AFTER + job: run-unit-tests + step_id: check-out-repository + content: + - name: Check out Repository + id: check-out-repository + uses: actions/checkout@v6 + with: + fetch-depth: 0 + + +* ``name``: Name of the GitHub workflow to customize. The PTB supports these workflows: + `exasol/toolbox/templates/github/workflows `__. +* ``remove_jobs``: List of job names to remove from the workflow. + This is useful when a job like ``publish-docs`` is not applicable for a project. +* ``step_customizations``: List of customizations: + + * ``action``: Type of customization + + * ``REPLACE``: Replace an existing step with the new content + * ``INSERT_AFTER``: Insert the content **after** the specified step + + * ``job``: Name of the job inside the workflow that should be modified, e.g. ``run-unit-tests``. + * ``step_id``: ID of the step to replace or after which to insert the new step + * ``content``: Content of the new step. The PTB does not validate that this will work on + GitHub, but it does validate that it is valid YAML content. + +.. note:: + + For more information, see the API documentation for + :class:`exasol.toolbox.util.workflows.patch_workflow.WorkflowPatcherConfig`. diff --git a/exasol/toolbox/config.py b/exasol/toolbox/config.py index 061d2b1e8..3cd02df43 100644 --- a/exasol/toolbox/config.py +++ b/exasol/toolbox/config.py @@ -161,9 +161,10 @@ class BaseConfig(BaseModel): default=(), description=""" This is used to provide hooks to extend one or more of the Nox sessions provided - by the python-toolbox. As described on the plugins pages: - - https://exasol.github.io/python-toolbox/main/user_guide/customization.html#plugins - - https://exasol.github.io/python-toolbox/main/developer_guide/plugins.html, + by the python-toolbox. As described on the plugins pages:\n\n + * https://exasol.github.io/python-toolbox/main/user_guide/customization.html#plugins + * https://exasol.github.io/python-toolbox/main/developer_guide/plugins.html + possible plugin options are defined in ``exasol.toolbox.nox.plugins.NoxTasks``. """, ) @@ -172,8 +173,8 @@ class BaseConfig(BaseModel): description=""" This is used to define which dependency manager is used to install dependencies in the CI. For more details on which PTB version pairs with which - dependency manager, see: - https://exasol.github.io/python-toolbox/main/user_guide/dependencies.html + dependency manager, see:\n\n + * https://exasol.github.io/python-toolbox/main/user_guide/dependencies.html """, ) os_version: str = Field( diff --git a/exasol/toolbox/util/workflows/patch_workflow.py b/exasol/toolbox/util/workflows/patch_workflow.py index ad80f9865..322efeeec 100644 --- a/exasol/toolbox/util/workflows/patch_workflow.py +++ b/exasol/toolbox/util/workflows/patch_workflow.py @@ -32,8 +32,8 @@ class StepContent(BaseModel): The :class:`StepContent` is used to lightly validate the content which would be used to REPLACE or INSERT_AFTER the specified step in the GitHub workflow. - With the value `ConfigDict(extra="allow")`, this model allows for further fields - (e.g. `dummy`) to be specified without any validation. This design choice was + With the value ``ConfigDict(extra="allow")``, this model allows for further fields + (e.g. ``dummy``) to be specified without any validation. This design choice was intentional, as GitHub already allows additional fields and may specify more fields than what has been specified in this model. @@ -54,12 +54,13 @@ class StepContent(BaseModel): class StepCustomization(BaseModel): """ - The :class:`StepCustomization` is used to specify the desired modification: - * REPLACE - means that the contents of the specified `step_id` should be replaced - with whatever `content` is provided. - * INSERT_AFTER - means that the specified `content` should be inserted after - the specified `step_id`. - For a given step + The :class:`StepCustomization` is used to specify the desired modification. + An ``action`` of ``ActionType``: + + * ``REPLACE`` - means that the contents of the specified ``step_id`` should be + replaced with whatever ``content`` is provided. + * ``INSERT_AFTER`` - means that the specified `content` should be inserted after + the specified ``step_id``. """ action: ActionType @@ -69,6 +70,10 @@ class StepCustomization(BaseModel): def validate_workflow_name(workflow_name: str) -> str: + """ + Validates that the given ``workflow_name`` is a valid workflow name provided by + the PTB. + """ if workflow_name not in WORKFLOW_TEMPLATE_OPTIONS.keys(): raise ValueError( f"Invalid workflow: {workflow_name}. Must be one of {WORKFLOW_TEMPLATE_OPTIONS.keys()}" @@ -82,10 +87,12 @@ def validate_workflow_name(workflow_name: str) -> str: class Workflow(BaseModel): """ The :class:`Workflow` is used to specify which workflow should be modified. - This is determined by the workflow `name`. A workflow can be modified by specifying: - * `remove_jobs` - job names in this list will be removed from the workflow. - * `step_customization` - items in this list indicate which job's step - should be modified. + This is determined by the workflow ``name``. A workflow can be modified by + specifying: + + * ``remove_jobs`` - job names in this list will be removed from the workflow. + * ``step_customization`` - items in this list indicate which job's step + should be modified. """ name: WorkflowName @@ -95,9 +102,9 @@ class Workflow(BaseModel): class WorkflowPatcherConfig(BaseModel): """ - The :class:`WorkflowPatcherConfig` is used to validate the expected format for - the `.workflow-patcher.yml`, which is used to modify the workflow templates provided - by the PTB. + The :class:`WorkflowPatcherConfig` is used to validate the expected format + for the ``.workflow-patcher.yml``, which is used to modify the workflow + templates provided by the PTB. """ workflows: list[Workflow] From 217d4d65fd687ed8735f380fce57e61204734d09 Mon Sep 17 00:00:00 2001 From: Ariel Schulz Date: Tue, 24 Feb 2026 16:16:08 +0100 Subject: [PATCH 09/11] Modify changelog --- doc/changes/unreleased.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/doc/changes/unreleased.md b/doc/changes/unreleased.md index c41100a9a..9d608c2c7 100644 --- a/doc/changes/unreleased.md +++ b/doc/changes/unreleased.md @@ -2,19 +2,28 @@ ## Summary +In this major version: +* the Nox session `workflow:generate` has been added to replace the deprecated +`tbx workflow install` and `tbx workflow update`. It has the additional feature +that users may customize the PTB provided workflows with a `.workflow-patcher.yml` +file. +* the GitHub workflow templates have been modified to include step_ids and to follow +an AP-format naming convention, as such it is anticipated that updating the workflows +results in several small changes. + ## Feature * #691: Started customization of PTB workflows by defining the YML schema * #712: Added basic logging to workflow processing -* #714: Added logic to modify a workflow using the .workflow-patcher.yml -* #717: Restricted workflow names in .workflow-patcher.yml to template workflow names +* #714: Added logic to modify a workflow using the `.workflow-patcher.yml` +* #717: Restricted workflow names in `.workflow-patcher.yml` to template workflow names * #719: Added Nox session `workflow:generate` to generate/update workflows using the `.workflow-patcher.yml` (if desired) ## Documentation * #705: Described how the versions of poetry and python are retrieved * #706: Added description how to ignore findings to the User Guide -* #720: Added documentation for Nox session `workflow:generate` +* #721: Added documentation for Nox session `workflow:generate` ## Refactoring From 4019d3fba6032fd3550a4cd9eac3938bf9151c7e Mon Sep 17 00:00:00 2001 From: Ariel Schulz Date: Wed, 25 Feb 2026 09:30:47 +0100 Subject: [PATCH 10/11] Remove addition as not so useful without the field annotation --- doc/conf.py | 1 - poetry.lock | 67 ++------------------------------------------------ pyproject.toml | 1 - 3 files changed, 2 insertions(+), 67 deletions(-) diff --git a/doc/conf.py b/doc/conf.py index 95375d240..afc9576f4 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -29,7 +29,6 @@ extensions = [ "sphinx.ext.todo", "sphinx.ext.autodoc", - "sphinxcontrib.autodoc_pydantic", "sphinx.ext.viewcode", "sphinx.ext.napoleon", "sphinx.ext.intersphinx", diff --git a/poetry.lock b/poetry.lock index 7b3bf768e..634301c0c 100644 --- a/poetry.lock +++ b/poetry.lock @@ -143,30 +143,6 @@ files = [ {file = "attrs-25.4.0.tar.gz", hash = "sha256:16d5969b87f0859ef33a48b35d55ac1be6e42ae49d5e853b597db70c35c57e11"}, ] -[[package]] -name = "autodoc-pydantic" -version = "2.2.0" -description = "Seamlessly integrate pydantic models in your Sphinx documentation." -optional = false -python-versions = "<4.0.0,>=3.8.1" -groups = ["main"] -files = [ - {file = "autodoc_pydantic-2.2.0-py3-none-any.whl", hash = "sha256:8c6a36fbf6ed2700ea9c6d21ea76ad541b621fbdf16b5a80ee04673548af4d95"}, -] - -[package.dependencies] -pydantic = ">=2.0,<3.0.0" -pydantic-settings = ">=2.0,<3.0.0" -Sphinx = ">=4.0" - -[package.extras] -docs = ["myst-parser (>=3.0.0,<4.0.0)", "sphinx-copybutton (>=0.5.0,<0.6.0)", "sphinx-rtd-theme (>=2.0.0,<3.0.0)", "sphinx-tabs (>=3,<4)", "sphinxcontrib-mermaid (>=0.9.0,<0.10.0)"] -erdantic = ["erdantic (<2.0)"] -linting = ["ruff (>=0.4.0,<0.5.0)"] -security = ["pip-audit (>=2.7.2,<3.0.0)"] -test = ["coverage (>=7,<8)", "defusedxml (>=0.7.1)", "pytest (>=8.0.0,<9.0.0)", "pytest-sugar (>=1.0.0,<2.0.0)"] -type-checking = ["mypy (>=1.9,<2.0)", "types-docutils (>=0.20,<0.21)", "typing-extensions (>=4.11,<5.0) ; python_version <= \"3.9\""] - [[package]] name = "autodocsumm" version = "0.2.14" @@ -690,7 +666,7 @@ files = [ {file = "colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6"}, {file = "colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44"}, ] -markers = {main = "sys_platform == \"win32\" or platform_system == \"Windows\"", dev = "platform_system == \"Windows\""} +markers = {main = "platform_system == \"Windows\" or sys_platform == \"win32\"", dev = "platform_system == \"Windows\""} [[package]] name = "colorlog" @@ -2531,30 +2507,6 @@ files = [ [package.dependencies] typing-extensions = ">=4.14.1" -[[package]] -name = "pydantic-settings" -version = "2.13.1" -description = "Settings management using Pydantic" -optional = false -python-versions = ">=3.10" -groups = ["main"] -files = [ - {file = "pydantic_settings-2.13.1-py3-none-any.whl", hash = "sha256:d56fd801823dbeae7f0975e1f8c8e25c258eb75d278ea7abb5d9cebb01b56237"}, - {file = "pydantic_settings-2.13.1.tar.gz", hash = "sha256:b4c11847b15237fb0171e1462bf540e294affb9b86db4d9aa5c01730bdbe4025"}, -] - -[package.dependencies] -pydantic = ">=2.7.0" -python-dotenv = ">=0.21.0" -typing-inspection = ">=0.4.0" - -[package.extras] -aws-secrets-manager = ["boto3 (>=1.35.0)", "boto3-stubs[secretsmanager]"] -azure-key-vault = ["azure-identity (>=1.16.0)", "azure-keyvault-secrets (>=4.8.0)"] -gcp-secret-manager = ["google-cloud-secret-manager (>=2.23.1)"] -toml = ["tomli (>=2.0.1)"] -yaml = ["pyyaml (>=6.0.1)"] - [[package]] name = "pyfakefs" version = "5.9.3" @@ -2727,21 +2679,6 @@ files = [ [package.dependencies] six = ">=1.5" -[[package]] -name = "python-dotenv" -version = "1.2.1" -description = "Read key-value pairs from a .env file and set them as environment variables" -optional = false -python-versions = ">=3.9" -groups = ["main"] -files = [ - {file = "python_dotenv-1.2.1-py3-none-any.whl", hash = "sha256:b81ee9561e9ca4004139c6cbba3a238c32b03e4894671e181b671e8cb8425d61"}, - {file = "python_dotenv-1.2.1.tar.gz", hash = "sha256:42667e897e16ab0d66954af0e60a9caa94f0fd4ecf3aaf6d2d260eec1aa36ad6"}, -] - -[package.extras] -cli = ["click (>=5.0)"] - [[package]] name = "python-slugify" version = "8.0.4" @@ -4016,4 +3953,4 @@ type = ["pytest-mypy"] [metadata] lock-version = "2.1" python-versions = ">=3.10,<4.0" -content-hash = "a37eb8ffe43f83a51a784ed9deb85ca6f82f5f5779e0ae2377006cbc4fef6ec8" +content-hash = "231df9e065279a52f02698bee3a9eab2706b8be77ec1b72d8ef61ff4a9e6af75" diff --git a/pyproject.toml b/pyproject.toml index 886453e24..d01a5f05b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -27,7 +27,6 @@ classifiers = [ "Programming Language :: Python :: 3.14", ] dependencies = [ - "autodoc-pydantic (>=2.2.0,<3.0.0)", "bandit[toml]>=1.7.9,<2", "black>=24.1.0,<26", "coverage>=6.4.4,<8.0.0", From 55892cb5a061cd1c55cf9b5555a3b9fdc5b734e3 Mon Sep 17 00:00:00 2001 From: Ariel Schulz Date: Wed, 25 Feb 2026 15:33:43 +0100 Subject: [PATCH 11/11] Apply review comments & fix another date --- .../features/github_workflows/create_and_update.rst | 4 ++-- .../github_workflows/github_project_configuration.rst | 2 +- doc/user_guide/features/github_workflows/index.rst | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/user_guide/features/github_workflows/create_and_update.rst b/doc/user_guide/features/github_workflows/create_and_update.rst index 3a7f401e6..715796771 100644 --- a/doc/user_guide/features/github_workflows/create_and_update.rst +++ b/doc/user_guide/features/github_workflows/create_and_update.rst @@ -76,7 +76,7 @@ used when you :ref:`install or update your workflows `. The pydantic validation checks that the yml file is in the expected format and that the specified workflow names exist. However, when a workflow is being generated, each removed job or modified step_id is checked to see if it exists. - If it does not an exist, an exception will be raised (:ref:`workflow_exceptions`). + If it does not exist, an exception will be raised (:ref:`workflow_exceptions`). .. _update_workflows: @@ -102,7 +102,7 @@ Add all Workflows to Your Project **Deprecation Notice:** These ``tbx`` endpoints are marked as **deprecated** and are scheduled for removal - by **April 22nd, 2025**. + by **April 22nd, 2026**. Please note that these legacy commands do not allow users to use their specified ``.workflow-patcher.yml`` file to further customise or patch workflows. Users diff --git a/doc/user_guide/features/github_workflows/github_project_configuration.rst b/doc/user_guide/features/github_workflows/github_project_configuration.rst index 3696f26ca..c3ed7eeab 100644 --- a/doc/user_guide/features/github_workflows/github_project_configuration.rst +++ b/doc/user_guide/features/github_workflows/github_project_configuration.rst @@ -8,7 +8,7 @@ Branch Protection The best and most maintainable way to have solid branch protection (:code:`Settings/Branches/main`) is to require the workflow :code:`merge-gate / Allow -Merge` to pass successfully. Additionally, if it make sense for your project, +Merge` to pass successfully. Additionally, if it makes sense for your project, you can further require that ``SonarCloud Code Analysis`` passes. .. note:: diff --git a/doc/user_guide/features/github_workflows/index.rst b/doc/user_guide/features/github_workflows/index.rst index 87e37228f..6393a1b1b 100644 --- a/doc/user_guide/features/github_workflows/index.rst +++ b/doc/user_guide/features/github_workflows/index.rst @@ -29,7 +29,7 @@ workflows from the templates. consistent interdependencies. For more details, see :ref:`ci_actions`. The deprecated alternate is to use the CLI provided by - ``poetry run -- tbx workflow --help``. This will be removed by April 22nd, 2025. + ``poetry run -- tbx workflow --help``. This will be removed by April 22nd, 2026. Workflows ---------