Merge branch 'main' of https://github.com/NHSDigital/data-validation-engine into release_v09

Author: georgeRobertson

CHANGELOG.md

@@ -1,3 +1,20 @@
+## v0.8.1 (2026-06-17)
+
+### Build
+- Update lxml from v4.6.4 to 6.1.1
+- Update pyarrow from 17.0.0 to 23.0.1
+
+## v0.8.0 (2026-06-10)
+
+### Feat
+
+- add additional fields check into csv readers (#109)
+
+### Fix
+
+- add greater error handling around polars and duckdb csv reader (#112)
+- adjust csv header check feedback message to be more detailed
+
 ## v0.7.6 (2026-04-30)
 
 ### Fix

README.md

@@ -4,7 +4,8 @@
 </h1>
 
 ![License](https://img.shields.io/github/license/NHSDigital/data-validation-engine)
-![Version](https://img.shields.io/github/v/release/NHSDigital/data-validation-engine)
+![PyPi](https://img.shields.io/pypi/v/data-validation-engine)
+![Conda](https://anaconda.org/nhs/data-validation-engine/badges/version.svg)
 [![CI Unit Tests](https://github.com/NHSDigital/data-validation-engine/actions/workflows/ci_testing.yml/badge.svg)](https://github.com/NHSDigital/data-validation-engine/actions/workflows/ci_testing.yml)
 [![CI Formatting & Linting](https://github.com/NHSDigital/data-validation-engine/actions/workflows/ci_linting.yml/badge.svg)](https://github.com/NHSDigital/data-validation-engine/actions/workflows/ci_linting.yml)
 
@@ -60,13 +61,16 @@ Below is a list of features that we would like to implement or have been request
 | ------------------------------------------------------------------------------- | ----------------- | --------- |
 | Open source release                                                             | 0.1.0             | Yes       |
 | Uplift to Python 3.11                                                           | 0.2.0             | Yes       |
-| Uplift Pyspark to 3.5                                                           | TBA               | No        |
-| Allow DVE to run on Python 3.12+                                                | TBA               | No        |
-| Upgrade to Pydantic 2.0                                                         | TBA               | No        |
+| Uplift Pyspark to 3.5                                                           | 0.8.0             | Yes       |
+| Allow DVE to run on Python 3.12+                                                | 0.8.0             | Yes       |
+| Upgrade to Pydantic 2.0                                                         | 0.9.0             | No        |
 | Uplift Pyspark to 4.0+                                                          | TBA               | No        |
-| Create a more user friendly interface for building and modifying dischema files | Not yet confirmed | No        |
+| Polars upgrade to v1+ | TBA | No |
+| DuckDB upgrade to v1.5+ | TBA | No |
+| Python 3.13 & 3.14 upgrade | TBA | No |
+| Create a more user friendly interface for building and modifying dischema files | TBA | No        |
 
-Beyond the Python and Pydantic upgrade, we cannot confirm the other features will be made available anytime soon. Therefore, if you have the interest and desire to make these features available, then please read the [Contributing](#Contributing) section and get involved.
+If you are interested in getting any of the unreleased features listed above available, then please read the [Contributing](#Contributing) section and then submit us a pull request.
 
 ## Contributing
 Please see guidance [here](https://github.com/NHSDigital/data-validation-engine/blob/main/CONTRIBUTE.md).

docs/user_guidance/data_contract.md

@@ -6,11 +6,11 @@ tags:
     - Domain Types
 ---
 
-The Data Contract defines the structure (models) of your data and controls how it is typecast. We use [Pydantic](https://docs.pydantic.dev/1.10/) to generate and validate the models. This page is meant to give you greater details on how you should write your Data Contract. If you want a summary of how the Data Contract works, please refer to the [Getting Started](./getting_started.md#rules-configuration-introduction) page.
+The Data Contract defines the structure (models) of your data and controls how it is typecast. We use [Pydantic](https://pydantic.dev/docs/validation/1.10/overview/) to generate and validate the models. This page is meant to give you greater details on how you should write your Data Contract. If you want a summary of how the Data Contract works, please refer to the [Getting Started](./getting_started.md#rules-configuration-introduction) page.
 
 !!! Note
 
-    We plan to migrate to Pydantic v2+ in a future release. This page currently reflects what is available through Pydantic v1.
+    We plan to migrate to Pydantic v2+ in v0.9.0. This page currently reflects what is available through Pydantic v1.
 
 ## Models
 
@@ -206,7 +206,7 @@ If you want to read more about the readers, please see the [File Transformation]
 
 Within the `fields` section of the contract you must define what data type a given field should be. Depending on how strict/lenient you want your types to be, a number of types are available to use. The types available are:
 
-- [Built-in standard library](https://docs.python.org/3.11/library/stdtypes.html) types (such as `int`, `str`, `date`) available with your version of Python installed for the DVE.
+- [Built-in standard library](https://docs.python.org/3.12/library/stdtypes.html) types (such as `int`, `str`, `date`) available with your version of Python installed for the DVE.
 - [Pydantic v1 types](https://docs.pydantic.dev/1.10/usage/types/)
 - [Custom Types](./data_contract.md#custom-types)
 - [Domain types](./data_contract.md#domain-types)

docs/user_guidance/implementations/spark.md

@@ -25,7 +25,7 @@ def get_spark_session() -> SparkSession:
     os.environ["PYSPARK_SUBMIT_ARGS"] = " ".join(
         [
             "--packages",
-            "com.databricks:spark-xml_2.12:0.16.0,io.delta:delta-core_2.12:2.4.0",
+            "com.databricks:spark-xml_2.12:0.16.0,io.delta:delta-spark_2.12:3.2.0",
             "pyspark-shell",
         ]
     )

docs/user_guidance/install.md

@@ -8,7 +8,7 @@ tags:
 !!! warning
     **DVE is currently an unstable package. Expect breaking changes between every minor patch**. We intend to follow semantic versioning of `major.minor.patch` more strictly after a 1.0 release. Until then, we recommend that you pin your install to the latest version available and keep an eye on [future releases](https://github.com/NHSDigital/data-validation-engine/releases).
 
-    **Please note that we only support Python runtimes of 3.10 and 3.11.** In the future we will look to add support for Python versions greater than 3.11, but it's not an immediate priority.
+    **Please note that we only support Python runtimes of 3.10, 3.11 & 3.12.** In the future we will look to add support for Python versions greater than 3.12, but it's not an immediate priority.
 
     If working on Python 3.7, the `0.1` release supports this (and only this) version of Python. However, we have not been updating that version with any bugfixes, performance improvements etc. There are also a number of vulnerable dependencies on version `0.1` release due to [Python 3.7 being depreciated](https://devguide.python.org/versions/) and a number of packages dropping support. **If you choose to install `0.1`, you accept the risks of doing so and additional support will not be provided.**
 
@@ -71,8 +71,11 @@ You can install the DVE package through python package managers such as [pip](ht
     poetry install
     ```
 
-!!! info
-    We are working on getting the DVE available via Conda. We will update this page with the relevant instructions once this has been successfully setup.
+=== "conda"
+
+    ```sh
+    conda install nhs::data-validation-engine
+    ```
 
 Python dependencies are listed in the [`pyproject.toml`](https://github.com/NHSDigital/data-validation-engine/blob/main/pyproject.toml). Many of the dependencies are locked to quite restrictive versions due to complexity of this package. Core packages such as Pydantic, Pyspark and DuckDB are unlikely to receive flexible version constraints as changes in those packages could cause the DVE to malfunction. For less important dependencies, we have tried to make the contraints more flexible. Therefore, we would advise you to install the DVE into a seperate environment rather than trying to integrate it into an existing Python environment.
 
@@ -83,6 +86,7 @@ Once you have installed the DVE you are almost ready to use it. To be able to ru
 
 | DVE Version  | Python Version | DuckDB Version | Spark Version | Pydantic Version |
 | ------------ | -------------- | -------------- | ------------- | ---------------- |
+| >=0.8.0      | >=3.10,<3.13   | 1.1.3          | 3.5.2         | 1.10.19          |
 | >=0.7.2      | >=3.10,<3.12   | 1.1.*          | 3.4.*         | 1.10.16          |
 | >=0.6        | >=3.10,<3.12   | 1.1.*          | 3.4.*         | 1.10.15          |
 | >=0.2,<0.6   | >=3.10,<3.12   | 1.1.0          | 3.4.4         | 1.10.15          |