release: v0.8.0 (#103)

Release v0.8.0

Features:
    Add additional field check into csv readers

Fixes:
    add greater error handling around polars and duckdb csv reader

Build:
    Upgrade supported Python versions to 3.12
    Upgrade Pyspark to 3.5.2
    Upgrade idna from 3.13 to 3.15 to resolve vulns
    Upgrade pymdown-extensions from 10.21.2 to 10.21.3 to resolve vulns
    Upgrade urllib3 from 2.6.3 to 2.7.0 to resolve vulns
    Upgrade pytest from 8.4.2 to 9.0.3 to resolve vulns

Author: georgeRobertson

.mise.toml

@@ -1,4 +1,4 @@
 [tools]
-python="3.11"
+python="3.12"
 poetry="2.3.3"
 java="liberica-1.8.0"

CHANGELOG.md

@@ -1,3 +1,14 @@
+## v0.8.0 (2026-06-10)
+
+### Feat
+
+- add additional fields check into csv readers (#109)
+
+### Fix
+
+- add greater error handling around polars and duckdb csv reader (#112)
+- adjust csv header check feedback message to be more detailed
+
 ## v0.7.6 (2026-04-30)
 
 ### Fix

README.md

@@ -60,13 +60,16 @@ Below is a list of features that we would like to implement or have been request
 | ------------------------------------------------------------------------------- | ----------------- | --------- |
 | Open source release                                                             | 0.1.0             | Yes       |
 | Uplift to Python 3.11                                                           | 0.2.0             | Yes       |
-| Uplift Pyspark to 3.5                                                           | TBA               | No        |
-| Allow DVE to run on Python 3.12+                                                | TBA               | No        |
-| Upgrade to Pydantic 2.0                                                         | TBA               | No        |
+| Uplift Pyspark to 3.5                                                           | 0.8.0             | Yes       |
+| Allow DVE to run on Python 3.12+                                                | 0.8.0             | Yes       |
+| Upgrade to Pydantic 2.0                                                         | 0.9.0             | No        |
 | Uplift Pyspark to 4.0+                                                          | TBA               | No        |
-| Create a more user friendly interface for building and modifying dischema files | Not yet confirmed | No        |
+| Polars upgrade to v1+ | TBA | No |
+| DuckDB upgrade to v1.5+ | TBA | No |
+| Python 3.13 & 3.14 upgrade | TBA | No |
+| Create a more user friendly interface for building and modifying dischema files | TBA | No        |
 
-Beyond the Python and Pydantic upgrade, we cannot confirm the other features will be made available anytime soon. Therefore, if you have the interest and desire to make these features available, then please read the [Contributing](#Contributing) section and get involved.
+If you are interested in getting any of the unreleased features listed above available, then please read the [Contributing](#Contributing) section and then submit us a pull request.
 
 ## Contributing
 Please see guidance [here](https://github.com/NHSDigital/data-validation-engine/blob/main/CONTRIBUTE.md).

docs/user_guidance/data_contract.md

@@ -6,11 +6,11 @@ tags:
     - Domain Types
 ---
 
-The Data Contract defines the structure (models) of your data and controls how it is typecast. We use [Pydantic](https://docs.pydantic.dev/1.10/) to generate and validate the models. This page is meant to give you greater details on how you should write your Data Contract. If you want a summary of how the Data Contract works, please refer to the [Getting Started](./getting_started.md#rules-configuration-introduction) page.
+The Data Contract defines the structure (models) of your data and controls how it is typecast. We use [Pydantic](https://pydantic.dev/docs/validation/1.10/overview/) to generate and validate the models. This page is meant to give you greater details on how you should write your Data Contract. If you want a summary of how the Data Contract works, please refer to the [Getting Started](./getting_started.md#rules-configuration-introduction) page.
 
 !!! Note
 
-    We plan to migrate to Pydantic v2+ in a future release. This page currently reflects what is available through Pydantic v1.
+    We plan to migrate to Pydantic v2+ in v0.9.0. This page currently reflects what is available through Pydantic v1.
 
 ## Models
 
@@ -206,7 +206,7 @@ If you want to read more about the readers, please see the [File Transformation]
 
 Within the `fields` section of the contract you must define what data type a given field should be. Depending on how strict/lenient you want your types to be, a number of types are available to use. The types available are:
 
-- [Built-in standard library](https://docs.python.org/3.11/library/stdtypes.html) types (such as `int`, `str`, `date`) available with your version of Python installed for the DVE.
+- [Built-in standard library](https://docs.python.org/3.12/library/stdtypes.html) types (such as `int`, `str`, `date`) available with your version of Python installed for the DVE.
 - [Pydantic v1 types](https://docs.pydantic.dev/1.10/usage/types/)
 - [Custom Types](./data_contract.md#custom-types)
 - [Domain types](./data_contract.md#domain-types)

docs/user_guidance/implementations/spark.md

@@ -25,7 +25,7 @@ def get_spark_session() -> SparkSession:
     os.environ["PYSPARK_SUBMIT_ARGS"] = " ".join(
         [
             "--packages",
-            "com.databricks:spark-xml_2.12:0.16.0,io.delta:delta-core_2.12:2.4.0",
+            "com.databricks:spark-xml_2.12:0.16.0,io.delta:delta-spark_2.12:3.2.0",
             "pyspark-shell",
         ]
     )

docs/user_guidance/install.md

@@ -8,7 +8,7 @@ tags:
 !!! warning
     **DVE is currently an unstable package. Expect breaking changes between every minor patch**. We intend to follow semantic versioning of `major.minor.patch` more strictly after a 1.0 release. Until then, we recommend that you pin your install to the latest version available and keep an eye on [future releases](https://github.com/NHSDigital/data-validation-engine/releases).
 
-    **Please note that we only support Python runtimes of 3.10 and 3.11.** In the future we will look to add support for Python versions greater than 3.11, but it's not an immediate priority.
+    **Please note that we only support Python runtimes of 3.10, 3.11 & 3.12.** In the future we will look to add support for Python versions greater than 3.12, but it's not an immediate priority.
 
     If working on Python 3.7, the `0.1` release supports this (and only this) version of Python. However, we have not been updating that version with any bugfixes, performance improvements etc. There are also a number of vulnerable dependencies on version `0.1` release due to [Python 3.7 being depreciated](https://devguide.python.org/versions/) and a number of packages dropping support. **If you choose to install `0.1`, you accept the risks of doing so and additional support will not be provided.**
 
@@ -83,6 +83,7 @@ Once you have installed the DVE you are almost ready to use it. To be able to ru
 
 | DVE Version  | Python Version | DuckDB Version | Spark Version | Pydantic Version |
 | ------------ | -------------- | -------------- | ------------- | ---------------- |
+| >=0.8.0      | >=3.10,<3.13   | 1.1.3          | 3.5.2         | 1.10.19          |
 | >=0.7.2      | >=3.10,<3.12   | 1.1.*          | 3.4.*         | 1.10.16          |
 | >=0.6        | >=3.10,<3.12   | 1.1.*          | 3.4.*         | 1.10.15          |
 | >=0.2,<0.6   | >=3.10,<3.12   | 1.1.0          | 3.4.4         | 1.10.15          |