`unevaluatedProperties` do not treat "in-place applicators" per the spec

[hackowitz-af]

unevaluatedProperies should be applied after validating all subschemas, such that ValidationErrors from the subschemas are raised/yielded before raising errors for unevaluatedProperies

From https://json-schema.org/draft/2020-12/draft-bhutton-json-schema-01#name-unevaluatedproperties:

"properties", "patternProperties", "additionalProperties", and all in-place applicators MUST be evaluated before this keyword can be evaluated.

The reference to "in-place applicators" includes logical keywords like allOf and conditional keywords like if and then.

From this documentation, it proceeds that:

• Properties from valid subschemas (recursively) are considered "evaluated"
• Properties from invald subschemas are evaluated first, raising/yielding any ValidationErrors, before being considered "unevaluated".

This makes a significant difference when failing on the first error encountered, where the "unevaluated properties are not allowed" message is misleading for a property that is present in the schema but fails validation - especially when the validation error is nested deeply within the "unevaluated" property.

Environment

$ pip list

Package                   Version
------------------------- --------
attrs                     25.3.0
jsonschema                4.24.0
jsonschema-specifications 2025.4.1
pip                       25.1.1
referencing               0.36.2
rpds-py                   0.25.1
setuptools                80.3.1
typing_extensions         4.14.0
wheel                     0.45.1

Example

The (simplified) example schema attempts to define a generic "signal" schema, which may have different properties depending on it's "data type". This is arguably a bad idea but, good or bad, is possible in draft 2020-12:

{
  "$id": "signal.json",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "unevaluatedProperties": false,
  "required": ["source","destination","data type"],
  "properties": {
    "data type": {"enum": ["float","string"]},
    "source": true,
    "destination": true,
  },
  "allOf": [
    {
      "if": {"properties": {"data type": {"const": "float"}}},
      "then": {"properties": {"precision": {"type": "number"}}}
    },
    {
      "if": {"properties": {"data type": {"const": "string"}}},
      "then": {"properties": {"encoding": {"enum": ["ascii", "utf-8"]}}}
    }
  ]
}

This works as expected for well-formed objects:

validator = jsonschema.Draft202012Validator(schema)
validator.validate({"source": "A/C", "destination": "Store", "data type": "string", "encoding": "utf-8"})
validator.validate({"source": "A/C", "destination": "Store", "data type": "float", "precision": 64})

When a sub-schema gives unexpected results, the unevaluatedProperties error is wrongly printed, instead

validator.validate({"source": "A/C", "destination": "Store", "data type": "string", "encoding": "utf-16"})

ValidationError: Unevaluated properties are not allowed ('encoding' was unexpected)

Failed validating 'unevaluatedProperties' in schema:
    {'$id': 'signal.json',
     '$schema': 'https://json-schema.org/draft/2020-12/schema',
     'unevaluatedProperties': False,
     'required': ['source', 'destination', 'data type'],
     'properties': {'data type': {'enum': ['float', 'string']},
                    'source': True,
                    'destination': True},
     'allOf': [{'if': {'properties': {'data type': {'const': 'float'}}},
                'then': {'properties': {'precision': {'type': 'number'}}}},
               {'if': {'properties': {'data type': {'const': 'string'}}},
                'then': {'properties': {'encoding': {'enum': ['ascii',
                                                              'utf-8']}}}}]}

On instance:
    {'source': 'A/C',
     'destination': 'Store',
     'data type': 'string',
     'encoding': 'utf-16'}

This is not incorrect behavior, and a somewhat misleading message. The real problem is that utf-16 is not an allowed value, not that encoding is unevaluated. Only when we print all errors, we find the real (or at least original) error:

import traceback
for error in validator.iter_errors({"source": "A/C", "destination": "Store", "data type": "string", "encoding": "utf-16"}):
    traceback.print_exception(error)

jsonschema.exceptions.ValidationError: Unevaluated properties are not allowed ('encoding' was unexpected)

Failed validating 'unevaluatedProperties' in schema:
    {'$id': 'signal.json',
     '$schema': 'https://json-schema.org/draft/2020-12/schema',
     'unevaluatedProperties': False,
     'required': ['source', 'destination', 'data type'],
     'properties': {'data type': {'enum': ['float', 'string']},
                    'source': True,
                    'destination': True},
     'allOf': [{'if': {'properties': {'data type': {'const': 'float'}}},
                'then': {'properties': {'precision': {'type': 'number'}}}},
               {'if': {'properties': {'data type': {'const': 'string'}}},
                'then': {'properties': {'encoding': {'enum': ['ascii',
                                                              'utf-8']}}}}]}

On instance:
    {'source': 'A/C',
     'destination': 'Store',
     'data type': 'string',
     'encoding': 'utf-16'}
jsonschema.exceptions.ValidationError: 'utf-16' is not one of ['ascii', 'utf-8']

Failed validating 'enum' in schema['allOf'][1]['then']['properties']['encoding']:
    {'enum': ['ascii', 'utf-8']}

On instance['encoding']:
    'utf-16'

[hackowitz-af]

After a little bit of reading, this might be solved quickly by changing the order of the _validators by reordering (and unfortunately un-alphabetizing) the validators passed to jsonschema.validators.create. I've also noticed that the validator._validators are reversed from the validators function argument, although I can't understand how.

More complex, but I think better, would be to allow annotating (wrapped?) validation functions as dependent. If the function has a jsonschema_depends attribute, it is a list of keywords to validate before the current keyword. If not, or the attribute is falsy, the keyword is independent and the keyword can be validated right away. This allows fairly generic keyword dependence, for the several cases within json-schema drafts themselves as well as for extension to other validators. It also seems much more explicit than dictionary key order, which might change inadvertently without expected consequences.

I'm happy to make a PR myself - I'll plan to go the annotation route unless there is a strong opinion against

[Julian]

Hi there.

I do not want to go down either of those paths for now, the real notion of dependencies between keywords requires a full rewrite of the validator interface (in order to support statefulness, i.e. annotations in general, including beyond those defined in the spec -- though this is in flux, as the next versions of the spec are going to make changes here, so there are chickens and eggs to be had.)

Properties from invald subschemas are evaluated first, raising/yielding any ValidationErrors, before being considered "unevaluated".

This library doesn't have any notion of a guarantee for which errors are raised in which order (and the spec does not have any such notion either, especially given JSON objects are not ordered). The line you quoted is funny language, where the MUST means "you need to know the answer to the evaluation result of the other properties to implement this one" (and funnily, we don't, we just do the work twice, which is another story.) But it doesn't mean anything about which error comes first, there's no such notion.

All that being said I don't disagree the error is confusing, but I don't have time to do the bigger refactor right now. If you want to improve the error message I suspect a one word change (of "was not expected" to "was not evaluated") already might do it, though even nicer would be mentioning "because of errors in a subschema" which probably is doable without any refactoring. A PR to do that would be accepted.

Similarly, a PR to make by_relevance prefer the other error likely also is an improvement, though #1258 might be relevant for that -- i.e. PRs to improve error relevance are also on the table, but look there to see whether it changes the result.

Appreciate the issue.

[hackowitz-af]

Thanks Julian!

... the real notion of dependencies between keywords requires ... statefulness, i.e. annotations in general, including beyond those defined in the spec ...

I totally agree! I think that would be a great change and speed things up a bit since it would also solve the repeat evaluations. Since it seems this is eventually useful (or at least not harmful) in some way or another, beyond this issue, would there be value in a PR or would I be wasting my time?

[hackowitz-af]

Re:

This library doesn't have any notion of a guarantee for which errors are raised in which order (and the spec does not have any such notion either, especially given JSON objects are not ordered).

In the context of the order of properties in the schema and instance, I totally agree and didn't mean to communicate otherwise. I only meant the order of evaluation of interdependent schema keywords. Thanks for clarifying.

"properties", "patternProperties", "additionalProperties", and all in-place applicators MUST be evaluated before this keyword can be evaluated.

The line you quoted ... doesn't mean anything about which error comes first, there's no such notion.

I think this depends on the interpretation of "evaluated" - the spec does communicate a sequence of evaluation between certain dependent keywords (described in 10.1 and 11.1), and that the sequence MUST (as per RFC2119) be followed, at least in this case from 11.3

My interpretation from section7 is that "evaluating" a keyword means processing it and gathering the results. This would mean that any behind-the-scenes processing that does not gather the results is not "evaluation" in it's intended definition. I also assume that the sequence of evaluation is presented via the sequence of gathered results - especially when evaluation is interrupted by raising a ValidationError.

Per this vocabulary, it seems that:

• The indirect processing of anyOf the first time, inside unevaluatedProperties, is not "evaluation" because it does not gather the results.
• The processing of unevaluatedProperties, as a keyword, is "evaluation" because it is processed and it's results are gathered.
• The processing of anyOf the second time, as a keyword, is "evaluation" because it is processed and it's results are gathered.
• The spec is not followed because the in-place applicator anyOf is "evaluated" after unevaluatedProperties, which violates the quoted MUST imperative.

I'm happy to be corrected if my understanding of the terms is wrong.

I don't have time to do the bigger refactor right now.

No problem - I might take a stab at it sometime soon, and make a PR if I think the result is anything you might find useful. All in all, I think a stateful validator would be a performance and capability improvement regardless of if/how this issue is resolved.

Thanks for taking the time to discuss,

[hackowitz-af]

I got clarification from the json-schema-org/json-schema-spec and I did misunderstand the definitions of [un]evaluated the spec uses - which apparently had been a debate long before I joined it. I am proposing a few changes to the spec, but until/unless they are accepted - there is nothing needed to do here