[Python] Using Python how do I validate JSON against a JSON schema in a streaming fashion,...

I have a large JSON that I do not want to load into memory. I would like to validate it against a JSON schema in a streaming fashion. All libraries I could find so far, only validate completely loaded JSON objects (like Pydantic or https://github.com/python-jsonschema/jsonschema). What I rather neeed is some way to validate it feeding the original JSON chunk by chunk, i.e, control the size of the buffer.

This could look like this:

import pydantic # I use V2
import ijson
import pathlib

class A(pydantic.BaseModel):
i: int
a: list[int]
s: str

jsonpath = pathlib.Path("some.json")
validator = MyValidator(schema=A.model_json_schema())
with jsonpath.open("rb") as file:
for prefix, event, value in ijson.parse(file, use_float=True):
validator.event((prefix, event, value))

print(validator.errors)


Imagine some.json file is ~50 MB large A instance with a very long array. I do not want to load the whole object into memory (this is what Pydantic would do), but I want to make sure that some.json complies with the schema of A. The validator.errors could give me a list of errors which is going to be empty in cases none where discovered.

Continue reading...