I'm experiencing slow performance when parsing long JSON strings retrieved from Redis in Python. The process involves fetching a large string (a list of dicts with same structure) stored in Redis, then deserializing it into json for further processing. Some lists contains over 100000 items and have size more then 70mb. For such lists i observe time, consumed for converting string to json around 1.5-4 seconds. I tried ujson, orjson, msgspec, msgpack, simdjson, redis-json, but all are giving approximately same result. The best timing of 1.5s i got with msgspec.json.Decoder and providing struct for data, but it is still more then i need. What can be another options, assuming that i have list of dicts with over 100000 items and size over 70mb, which i need to store, and then it should be used in apps from different containers (including highly loaded fastApi service). There are over 10k such lists, but the most of them are small and there are no issues with them, only like 10-15% are big enough to slower overall performance. Any ideas are appreciated, even completely different approach for solving this task.Structure of stored data: to_save = List[Route] class Route(msgspec.Struct): poos: List[str] asts: List[str] has_limit: str items: List[Item] is_v5: bool datas: int actions: int class Item(msgspec.Struct): fro: int to: int header: int type: str actions: int datas: int params: str Continue reading...
