[Python] Handling Large JSON Responses in Python: Using cURL and Exporting Compressed Files

I’m working with the API of a project management web application (similar to a Kanban system). The app allows users to create and manage cards, where each card can have custom attributes. Through the API, i can also create/manage those cards, but i need their attributes & id. Here’s an example of the data structure (oversimplified of course):

Card 1:
CAT_A: info_A
CAT_B: info_B
Card 2:
CAT_A: info_C
CAT_B: info_D
CAT_C: info_E


I’ve written a Python script to retrieve all the attributes (CAT_*). The script works by querying several endpoints because the API design is suboptimal. It works perfectly for most cases, except for two specific URLs.

The URL format I’m using is as follows:

[base_url]/projects/[project_id]/[endpoint]


For one specific project_id/endpoint pair, the request fails (as error 500), even though:

• The same project_id works with other endpoints.
• The same endpoint works with other project_id values.

What I’ve Tried:

• Tested the problematic pair with cURL, and it worked. However, the returned JSON was almost 4 million lines long.
• Discussed with colleagues who suggested the issue might be related to the size of the JSON response.

My Questions:

1. How can I properly implement cURL in Python to handle such large responses?
2. How can I directly extract the JSON response from the API into a compressed file (e.g., .gz) and then work with it in Python?

Additional Context:

• I’m using Python for this project.
• The API doesn’t seem to offer direct support for compressing responses.
• Efficiency is important since the JSON size is enormous.
• Im in an company network, so i need to specify some things like the proxy etc.
• Im not sure if i can give the name of the app nor the API docs link, but tbh, the API docs is really bad
• Im not sure if i can show some code, but im working on a way to anonymise it and make it safe for me

Thank you in advance for your help!

Continue reading...