[Python] How to improve performance when processing large datasets with loops in Python?

I’m working with a large dataset that I fetch using this code:

response = await client.get(tenant_url, headers=headers, params=params)
response.raise_for_status()
data = response.json()
numpy_2d_arrays = np.array([[device["id"]["id"], device["name"]] for device in data["data"]])


To improve performance, I’m considering converting the data to a Polar DataFrame before extracting the id and name. Will this help improve performance, or is there a better way to handle large datasets in loops?

I’d appreciate any suggestions or performance tips!

data is

{
"data": [
{
"id": {
"entityType": "DEVICE",
"id": "7c4145c0-e533-11ef-9681-df1aaf416822"
},
"name": "C4DEE264E540-002"
},
{
"id": {
"entityType": "DEVICE",
"id": "be7b4b90-b36a-11ed-9188-71dcc7f44f0f"
},
"name": "C4DEE264E540"
},
{
"id": {
"entityType": "DEVICE",
"id": "fbcded60-de0e-11ef-9681-df1aaf416822"
},
"name": "C4DEE264E540-001"
}
],
"totalPages": 1,
"totalElements": 3,
"hasNext": false
}

Continue reading...