[Python] Python json.loads() truncating inner data

I'm having issues when parsing JSON data from an API request. I get all pages from the API request using this code:

try:
limit = 100 # Max limit per page, can be reduced as needed
page = 1 # Start with first page
headers = {
'Authorization': 'Token {}'.format(api_token),
'Content-Type': 'application/json'}

results = []
while True:
headers['X-Page-Limit'] = str(limit)
headers['X-Page-Page'] = str(page)

response = requests.get(url, headers=headers)
counter += 1

if 200 <= response.status_code < 300:
results.extend(response.json())
response_headers = response.headers
limit = int(response_headers.get('X-Page-Limit', limit))
page = int(response_headers.get('X-Page-Page', page)) + 1
total = int(response_headers.get('X-Page-Total', 0))

if page > (total // limit) + (total % limit > 0): ## Check if all pages have been fetched
print ('Before loads: ',len(results))
print ('Before loads: ',results)
results = json.loads(response.content)
print ('After loads: ',len(results))
print ('After loads: ',results)

return results, counter
break


The 1st print(results) before the json.loads() shows 169 registries, but the 2nd print (results) after json.loads() only shows 69. All records returned have the same structure, but few have a description key with value including quotes or single quotes, and I don't know if that maybe breaking the loads() processs.

Additionally, see the differences in the content in "results" before and after:

BEFORE only shows keys with no values:
['auto_upgrade', 'rtsa', 'wifi', 'switch_mgmt', 'wootcloud', 'rogue', 'mgmt', 'persist_config_on_device', 'uplink_port_config', 'engagement', 'analytic', 'ssh_keys', 'vars', 'wids', 'mxtunnel', 'occupancy', 'public_zone_occupancy', 'zone_occupancy_alert', 'gateway_mgmt', 'ssr', 'mxedge', 'synthetic_test', 'led', 'status_portal', 'config_auto_revert', 'enable_unii_4', 'id', 'for_site', 'site_id', 'org_id', 'created_time', 'modified_time']

AFTER shows all the data:
{'auto_upgrade': {'enabled': True, 'version': 'custom', 'time_of_day': '23:00', 'custom_versions': {'AP12': '0.12.27139', 'AP21': '', 'AP24': '0.12.27139', 'AP32': '0.12.27139', 'AP32E': '0.12.27139', 'AP33': '0.12.27139', 'AP34': '0.12.27139', 'AP41': '', 'AP41E': '', 'AP43': '', 'AP43E': '', 'AP45': '0.12.27139', 'AP45E': '0.12.27139', 'AP61': '', 'AP61E': '', 'AP63': '', 'AP63E': '', 'AP64': '0.12.27139'}, 'day_of_week': 'thu'}, 'rtsa': {'enabled': True, 'track_asset': True, 'app_waking': True}, 'wifi': {'enabled': True, 'locate_unconnected': True, 'mesh_enabled': False, 'mesh_allow_dfs': False, 'detect_interference': False, 'locate_connected': False}, 'switch_mgmt': {'use_mxedge_proxy': False, 'root_password': 'some'}, 'wootcloud': None, 'rogue': {'min_rssi': -85, 'min_duration': 10, 'enabled': True, 'honeypot_enabled': True, 'whitelisted_bssids': [], 'whitelisted_ssids': []}, 'mgmt': {'use_wxtunnel': False}, 'persist_config_on_device': True, 'uplink_port_config': {'keep_wlans_up_if_down': False}, 'engagement': {'dwell_tags': {'passerby': '1-300', 'bounce': '301-14400', 'engaged': '14401-28800', 'stationed': '28801-42000'}, 'dwell_tag_names': {'passerby': '', 'bounce': '', 'engaged': '', 'stationed': ''}, 'hours': {'sun': None, 'mon': None, 'tue': None, 'wed': None, 'thu': None, 'fri': None, 'sat': None}}, 'analytic': {'enabled': True}, 'ssh_keys': [], 'vars': {}, 'wids': {}, 'mxtunnel': {'enabled': False}, 'occupancy': {'min_duration': 120, 'clients_enabled': True, 'sdkclients_enabled': True, 'assets_enabled': True, 'unconnected_clients_enabled': True}, 'public_zone_occupancy': {'enabled': True, 'client_density_enabled': False, 'rssi_zones_enabled': False, 'url': 'https://app.eu.mist.com/something'}, 'zone_occupancy_alert': {'enabled': False, 'threshold': 5}, 'gateway_mgmt': {'app_usage': False, 'security_log_source_interface': '', 'auto_signature_update': {'enable': True, 'time_of_day': '02:00', 'day_of_week': ''}}, 'ssr': {}, 'mxedge': {'radsec': {'enabled': False, 'server_selection': 'ordered', 'match_ssid': False, 'auth_servers': [], 'acct_servers': [], 'src_ip_source': 'any'}, 'mist_nac': {'enabled': False, 'client_ips': {}, 'auth_server_port': 1812, 'acct_server_port': 1812}, 'mist_nacedge': {'enabled': False, 'auth_ttl': 604800, 'mxedge_hosts': []}}, 'synthetic_test': {'-wan_speedtest': ''}, 'led': {'enabled': True, 'brightness': 255}, 'status_portal': {'enabled': False, 'hostnames': []}, 'config_auto_revert': True, 'enable_unii_4': False, 'id': 'portalid', 'for_site': True, 'site_id': 'siteid', 'org_id': 'orgid', 'created_time': 0, 'modified_time': 10}


However, this second method to fetch all pages from API request is returnign the correct "result" with all data, and for the 169 records:

try:
results = []
page = 1
limit = 100
params = {'X-Page-Page': 6, 'X-Page-Total': 109}

pages_remaining = True
while pages_remaining:
response = requests.get(check_devices_stats_url, headers=headers, params=params).json()
if response:
[results.append(i) for i in (response)]
page += 1
params = {'X-Page-Page' : page}
if not len(response) == 100:
pages_remaining = False

Continue reading...