[Python] How do I get all data from an API when I don't know the max number of pages

I am pulling data using an API, Python, and the requests package. I want to pull all the data, but have only been able to pull 4,000 rows. How do I pull all of the data? The number of pages is not present in the response. I don't know how many rows are in the data, but it's more than 4,000.

Here is the working code that can pull 4,000 rows, though some of the details need to remain private:

headers = {
'accept': '*/*',
'Authorization': 'Bearer <generated_token_put_here>',
'Content-Type': 'application/json',
'verify':'/etc/ssl/certs/ca-certificates.crt'
}

data = '{"pageSize": 2000, "pageNumber": 100}' #his is the largest pageSize and pageNumber values that will still return data.

response = requests.post('<api_endpoint_put_here>', headers=headers, verify=True, data=data)

Continue reading...