[Python] scrape a website which has the same url for multiple pages? with the page jump being...

I've been at this for days, I'm trying to scrape this website: "https://careers.ispor.org/jobseeker/search/results/" I've got everything covered, from the script that will extract the information in every job page to the script that counts how many pages are there, but in order to get the individual links of every job posting, I need to loop through the pages containing 25 listings each, but THAT is the problem:

1. it's all under the same URL
2. forget about Selenium, it gets detected and blocked after 2 tries
3. the data loads with javascript so the regular requests method doesn't work, the only working method I found is by using Zenrows' free trial to make the request

Here's my code so far, I got the first page's data successfully at this point:

from zenrows import ZenRowsClient

client = ZenRowsClient("the_api_key_i_got_from_the_trial_account")
url = "https://careers.ispor.org/jobseeker/search/results"
params = {"js_render":"true","premium_proxy":"true"}

response = client.get(url, params=params)
soup = BeautifulSoup(response.content, 'html.parser')


after that I've been trying to replicate the ajax request that I inspected when I clicked on the "next page" button, but the result I keep getting only mimics having gone to the next page while the job postings on the returned result are of the first page:

data = {
'page': '2',
'pos_flt': '0',
'location_autocomplete': 'true',
'radius': '320',
'ajaxRequest': '1',
'user_latlong': 'lat=33.874698638916$long=10.102299690247',
"js_render": "true",
"premium_proxy": "true"
}
cookies = {
'AWSALB': "e6+c5w9IR/N4+ERov3onMB85zlZbl+mughxR4zfjLRLMoq9SJwBHTesVwdSAoTLuK88spU0tbqTVZ8jI7NGHLxMo/7Q+DefZBboxMZDGRMLBY60+HRQaBnKOYDhJ",
'AWSALBCORS': "e6+c5w9IR/N4+ERov3onMB85zlZbl+mughxR4zfjLRLMoq9SJwBHTesVwdSAoTLuK88spU0tbqTVZ8jI7NGHLxMo/7Q+DefZBboxMZDGRMLBY60+HRQaBnKOYDhJ",
'JTSUBREF': "careers.ispor.org",
'datadome': "pn970laC_lalBETD5NWHB~pVKYYLrP2fg9_1JlfW1POc~Ny5Usr37BfuNP1UiAl3kCxoOA7z0Pvlwo69rK5WBre5T9znj0U3p55vC_mMGn1w56eqcSU1eWpla3DYLyJb"
}
headers = {
'authority': 'careers.ispor.org',
'method': 'GET',
'path': '/c/@search_results/controller/includes/search_jobs.cfm?page=2&pos_flt=0&location_autocomplete=true&radius=320&ajaxRequest=1&user_latlong=lat%3D33.874698638916%24long%3D10.102299690247',
'scheme': 'https'
}
response = client.get(url, params=params, headers=headers, cookies=cookies, data=data)
soup2 = BeautifulSoup(response.content, 'html.parser')

Continue reading...