[Python] How can i avoid rate limit problems in this case?

I'm new in scrapping and I'm fighting with Rate Limit.

The data that I am trying to scrape is found in web pages that share a url except for a part that I will call CODE.

The problem that I found is that having the amount of different code I have I can only download around the first one hundred before being limited by the web.

for code in codes:
url=url_partido.format(code)
data=requests.get(url)
with open("nbadata/games22/{}.html".format(code),"w+") as f:
f.write(data.text)


This way I'm extracting first data, the problem is that I have around 1000 different pieces of code and I get blocked around 100.

I read about different ways of dodge rate limit and one was changing the header to make the server think we come from a navigator:

data = requests.get(url,headers=headers)
for code in codes:
url=url_partido.format(code)
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36'}

data = requests.get(url,headers=headers)
with open("nbadata/games22/{}.html".format(code),"w+") as f:
f.write(data.text)


Fail, same problem. Also tried to add timesleep(1) to reduce the number of demands.

The most useful thing I read was that changing between different servers will be the final answer. I know there are free servers or premium ones. I found some of them and I tried this but IT ALSO FAIL.

servers = ['95.216.194.46:1081', '45.174.87.18:999', ' 45.6.4.60:8080']
for code in codes:
for server in servers:
try:
url = url_partido.format(code).replace('{server}', server)
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36'}
data = requests.get(url,headers=headers)
with open("nbadata/games22/{}.html".format(code),"w+") as f:
f.write(data.text)
break
except Exception as e:
print(f"Error {url}: {e}")
continue


Can you please give me some ideas or tell me what is wrong?

THANK YOU!

Continue reading...