[Python] Why does OpenStreetMap give a 403 error through Python's urllib.request.urlretrieve,...

I'm trying to grab tiles from OpenStreetMap. I learned the syntax of their API. I have an "old way" of doing things, using Requests, and a "new way", which is being used by someone else's github project. I'm trying to use their project, but it's failing for me in one spot. I then put together this short script as a minimal example:

from PIL import Image
import requests
import urllib.request

#A tile we want to grab from Open Street Map
tile_url = "https://a.tile.openstreetmap.org/16/19299/24629.png"


#Old way: Get the bytes via requests.get, then parse as image and save.
old_way = Image.open(requests.get(tile_url,stream=True).raw)
old_way.save("oldway.png")


#New way: use urlretrieve to directly copy the file over to newway.png
destination = "newway.png"
try:
path, response = urllib.request.urlretrieve(tile_url, destination)
except urllib.error.URLError as e:
print("URL error!")
print(e.code)

#Expected behavior: Two identical images are created,
#named oldway.png and newway.png
#Actual behavior: We get oldway, but newway gives a 403 error.


What's going wrong here? What's the difference between these two HTTP GET requests which results in the first one working fine, and the second one giving a 403 error? I've tried digging into the source code of the respective Python libraries, but it turns out to be a pretty long mess of functions calling each other. And of course since it's HTTPS, I can't monitor the network connection to evaluate the raw bytes being transferred and figure things out that way.

Is there something wrong with the user-agent? Something with the headers? Please note that I'm not just trying to solve this problem ("What's the issue? You have a method that works. Ignore the broken one"), but I'm trying to learn about the details here and hoping that this odd edge case can benefit me in the future.

Continue reading...