I am trying to collect all image filenames from this website: https://www.shipspotting.com/ I have already collected a python dict cat_dict of all the category names and their id numbers. So my strategy is to iterate through every category page, call the data loading API and save it's response for every page. I have identified https://www.shipspotting.com/ssapi/gallery-search as the request URL which loads the next page of content. However, when I request this URL with the requests library, I get a 404. What do I need to do to obtain the correct response in loading the next page of content? import requests from bs4 import BeautifulSoup cat_page = 'https://www.shipspotting.com/photos/gallery?category=' for cat in cat_dict: cat_link = cat_page + str(cat_dict[cat]) headers = { "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:96.0) Gecko/20100101 Firefox/96.0", "Referer": cat_link } response = requests.get('https://www.shipspotting.com/ssapi/gallery-search', headers=headers) soup = BeautifulSoup(response.text, 'html.parser') https://www.shipspotting.com/photos/gallery?category=169 is an example page (cat_link) Continue reading...
