Wrote code to scrape a website which lists daycares. Code able to get the relevant information one first of 100+ pages. No clue how to accommodate code to scrape the site past page 1. Each page lists only 25 daycares and there are 150+ pages to scrape. I read something about looking for Post or Get methods that are referenced when clicking on page numbers, but wasn't able to fix my code using that information (believe it was a user named "manhattan" who shared a solution.) Here's a copy of the current code. Appreciate some help: import requests, bs4, re cc=requests.get('https://data.nj.gov/childcare_results?center=&phone=&capacity_low=&capacity_high=&county=&city=&addr1=&zip=') cc1=bs4.BeautifulSoup(cc.text,"html.parser") #Trying to get all instances of <a href ="[website for inspection report for each daycare]> <a>, but line below returns many more instances of href. Will have to fix that. #Code below does that for the first page in the above website, but no idea how to make it do the same for the remaining 150+ pages of listings cc2 = cc1.select('div a') print(len(cc2)) print (cc2) Continue reading...
