I'm doing a web scrape of a website with 122 different pages with 10 entries per page. The code breaks on random pages, on random entries each time it is ran. I can run the code on a url one time and it works while other times it does not. def get_soup(url): soup = BeautifulSoup(requests.get(url).content, 'html.parser') return soup def from_soup(soup, myCellsList): cellsList = soup.find_all('li', {'class' : 'product clearfix'}) for i in range (len(cellsList)): ottdDict = {} ottdDict['Name'] = cellsList.h3.text.strip() This is only a piece of my code, but this is where the error is occurring. The problem is that when I use this code, the h3 tag is not always appearing in each item in the cellsList. This results in a NoneType error when the last line of the code is ran. However, the h3 tag is always there in the HTML when I inspect the webpage. cellsList vs html 1 same comparison made from subsequent soup request What could be causing these differences and how can I avoid this problem? I was able to run the code successfully for a time, and it seems to have all of a sudden stopped working. The code is able to scrape some pages without problem but it randomly does not register the h3 tags on random entries on random pages. Continue reading...
