I am trying to get the gold price and it's percentage increase / decrease (using web scraping). I am new to web scraping. The main issue is that the scraped html is dissimilar to the websites html (on google) Hello I am trying to make a python program to scrape the gold price off this website - https://goldprice.org/. I am completely new to the world of web-scraping, but have managed to come up with the following lines of python: # this should print the website's html import requests from bs4 import BeautifulSoup url = "https://goldprice.org/" req = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'}) data = req.content soup = BeautifulSoup(data, "html.parser") print(soup) this works and outputs html after running. The next step is to parse the data and find the following pieces of information: data to find (the gold price and the percentage increase / decrease) I have no idea how to do this, but I tried to implement a searching algorithm like so: def SearchSoup(toFind): """Finds all the indexes of toFind in the soup""" mentionIdx = [] current = "" strLen = len(toFind) start = 0 for i in range(len(soup)): if i - start == strLen: start = i if soup != toFind[i - start]: start = i + 1 current = "" continue current += soup if toFind == current: mentionIdx.append(start) return mentionIdx this does give me a traceback error when parsing the soup: KeyError: 0" - stems from the "if soup != toFind[i - start]:" and line 1573, in __getitem__ return self.attrs[key] ~~~~~~~~~~^^^^^ The main problem is that in the scraped html code, the gold price / gold %inc/dec does not seem to be included. I manually read through every line in it, but it wasn't there. Furthermore, when I went onto the actual website and checked the html on it (by pressing fn + f12), the scraped html differed from it. I am quite lost here Continue reading...
