My script uses BS4 and Requests to scrape website data into dictionaries in a list. The same key/value pairs exist for every list item. It's something like this: [{'country': 'us', 'state':'new york', 'people':50},{'country':'us', 'state':'california','people':30']} I turn it into a Pandas dataframe with hundreds of dictionaries in the list. I scrape every hour for 5 hours; only the people key's value changes. I'm not sure a list of lists of dictionaries is a great idea. I only need updated people values. Is this realistically done with built in Python lists and dictionaries? I don't know much about databases, but I think SQLite might be suitable. After several scrapes, I would have easy data access to see people in new york over time or what time california had the most people. I could plot the data in different ways! Continue reading...
