[Python] How can I scrape a page with dynamic content (created by JavaScript) in Python?

I'm trying to develop a simple web scraper. I want to extract plain text without HTML markup. My code works on plain (static) HTML, but not when content is generated by JavaScript embedded in the page.

In particular, when I use urllib2.urlopen(request) to read the page content, it doesn't show anything that would be added by the JavaScript code, because that code isn't executed anywhere. Normally it would be run by the web browser, but that isn't a part of my program.

How can I access this dynamic content from within my Python code?


See also Can scrapy be used to scrape dynamic content from websites that are using AJAX? for answers specific to Scrapy.

See also How can I scroll a web page using selenium webdriver in python? for handling a specific sort of dynamic content via Selenium.

Continue reading...