I would like to scrape all posts from Twitter/X that contain at least one of the two tags, posted from 01.15.2025 until today (my project is going to last until 18.05) I've been making a bot with playwright, which works pretty well overall, aside from the times when X shows the "Try reloading" error, but I decided to simply scrape several times going backwards in time, and then put the csv files together. I scraped the last week recently, but now when I set it to "until 03.03", the bot only saved 03.03 to the csv, while I see that it scrolls unil 28.02 when it's working. I'm not sure what could be the problem. query = "%20OR%20".join([f"%23{tag}" for tag in hashtags]) search_url = f"https://twitter.com/search?q={query}%20until%3A{until_date}&src=typed_query&f=live" print(f"Search URL: {search_url}") await page.goto(search_url) await page.wait_for_selector('[data-testid="tweet"]', timeout=20000) processor_task = asyncio.create_task( scraper_queue.process_tweets(page, tweets_data) ) last_height = 0 while not scraper_queue.stop_event.is_set(): if (datetime.now() - scraper_queue.last_tweet_time).seconds > timeout: print(f"No new tweets for {timeout} seconds. Saving and exiting...") scraper_queue.stop_event.set() break if await check_reload_prompt(page): continue await page.evaluate("window.scrollTo(0, document.body.scrollHeight)") await page.wait_for_timeout(random.uniform(3000, 5000)) current_height = await page.evaluate("document.body.scrollHeight") if current_height == last_height: print("No more new tweets detected. Exiting...") break last_height = current_height # Collect all visible tweets on the current page tweets = await page.query_selector_all('[data-testid="tweet"]') for tweet in tweets: await scraper_queue.queue.put(tweet) await scraper_queue.queue.put(None) await processor_task await browser.close()``` Continue reading...
