[Python] Python aiohttp web scraping: FL.ru and Kwork parsers not returning results despite...

I'm developing a Python web scraper to parse job listings from FL.ru and Kwork freelance platforms using aiohttp for async HTTP requests and BeautifulSoup4 for HTML parsing. Here is the repo: https://github.com/dias-2008/freelance-parser.git
While the program successfully logs in, it's not finding any job listings in the HTML.

Debug logs show:

2025-03-25 00:49:54 - INFO - Starting freelance job parser... 2025-03-25 00:49:54 - INFO - Starting parser: KworkParser 2025-03-25 00:49:55 - INFO - Found 0 project containers on Kwork page. 2025-03-25 00:49:55 - ERROR - No project containers found on Kwork page.

2025-03-25 00:49:55 - INFO - Starting parser: FreelancehuntParser 2025-03-25 00:49:56 - INFO - Found 0 new jobs from FreelancehuntParser

2025-03-25 00:49:56 - INFO - Starting parser: FLRuParser 2025-03-25 00:49:57 - INFO - Found 30 project containers on FL.ru page. 2025-03-25 00:49:57 - WARNING - Title element not found in job container. (30 occurrences) 2025-03-25 00:49:57 - INFO - Parsed 0 jobs from FL.ru.

What I've Tried:

1. Login Process:

• Implemented proper session handling with aiohttp
• Added necessary headers and cookies
• Added CSRF token handling
• Verified login success through profile indicators

1. HTML Parsing:

• Tried multiple BeautifulSoup parsers (html.parser, lxml)
• Attempted different selector combinations for project containers
• Added debug logging and saved HTML for inspection
• Verified HTML structure manually in debug files

1. Request Handling:

• Added delays between requests (2 seconds after login)
• Implemented error handling and retries
• Added session persistence
• Verified response status codes (all 200 OK) Expected vs Actual Results:

Expected:

1. ✓ Successful login
2. ✓ Fetch projects page HTML
3. ❌ Parse 20-30 job listings per page
4. ❌ Extract job details (title, description, price) Actual:

• Login successful (verified through response)
• HTML page loads successfully (312KB+)
• BeautifulSoup finds 0 project containers
• No parsing errors, just empty results The HTML content is being received (verified in debug files), but BeautifulSoup isn't finding any elements with the selectors. Could this be related to JavaScript rendering or am I missing something in the request headers?

Environment:

• Python 3.11
• aiohttp 3.8.5
• beautifulsoup4 4.12.2
• Windows 10

Continue reading...