[Python] How can I reliably automate a FlareSolverr-based scraper in AWS ECS?

I have a Python-based scraper that uses FlareSolverr to bypass JavaScript challenges (e.g. Cloudflare). Locally, everything works fine using Docker Compose:

✅ Local setup (working):

services:
flaresolverr:
image: ghcr.io/flaresolverr/flaresolverr:latest
ports:
- 8191:8191

scraper:
build: .
environment:
FLARESOLVERR_URL: http://flaresolverr:8191
depends_on:
- flaresolverr


In Python:

FLARESOLVERR_URL = os.environ.get("FLARESOLVERR_URL", "http://localhost:8191")
requests.post(f"{FLARESOLVERR_URL}/v1", ...)


Problem in AWS Fargate

I tried deploying both scraper and flaresolverr containers into the same ECS Fargate task definition like this:

{
"family": "scraper-task",
"requiresCompatibilities": ["FARGATE"],
"networkMode": "awsvpc",
"containerDefinitions": [
{
"name": "scraper",
"image": "...",
"essential": true,
"environment": [
{ "name": "FLARESOLVERR_URL", "value": "http://flaresolverr:8191" }
]
},
{
"name": "flaresolverr",
"image": "ghcr.io/flaresolverr/flaresolverr:latest",
"essential": false
}
]
}


But I get this error during execution:

FlareSolverr request failed:
HTTPConnectionPool(host='flaresolverr', port=8191):
Name or service not known


Apparently, the scraper container starts before FlareSolverr is fully up – or DNS resolution fails within Fargate.

things I tried:

• curl polling in entrypoint to wait for FlareSolverr → still fails
• Using localhost, IPs, or container name → doesn’t work in Fargate
• Public FlareSolverr endpoint is not an option (privacy and rate limits)

What I’m trying to achieve:

• Run the scraper automatically (e.g. every 12h) without keeping my laptop on
• Avoid using a public FlareSolverr
• Ideally stay serverless (Fargate preferred), but open to alternatives

Continue reading...