[Python] Issue Consuming API on Port 448 with Azure WAF: Request Rate Limiting and Proxy...

Context and Problem: I'm developing a service that consumes an API hosted on port 448. The API is protected by Azure's WAF V2, which limits requests: it allows only 150 consecutive requests, after which it blocks further requests for 5 minutes before allowing new ones.

I've tried various solutions to avoid these blocks, including:

Residential Proxies: I tested services like Smart Proxy, but these only support ports 443 and 80, so they're not applicable for port 448.

Tor: Tor IPs are immediately blocked by the API.

Azure Application: I deployed an application in Azure to route the requests through a proxy, but the request still goes out with the server's IP instead of the proxy's.

Important: The Azure-protected API is not my own; I am only consuming it for a scraping process.

from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import JSONResponse
import logging
import uvicorn
import random
import asyncio
from fake_useragent import UserAgent
from curl_cffi import requests as cffi_requests # pip install curl-cffi

# Logging configuration
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)

ua = UserAgent()
app = FastAPI()

def get_random_headers():
"""Generates dynamic headers with a random User-Agent"""
return {
"Authority": "example.domain.com:448",
"Accept": "application/json, text/plain, */*",
"Accept-Language": "en-US,en;q=0.9",
"Origin": "https://example.domain.com",
"Referer": "https://example.domain.com/",
"User-Agent": ua.random,
"Priority": "u=1, i",
"Alt-Used": "example.domain.com",
"Upgrade-Insecure-Requests": "1"
}

async def make_request(url, params=None):
"""Performs the request with headers and mimics browser fingerprinting"""
await asyncio.sleep(random.uniform(1, 4)) # Add random delay
try:
async with cffi_requests.AsyncSession() as s:
response = await s.get(
url,
params=params,
headers={
**get_random_headers(),
"Sec-Ch-Ua": '"Not_A Brand";v="8", "Chromium";v="120"',
"Sec-Ch-Ua-Platform": '"Windows"',
"Sec-Fetch-Dest": "empty",
"Sec-Fetch-Mode": "cors",
"Sec-Fetch-Site": "same-site"
},
impersonate="chrome120"
)

if response.status_code != 200:
logging.error(f"Error {response.status_code}: {response.text}")
raise HTTPException(status_code=response.status_code, detail=response.text)

return JSONResponse(content=response.json())
except HTTPException as http_exc:
raise http_exc
except Exception as e:
logging.error(f"Critical error: {str(e)}")
raise HTTPException(status_code=500, detail="Internal server error")

# Endpoints
@app.get("/api/v2/Query")
async def query_api(request: Request, number: str, only_active: bool = False, page: int = 1):
params = {
"number": number,
"onlyActive": str(only_active).lower(),
"page": page
}
return await make_request("https://example.domain.com:448/api/v2/Query", params)

if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=8080)



What I Have Tried So Far: 1. Residential Proxies: These don't work as they only support ports 443 and 80. 2. Tor: IPs used by Tor get blocked immediately. 3. Routing Requests Through an Azure Application: Even when trying to route the request via a proxy, the outgoing request still shows the server's IP instead of the proxy's.

Questions:

1. 
   Has anyone encountered a similar issue consuming an API with these characteristics and request limitations?
   
   
2. 
   What strategies do you recommend for sending requests to a service on a non-standard port (like 448) when proxy usage or IP rotation is required?
   
   
3. 
   Is there any configuration in Azure (or other environments) that allows outgoing requests to be routed such that the proxy's IP is used instead of the server's?
   

Additional Information: • The API is protected with Azure's WAF V2, limiting the number of consecutive requests to 150, then blocking for 5 minutes. • The Azure API is not my own; I am merely consuming it for a scraping process.

Any suggestions or architectural recommendations to overcome these issues would be greatly appreciated.

Continue reading...