[Python] Not receiving headers Scrapy ProxyMesh

Discussão em 'Python' iniciado por Stack, Fevereiro 7, 2025.

  1. Stack

    Stack Membro Participativo

    I am quite new to Scrapy / ProxyMesh. My request to the Proxymesh server seems to be working as I see my bandwith consumption on the ProxyMesh website, and the meta.proxy is correct in my logs. However, when I log the response headers in Scrapy, I do not receive the X-Proxymesh-IP that I am supposed to receive. Here is my code. What am I doing wrong?

    This is my middleware

    class Proxymesh(object):

    def __init__(self):

    logging.debug('Initialized Proxymesh middleware')

    self.proxy_ip = 'http://host:port'

    def process_request(self, request, spider):

    logging.debug('Processing request through proxy IP: ' + self.proxy_ip)

    request.meta['proxy'] = self.proxy_ip


    These are my settings in my spider

    custom_settings = {
    "DOWNLOADER_MIDDLEWARES": {
    "projectName.middlewares.proxymesh.Proxymesh" : 1,
    }


    This is what the response headers look like

    ['Set-Cookie']:['__cfduid=d88d4e4cb7... HttpOnly']
    ['Vary']:['User-Agent,Accept-Encoding']
    ['Server']:['cloudflare-nginx']
    ['Date']:['Thu, 19 Oct 2017 10...38:10 GMT']
    ['Cf-Ray']:['3b031b30cbef1565-CDG']
    ['Content-Type']:['text/html; charset=UTF-8']


    Thank you for your help

    Continue reading...

Compartilhe esta Página