I'm using Selenium for automation. If I use a normal Selenium driver (with/without headless) then my request is blocked because the website detects the selenium web driver. But if I add in only this line chrome_options.add_argument('--disable-blink-features=AutomationControlled') to the options then the website no longer is able to detect the driver, and my requests succeed. I am trying to use a proxy (requires auth) for the web driver. The easiest way I found is to use Selenium Wire. But when I use the Selenium Wire driver (with/without a proxy) my browser is detected by the site. Using the disable blink option doesn't help. And I'm running into this issue with/without headless (tried "--headless=new" and "--headless"). It seems that Selenium Wire's driver is doing something different. From this link, I noticed this warning message "Selenium Wire is no longer maintained, and the library has a dependency on blinker==1.7.0..." I'm not sure if this affects the option shown above. I tried every solution on this Stack Overflow thread but nothing worked. What is the Selenium Wire driver doing differently? My goal is to be able to use a proxy with the driver. Maybe using the normal selenium driver is better, but I have been facing issues on this SO thread too. def get_chromedriver(proxy): host, port, username, password = parse_proxy(proxy) if host == "": chrome_options = webdriver.ChromeOptions() # chrome_options.add_argument("--headless") else: chrome_options = Options() # chrome_options.add_argument("--headless=new") chrome_options.add_argument(f'user-agent={userAgent}') chrome_options.add_argument('--disable-blink-features=AutomationControlled') chrome_options.add_experimental_option("excludeSwitches", ['enable-automation']) chrome_options.add_experimental_option('useAutomationExtension', False) chrome_options.add_argument("start-maximized") if host != "": proxy_url = f"http://{username}:{password}@{host}:{port}" seleniumwire_options = { "proxy": { "http": proxy_url, "https": proxy_url }, } driver = seleniumwire_webdriver.Chrome( service=Service(ChromeDriverManager().install()), seleniumwire_options=seleniumwire_options, options=chrome_options ) # driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => false})") # tried false and undefined # driver.execute_cdp_cmd("Network.setUserAgentOverride", {"userAgent": userAgent}) # tried multiple useragents else: driver = webdriver.Chrome(service=Service(), options=chrome_options) return driver Continue reading...
