[Python] Python Amazon Price scrapper linux vs windows, different behaviour

I wrote an amazon price scraper in python (3.8) using beautifulsoup and I made it works after playing with the header using something like

headers = {
'dnt': '1',
'upgrade-insecure-requests': '1',
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'sec-fetch-site': 'none',
'sec-fetch-mode': 'navigate',
'sec-fetch-user': '?1',
'sec-fetch-dest': 'document',
'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',}


My development environment is windows. I have a linux vm (Ubuntu 20 with python 3.8) running on azure and I was planning to deploy my code on it. However, it is not working ont it and when my script access to the url, there is the page with the captcha. If I run again my code on my windows environment it works fine. I also try to use proxy without success.

What is the reason between this 2 differents behaviours ? Is there something I could change to make it work with linux ?

Thanks for your help

Here is an exmaple of the code I am trying

import pandas as pd
import requests
from bs4 import BeautifulSoup
import json
from time import sleep



url = "https://www.amazon.ca/Nintendo-Switch-Neon-Blue-Joy%E2%80%91/dp/B07VGRJDFY/?_encoding=UTF8&pd_rd_w=kxkmL&content-id=amzn1.sym.a33d7cb3-a403-494e-b920-61e26e386e19&pf_rd_p=a33d7cb3-a403-494e-b920-61e26e386e19&pf_rd_r=HR1V9EKMHF8PK0SENZNR&pd_rd_wg=hEc51&pd_rd_r=3552437c-bf9e-4b4e-9585-8aeec4893310&ref_=pd_gw_crs_zg_bs_3198031"

headers = {
'dnt': '1',
'upgrade-insecure-requests': '1',
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'sec-fetch-site': 'none',
'sec-fetch-mode': 'navigate',
'sec-fetch-user': '?1',
'sec-fetch-dest': 'document',
'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8'}

def get_response(url):
response = requests.get(url, headers=headers)
return response

response = get_response(url)

soup = BeautifulSoup(response.content, 'html.parser')

# change the encoding to utf-8
soup.encode('utf-8')

print(soup.prettify())

def check_price():
title = soup.find(id= "productTitle").get_text().strip()

print(title)

check_price()


It works fine in my windows environement : i get the title of the product but not with linux I get the following error:

title = soup.find(id= "productTitle").get_text().strip() AttributeError: 'NoneType' object has no attribute 'get_text'
​

Continue reading...