[Python] BeautifulSoup script gives different output depending on how it is called

I have a problem similar to this one with my Python script.

I have a "script.py" file in which I parse the HTML code of a webpage and look for different elements, among which this one:

<div class="stock available" title="Disponibilité">
<span>
En stock
</span>
</div>


Here is the part of my code looking for this element in "script.py":

import requests
from bs4 import BeautifulSoup
headers = {
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36 Edg/89.0.774.57'
}
page = requests.get("target_website.com", headers = headers, verify = False)
html = BeautifulSoup(page.text, "html.parser")

element_dispo = html.find('div', {'title':'Disponibilité'})
element_dispo = element_dispo.get('class') if element_dispo else []
dispo = 'Dispo' if 'available' in element_dispo else 'Non dispo'


When running the script by itself, everything works as expected, but if I try to execute the "script.py" file by calling it from the "main_script.py" file, with the code below, then the wanted element is not found and dispo always get the value 'Non dispo'.

with open("script.py") as file:
exec(file.read())


I tried to save the parsed HTML codes in .txt files to compare them, and they are strictly identical. I use Microsoft Visual Studio Code and Python 3.9.

Does anyone have any idea of what's happening?

PS: the reason why I call the "script.py" with the exec function is that this file scraps a lot of URLs and create a CSV file with the extracted data. Thus I need it to be executed before opening the output CSV file. Also this is me trying to use one of my collegues' script into my "main_script.py".

Continue reading...