I have a code that scrapes data out of a certain website using BeautifulSoup, I wanted to deploy that code on the cloud so I deployed it on AWS lambda. the lambda function should use the scraping script to scrape data store that data into arrays, and then use the open() function to create a file, I stored the array data into a CSV file and uploaded that file from aws lambda temp into an S3 Bucket. after event testing, it seems that the CSV file was created and uploaded to an s3 bucket successfully but it's empty, even printing the arrays into the output console shows it's empty. the code works perfectly on my pc, it scrapes the data and saves it on a CSV file on my pc. testing some code changes, I think that the problem is in the requesting data part, maybe it fails to request or receive the data, or maybe there is something wrong with the HTML parser or converting data into a text file. maybe I need to put this part out lambda_handler and pass result text into the handler later. idk, maybe that can help you figure it out for me. requesting data: note that i didnt specify the website iam scraping data out of it as it might be bad way of presenting it. so its **** instead. url = f"https://www.****" # Fetch data from the URL result = requests.get(url) # requesting data from the link doc = BeautifulSoup(result.text, "html.parser") the rest of the code is for scraping data, I dont want to share the full code as it works fine on my pc and i think nothing wrong with it, but mainly i dont want to increase the question length. note: i though it might be a permetion issue but it seems that everything is fine and lambda by default has internet access as fas as i know. Continue reading...
