[Python] Python image scraping tool giving errors every couple image downloads

So, I'm in way over my head with this stuff. I haven't used Python since middle school and the extent of my coding practice is in MediaWiki and HTML. I'm trying to scrape my Fandom wiki of its images in full resolution, and I'm working with this script I found online that I've somewhat managed to revive. It technically works, but there are two major issues.

It uses a list of filenames to navigate urls, pulling an image from each one. I need the filenames unchanged, but I can't get it to write the image without having a filename in the save path directory, which results in each of my files having a prefix.

The second, much bigger problem is that every couple images I'll get a

TypeError: 'NoneType' is not subscriptable


message, and it will skip that image. I have no clue why it's doing this if the script will work the other half of the time.

TypeError traceback

This is the full script:

import requests
import lxml
import bs4
import time
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup

FILE_URL_BASE = "https://destinycollect.fandom.com/wiki/"

USERNAME = "username" # Input valid Fandom login credentials
PASSWORD = "password"

filenames = []

# Log in to your Fandom account.
def auto_login(d, username, password):
d.get("https://www.fandom.com/signin")
uname = driver.find_element(By.ID, "identifier")
uname.click()
uname.send_keys(username)
pswrd = driver.find_element(By.ID, "password")
pswrd.click()
pswrd.send_keys(password)
time.sleep(5)
print("Success!")

# Download the image at the given filename
def download_file(filename):
try:
url_string = FILE_URL_BASE + filename
driver.get(url_string)
source = driver.page_source
soup = BeautifulSoup(source, 'html.parser')
url = soup.find(property="og:image")['content']
r = requests.get(url)
print(filename)
file_path = save_path + filename[5:]
with open(file_path.replace('"', '_').replace('?', '_'), "wb") as f:
f.write(r.content)
f.close()
except:
return download_file

with open("Filenames.txt", "r", encoding="utf-8") as f:
filenames = f.read().splitlines()
f.close()

driver = webdriver.Chrome()
save_path = 'F:/Wiki Images/REMOVE'
auto_login(driver, USERNAME, PASSWORD)

for item in filenames:
download_file(item)



I spent my whole day figuring out how the original script worked, and how to mend the various deprecated aspects of it. I mostly understand how the login and file locating parts work, though I couldn't get the script to hit the sign in button (you need to be signed in to access each image's url). I'm still not 100% on how the parts referencing 'filenames' works.

In any case, I'm at a loss as to how to make the code more airtight to prevent the copious errors. I added the try/except statement to skip over the failures, but it is missing way too many to be effective. If anyone could give me some pointers as to how I can make this work for me, I would be very grateful.

Here is a small selection of valid filenames to be scraped for potential testing purposes:

File:McFarlane Deluxe Lord Saladin 1.png
File:McFarlane Deluxe Lord Saladin 10.jpg
File:McFarlane Deluxe Lord Saladin 11.jpg
File:McFarlane Deluxe Lord Saladin 12.jpg
File:McFarlane Deluxe Lord Saladin 2.jpg
File:McFarlane Deluxe Lord Saladin 3.jpg
File:McFarlane Deluxe Lord Saladin 4.jpg
File:McFarlane Deluxe Lord Saladin 5.jpg
File:McFarlane Deluxe Lord Saladin 6.jpg
File:McFarlane Deluxe Lord Saladin 7.jpg
File:McFarlane Deluxe Lord Saladin 8.jpg
File:McFarlane Deluxe Lord Saladin 9.jpg

Continue reading...