[Python] Scrapy-Splash Waiting for Page to Load

I'm new to scrapy and splash, and I need to scrape data from single page and regular web apps.

A caveat, though, is I'm mostly scraping data from internal tools and applications, so some require authentication and all of them require at least a couple of seconds loading time before the page fully loads.

I naively tried a Python time.sleep(seconds) and it didn't work. It seems like SplashRequest and scrapy.Request both run and yield results, basically. I then learned about LUA scripts as arguments to these requests, and attempted a LUA script with various forms of wait(), but it looks like the requests never actually run the LUA scripts. It finishes right away and my HTMl selectors don't find anything I'm looking for.

I'm following directions from here https://github.com/scrapy-plugins/scrapy-splash, and have their docker instance running on localhost:8050 and created a settings.py.

Anyone with experience here know what I might be missing?

Thanks!

spider.py

# -*- coding: utf-8 -*-
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy_splash import SplashRequest
import logging
import base64
import time
# from selenium import webdriver

# lua_script="""
# function main(splash)
# splash:set_user_agent(splash.args.ua)
# assert(splash:go(splash.args.url))
# splash:wait(5)

# -- requires Splash 2.3
# -- while not splash:select('#user-form') do
# -- splash:wait(5)
# -- end
# repeat
# splash:wait(5))
# until( splash:select('#user-form') ~= nil )

# return {html=splash:html()}
# end
# """

load_page_script="""
function main(splash)
splash:set_user_agent(splash.args.ua)
assert(splash:go(splash.args.url))
splash:wait(5)

function wait_for(splash, condition)
while not condition() do
splash:wait(0.5)
end
end

local result, error = splash:wait_for_resume([[
function main(splash) {
setTimeout(function () {
splash.resume();
}, 5000);
}
]])

wait_for(splash, function()
return splash:evaljs("document.querySelector('#user-form') != null")
end)

-- repeat
-- splash:wait(5))
-- until( splash:select('#user-form') ~= nil )

return {html=splash:html()}
end
"""

class HelpSpider(scrapy.Spider):
name = "help"
allowed_domains = ["secet_internal_url.com"]
start_urls = ['https://secet_internal_url.com']

# http_user = 'splash-user'
# http_pass = 'splash-password'


def start_requests(self):
logger = logging.getLogger()
login_page = 'https://secet_internal_url.com/#/auth'

splash_args = {
'html': 1,
'png': 1,
'width': 600,
'render_all': 1,
'lua_source': load_page_script
}

#splash_args = {
# 'html': 1,
# 'png': 1,
# 'width': 600,
# 'render_all': 1,
# 'lua_source': lua_script
#}

yield SplashRequest(login_page, self.parse, endpoint='execute', magic_response=True, args=splash_args)

def parse(self, response):
# time.sleep(10)
logger = logging.getLogger()

html = response._body.decode("utf-8")

# Looking for a form with the ID 'user-form'
form = response.css('#user-form')

logger.info("####################")
logger.info(form)
logger.info("####################")

Continue reading...