[Python] Scrapy spider can't connect when running through main, but has no problem when done...

Basically the title. For some unknown reason, one of my spiders has trouble connecting to an url when I add it to my script, but none when doing so using the scrapy crawl command on the terminal. On both occasions I'm trying to scrape the same amount of data, using the same settings and I even tried to remove my other spiders from the script to see if they weren't not interfering with each other, but couldn't solve it

Those two are the example codes for my defective spider and my main file

Main:

import scrapy
from scrapy.utils.project import get_project_settings
from scrapy.crawler import CrawlerProcess
from ProjectFolder import Examplespider1
from ProjectFolder import Examplespider2


if __name__ == "__main__":


shared_data = {
"names": list_of_names,
"results": {
"spider1": {},
"spider2": {},
}
}


search_term = 'example'


settings = get_project_settings()
process = CrawlerProcess(settings)


process.crawl(Examplespider1, shared_data=shared_data, process_name=search_term)
process.crawl(Examplespider2, shared_data=shared_data, process_name=search_term)


process.start()


spider1_results = shared_data["results"]["spider1"]
spider2_results = shared_data["results"]["spider2"]

#do something with the data


Problem Spider:

import scrapy

class Examplespider1(scrapy.Spider):
name = "spider1"
allowed_domains = ["exampledomain.com"]
start_urls = ["https:exampledomain.com/something"]

def __init__(self, shared_data, process_name, *args, **kwargs):
super().__init__(*args, **kwargs)
self.shared_data = shared_data
self.process_name = process_name

def parse(self, response):
view_state = response.xpath('//input[@name="javax.faces.ViewState"]/@value').get()
if not view_state:
self.log("Não foi possível encontrar javax.faces.ViewState na página inicial.", level=scrapy.log.ERROR)
return


names = self.shared_data["names"]

for name in names:

form_data = {
formdata
}

yield scrapy.FormRequest(
url=self.start_urls[0],
formdata=form_data,
callback=self.parsesearch,
cb_kwargs={'name': name}
)

def parsesearch(self, response, nome_da_parte):
namespaces = {'x': 'http://www.w3.org/1999/xhtml'}
processos = response.xpath(
'//x:tbody[@id="fPProcessosTable:tb"]//x:b[@class="btn-block"]/text()',
namespaces=namespaces
).getall()

self.shared_data["results"]["spider1"][name] = processos
yield {
"Name": name,
"processos": processos,
}


When I'm using the spider alone on the terminal, I just comment the whole init function and just create a local variable with all the names and comment the parts related to the shared_date, but using a local variable for the names in the spider by default obviously solved nothing, so I'm completely out of ideas here

Don't think it'll be particularly useful, but just in case, here are the logs when my spider succeeds in the terminal and when it fails on script:

2025-01-23 11:42:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://example.com/something> (referer: None)

ERROR: Gave up retrying <GET https:example.com/something> (failed 3 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>]

Continue reading...