[Python] Python Flask Celery [WinError 10061] No connection could be made because the target...

I am using Flask with celery. I have a celery task that runs botasaurus code and it seems that botasaurus is using websocket you can see in the celery logs. The function works well and get executed but when the task is done botasaurus close the websocket i am assuming. Which is causing celery to log this error continously.

[WinError 10061] No connection could be made because the target machine actively refused it - goodbye

How to avoid this because it keep on logging the error

Celery Logs

-------------- celery@Samar v5.3.6 (emerald-rush)
--- ***** -----
-- ******* ---- Windows-11-10.0.22631-SP0 2024-10-03 17:13:41
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: manage:0x20042f8af00
- ** ---------- .> transport: sqs://xxxxxxxxxx:**@localhost//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 8 (solo)
-- ******* ---- .> task events: ON
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery


[tasks]
. celery_task.run_datalog_task

[2024-10-03 17:13:41,683: WARNING/MainProcess] C:\Users\DELL\.virtualenvs\MondayDuo-dYwy_AZn\Lib\site-packages\celery\worker\consumer\consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
warnings.warn(

[2024-10-03 17:13:41,683: INFO/MainProcess] Connected to sqs://xxxxxxxxxxxx:**@localhost//
[2024-10-03 17:13:41,698: WARNING/MainProcess] C:\Users\DELL\.virtualenvs\MondayDuo-dYwy_AZn\Lib\site-packages\celery\worker\consumer\consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
warnings.warn(

[2024-10-03 17:13:44,184: INFO/MainProcess] celery@Samar ready.
[2024-10-03 17:13:44,808: INFO/MainProcess] Task celery_task.run_datalog_task[878898b3-3456-46c6-j4g8-3f899bb59cf3] received
[2024-10-03 17:13:44,823: WARNING/MainProcess] Running
[2024-10-03 17:13:45,399: INFO/MainProcess] Websocket connected
[2024-10-03 17:13:45,431: INFO/MainProcess] Websocket connected
[2024-10-03 17:14:18,571: WARNING/MainProcess] Current URL: https://xxxxx.monday.com/boards/4904419343/views/115175155
[2024-10-03 17:14:24,592: WARNING/MainProcess] Clicked options button
[2024-10-03 17:14:27,568: WARNING/MainProcess] Clicked more actions button
[2024-10-03 17:14:28,714: WARNING/MainProcess] Clicked export to excel button
[2024-10-03 17:14:32,480: WARNING/MainProcess] Clicked include subitems checkbox
[2024-10-03 17:14:34,608: WARNING/MainProcess] Downloading file...
[2024-10-03 17:14:52,611: WARNING/MainProcess] Latest file downloaded: C:\Users\DELL\Downloads\Commission_Calculator_1727957674.xlsx
[2024-10-03 17:14:52,611: WARNING/MainProcess] Deleted existing file
[2024-10-03 17:14:52,611: WARNING/MainProcess] Moved file to: E:\salesduo\Codes\MondayDuo\server\routes\xxxx\functions\wxxx_tx\Commission_Calculator_1727957674.xlsx
[2024-10-03 17:14:52,664: ERROR/MainProcess] Connection to remote host was lost. - goodbye
[2024-10-03 17:14:53,061: WARNING/MainProcess] Written
[2024-10-03 17:14:53,062: WARNING/MainProcess]
[2024-10-03 17:14:53,062: WARNING/MainProcess]
[2024-10-03 17:14:53,062: WARNING/MainProcess] output/export_commission_calc_board.json
[2024-10-03 17:14:54,702: ERROR/MainProcess] [WinError 10061] No connection could be made because the target machine actively refused it - goodbye
[2024-10-03 17:14:55,339: WARNING/MainProcess] Item 15040 Avenida Venusto Unit 188, San Diego, CA 92128 already exists
[2024-10-03 17:14:56,746: ERROR/MainProcess] [WinError 10061] No connection could be made because the target machine actively refused it - goodbye
[2024-10-03 17:14:57,002: WARNING/MainProcess] Item 1401 El Norte Pkwy Spc 324, San Marcos, CA 92069 already exists
[2024-10-03 17:14:58,780: ERROR/MainProcess] [WinError 10061] No connection could be made because the target machine actively refused it - goodbye
[2024-10-03 17:14:58,809: WARNING/MainProcess] Item 2232 Cliff St, San Diego, CA 92116 already exists
[2024-10-03 17:15:00,404: WARNING/MainProcess] Item 3311-13 Clairemont Mesa Blvd already exists
[2024-10-03 17:15:00,841: ERROR/MainProcess] [WinError 10061] No connection could be made because the target machine actively refused it - goodbye
[2024-10-03 17:15:02,105: WARNING/MainProcess] Item 142 Agostino, Irvine, CA 92614 already exists
[2024-10-03 17:15:02,901: ERROR/MainProcess] [WinError 10061] No connection could be made because the target machine actively refused it - goodbye
[2024-10-03 17:15:04,278: WARNING/MainProcess] Item SELLER - 2271 Laconia St. , San Diego, CA 92114 already exists
[2024-10-03 17:15:04,949: ERROR/MainProcess] [WinError 10061] No connection could be made because the target machine actively refused it - goodbye
[2024-10-03 17:15:06,528: WARNING/MainProcess] Item 26896 Chance Dr, Menifee, CA 92585 already exists
[2024-10-03 17:15:07,002: ERROR/MainProcess] [WinError 10061] No connection could be made because the target machine actively refused it - goodbye
[2024-10-03 17:15:09,043: ERROR/MainProcess] [WinError 10061] No connection could be made because the target machine actively refused it - goodbye
[2024-10-03 17:15:09,215: WARNING/MainProcess] Item 1599 Buckeye Dr, , San Diego, CA 92021 already exists
[2024-10-03 17:15:11,074: ERROR/MainProcess] [WinError 10061] No connection could be made because the target machine actively refused it - goodbye
[2024-10-03 17:15:11,090: WARNING/MainProcess] Item 19985 Sunset Oaks Dr, Ramona, CA 92065 already exists
[2024-10-03 17:15:12,871: WARNING/MainProcess] Item 1649 Palomar Dr, San Marcos, CA 92069 already exists
[2024-10-03 17:15:13,107: ERROR/MainProcess] [WinError 10061] No connection could be made because the target machine actively refused it - goodbye
[2024-10-03 17:15:14,402: WARNING/MainProcess] Item BUYER- 8020 Caminto Gianna , La Jolla, CA 92037 already exists
[2024-10-03 17:15:14,402: WARNING/MainProcess] No new items to upload
[2024-10-03 17:15:14,402: INFO/MainProcess] Task celery_task.run_datalog_task[878898b3-2668-46c6-b9b4-3f899bb59cf3] succeeded in 89.59399999998277s: 'Datalog Task completed successfully'
[2024-10-03 17:15:15,152: ERROR/MainProcess] [WinError 10061] No connection could be made because the target machine actively refused it - goodbye
[2024-10-03 17:15:17,214: ERROR/MainProcess] [WinError 10061] No connection could be made because the target machine actively refused it - goodbye
[2024-10-03 17:15:19,261: ERROR/MainProcess] [WinError 10061] No connection could be made because the target machine actively refused it - goodbye
[2024-10-03 17:15:21,293: ERROR/MainProcess] [WinError 10061] No connection could be made because the target machine actively refused it - goodbye
[2024-10-03 17:15:23,356: ERROR/MainProcess] [WinError 10061] No connection could be made because the target machine actively refused it - goodbye
[2024-10-03 17:15:25,402: ERROR/MainProcess] [WinError 10061] No connection could be made because the target machine actively refused it - goodbye
[2024-10-03 17:15:27,433: ERROR/MainProcess] [WinError 10061] No connection could be made because the target machine actively refused it - goodbye
[2024-10-03 17:15:29,481: ERROR/MainProcess] [WinError 10061] No connection could be made because the target machine actively refused it - goodbye
[2024-10-03 17:15:31,513: ERROR/MainProcess] [WinError 10061] No connection could be made because the target machine actively refused it - goodbye
[2024-10-03 17:15:33,560: ERROR/MainProcess] [WinError 10061] No connection could be made because the target machine actively refused it - goodbye


celery_task.py

import time
from celery import shared_task

from flask import current_app as app
from routes.Whissel.functions.datalogger import run_datalog


@shared_task(acks_late=False, retry=False)
def run_datalog_task(incoming_item):
run_datalog(app, incoming_item)
return "Datalog Task completed successfully"



datalogger.py

import os
from routes.Whissel.functions.excel_parser import main
from .botasaurus_datalogger import export_commission_calc_board

def run_datalog(app, incoming_item):
try:
try:
os.environ['PATH'] += ':/usr/bin'
export_commission_calc_board()
except Exception as error:
app.logger.error(f'Failed to scrape from Monday: {error}', exc_info=True)

# This code will be executed after the botasaurus function export_commission_calc_board()
incoming_item_id = incoming_item['event']["pulseId"]
main(app, incoming_item_id)

except Exception as error:
app.logger.error(f'Failed to scrape from Monday: {error}', exc_info=True)


botasaurus_datalogger.py

import time
import os
import shutil
from botasaurus.browser import browser, Driver


def get_latest_downloaded_file(downloads_folder):
files = os.listdir(downloads_folder)
files = [f for f in files if os.path.isfile(os.path.join(downloads_folder, f))]
latest_file = max(files, key=lambda f: os.path.getmtime(os.path.join(downloads_folder, f)))
latest_file_path = os.path.join(downloads_folder, latest_file)

print(f"Latest file downloaded: {latest_file_path}")
return latest_file_path

def move_file_to_whissel_tx(latest_file_path):
current_directory = os.getcwd()
whissel_tx_directory = os.path.join(current_directory, "routes", "Whissel", "functions", "whissel_tx")
# whissel_tx_directory = os.path.join(current_directory, "whissel_tx")

for file_name in os.listdir(whissel_tx_directory):
file_path = os.path.join(whissel_tx_directory, file_name)
if os.path.isfile(file_path):
os.remove(file_path)
print("Deleted existing file")

latest_file_name = os.path.basename(latest_file_path)
destination_path = os.path.join(whissel_tx_directory, latest_file_name)
shutil.move(latest_file_path, destination_path)

print(f"Moved file to: {destination_path}")
return destination_path

@browser(
tiny_profile=True,
profile="samar",
headless=True
)
def export_commission_calc_board(driver: Driver, data):
board_url = "https://whissel.monday.com/boards/4904419343/views/115175155"
login_url = "https://whissel.monday.com/auth/login_monday/email_password"
driver.get(board_url)

# Check if the page redirects to the login page
if driver.current_url == login_url:
username = "xxxxxxxxx"
password = "xxxxxxxx"
driver.type("input[id='user_email']", username)
driver.type("input[id='user_password']", password)
print("Logging In")
driver.click("button[aria-label='Log in']")

print("Current URL: " + str(driver.current_url))
time.sleep(5)

driver.wait_for_element("button[aria-label='Options']", 90)
driver.humane_click("button[aria-label='Options']")
print("Clicked options button")

time.sleep(2)
driver.humane_click("svg[aria-label='More actions']")
print("Clicked more actions button")
driver.humane_click("svg[aria-label='Export board to Excel']")
print("Clicked export to excel button")

time.sleep(3)
driver.wait_for_element("input[aria-label='Include subitems']", 90)
driver.click("input[aria-label='Include subitems']")
print("Clicked include subitems checkbox")
time.sleep(1)
driver.wait_for_element("button[class='export-button button_76fe792e8c sizeMedium_7c2cbf729b kindPrimary_b6eee345a9 colorPrimary_8b69792663']", 90)
driver.click(
"button[class='export-button button_76fe792e8c sizeMedium_7c2cbf729b kindPrimary_b6eee345a9 colorPrimary_8b69792663']")

print("Downloading file...")
time.sleep(18)
driver.close()

# downloads_folder = "/home/salesduo/Downloads/" # for server
downloads_folder = r"C:\Users\DELL\Downloads" # for local development samar

latest_file_path = get_latest_downloaded_file(downloads_folder)
move_file_to_whissel_tx(latest_file_path)

return {
"heading": "Successfully downloaded and moved"
}


Thank you for any help

I tried to look through botasaurus docs but there wasn't anything related to websocket it creates.

I tried to change retry option to false in celery but it doesn't seem to work @shared_task(acks_late=False, retry=False)

I am expecting a solution where it leaves the connection with botasaurus without raising any error

Continue reading...