[Python] Selenium with chromedriver not working in AWS EKS but works fine locally

I am scraping data from many web pages. I am processing the web pages in batches. It works fine when I run it locally as devcontainer in vscode, but when I run it in AWS EKS as a job, its processing some URLs (40 or 50 out of 300) and then its giving the below error and crashing:

Error starting WebDriver: Message: session not created: probably user data directory is already in use, please specify a unique value for --user-data-dir argument, or don't use --user-data-dir


I tried adding the --user-data-dir argument and setting its value to a unique directory for every time the web driver is initialized, but that is also giving the same issue.

self.tmp_dir = os.path.join(os.getcwd(), f"chrome_data_{int(time.time())}")
os.makedirs(self.tmp_dir, exist_ok=True)

options.add_argument("--headless")
options.add_argument("--disable-gpu")
options.add_argument("--no-sandbox")
options.add_argument(
"--disable-dev-shm-usage"
)
options.add_argument(f"user-data-dir={self.tmp_dir}") # Use the temp directory


In the logs from EKS job, I do see a lot of zombie processes

2025-04-11 12:32:25,630 ERROR [root]: init_webdriver - Error starting WebDriver: Message: session not created: probably user data directory is already in use, please specify a unique value for --user-data-dir argument, or don't use --user-data-dir
Stacktrace:
#0 0x556313a37d0a <unknown>
#1 0x5563134e85f0 <unknown>
#2 0x5563135220db <unknown>
#3 0x55631351de2f <unknown>
#4 0x55631356df9e <unknown>
#5 0x55631356d4c6 <unknown>
#6 0x55631355f433 <unknown>
#7 0x55631352bea3 <unknown>
#8 0x55631352cb01 <unknown>
#9 0x5563139fcb5b <unknown>
#10 0x556313a00a41 <unknown>
#11 0x5563139e3c52 <unknown>
#12 0x556313a015b4 <unknown>
#13 0x5563139c7f0f <unknown>
#14 0x556313a25db8 <unknown>
#15 0x556313a25f96 <unknown>
#16 0x556313a36b56 <unknown>
#17 0x7f549a5a6ea7 start_thread
Attempting to close chrome...
2025-04-11 12:32:25,641 INFO [root]: init_webdriver - Running Chrome processes: 44, 46, 51, 52, 629, 631, 636, 637, 1183, 1185, 1190, 1191, 1647, 1649, 1654, 1655, 1782, 1784, 1789, 1790, 2046, 2048, 2053, 2054, 2162, 2164, 2169, 2170, 2748, 2750, 2755, 2756, 2891, 2893, 2898, 2899, 3434, 3436, 3441, 3442, 3570, 3572, 3577, 3578, 3688, 3690, 3695, 3696, 3822, 3824, 3829, 3830, 3984, 3986, 3991, 3992, 4127, 4129, 4134, 4135, 4306, 4308, 4313, 4314, 4431, 4859, 4861, 4866, 4867, 5002, 5004, 5009, 5010, 5254, 5256, 5261, 5262, 5563, 5565, 5570, 5571, 5862, 5864, 5869, 5870, 6006, 6008, 6013, 6014, 6116, 6118, 6123, 6124, 6195, 6197
2025-04-11 12:32:25,641 INFO [root]: quit_driver - Starting to close WebDriver
2025-04-11 12:32:25,642 INFO [root]: quit_driver - WebDriver is None.
2025-04-11 12:32:25,642 INFO [root]: kill_chrome_driver - Starting to terminate Chrome Driver and Chrome.
2025-04-11 12:32:25,653 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 44
2025-04-11 12:32:25,653 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 46
2025-04-11 12:32:25,654 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 51
2025-04-11 12:32:25,654 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 52
2025-04-11 12:32:25,654 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 629
2025-04-11 12:32:25,654 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 631
2025-04-11 12:32:25,655 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 636
2025-04-11 12:32:25,655 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 637
2025-04-11 12:32:25,655 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 1183
2025-04-11 12:32:25,655 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 1185
2025-04-11 12:32:25,656 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 1190
2025-04-11 12:32:25,656 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 1191
2025-04-11 12:32:25,656 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 1647
2025-04-11 12:32:25,656 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 1649
2025-04-11 12:32:25,657 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 1654
2025-04-11 12:32:25,657 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 1655
2025-04-11 12:32:25,657 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 1782
2025-04-11 12:32:25,658 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 1784
2025-04-11 12:32:25,658 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 1789
2025-04-11 12:32:25,658 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 1790
2025-04-11 12:32:25,658 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 2046
2025-04-11 12:32:25,659 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 2048
2025-04-11 12:32:25,659 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 2053
2025-04-11 12:32:25,659 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 2054
2025-04-11 12:32:25,659 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 2162
2025-04-11 12:32:25,660 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 2164
2025-04-11 12:32:25,660 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 2169
2025-04-11 12:32:25,660 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 2170
2025-04-11 12:32:25,660 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 2748
2025-04-11 12:32:25,661 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 2750
2025-04-11 12:32:25,661 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 2755
2025-04-11 12:32:25,661 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 2756
2025-04-11 12:32:25,661 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 2891
2025-04-11 12:32:25,662 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 2893
2025-04-11 12:32:25,662 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 2898
2025-04-11 12:32:25,662 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 2899
2025-04-11 12:32:25,662 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 3434
2025-04-11 12:32:25,663 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 3436
2025-04-11 12:32:25,663 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 3441
2025-04-11 12:32:25,666 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 3442
2025-04-11 12:32:25,666 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 3570
2025-04-11 12:32:25,666 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 3572
2025-04-11 12:32:25,667 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 3577
2025-04-11 12:32:25,667 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 3578
2025-04-11 12:32:25,667 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 3688
2025-04-11 12:32:25,667 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 3690
2025-04-11 12:32:25,668 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 3695
2025-04-11 12:32:25,668 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 3696
2025-04-11 12:32:25,668 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 3822
2025-04-11 12:32:25,668 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 3824
2025-04-11 12:32:25,669 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 3829
2025-04-11 12:32:25,669 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 3830
2025-04-11 12:32:25,670 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 3984
2025-04-11 12:32:25,670 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 3986
2025-04-11 12:32:25,670 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 3991
2025-04-11 12:32:25,670 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 3992
2025-04-11 12:32:25,671 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 4127
2025-04-11 12:32:25,671 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 4129
2025-04-11 12:32:25,671 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 4134
2025-04-11 12:32:25,671 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 4135
2025-04-11 12:32:25,672 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 4306
2025-04-11 12:32:25,672 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 4308
2025-04-11 12:32:25,672 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 4313
2025-04-11 12:32:25,673 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 4314
2025-04-11 12:32:25,673 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 4431
2025-04-11 12:32:25,673 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 4859
2025-04-11 12:32:25,673 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 4861
2025-04-11 12:32:25,674 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 4866
2025-04-11 12:32:25,674 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 4867
2025-04-11 12:32:25,674 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 5002
2025-04-11 12:32:25,674 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 5004
2025-04-11 12:32:25,675 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 5009
2025-04-11 12:32:25,675 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 5010
2025-04-11 12:32:25,675 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 5254
2025-04-11 12:32:25,675 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 5256
2025-04-11 12:32:25,676 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 5261
2025-04-11 12:32:25,676 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 5262
2025-04-11 12:32:25,676 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 5563
2025-04-11 12:32:25,676 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 5565
2025-04-11 12:32:25,677 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 5570
2025-04-11 12:32:25,677 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 5571
2025-04-11 12:32:25,677 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 5862
2025-04-11 12:32:25,677 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 5864
2025-04-11 12:32:25,678 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 5869
2025-04-11 12:32:25,678 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 5870
2025-04-11 12:32:25,678 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 6006
2025-04-11 12:32:25,679 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 6008
2025-04-11 12:32:25,679 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 6013
2025-04-11 12:32:25,679 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 6014
2025-04-11 12:32:25,679 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 6116
2025-04-11 12:32:25,680 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 6118
2025-04-11 12:32:25,680 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 6123
2025-04-11 12:32:25,680 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 6124
2025-04-11 12:32:25,680 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 6195
2025-04-11 12:32:25,681 INFO [root]: kill_chrome_driver - Killed Chrome process with PID: 6197
2025-04-11 12:32:25,691 ERROR [root]: kill_chrome_driver - Failed to kill ChromeDriver processes: Command '['pgrep', '-f', 'chromedriver']' returned non-zero exit status 1.
2025-04-11 12:32:26,705 INFO [root]: kill_chrome_driver - Remaining Chrome-related processes after kill attempts:
2025-04-11 12:32:26,705 INFO [root]: kill_chrome_driver - appuser 44 0.0 0.0 0 0 ? Z 12:14 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,706 INFO [root]: kill_chrome_driver - appuser 46 0.0 0.0 0 0 ? Z 12:14 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,706 INFO [root]: kill_chrome_driver - appuser 51 0.0 0.0 0 0 ? Z 12:14 0:00 [chrome] <defunct>
2025-04-11 12:32:26,707 INFO [root]: kill_chrome_driver - appuser 52 0.0 0.0 0 0 ? Z 12:14 0:00 [chrome] <defunct>
2025-04-11 12:32:26,708 INFO [root]: kill_chrome_driver - appuser 629 0.0 0.0 0 0 ? Z 12:16 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,709 INFO [root]: kill_chrome_driver - appuser 631 0.0 0.0 0 0 ? Z 12:16 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,711 INFO [root]: kill_chrome_driver - appuser 636 0.0 0.0 0 0 ? Z 12:16 0:00 [chrome] <defunct>
2025-04-11 12:32:26,714 INFO [root]: kill_chrome_driver - appuser 637 0.0 0.0 0 0 ? Z 12:16 0:00 [chrome] <defunct>
2025-04-11 12:32:26,715 INFO [root]: kill_chrome_driver - appuser 1183 0.0 0.0 0 0 ? Z 12:20 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,715 INFO [root]: kill_chrome_driver - appuser 1185 0.0 0.0 0 0 ? Z 12:20 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,716 INFO [root]: kill_chrome_driver - appuser 1190 0.0 0.0 0 0 ? Z 12:20 0:00 [chrome] <defunct>
2025-04-11 12:32:26,716 INFO [root]: kill_chrome_driver - appuser 1191 0.0 0.0 0 0 ? Z 12:20 0:00 [chrome] <defunct>
2025-04-11 12:32:26,716 INFO [root]: kill_chrome_driver - appuser 1647 0.0 0.0 0 0 ? Z 12:21 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,717 INFO [root]: kill_chrome_driver - appuser 1649 0.0 0.0 0 0 ? Z 12:21 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,719 INFO [root]: kill_chrome_driver - appuser 1654 0.0 0.0 0 0 ? Z 12:21 0:00 [chrome] <defunct>
2025-04-11 12:32:26,719 INFO [root]: kill_chrome_driver - appuser 1655 0.0 0.0 0 0 ? Z 12:21 0:00 [chrome] <defunct>
2025-04-11 12:32:26,720 INFO [root]: kill_chrome_driver - appuser 1782 0.0 0.0 0 0 ? Z 12:21 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,721 INFO [root]: kill_chrome_driver - appuser 1784 0.0 0.0 0 0 ? Z 12:21 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,722 INFO [root]: kill_chrome_driver - appuser 1789 0.0 0.0 0 0 ? Z 12:21 0:00 [chrome] <defunct>
2025-04-11 12:32:26,723 INFO [root]: kill_chrome_driver - appuser 1790 0.0 0.0 0 0 ? Z 12:21 0:00 [chrome] <defunct>
2025-04-11 12:32:26,724 INFO [root]: kill_chrome_driver - appuser 2046 0.0 0.0 0 0 ? Z 12:22 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,724 INFO [root]: kill_chrome_driver - appuser 2048 0.0 0.0 0 0 ? Z 12:22 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,725 INFO [root]: kill_chrome_driver - appuser 2053 0.0 0.0 0 0 ? Z 12:22 0:00 [chrome] <defunct>
2025-04-11 12:32:26,727 INFO [root]: kill_chrome_driver - appuser 2054 0.0 0.0 0 0 ? Z 12:22 0:00 [chrome] <defunct>
2025-04-11 12:32:26,727 INFO [root]: kill_chrome_driver - appuser 2162 0.0 0.0 0 0 ? Z 12:22 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,728 INFO [root]: kill_chrome_driver - appuser 2164 0.0 0.0 0 0 ? Z 12:22 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,728 INFO [root]: kill_chrome_driver - appuser 2169 0.0 0.0 0 0 ? Z 12:22 0:00 [chrome] <defunct>
2025-04-11 12:32:26,729 INFO [root]: kill_chrome_driver - appuser 2170 0.0 0.0 0 0 ? Z 12:22 0:00 [chrome] <defunct>
2025-04-11 12:32:26,730 INFO [root]: kill_chrome_driver - appuser 2748 0.0 0.0 0 0 ? Z 12:25 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,731 INFO [root]: kill_chrome_driver - appuser 2750 0.0 0.0 0 0 ? Z 12:25 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,733 INFO [root]: kill_chrome_driver - appuser 2755 0.0 0.0 0 0 ? Z 12:25 0:00 [chrome] <defunct>
2025-04-11 12:32:26,733 INFO [root]: kill_chrome_driver - appuser 2756 0.0 0.0 0 0 ? Z 12:25 0:00 [chrome] <defunct>
2025-04-11 12:32:26,734 INFO [root]: kill_chrome_driver - appuser 2891 0.0 0.0 0 0 ? Z 12:25 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,734 INFO [root]: kill_chrome_driver - appuser 2893 0.0 0.0 0 0 ? Z 12:25 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,735 INFO [root]: kill_chrome_driver - appuser 2898 0.0 0.0 0 0 ? Z 12:25 0:00 [chrome] <defunct>
2025-04-11 12:32:26,735 INFO [root]: kill_chrome_driver - appuser 2899 0.0 0.0 0 0 ? Z 12:25 0:00 [chrome] <defunct>
2025-04-11 12:32:26,736 INFO [root]: kill_chrome_driver - appuser 3434 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,736 INFO [root]: kill_chrome_driver - appuser 3436 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,739 INFO [root]: kill_chrome_driver - appuser 3441 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome] <defunct>
2025-04-11 12:32:26,741 INFO [root]: kill_chrome_driver - appuser 3442 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome] <defunct>
2025-04-11 12:32:26,741 INFO [root]: kill_chrome_driver - appuser 3570 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,742 INFO [root]: kill_chrome_driver - appuser 3572 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,742 INFO [root]: kill_chrome_driver - appuser 3577 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome] <defunct>
2025-04-11 12:32:26,743 INFO [root]: kill_chrome_driver - appuser 3578 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome] <defunct>
2025-04-11 12:32:26,743 INFO [root]: kill_chrome_driver - appuser 3688 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,743 INFO [root]: kill_chrome_driver - appuser 3690 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,744 INFO [root]: kill_chrome_driver - appuser 3695 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome] <defunct>
2025-04-11 12:32:26,744 INFO [root]: kill_chrome_driver - appuser 3696 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome] <defunct>
2025-04-11 12:32:26,745 INFO [root]: kill_chrome_driver - appuser 3822 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,745 INFO [root]: kill_chrome_driver - appuser 3824 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,746 INFO [root]: kill_chrome_driver - appuser 3829 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome] <defunct>
2025-04-11 12:32:26,747 INFO [root]: kill_chrome_driver - appuser 3830 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome] <defunct>
2025-04-11 12:32:26,747 INFO [root]: kill_chrome_driver - appuser 3984 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,748 INFO [root]: kill_chrome_driver - appuser 3986 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,748 INFO [root]: kill_chrome_driver - appuser 3991 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome] <defunct>
2025-04-11 12:32:26,748 INFO [root]: kill_chrome_driver - appuser 3992 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome] <defunct>
2025-04-11 12:32:26,749 INFO [root]: kill_chrome_driver - appuser 4127 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,749 INFO [root]: kill_chrome_driver - appuser 4129 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,750 INFO [root]: kill_chrome_driver - appuser 4134 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome] <defunct>
2025-04-11 12:32:26,750 INFO [root]: kill_chrome_driver - appuser 4135 0.0 0.0 0 0 ? Z 12:27 0:00 [chrome] <defunct>
2025-04-11 12:32:26,750 INFO [root]: kill_chrome_driver - appuser 4306 0.0 0.0 0 0 ? Z 12:28 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,751 INFO [root]: kill_chrome_driver - appuser 4308 0.0 0.0 0 0 ? Z 12:28 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,751 INFO [root]: kill_chrome_driver - appuser 4313 0.0 0.0 0 0 ? Z 12:28 0:00 [chrome] <defunct>
2025-04-11 12:32:26,751 INFO [root]: kill_chrome_driver - appuser 4314 0.0 0.0 0 0 ? Z 12:28 0:00 [chrome] <defunct>
2025-04-11 12:32:26,752 INFO [root]: kill_chrome_driver - appuser 4431 0.0 0.0 0 0 ? Z 12:28 0:00 [chrome] <defunct>
2025-04-11 12:32:26,752 INFO [root]: kill_chrome_driver - appuser 4859 0.0 0.0 0 0 ? Z 12:29 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,752 INFO [root]: kill_chrome_driver - appuser 4861 0.0 0.0 0 0 ? Z 12:29 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,753 INFO [root]: kill_chrome_driver - appuser 4866 0.0 0.0 0 0 ? Z 12:29 0:00 [chrome] <defunct>
2025-04-11 12:32:26,753 INFO [root]: kill_chrome_driver - appuser 4867 0.0 0.0 0 0 ? Z 12:29 0:00 [chrome] <defunct>
2025-04-11 12:32:26,753 INFO [root]: kill_chrome_driver - appuser 5002 0.0 0.0 0 0 ? Z 12:29 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,754 INFO [root]: kill_chrome_driver - appuser 5004 0.0 0.0 0 0 ? Z 12:29 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,754 INFO [root]: kill_chrome_driver - appuser 5009 0.0 0.0 0 0 ? Z 12:29 0:00 [chrome] <defunct>
2025-04-11 12:32:26,755 INFO [root]: kill_chrome_driver - appuser 5010 0.0 0.0 0 0 ? Z 12:29 0:00 [chrome] <defunct>
2025-04-11 12:32:26,755 INFO [root]: kill_chrome_driver - appuser 5254 0.0 0.0 0 0 ? Z 12:29 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,755 INFO [root]: kill_chrome_driver - appuser 5256 0.0 0.0 0 0 ? Z 12:29 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,756 INFO [root]: kill_chrome_driver - appuser 5261 0.0 0.0 0 0 ? Z 12:29 0:00 [chrome] <defunct>
2025-04-11 12:32:26,756 INFO [root]: kill_chrome_driver - appuser 5262 0.0 0.0 0 0 ? Z 12:29 0:00 [chrome] <defunct>
2025-04-11 12:32:26,756 INFO [root]: kill_chrome_driver - appuser 5563 0.0 0.0 0 0 ? Z 12:30 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,756 INFO [root]: kill_chrome_driver - appuser 5565 0.0 0.0 0 0 ? Z 12:30 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,757 INFO [root]: kill_chrome_driver - appuser 5570 0.0 0.0 0 0 ? Z 12:30 0:00 [chrome] <defunct>
2025-04-11 12:32:26,757 INFO [root]: kill_chrome_driver - appuser 5571 0.0 0.0 0 0 ? Z 12:30 0:00 [chrome] <defunct>
2025-04-11 12:32:26,757 INFO [root]: kill_chrome_driver - appuser 5862 0.0 0.0 0 0 ? Z 12:31 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,758 INFO [root]: kill_chrome_driver - appuser 5864 0.0 0.0 0 0 ? Z 12:31 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,758 INFO [root]: kill_chrome_driver - appuser 5869 0.0 0.0 0 0 ? Z 12:31 0:00 [chrome] <defunct>
2025-04-11 12:32:26,759 INFO [root]: kill_chrome_driver - appuser 5870 0.0 0.0 0 0 ? Z 12:31 0:00 [chrome] <defunct>
2025-04-11 12:32:26,759 INFO [root]: kill_chrome_driver - appuser 6006 0.0 0.0 0 0 ? Z 12:31 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,760 INFO [root]: kill_chrome_driver - appuser 6008 0.0 0.0 0 0 ? Z 12:31 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,760 INFO [root]: kill_chrome_driver - appuser 6013 0.0 0.0 0 0 ? Z 12:31 0:00 [chrome] <defunct>
2025-04-11 12:32:26,760 INFO [root]: kill_chrome_driver - appuser 6014 0.0 0.0 0 0 ? Z 12:31 0:00 [chrome] <defunct>
2025-04-11 12:32:26,760 INFO [root]: kill_chrome_driver - appuser 6116 0.0 0.0 0 0 ? Z 12:31 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,761 INFO [root]: kill_chrome_driver - appuser 6118 0.0 0.0 0 0 ? Z 12:31 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,761 INFO [root]: kill_chrome_driver - appuser 6123 0.0 0.0 0 0 ? Z 12:31 0:00 [chrome] <defunct>
2025-04-11 12:32:26,762 INFO [root]: kill_chrome_driver - appuser 6124 0.1 0.0 0 0 ? Z 12:31 0:00 [chrome] <defunct>
2025-04-11 12:32:26,762 INFO [root]: kill_chrome_driver - appuser 6195 0.0 0.0 0 0 ? Z 12:31 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,763 INFO [root]: kill_chrome_driver - appuser 6197 0.0 0.0 0 0 ? Z 12:31 0:00 [chrome_crashpad] <defunct>
2025-04-11 12:32:26,763 ERROR [root]: init_webdriver - Max attempts reached. Exiting.


I am quitting the driver at the end of every batch and killing the chrome processes recursively to kill all the child processes as well.

Does anybody has any idea why it might not be working on AWS EKS but works locally and how can I run it on EKS without any issue?

Continue reading...