[Python] Selenium Unable to locate element only when using headless chrome (Python)

I just started learning Selenium and need to verify a login web-page using a jenkins machine in the cloud, which doesn't have a GUI. I managed to run the script successfully on my system which has a UI. However when I modified the script to run headless, it fails saying unable to locate element. My script is as follows:

#!/usr/bin/env python3

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager
import time
import argparse


chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--window-size=1120, 550')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--allow-running-insecure-content')

driver = webdriver.Chrome(ChromeDriverManager().install(), chrome_options=chrome_options)
driver.implicitly_wait(5)

lhip = '13.14.15.16'
user = 'username'
paswd = 'password'


parser = argparse.ArgumentParser()

parser.add_argument('-i', '--lh_ip', type=str, metavar='', default=lhip, help='Public IP of VM' )
parser.add_argument('-u', '--usr', type=str, metavar='', default=user, help='Username for VM')
parser.add_argument('-p', '--pwd', type=str, metavar='', default=paswd, help='Password for VM')

args = parser.parse_args()


lh_url = 'https://' + args.lh_ip + '/login/'
driver.get(lh_url)
try:
if driver.title == 'Privacy error':
driver.find_element_by_id('details-button').click()
driver.find_element_by_id('proceed-link').click()
except:
pass

driver.find_element_by_id('username').send_keys(args.usr)
driver.find_element_by_id('password').send_keys(args.pwd)
driver.find_element_by_id('login-btn').click()
driver.implicitly_wait(10)
try:
if driver.find_element_by_tag_name('span'):
print('Login Failed')
except:
print('Login Successful')
driver.close()



The python script works fine on my system when used without the chrome_options. However upon adding them to run in headless mode, it fails with the following output:

[WDM] - Current google-chrome version is 85.0.4183
[WDM] - Get LATEST driver version for 85.0.4183
[WDM] - Driver [/home/ramesh/.wdm/drivers/chromedriver/linux64/85.0.4183.87/chromedriver] found in cache
Traceback (most recent call last):
File "/home/ramesh/practice_python/test_headless.py", line 44, in <module>
driver.find_element_by_id('username').send_keys(args.usr)
File "/home/ramesh/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 360, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "/home/ramesh/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element
'value': value})['value']
File "/home/ramesh/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/home/ramesh/.local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="username"]"}
(Session info: headless chrome=85.0.4183.121)


Since I have about one day's learning of Selenium, I may be doing something rather silly, so would be very grateful if someone showed me what I've done wrong. I've googled a lot and tried many things but none worked. Also why is it saying "css selector" when I have only used id for username?

Continue reading...