[Python] ERROR - SSH operator error: timed out in Airflow while connecting with ec2 instance

I am getting a timeout error while trying to connect with ec2 instance. I can connect to the machine through ssh command:

ssh -i keypair.pem myuser@ec2IPaddress
​

My connection file has the following things:

conn Id: ssh_custom Conn Type: SSH Host: ec2IPaddress Username: myuser Port: 8888 Extra: {"key_file":"/home/ubuntu/keypair.pem", "no_host_key_check": "false", "allow_host_key_change": "true", "timeout": "45"}
​

My dag file looks like:

from airflow import DAG
import datetime as dt
from airflow.operators.bash_operator import BashOperator
from airflow.contrib.hooks.ssh_hook import SSHHook
from airflow.contrib.operators.ssh_operator import SSHOperator

sshHook = SSHHook(ssh_conn_id='ssh_custom')

default_args = {
'owner': 'arpita',
'start_date': dt.datetime(2019, 11, 20),
'retries': 1,
'retry_delay': dt.timedelta(minutes=5),
'depends_on_past': False,
'email': ['example@abc.com'],
'email_on_failure': True,
'email_on_retry': True,
}

with DAG('sample',
default_args=default_args,
schedule_interval='30 * * * *',
) as dag:
task1 = SSHOperator(task_id="task1",
command='echo $HOSTNAME',
ssh_hook=sshHook)

task1


Getting the error:

ERROR - SSH operator error: timed out
Traceback (most recent call last):
File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/contrib/operators/ssh_operator.py", line 101, in execute
with self.ssh_hook.get_conn() as ssh_client:
File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/contrib/hooks/ssh_hook.py", line 180, in get_conn
sock=self.host_proxy)
File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/paramiko/client.py", line 349, in connect
retry_on_signal(lambda: sock.connect(addr))
File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/paramiko/util.py", line 280, in retry_on_signal
return function()
File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/paramiko/client.py", line 349, in <lambda>
retry_on_signal(lambda: sock.connect(addr))
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/models/taskinstance.py", line 926, in _run_raw_task
result = task_copy.execute(context=context)
File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/contrib/operators/ssh_operator.py", line 173, in execute
raise AirflowException("SSH operator error: {0}".format(str(e)))
airflow.exceptions.AirflowException: SSH operator error: timed out
[2019-11-26 15:23:20,254] {taskinstance.py:1080} INFO - All retries failed; marking task as FAILED
[2019-11-26 15:23:20,284] {logging_mixin.py:95} INFO - [[34m2019-11-26 15:23:20,284[0m] {[34mconfiguration.py:[0m299} WARNING[0m - section/key [[1msmtp[0m/[1msmtp_user[0m] not found in config[0m
[2019-11-26 15:23:20,284] {taskinstance.py:1086} ERROR - Failed to send email to: ['example@abc.com']
[2019-11-26 15:23:20,285] {taskinstance.py:1087} ERROR - [Errno 111] Connection refused
Traceback (most recent call last):
File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/contrib/operators/ssh_operator.py", line 101, in execute
with self.ssh_hook.get_conn() as ssh_client:
File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/contrib/hooks/ssh_hook.py", line 180, in get_conn
sock=self.host_proxy)
File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/paramiko/client.py", line 349, in connect
retry_on_signal(lambda: sock.connect(addr))
File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/paramiko/util.py", line 280, in retry_on_signal
return function()
File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/paramiko/client.py", line 349, in <lambda>
retry_on_signal(lambda: sock.connect(addr))
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/models/taskinstance.py", line 926, in _run_raw_task
result = task_copy.execute(context=context)
File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/contrib/operators/ssh_operator.py", line 173, in execute
raise AirflowException("SSH operator error: {0}".format(str(e)))
airflow.exceptions.AirflowException: SSH operator error: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/models/taskinstance.py", line 1084, in handle_failure
self.email_alert(error)
File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/models/taskinstance.py", line 1307, in email_alert
send_email(self.task.email, subject, html_content)
File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/utils/email.py", line 55, in send_email
mime_subtype=mime_subtype, mime_charset=mime_charset, **kwargs)
File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/utils/email.py", line 101, in send_email_smtp
send_MIME_email(smtp_mail_from, recipients, msg, dryrun)
File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/utils/email.py", line 121, in send_MIME_email
s = smtplib.SMTP_SSL(SMTP_HOST, SMTP_PORT) if SMTP_SSL else smtplib.SMTP(SMTP_HOST, SMTP_PORT)
File "/usr/lib/python3.5/smtplib.py", line 251, in __init__
(code, msg) = self.connect(host, port)
File "/usr/lib/python3.5/smtplib.py", line 335, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "/usr/lib/python3.5/smtplib.py", line 306, in _get_socket
self.source_address)
File "/usr/lib/python3.5/socket.py", line 711, in create_connection
raise err
File "/usr/lib/python3.5/socket.py", line 702, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

Continue reading...