I am a fresher python dev and trying to build a data pipeline that monitors a gmail, when new mail arrives, we downloads the attachments -> process them and push to database. Processing and pushing to db is complete but I need help in understanding how do I monitor my gmail. I want to build this pipeline event driven and not using polling. Can someone explain me the work flow of the pipeline using these tools:- Flask, Apache airflow? I need help in understanding the structure (monitoring gmail, using Flask and airflow to build this) Python version==3.9 I tried polling, so a simple flask app in python that uses APScheduler to run a cron job.(/gmail). This would check my mail after 4:00PM, if I have unread mails it has relevant subject, I would download the data and pass it to next function for processing. Now I expect this system to use Apache Airflow and it should be event driven, so whenever there is a new mail that arrives my inbox, it should immediately(latency is not any issue) download the attachments and send to next python functions. Continue reading...
