I wrote a script to get all the data in the Airflow database using psycopg2 which was consistent enough but a team mate of mine suggests asyncpg. The asyncpg does seem faster than it but still only seems faster on the paper. For me I had 4 different Airflow instance database connections for which it'd take around 7 seconds to get 303 rows of data. Now the project has expanded and is supposed to include 16 instances. Considering the amount of connections that need to be made using psycopg2 it is bound to make the application slow. How reliable is the asyncpg? Since if I'd need to migrate the code it'll require lots of work and should not prove to be faster than psycopg2 by 2 seconds. It'll annoy not only me but the team as well. Continue reading...
