We (data team) are considering using an alternative data ingestion tool to the in-house tool we used for years. Our main candidate was Airbyte, because of how simple it is to use and deploy. Installation: I used the official Documentation#QuickStart to install airbyte using the abctl local install Issue The struggle starts when we wanted to experement with building a custom connectors to some source systems. Since The ingestion logic is a bit unusual and requires some python scripting. So we wanted to use the PythonCDK they provide. but I was unable to get to the connector into the running instance of airbyte. What I tried already I followed the tuttorial they have on their official website PythonCDK. I cloned the repo. I created a functional connector in the dedicated spot (funcational = tests are green). But still I didn't get how to get in into my running instance, or how should the service be deployed so we can use our private connectors. Continue reading...
