Airflow and Mongo

[openp2pdesign]

The current version of hello_world.py at d6790e2 does not work well, it seems that within Airflow data must be saved to a database through a hook function, but there is currently no function for Mongo. There's a plan for developing it, and a couple of gists, but nothing really usable yet. It's not possible to save data to Mongo yet.
There's a hook and a sensor for Redis, though.

One option would be to save data as JSON or JSONB to Postgres, and then export it to Mongo in the end, hopefully with a specific operator in Airflow or with something else (NiFi #5 ? which could also be used for streaming data, which airflow cannot handle).

[openp2pdesign]

With 0cf367d it is now possible to:

1. save JSON data to postgres with a PythonOperator (to be improved, not fully working yet)
2. launch an external python file with a BashOperator (in the same folder) that can save to Mongo

This is partially tested, seems to be working, it would be an hack that for the moment let us saving data to mongo even without an hook.

Next steps would be:

1. improve saving data to postgres
2. the table name cannot be easily passed as a string to the sql command, so the best thing (also for organizing everything in a clearer way) would be to create a database for each DAG (by checking first if it already exists) and then the tables for raw, clean, meteor
3. rewrite the external python script as a generic postgre2mongo connector, so that we have just to tell him (with bash arguments) which table load from postgres and how to save it in mongo

[openp2pdesign]

The connection with postgres:

[openp2pdesign]

The creation of a connection to PostgreSQL in Airflow has been solved with https://github.com/fablabbcn/DataVisPlanner/blob/3237dc535dbbe5dd1410af12d94125c5d8b22815/airflow/entrypoint.sh#L61

[openp2pdesign]

The MongoDB hook and operator was added to Airflow: https://issues.apache.org/jira/browse/AIRFLOW-83?filter=-3

This should be included with version 2.0.0 when it will be out