Apache Airflow with CRDB

General
#1

Hello team,
We would to use CRDB with Apache Airflow workflow engine. Currently, PostgresSQL DB works seamlessly with Airflow but we prefer CRDB instead. We see the following error in our logs:
psycopg2.errors.FeatureNotSupported: unimplemented: ALTER COLUMN TYPE from int to float is prohibited until v21.1

Error log:
INFO [alembic.runtime.migration] Context impl CockroachDBImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade → e3a246e0dc1, current schema
INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 → 1507a7289a2f, create is_encrypted
affinity <class ‘sqlalchemy.sql.sqltypes.INTEGER’>
INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f → 13eb55f81627, maintain history for compatibility with earlier migrations
INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 → 338e90f54d61, More logging into task_instance
INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 → 52d714495f0, job_id indices
INFO [alembic.runtime.migration] Running upgrade 52d714495f0 → 502898887f84, Adding extra to Log
INFO [alembic.runtime.migration] Running upgrade 502898887f84 → 1b38cef5b76e, add dagrun
INFO [alembic.runtime.migration] Running upgrade 1b38cef5b76e → 2e541a1dcfed, task_duration
Traceback (most recent call last):
File “/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py”, line 1277, in _execute_context
cursor, statement, parameters, context
File “/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/default.py”, line 593, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.FeatureNotSupported: unimplemented: ALTER COLUMN TYPE from int to float is prohibited until v21.1
HINT: You have attempted to use a feature that is not yet implemented.
See: https://go.crdb.dev/issue-v/54844/v20.2

CockroachDB Version:
Build Tag: v20.2.2
Build Time: 2020/11/25 14:45:44
Distribution: CCL
Platform: linux amd64 (x86_64-unknown-linux-gnu)
Go Version: go1.13.14
C Compiler: gcc 6.3.0
Build Commit ID: 92d94952060fd24400906dc06e392d7a7419b392
Build Type: release

Airflow Version:
apache-airflow==1.10.14

Does CRDB work with Airflow? If yes, can you please share the documentation/presentation with me? If no, do you have a plan to make CRDB work with Airflow? The error logs says v.21 will support ALTER column.

Thank you in advance!

#2

CockroachDB does not have full support yet for Airflow. You can see some open issues here and here. There are no immediate plans to add support but your feedback helps support prioritization.

The error you are seeing is fixed in our next release v21.1, see issue. You can try using Airflow with the v21.1 alpha release

#4

@vy-ton Yes, I upgraded CRDB to alpha v21 and enabled enable_experimental_alter_column_type_general = true flag, I don’t see the ALTER issue anymore in the error log. However, the current issue is that Airflow fails to create Sequences and assign them to PK in tables to auto generate PK values on record insertions. Therefore, all the insertion are failing. I manually created sequences and assigned them to Airflow tables, it started working now, still testing and collect error logs if any error show up.