I'm trying to create a trigger in my database by executing a raw SQL string in alembic's upgrade function. However I get a sqlalchemy.exc.ProgrammingError when running upgrade:
Running this in my migration script:
def upgrade():
connection = op.get_bind()
connection.execute('DELIMITER #')
Produces this traceback:
app/migrate.py:10: ExtDeprecationWarning: Importing flask.ext.script is deprecated, use flask_script instead.
from flask.ext.script import Manager
app/migrate.py:11: ExtDeprecationWarning: Importing flask.ext.migrate is deprecated, use flask_migrate instead.
from flask.ext.migrate import Migrate, MigrateCommand
/home/my_project/app/database/models.py:2: ExtDeprecationWarning: Importing flask.ext.whooshalchemy is deprecated, use flask_whooshalchemy instead.
import flask.ext.whooshalchemy
/home/my_project/venv/local/lib/python2.7/site-packages/flask_whooshalchemy.py:18: ExtDeprecationWarning: Importing flask.ext.sqlalchemy is deprecated, use flask_sqlalchemy instead.
import flask.ext.sqlalchemy as flask_sqlalchemy
/home/my_project/venv/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py:1518: Warning: '##tx_isolation' is deprecated and will be removed in a future release. Please use '##transaction_isolation' instead
cursor.execute('SELECT ##tx_isolation')
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade 60f73629ea28 -> 48dd7624f294, add trigger for adding to history table
Traceback (most recent call last):
File "app/migrate.py", line 25, in <module>
manager.run()
File "/home/my_project/venv/local/lib/python2.7/site-packages/flask_script/__init__.py", line 412, in run
result = self.handle(sys.argv[0], sys.argv[1:])
File "/home/my_project/venv/local/lib/python2.7/site-packages/flask_script/__init__.py", line 383, in handle
res = handle(*args, **config)
File "/home/my_project/venv/local/lib/python2.7/site-packages/flask_script/commands.py", line 216, in __call__
return self.run(*args, **kwargs)
File "/home/my_project/venv/local/lib/python2.7/site-packages/flask_migrate/__init__.py", line 244, in upgrade
command.upgrade(config, revision, sql=sql, tag=tag)
File "/home/my_project/venv/local/lib/python2.7/site-packages/alembic/command.py", line 254, in upgrade
script.run_env()
File "/home/my_project/venv/local/lib/python2.7/site-packages/alembic/script/base.py", line 416, in run_env
util.load_python_file(self.dir, 'env.py')
File "/home/my_project/venv/local/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
module = load_module_py(module_id, path)
File "/home/my_project/venv/local/lib/python2.7/site-packages/alembic/util/compat.py", line 75, in load_module_py
mod = imp.load_source(module_id, path, fp)
File "migrations/env.py", line 87, in <module>
run_migrations_online()
File "migrations/env.py", line 80, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/home/my_project/venv/local/lib/python2.7/site-packages/alembic/runtime/environment.py", line 817, in run_migrations
self.get_context().run_migrations(**kw)
File "/home/my_project/venv/local/lib/python2.7/site-packages/alembic/runtime/migration.py", line 323, in run_migrations
step.migration_fn(**kw)
File "/home/my_project/migrations/versions/48dd7624f294_add_trigger_for_adding_to_history_table.py", line 22, in upgrade
connection.execute('DELIMITER #')
File "/home/my_project/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 939, in execute
return self._execute_text(object, multiparams, params)
File "/home/my_project/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1097, in _execute_text
statement, parameters
File "/home/my_project/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
context)
File "/home/my_project/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1393, in _handle_dbapi_exception
exc_info
File "/home/my_project/venv/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/my_project/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
context)
File "/home/my_project/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 469, in do_execute
cursor.execute(statement, parameters)
File "/home/my_project/venv/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
self.errorhandler(self, exc, value)
File "/home/my_project/venv/local/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
sqlalchemy.exc.ProgrammingError: (_mysql_exceptions.ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'DELIMITER #' at line 1") [SQL: 'DELIMITER #']
Can DELIMETER not be used in connection.execute for some reason or am I doing something wrong?
Relevant versions:
pip:
- alembic==0.9.1
- Flask-SQLAlchemy==2.1
- MySQL-python==1.2.5
- SQLAlchemy==1.1.4
- SQLAlchemy-Utils==0.32.12
MySQL: 5.7.21
OS: ubuntu 16
Related
After updating superset to 2.0.0 I cannot access sqllab nor view databases (Oracle DB).
I'm running superset on openshift with custom image built from apache/superset:2.0.0
Installed dependencies:
cx_Oracle certifi flask-oidc==1.4.0 flask_openid gevent psycopg2 psycopg2-binary==2.9.1 redis==3.5.3 itsdangerous==2.0.1
I tried running superset db upgrade with following result:
$ superset db upgrade
Loaded your LOCAL configuration at [/app/pythonpath/superset_config.py]
logging was configured successfully
2022-11-21 14:21:41,835:INFO:superset.utils.logging_configurator:logging was configured successfully
2022-11-21 14:21:41,841:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
and superset init throws error:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.UndefinedColumn: column dbs.allow_multi_schema_metadata_fetch does not exist
LINE 1: ..., dbs.force_ctas_schema AS dbs_force_ctas_schema, dbs.allow_...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/superset", line 33, in \<module\>
sys.exit(load_entry_point('apache-superset', 'console_scripts', 'superset')())
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
return self.main(\*args, \*\*kwargs)
File "/usr/local/lib/python3.8/site-packages/flask/cli.py", line 601, in main
return super().main(\*args, \*\*kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return \_process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, \*\*ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return \__callback(\*args, \*\*kwargs)
File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), \*args, \*\*kwargs)
File "/usr/local/lib/python3.8/site-packages/flask/cli.py", line 445, in decorator
return \__ctx.invoke(f, \*args, \*\*kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return \__callback(\*args, \*\*kwargs)
File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), \*args, \*\*kwargs)
File "/usr/local/lib/python3.8/site-packages/flask/cli.py", line 445, in decorator
return \__ctx.invoke(f, \*args, \*\*kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return _callback(\*args, \*\*kwargs)
File "/app/superset/cli/main.py", line 62, in init
security_manager.sync_role_definitions()
File "/app/superset/security/manager.py", line 731, in sync_role_definitions
self.create_missing_perms()
File "/app/superset/security/manager.py", line 679, in create_missing_perms
merge_pv("datasource_access", datasource.get_perm())
File "/app/superset/connectors/sqla/models.py", line 826, in get_perm
if self.database is None:
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 294, in __get__
return self.impl.get(instance_state(instance), dict_)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 730, in get
value = self.callable(state, passive)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/strategies.py", line 759, in \_load_for_state
return self.\_emit_lazyload(
File "\<string\>", line 1, in \<lambda\>
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/strategies.py", line 847, in \_emit_lazyload
q(session)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/ext/baked.py", line 615, in \_load_on_pk_identity
result = list(bq.for_session(self.session).params(\*\*params))
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/ext/baked.py", line 444, in __iter__
return q.\_execute_and_instances(context)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3560, in \_execute_and_instances
result = conn.execute(querycontext.statement, self.\_params)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in \_execute_on_connection
return connection.\_execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in \_execute_clauseelement
ret = self.\_execute_context(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in \_execute_context
self._handle_dbapi_exception(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in handle_dbapi_exception
util.raise_(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise
raise exception
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column dbs.allow_multi_schema_metadata_fetch does not exist
LINE 1: ..., dbs.force_ctas_schema AS dbs_force_ctas_schema, dbs.allow_...
^
\[SQL: SELECT dbs.uuid AS dbs_uuid, dbs.created_on AS dbs_created_on, dbs.changed_on AS dbs_changed_on, dbs.id AS dbs_id, dbs.verbose_name AS dbs_verbose_name, dbs.database_name AS dbs_database_name, dbs.sqlalchemy_uri AS dbs_sqlalchemy_uri, dbs.password AS dbs_password, dbs.cache_timeout AS dbs_cache_timeout, dbs.select_as_create_table_as AS dbs_select_as_create_table_as, dbs.expose_in_sqllab AS dbs_expose_in_sqllab, dbs.configuration_method AS dbs_configuration_method, dbs.allow_run_async AS dbs_allow_run_async, dbs.allow_file_upload AS dbs_allow_file_upload, dbs.allow_ctas AS dbs_allow_ctas, dbs.allow_cvas AS dbs_allow_cvas, dbs.allow_dml AS dbs_allow_dml, dbs.force_ctas_schema AS dbs_force_ctas_schema, dbs.allow_multi_schema_metadata_fetch AS dbs_allow_multi_schema_metadata_fetch, dbs.extra AS dbs_extra, dbs.encrypted_extra AS dbs_encrypted_extra, dbs.impersonate_user AS dbs_impersonate_user, dbs.server_cert AS dbs_server_cert, dbs.is_managed_externally AS dbs_is_managed_externally, dbs.external_url AS dbs_external_url, dbs.created_by_fk AS dbs_created_by_fk, dbs.changed_by_fk AS dbs_changed_by_fk
FROM dbs
WHERE dbs.id = %(param_1)s\]
\[parameters: {'param_1': 1}\]
(Background on this error at: http://sqlalche.me/e/13/f405)
Anyone found solution for this error?
Tried:
superset db upgrade
superset init
updating dependencies
restarting superset
Expect:
successfully migrate to 2.0.0 version
When I run this basic code to access my AWS redshift data warehouse, I get this error:
Traceback (most recent call last):
File "<ipython-input-40-dcebeb0708d9>", line 25, in <module>
engine = sqlalchemy.create_engine(URL)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\__init__.py", line 479, in create_engine
return strategy.create(*args, **kwargs)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\strategies.py", line 56, in create
plugins = u._instantiate_plugins(kwargs)
TypeError: _instantiate_plugins() missing 1 required positional argument: 'kwargs'
This is the code I'm running. I originally planned on using psycopg2 to access the database but psycopg2 returns the sql query as a list with funky formatting; I'd have to do too much work to extract the data into a dataframe. I'm hoping sqlalchemy produces and output that's easier to work with.
import sqlalchemy
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
import pandas as pd
from sqlalchemy.engine.url import URL
config = dict(
drivername='driver',
username= username_u,
password= password_u,
host='rs-data.prod-lde.bsp.gsa.gov',
port='5439',
database='edw',
query={'encoding': 'utf-8'}
)
url = URL(**config)
engine = sqlalchemy.create_engine(url) # Having "URL" instead of "url" caused the original problem
Thanks in advance.
>>>>>Update<<<<<:
I fixed my silly mistake and the new code is this:
import sqlalchemy
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
import pandas as pd
from sqlalchemy.engine.url import URL
config = dict(
drivername='postgresql',
username= username_u,
password= password_u,
host='rs-data.prod-lde.bsp.gsa.gov',
port='5439',
database='edw',
query={'encoding': 'utf-8'}
)
url = URL(**config)
engine = sqlalchemy.create_engine(url)
data_frame = pd.read_sql('select date_signed from edw.fpds.fpds_atom limit 5;', engine)
Now I'm getting this error:
ProgrammingError: (psycopg2.ProgrammingError) invalid dsn: invalid connection option "encoding"
(Background on this error at: http://sqlalche.me/e/f405)
I think the problem lies in the config dict values. I first had it as drivername='driver' and that caused this error:
Traceback (most recent call last):
File "<ipython-input-48-dcb4b19c1bd3>", line 25, in <module>
engine = sqlalchemy.create_engine(url)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\__init__.py", line 479, in create_engine
return strategy.create(*args, **kwargs)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\strategies.py", line 61, in create
entrypoint = u._get_entrypoint()
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\url.py", line 172, in _get_entrypoint
cls = registry.load(name)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\util\langhelpers.py", line 240, in load
"Can't load plugin: %s:%s" % (self.group, name)
NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:driver
so I changed it to drivername='postgresql' and that produced this error:
Traceback (most recent call last):
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 2276, in _wrap_pool_connect
return fn()
File "C:\Anaconda3\lib\site-packages\sqlalchemy\pool\base.py", line 363, in connect
return _ConnectionFairy._checkout(self)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\pool\base.py", line 773, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\pool\base.py", line 492, in checkout
rec = pool._do_get()
File "C:\Anaconda3\lib\site-packages\sqlalchemy\pool\impl.py", line 139, in _do_get
self._dec_overflow()
File "C:\Anaconda3\lib\site-packages\sqlalchemy\util\langhelpers.py", line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\util\compat.py", line 153, in reraise
raise value
File "C:\Anaconda3\lib\site-packages\sqlalchemy\pool\impl.py", line 136, in _do_get
return self._create_connection()
File "C:\Anaconda3\lib\site-packages\sqlalchemy\pool\base.py", line 308, in _create_connection
return _ConnectionRecord(self)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\pool\base.py", line 437, in __init__
self.__connect(first_connect_check=True)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\pool\base.py", line 652, in __connect
connection = pool._invoke_creator(self)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\strategies.py", line 114, in connect
return dialect.connect(*cargs, **cparams)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\default.py", line 489, in connect
return self.dbapi.connect(*cargs, **cparams)
File "C:\Anaconda3\lib\site-packages\psycopg2\__init__.py", line 126, in connect
dsn = _ext.make_dsn(dsn, **kwargs)
File "C:\Anaconda3\lib\site-packages\psycopg2\extensions.py", line 175, in make_dsn
parse_dsn(dsn)
ProgrammingError: invalid dsn: invalid connection option "encoding"
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<ipython-input-53-3b57714b0ad8>", line 28, in <module>
data_frame = pd.read_sql('select date_signed from edw.fpds.fpds_atom limit 5;', engine)
File "C:\Anaconda3\lib\site-packages\pandas\io\sql.py", line 438, in read_sql
chunksize=chunksize,
File "C:\Anaconda3\lib\site-packages\pandas\io\sql.py", line 1218, in read_query
result = self.execute(*args)
File "C:\Anaconda3\lib\site-packages\pandas\io\sql.py", line 1087, in execute
return self.connectable.execute(*args, **kwargs)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 2181, in execute
connection = self._contextual_connect(close_with_result=True)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 2242, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 2280, in _wrap_pool_connect
e, dialect, self
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1547, in _handle_dbapi_exception_noconnection
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\util\compat.py", line 398, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\util\compat.py", line 152, in reraise
raise value.with_traceback(tb)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 2276, in _wrap_pool_connect
return fn()
File "C:\Anaconda3\lib\site-packages\sqlalchemy\pool\base.py", line 363, in connect
return _ConnectionFairy._checkout(self)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\pool\base.py", line 773, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\pool\base.py", line 492, in checkout
rec = pool._do_get()
File "C:\Anaconda3\lib\site-packages\sqlalchemy\pool\impl.py", line 139, in _do_get
self._dec_overflow()
File "C:\Anaconda3\lib\site-packages\sqlalchemy\util\langhelpers.py", line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\util\compat.py", line 153, in reraise
raise value
File "C:\Anaconda3\lib\site-packages\sqlalchemy\pool\impl.py", line 136, in _do_get
return self._create_connection()
File "C:\Anaconda3\lib\site-packages\sqlalchemy\pool\base.py", line 308, in _create_connection
return _ConnectionRecord(self)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\pool\base.py", line 437, in __init__
self.__connect(first_connect_check=True)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\pool\base.py", line 652, in __connect
connection = pool._invoke_creator(self)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\strategies.py", line 114, in connect
return dialect.connect(*cargs, **cparams)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\default.py", line 489, in connect
return self.dbapi.connect(*cargs, **cparams)
File "C:\Anaconda3\lib\site-packages\psycopg2\__init__.py", line 126, in connect
dsn = _ext.make_dsn(dsn, **kwargs)
File "C:\Anaconda3\lib\site-packages\psycopg2\extensions.py", line 175, in make_dsn
parse_dsn(dsn)
ProgrammingError: (psycopg2.ProgrammingError) invalid dsn: invalid connection option "encoding"
(Background on this error at: http://sqlalche.me/e/f405)
The key question I guess is what should I be using for the drivername parameter?
It was so hard to put that right title. Ok, here it goes. I was following this tutorial to install Apache Airflow on my Mac (Mojave version) -
https://towardsdatascience.com/getting-started-with-apache-airflow-df1aa77d7b1b
Right at the first step after performing the pip install airflow task, when I run the airflow version command I am getting the following error and then the airflow version appears -
ERROR - Failed on pre-execution callback using Traceback (most recent call last):
File
"/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py",
line 1244, in _execute_context
cursor, statement, parameters, context File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/default.py",
line 552, in do_execute
cursor.execute(statement, parameters) sqlite3.OperationalError: no such table: log
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File
"/Users/karthikv/anaconda3/lib/python3.7/site-packages/airflow/utils/cli_action_loggers.py",
line 68, in on_pre_execution
cb(**kwargs) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/airflow/utils/cli_action_loggers.py",
line 99, in default_action_log
session.add(log) File "/Users/karthikv/anaconda3/lib/python3.7/contextlib.py", line 119, in
exit
next(self.gen) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/airflow/utils/db.py",
line 45, in create_session
session.commit() File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/session.py",
line 1026, in commit
self.transaction.commit() File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/session.py",
line 493, in commit
self._prepare_impl() File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/session.py",
line 472, in _prepare_impl
self.session.flush() File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/session.py",
line 2451, in flush
self._flush(objects) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/session.py",
line 2589, in _flush
transaction.rollback(_capture_exception=True) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py",
line 68, in exit
compat.reraise(exc_type, exc_value, exc_tb) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/compat.py",
line 129, in reraise
raise value File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/session.py",
line 2549, in _flush
flush_context.execute() File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py",
line 422, in execute
rec.execute(self) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py",
line 589, in execute
uow, File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py",
line 245, in save_obj
insert, File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py",
line 1120, in _emit_insert_statements
statement, params File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py",
line 988, in execute
return meth(self, multiparams, params) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/sql/elements.py",
line 287, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params) File
"/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py",
line 1107, in _execute_clauseelement
distilled_params, File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py",
line 1248, in _execute_context
e, statement, parameters, cursor, context File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py",
line 1466, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/compat.py",
line 383, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/compat.py",
line 128, in reraise
raise value.with_traceback(tb) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py",
line 1244, in _execute_context
cursor, statement, parameters, context File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/default.py",
line 552, in do_execute
cursor.execute(statement, parameters) sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such
table: log [SQL: INSERT INTO log (dttm, dag_id, task_id, event,
execution_date, owner, extra) VALUES (?, ?, ?, ?, ?, ?, ?)]
[parameters: ('2019-08-12 20:50:24.960006', None, None, 'cli_version',
None, 'karthikv', '{"host_name": "192-168-1-6.tpgi.com.au",
"full_command": "[\'/Users/karthikv/anaconda3/bin/airflow\',
\'version\']"}')]
(Background on this error at: http://sqlalche.me/e/e3q8)
Can someone help me what this error means and how to solve it? I understand from the instructions that by default SQLLite db gets installed and a single DAG restrictions would be in place before we get into setting up backend database say PostgreSQL.
I tried to uninstall using pip uninstall airflow to perform clean installation again. I get the following error -
WARNING: Skipping airflow as it is not installed.
Kindly help me in solving the issue (or) pointing me to resources where I can do further reading.
You need to perform initialization after installation:
$ export AIRFLOW_HOME=some/dir
$ airflow db init # or `airflow initdb` for the legacy 1.X
If AIRFLOW_HOME is unset, ~/airflow/ will be created and used. This is where the config and logs will be stored; if you want to reset the configuration, remove the dir stored in AIRFLOW_HOME and rerun airflow db init.
Now other commands should work, e.g.
$ airflow db check
[2021-02-06 23:35:21,577] {db.py:756} INFO - Connection successful.
Source: Initializing Airflow Database section from airflow docs. Kudos to Anas Tiour who notified about the command line interface change in 2.0.0 in this comment!
when I migrate my database, i see that error.
i did init. (python manager.py db init)
and. migrate (python manager.py db migrate)
i use google app engine, flask, mac, python
what should i do?
python manager.py db migrate
/Library/Python/2.7/site-packages/flask_sqlalchemy/__init__.py:839: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future. Set it to True or False to suppress this warning.
'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and '
Traceback (most recent call last):
File "manager.py", line 3, in <module>
manager.run()
File "/Library/Python/2.7/site-packages/flask_script/__init__.py", line 412, in run
result = self.handle(sys.argv[0], sys.argv[1:])
File "/Library/Python/2.7/site-packages/flask_script/__init__.py", line 383, in handle
res = handle(*args, **config)
File "/Library/Python/2.7/site-packages/flask_script/commands.py", line 216, in __call__
return self.run(*args, **kwargs)
File "/Library/Python/2.7/site-packages/flask_migrate/__init__.py", line 182, in migrate
version_path=version_path, rev_id=rev_id)
File "/Library/Python/2.7/site-packages/alembic/command.py", line 176, in revision
script_directory.run_env()
File "/Library/Python/2.7/site-packages/alembic/script/base.py", line 421, in run_env
util.load_python_file(self.dir, 'env.py')
File "/Library/Python/2.7/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
module = load_module_py(module_id, path)
File "/Library/Python/2.7/site-packages/alembic/util/compat.py", line 75, in load_module_py
mod = imp.load_source(module_id, path, fp)
File "migrations/env.py", line 87, in <module>
run_migrations_online()
File "migrations/env.py", line 70, in run_migrations_online
poolclass=pool.NullPool)
File "/Library/Python/2.7/site-packages/sqlalchemy/engine/__init__.py", line 428, in engine_from_config
return create_engine(url, **options)
File "/Library/Python/2.7/site-packages/sqlalchemy/engine/__init__.py", line 387, in create_engine
return strategy.create(*args, **kwargs)
File "/Library/Python/2.7/site-packages/sqlalchemy/engine/strategies.py", line 80, in create
dbapi = dialect_cls.dbapi(**dbapi_args)
File "/Library/Python/2.7/site-packages/sqlalchemy/dialects/mysql/gaerdbms.py", line 77, in dbapi
from google.storage.speckle.python.api import rdbms_googleapi
ImportError: cannot import name rdbms_googleapi
Trying to run Python3.2, SQLAlchemy0.8 and MySQL5.2 on Ubuntu using Eclispse but I keep getting the error below. Am using pymysql (pymysql3 actually) engine.
module monitor
from sqlalchemy import create_engine, MetaData
from sqlalchemy.ext.declarative import declarative_base
Engine = create_engine('mysql+pymysql://user:mypass#localhost/mydb')
Base = declarative_base(Engine)
Metadata = MetaData(bind=Engine)
from sqlalchemy.orm import sessionmaker
from sqlalchemy import Table, Column, Integer
Session = sessionmaker(bind=Engine)
session = Session()
class Student(Base):
__table__ = Table('student_name', Metadata,
Column('id', Integer, primary_key=True),
autoload=True)
With that, when I run the module it throws the error as indicated below. What am I doing wrong?
Traceback (most recent call last):
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/pool.py", line 757, in _do_get
return self._pool.get(wait, self._timeout)
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/util/queue.py", line 166, in get
raise Empty
sqlalchemy.util.queue.Empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/lukik/workspace/upark/src/monitor.py", line 12, in <module>
class Parking(Base):
File "/home/lukik/workspace/upark/src/monitor.py", line 15, in Parking
autoload=True)
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/schema.py", line 333, in __new__
table._init(name, metadata, *args, **kw)
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/schema.py", line 397, in _init
self._autoload(metadata, autoload_with, include_columns)
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/schema.py", line 425, in _autoload
self, include_columns, exclude_columns
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/engine/base.py", line 1604, in run_callable
with self.contextual_connect() as conn:
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/engine/base.py", line 1671, in contextual_connect
self.pool.connect(),
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/pool.py", line 272, in connect
return _ConnectionFairy(self).checkout()
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/pool.py", line 425, in __init__
rec = self._connection_record = pool._do_get()
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/pool.py", line 777, in _do_get
con = self._create_connection()
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/pool.py", line 225, in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/pool.py", line 322, in __init__
exec_once(self.connection, self)
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/event.py", line 381, in exec_once
self(*args, **kw)
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/event.py", line 398, in __call__
fn(*args, **kw)
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/engine/strategies.py", line 168, in first_connect
dialect.initialize(c)
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/dialects/mysql/base.py", line 2052, in initialize
default.DefaultDialect.initialize(self, connection)
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/engine/default.py", line 172, in initialize
self._get_default_schema_name(connection)
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/dialects/mysql/base.py", line 2019, in _get_default_schema_name
return connection.execute('SELECT DATABASE()').scalar()
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/engine/base.py", line 664, in execute
params)
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/engine/base.py", line 808, in _execute_text
statement, parameters
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/engine/base.py", line 871, in _execute_context
context)
File "/usr/local/lib/python3.2/dist-packages/SQLAlchemy-0.8.0b2-py3.2.egg/sqlalchemy/engine/default.py", line 322, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python3.2/dist-packages/pymysql/cursors.py", line 105, in execute
query = query % escaped_args
TypeError: unsupported operand type(s) for %: 'bytes' and 'tuple'
mysql-connector-python and oursql work fine for me under py3k.
How to install?
$ pip install mysql-connector-python
Usage
Engine = create_engine('mysql+mysqlconnector://<USERNAME>:<PASSWD>#<HOSTNAME>:<PORT>/<DBNAME>')
Stay on py2k or a different DB driver for the time being. This is a known SQLAlchemy bug: 2663.