I am stuck with a problem where when I try to delete a chart or a user or dashboard from Apache Superset throw the web UI (installed using helm on Kubernetes), the browser gets a 500 error and the logs indicated the following:
superset-prod-6785cd75df-zqp5j superset] [SQL: SELECT report_schedule.created_on AS report_schedule_created_on, report_schedule.changed_on AS report_schedule_changed_on, report_schedule.id AS report_schedule_id, report_schedule.type AS report_schedule_type, report_schedule.name AS report_schedule_name, report_schedule.description AS report_schedule_description, report_schedule.context_markdown AS report_schedule_context_markdown, report_schedule.active AS report_schedule_active, report_schedule.crontab AS report_schedule_crontab, report_schedule.sql AS report_schedule_sql, report_schedule.chart_id AS report_schedule_chart_id, report_schedule.dashboard_id AS report_schedule_dashboard_id, report_schedule.database_id AS report_schedule_database_id, report_schedule.last_eval_dttm AS report_schedule_last_eval_dttm, report_schedule.last_state AS report_schedule_last_state, report_schedule.last_value AS report_schedule_last_value, report_schedule.last_value_row_json AS report_schedule_last_value_row_json, report_schedule.validator_type AS report_schedule_validator_type, report_schedule.validator_config_json AS report_schedule_validator_config_json, report_schedule.log_retention AS report_schedule_log_retention, report_schedule.grace_period AS report_schedule_grace_period, report_schedule.working_timeout AS report_schedule_working_timeout, report_schedule.created_by_fk AS report_schedule_created_by_fk, report_schedule.changed_by_fk AS report_schedule_changed_by_fk
FROM report_schedule
WHERE report_schedule.chart_id = %(chart_id_1)s]
[parameters: {'chart_id_1': '51'}]
(Background on this error at: http://sqlalche.me/e/13/f405)
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.UndefinedColumn: column report_schedule.working_timeout does not exist
LINE 1: ...ule.grace_period AS report_schedule_grace_period, report_sch...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/api/__init__.py", line 84, in wraps
return f(self, *args, **kwargs)
File "/app/superset/views/base_api.py", line 80, in wraps
duration, response = time_function(f, self, *args, **kwargs)
File "/app/superset/utils/core.py", line 1484, in time_function
response = func(*args, **kwargs)
File "/app/superset/utils/log.py", line 125, in wrapper
value = f(*args, **kwargs)
File "/app/superset/charts/api.py", line 383, in delete
DeleteChartCommand(g.user, pk).run()
File "/app/superset/charts/commands/delete.py", line 49, in run
self.validate()
File "/app/superset/charts/commands/delete.py", line 64, in validate
reports = ReportScheduleDAO.find_by_chart_id(self._model_id)
File "/app/superset/reports/dao.py", line 45, in find_by_chart_id
.filter(ReportSchedule.chart_id == chart_id)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3373, in all
return list(self)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
distilled_params,
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column report_schedule.working_timeout does not exist
LINE 1: ...ule.grace_period AS report_schedule_grace_period, report_sch...
^
It seems like there is a mismatch between the backing database for Superset and the python code.
This issue seems to have been resolved after upgrading the superset docker image version to apache/superset:latest around 17 Jan 2021. Open to more in-depth answers, but felt it important to state this issue is no longer an issue for me.
Related
After updating superset to 2.0.0 I cannot access sqllab nor view databases (Oracle DB).
I'm running superset on openshift with custom image built from apache/superset:2.0.0
Installed dependencies:
cx_Oracle certifi flask-oidc==1.4.0 flask_openid gevent psycopg2 psycopg2-binary==2.9.1 redis==3.5.3 itsdangerous==2.0.1
I tried running superset db upgrade with following result:
$ superset db upgrade
Loaded your LOCAL configuration at [/app/pythonpath/superset_config.py]
logging was configured successfully
2022-11-21 14:21:41,835:INFO:superset.utils.logging_configurator:logging was configured successfully
2022-11-21 14:21:41,841:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
and superset init throws error:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.UndefinedColumn: column dbs.allow_multi_schema_metadata_fetch does not exist
LINE 1: ..., dbs.force_ctas_schema AS dbs_force_ctas_schema, dbs.allow_...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/superset", line 33, in \<module\>
sys.exit(load_entry_point('apache-superset', 'console_scripts', 'superset')())
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
return self.main(\*args, \*\*kwargs)
File "/usr/local/lib/python3.8/site-packages/flask/cli.py", line 601, in main
return super().main(\*args, \*\*kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return \_process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, \*\*ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return \__callback(\*args, \*\*kwargs)
File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), \*args, \*\*kwargs)
File "/usr/local/lib/python3.8/site-packages/flask/cli.py", line 445, in decorator
return \__ctx.invoke(f, \*args, \*\*kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return \__callback(\*args, \*\*kwargs)
File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), \*args, \*\*kwargs)
File "/usr/local/lib/python3.8/site-packages/flask/cli.py", line 445, in decorator
return \__ctx.invoke(f, \*args, \*\*kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return _callback(\*args, \*\*kwargs)
File "/app/superset/cli/main.py", line 62, in init
security_manager.sync_role_definitions()
File "/app/superset/security/manager.py", line 731, in sync_role_definitions
self.create_missing_perms()
File "/app/superset/security/manager.py", line 679, in create_missing_perms
merge_pv("datasource_access", datasource.get_perm())
File "/app/superset/connectors/sqla/models.py", line 826, in get_perm
if self.database is None:
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 294, in __get__
return self.impl.get(instance_state(instance), dict_)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 730, in get
value = self.callable(state, passive)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/strategies.py", line 759, in \_load_for_state
return self.\_emit_lazyload(
File "\<string\>", line 1, in \<lambda\>
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/strategies.py", line 847, in \_emit_lazyload
q(session)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/ext/baked.py", line 615, in \_load_on_pk_identity
result = list(bq.for_session(self.session).params(\*\*params))
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/ext/baked.py", line 444, in __iter__
return q.\_execute_and_instances(context)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3560, in \_execute_and_instances
result = conn.execute(querycontext.statement, self.\_params)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in \_execute_on_connection
return connection.\_execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in \_execute_clauseelement
ret = self.\_execute_context(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in \_execute_context
self._handle_dbapi_exception(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in handle_dbapi_exception
util.raise_(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise
raise exception
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column dbs.allow_multi_schema_metadata_fetch does not exist
LINE 1: ..., dbs.force_ctas_schema AS dbs_force_ctas_schema, dbs.allow_...
^
\[SQL: SELECT dbs.uuid AS dbs_uuid, dbs.created_on AS dbs_created_on, dbs.changed_on AS dbs_changed_on, dbs.id AS dbs_id, dbs.verbose_name AS dbs_verbose_name, dbs.database_name AS dbs_database_name, dbs.sqlalchemy_uri AS dbs_sqlalchemy_uri, dbs.password AS dbs_password, dbs.cache_timeout AS dbs_cache_timeout, dbs.select_as_create_table_as AS dbs_select_as_create_table_as, dbs.expose_in_sqllab AS dbs_expose_in_sqllab, dbs.configuration_method AS dbs_configuration_method, dbs.allow_run_async AS dbs_allow_run_async, dbs.allow_file_upload AS dbs_allow_file_upload, dbs.allow_ctas AS dbs_allow_ctas, dbs.allow_cvas AS dbs_allow_cvas, dbs.allow_dml AS dbs_allow_dml, dbs.force_ctas_schema AS dbs_force_ctas_schema, dbs.allow_multi_schema_metadata_fetch AS dbs_allow_multi_schema_metadata_fetch, dbs.extra AS dbs_extra, dbs.encrypted_extra AS dbs_encrypted_extra, dbs.impersonate_user AS dbs_impersonate_user, dbs.server_cert AS dbs_server_cert, dbs.is_managed_externally AS dbs_is_managed_externally, dbs.external_url AS dbs_external_url, dbs.created_by_fk AS dbs_created_by_fk, dbs.changed_by_fk AS dbs_changed_by_fk
FROM dbs
WHERE dbs.id = %(param_1)s\]
\[parameters: {'param_1': 1}\]
(Background on this error at: http://sqlalche.me/e/13/f405)
Anyone found solution for this error?
Tried:
superset db upgrade
superset init
updating dependencies
restarting superset
Expect:
successfully migrate to 2.0.0 version
I'm facing issue with my flask app that uses psycopg2 for postgres. Sometimes my API returns 500 server error in which it said in the log file:
[SQL: SELECT gallery.id AS gallery_id, gallery.caption AS gallery_caption, gallery.image AS gallery_image, gallery.created_date AS gallery_created_date, gallery.updated_date AS gallery_updated_date
FROM gallery ORDER BY gallery.id]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
[UID:1549][94744] Child process with pid: 98159 was killed by signal: 15, core dump: 0
[ERROR] [UID:1549][97456] wsgiAppHandler pApp->start_response() return NULL.
/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask_sqlalchemy/__init__.py:873: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future. Set it to True or False to suppress this warning.
'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and '
[2021-12-27 11:22:11,886] ERROR in app: Exception on /gallery/find-all [GET]
Traceback (most recent call last):
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1803, in _execute_context
cursor, statement, parameters, context
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
psycopg2.OperationalError: lost synchronization with server: got message type "G", length 1935229752
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask/app.py", line 1518, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask_cors/extension.py", line 165, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask_cors/extension.py", line 165, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/home/myapp/api/myapplication/auth.py", line 32, in decorated
return f(auth_data, *args, **kwargs)
File "/home/myapp/api/myapplication/gallery.py", line 71, in fetch_image
image_data = Gallery.query.filter().order_by(Gallery.id).all()
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2759, in all
return self._iter().all()
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2897, in _iter
execution_options={"_sa_orm_load_options": self.load_options},
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1692, in execute
result = conn._execute_20(statement, params or {}, execution_options)
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1614, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 326, in _execute_on_connection
self, multiparams, params, execution_options
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1491, in _execute_clauseelement
cache_hit=cache_hit,
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
e, statement, parameters, cursor, context
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2027, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1803, in _execute_context
cursor, statement, parameters, context
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) lost synchronization with server: got message type "G", length 1935229752
This API just fetch all the data from the table and returns them to the client. The table only have max 12 rows and contains base-64 string of image data. However, this also happens at other API that fetch a single data from other table and returns only text data-type.
These APIs works well in my local development env but went error in production (CPanel shared hosting)
Currently developing for python 3.9.6 with sqlalchemy version 1.4.23 and whenever I use sqlalchemy to connect to a database, I get an error with this stack trace:
Traceback (most recent call last):
File "/mnt/c/Importer/common/db.py", line 106, in db_cur
with engine.connect() as con:
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3166, in connect
return self._connection_cls(self, close_with_result=close_with_result)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
else engine.raw_connection()
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3245, in raw_connection
return self._wrap_pool_connect(self.pool.connect, _connection)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3212, in _wrap_pool_connect
return fn()
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 307, in connect
return _ConnectionFairy._checkout(self)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 767, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 425, in checkout
rec = pool._do_get()
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 146, in _do_get
self._dec_overflow()
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
return self._create_connection()
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 253, in _create_connection
return _ConnectionRecord(self)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 368, in __init__
self.__connect()
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 622, in __connect
pool.dispatch.connect.for_modify(
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/event/attr.py", line 329, in _exec_w_sync_on_first_run
self(*args, **kw)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/event/attr.py", line 343, in __call__
fn(*args, **kw)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 1691, in go
return once_fn(*arg, **kw)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 674, in first_connect
dialect.initialize(c)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/dialects/mysql/base.py", line 2961, in initialize
default.DefaultDialect.initialize(self, connection)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 407, in initialize
self.default_isolation_level = self.get_default_isolation_level(
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 473, in get_default_isolation_level
return self.get_isolation_level(dbapi_conn)
File "/home/jimmy/.local/share/virtualenvs/Importer-0-6a4YAh/lib/python3.9/site-packages/sqlalchemy/dialects/mysql/base.py", line 2720, in get_isolation_level
val = row[0]
KeyError: 0
I can fix the issue by manually changing the file from
val = row[0]
to
val = row['##transaction_isolation']
but I don't want to have to manually change dependency files after installing them. I have tried manually setting the transaction isolation when creating the engine, but that doesn't prevent get_isolation_level() from being called so the error persists. Is sqlalchemy not compatible with python 3.9.6? Does this have anything to do with the mysql server version?
Problem was with the DictCursor being used in the connection. Changing _fetch_type for SELECT ##transaction_isolation to be a tuple instead of a dictionary solved the issue.
I'm trying to catch an error when a key is missing in a dict and so fails to create a model. However although the error raised is DoesNotExist (as per the traceback) the below fails to catch it for some reason and I still get the error:
(aside: how do I see in a debugger what error has been raised before it hits the traceback?)
try:
fuel_type, created = FuelType.objects.get_or_create(name=tariff['serviceType'])
if created:
self.log_database_create(fuel_type)
except FuelType.DoesNotExist:
fuel_type, created = FuelType.objects.get_or_create(name=service_type)
if created:
self.log_database_create(fuel_type)
Error:
Traceback (most recent call last):
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/db/models/query.py", line 465, in get_or_create
return self.get(**lookup), False
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/db/models/query.py", line 387, in get
self.model._meta.object_name
etariff.models.DoesNotExist: FuelType matching query does not exist.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
psycopg2.IntegrityError: null value in column "name" violates not-null constraint
DETAIL: Failing row contains (5, null).
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1531, in <module>
globals = debugger.run(setup['file'], None, None, is_module)
File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 938, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/Applications/PyCharm.app/Contents/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/Users/Yunti/OneDrive/switcher5/manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
utility.execute()
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/core/management/__init__.py", line 345, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/core/management/base.py", line 348, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/core/management/base.py", line 399, in execute
output = self.handle(*args, **options)
File "/Users/Yunti/OneDrive/switcher5/etariff/management/commands/til_scraper.py", line 35, in handle
scraper.full_scrape(energy_usage, service_type, current_service_type, house_no, postcode, e7)
File "/Users/Yunti/OneDrive/switcher5/etariff/management/commands/til_scraper.py", line 347, in full_scrape
self.write_tariffs(tariff_data_list, region, service_type)
File "/Users/Yunti/OneDrive/switcher5/etariff/management/commands/til_scraper.py", line 460, in write_tariffs
tariff_instance = self.write_tariff(tariff, region, service_type)
File "/Users/Yunti/OneDrive/switcher5/etariff/management/commands/til_scraper.py", line 480, in write_tariff
fuel_type, created = FuelType.objects.get_or_create(name=tariff['serviceType'])
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/db/models/manager.py", line 122, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/db/models/query.py", line 467, in get_or_create
return self._create_object_from_params(lookup, params)
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/db/models/query.py", line 507, in _create_object_from_params
six.reraise(*exc_info)
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/utils/six.py", line 686, in reraise
raise value
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/db/models/query.py", line 499, in _create_object_from_params
obj = self.create(**params)
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/db/models/query.py", line 401, in create
obj.save(force_insert=True, using=self.db)
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/db/models/base.py", line 708, in save
force_update=force_update, update_fields=update_fields)
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/db/models/base.py", line 736, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/db/models/base.py", line 820, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/db/models/base.py", line 859, in _do_insert
using=using, raw=raw)
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/db/models/manager.py", line 122, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/db/models/query.py", line 1039, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 1060, in execute_sql
cursor.execute(sql, params)
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/db/backends/utils.py", line 79, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/db/utils.py", line 95, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/Users/Yunti/.virtualenvs/switcher5/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: null value in column "name" violates not-null constraint
DETAIL: Failing row contains (5, null).
My app started throwing this error last night. Users are complaining the site is slow as well. Still on Python 2.5. As far as I can tell the request itself is not unusual.
Any ideas?
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 703, in __call__
handler.post(*groups)
File "/base/data/home/apps/s~jupitersfolly/4.357517623694016771/controller/frequest.py", line 57, in post
request_types[request_type]()
File "/base/data/home/apps/s~jupitersfolly/4.357517623694016771/controller/frequest.py", line 78, in order
event, report, messages = Game.update(game_number, user_id, order)
File "/base/data/home/apps/s~jupitersfolly/4.357517623694016771/model/game.py", line 355, in update
event, report, game, universe, messages, updated, ended = db.run_in_transaction(Game.transactional_update, key_id, user_id, order)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 2433, in RunInTransaction
return RunInTransactionOptions(None, function, *args, **kwargs)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 2571, in RunInTransactionOptions
ok, result = _DoOneTry(new_connection, function, args, kwargs)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 2593, in _DoOneTry
result = function(*args, **kwargs)
File "/base/data/home/apps/s~jupitersfolly/4.357517623694016771/model/game.py", line 348, in transactional_update
game.put()
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 1074, in put
return datastore.Put(self._entity, **kwargs)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 579, in Put
return PutAsync(entities, **kwargs).get_result()
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 604, in get_result
return self.__get_result_hook(self)
File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1577, in __put_hook
self.check_rpc_success(rpc)
File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1212, in check_rpc_success
rpc.check_success()
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 570, in check_success
self.__rpc.CheckSuccess()
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 133, in CheckSuccess
raise self.exception
ArgumentError: An error occurred parsing (locally or remotely) the arguments to datastore_v3.Put()