My Pandas Dataframe named as data_five_minutes:
script_id
date_time
open
0
1
2019-01-11 09:35:00
25
1
1
2019-01-11 09:40:00
30
2
1
2019-01-11 09:45:00
48
Full ss:
Now I was trying to get only data which are having script_id as 1:
data = ps.sqldf(f"""SELECT d4.script_id,d4.date_time,d4.open,d4.high,d4.low,d4.close,d4.volume
FROM data_five_minutes d4 WHERE d4.script_id = {id}""")
this is just small condition but in original it has nested where clause so SQL query will be passed on original code as earlier i was passing the same query in database.
I am getting below error:
Traceback (most recent call last): File
"C:\Python\lib\site-packages\sqlalchemy\engine\base.py", line 1752, in
_e xecute_context
cursor, statement, parameters, context File "C:\Python\lib\site-packages\sqlalchemy\engine\default.py", line 714,
in do_executemany
cursor.executemany(statement, parameters) sqlite3.InterfaceError: Error binding parameter 2 - probably unsupported type.
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File
"D:\python_projects\auto\testing.py", line 70, in
FROM data_five_minutes d4 WHERE d4.script_id = 1""") File "C:\Python\lib\site-packages\pandasql\sqldf.py", line 156, in sqldf
return PandaSQL(db_uri)(query, env) File "C:\Python\lib\site-packages\pandasql\sqldf.py", line 58, in call
write_table(env[table_name], table_name, conn) File "C:\Python\lib\site-packages\pandasql\sqldf.py", line 121, in
write_table
index=not any(name is None for name in df.index.names)) # load index into d b if all levels are named File
"C:\Python\lib\site-packages\pandas\io\sql.py", line 518, in to_sql
method=method, File "C:\Python\lib\site-packages\pandas\io\sql.py", line 1320, in to_sql
table.insert(chunksize, method=method) File "C:\Python\lib\site-packages\pandas\io\sql.py", line 756, in insert
exec_insert(conn, keys, chunk_iter) File "C:\Python\lib\site-packages\pandas\io\sql.py", line 670, in
_execute_ins ert
conn.execute(self.table.insert(), data) File "C:\Python\lib\site-packages\sqlalchemy\engine\base.py", line 1263, in
ex ecute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS) File "C:\Python\lib\site-packages\sqlalchemy\sql\elements.py", line
324, in _e xecute_on_connection
self, multiparams, params, execution_options File "C:\Python\lib\site-packages\sqlalchemy\engine\base.py", line 1462, in
_e xecute_clauseelement
cache_hit=cache_hit, File "C:\Python\lib\site-packages\sqlalchemy\engine\base.py", line 1815, in
e xecute_context
e, statement, parameters, cursor, context File "C:\Python\lib\site-packages\sqlalchemy\engine\base.py", line 1996, in
h andle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from=e File "C:\Python\lib\site-packages\sqlalchemy\util\compat.py", line 207, in
rai se
raise exception File "C:\Python\lib\site-packages\sqlalchemy\engine\base.py", line 1752, in
_e xecute_context
cursor, statement, parameters, context File "C:\Python\lib\site-packages\sqlalchemy\engine\default.py", line 714,
in do_executemany
cursor.executemany(statement, parameters) sqlalchemy.exc.InterfaceError: (sqlite3.InterfaceError) Error binding
parameter 2 - probably unsupported type. [SQL: INSERT INTO
data_five_minutes (script_id, date_time, open, high, low, clos e,
volume) VALUES (?, ?, ?, ?, ?, ?, ?)] [parameters: ((1, '2021-05-25
10:30:00.000000', Decimal('1978.6'), Decimal('1985 '),
Decimal('1978.1'), Decimal('1985'), Decimal('323')), (1, '2021-05-25
10:35:0
0.000000', Decimal('1985'), Decimal('1986.05'), Decimal('1982.75'), Decimal('198
3.85'), Decimal('954')), and so on...] (Background on this error at: https://sqlalche.me/e/14/rvf5)
Process returned 1 (0x1) execution time : 2.672 s Press any key
to continue . . .
Related
I have a inquiry from my job to create 2 tables related one-to-one and insert some rows on the table. I have a .CSV with the data dictionary (column name and data type) from the table and I am wondering to know how to declare the tables columns automatically (declarative syntax) without write one by one column (there are 260 columns). Same thing for the insert, how to add rows to the multiple columns table without write column by column?
I have the data in a Data frame but I was not able to insert it using df.to_sql from pandas. Do you guys have any similar example?
The database used is MySQL. The table structure in the database is showed below:
enter image description here
Below is what I did.
I have created a function to define the table constraints.
def create_mysql_tables(dataframe, engine):
# print(dataframe.iloc)
if 'Active' in dataframe.columns:
start = time.time()
engine.execute('DROP TABLE IF EXISTS com_treb;')
engine.execute('DROP TABLE IF EXISTS cnd_treb;')
engine.execute('DROP TABLE IF EXISTS res_treb;')
engine.execute('DROP TABLE IF EXISTS mls_treb;')
dataframe.to_sql("mls_treb", if_exists='replace', con=engine,
dtype={'mls_number': VARCHAR(dataframe.index.get_level_values('mls_number').str.len().max())})
with engine.connect() as con:
con.execute('ALTER TABLE `mls_treb` ADD PRIMARY KEY (`mls_number`);')
end = time.time()
print(end - start)
First questions is, there is a better way to define a primary key using the SQLALCHEMY avoiding to write the query?
After I have created the table I have to insert some row if it doesn't exist. I was trying to do it by using the function below.
def crud_database(engine, mls_full_dataframe, res_full_dataframe):
for index, row in mls_full_dataframe.iterrows():
row_db_mls = pd.read_sql_query("SELECT * FROM mls_treb WHERE `mls_number` LIKE %(mlsnumber)s ", engine, params={'mlsnumber' : row.at['mls_number']})
if row_db_mls.empty:
row.to_sql("mls_treb", if_exists='append', con=engine, index_label='mls_number' )
if row.at['Class'][0] == 'RES':
row_res = res_full_dataframe.iloc[row.at['mls_number']]
else:
print(0)
but I am getting the error below when I am trying to insert the row from picture below:
image from row to be inserted with index 0
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1782, in _execute_context
self.dialect.do_executemany(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 729, in do_executemany
cursor.executemany(statement, parameters)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/mysql/connector/cursor.py", line 670, in executemany
return self.execute(stmt)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/mysql/connector/cursor.py", line 568, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/mysql/connector/connection.py", line 854, in cmd_query
result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/mysql/connector/connection.py", line 664, in _handle_result
raise errors.get_exception(packet)
mysql.connector.errors.ProgrammingError: 1054 (42S22): Unknown column '3' in 'field list'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/io/sql.py", line 1419, in to_sql
raise err
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/io/sql.py", line 1411, in to_sql
table.insert(chunksize, method=method)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/io/sql.py", line 845, in insert
exec_insert(conn, keys, chunk_iter)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/io/sql.py", line 762, in _execute_insert
conn.execute(self.table.insert(), data)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1289, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
return connection._execute_clauseelement(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1481, in _execute_clauseelement
ret = self._execute_context(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1845, in _execute_context
self._handle_dbapi_exception(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2026, in _handle_dbapi_exception
util.raise_(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1782, in _execute_context
self.dialect.do_executemany(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 729, in do_executemany
cursor.executemany(statement, parameters)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/mysql/connector/cursor.py", line 670, in executemany
return self.execute(stmt)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/mysql/connector/cursor.py", line 568, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/mysql/connector/connection.py", line 854, in cmd_query
result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/mysql/connector/connection.py", line 664, in _handle_result
raise errors.get_exception(packet)
sqlalchemy.exc.ProgrammingError: (mysql.connector.errors.ProgrammingError) 1054 (42S22): Unknown column '3' in 'field list'
[SQL: INSERT INTO mls_treb (mls_number, `3`) VALUES (%(mls_number)s, %(3)s)]
[parameters: ({'mls_number': 'mls_number', '3': 'N5404949'}, {'mls_number': 'active', '3': True}, {'mls_number': 'class_name', '3': 'RES'}, {'mls_number': 'active_date', '3': datetime.date(2022, 1, 16)})]
(Background on this error at: https://sqlalche.me/e/14/f405)
With Gord Thompsom help I could find the answer for problem. Prior to use the row to insert
row_to_insert = mls_full_dataframe.loc[mls_full_dataframe['mls_number'] == row.at['mls_number']]
row_to_insert.to_sql("mls_treb", if_exists='append', con=engine, index=False)
I was getting a series instead of row as dataframe. So, when I was trying to insert the row as serie, the number '3' was the index used as key and the compiler did not find the column name properly with that name "3". To fix it, I was retrieving the row_to_insert as dataframe and trying to insert it into database.
I have a Flask based backend application and when running the code locally, this error does not happen, but it does happen when deployed in my server in a kubernetes pod.
Interestingly enough, when you run the code for the first time, it fails instantly (when running the Future) but when you run it again the error still happens, but the code continues (and completes successfully). Pretty bizarre.
The error I am getting is:
2021-07-30 14:59:23,537 - mit_backend.logic.optimize - INFO - Inside first pass run code
2021-07-30 14:59:23,587 - mit_backend.logic.optimize - INFO - Optimising region ML10F
2021-07-30 14:59:23,588 - mit_backend.logic.optimize - INFO - Optimising region ML47F
2021-07-30 14:59:23,589 - mit_backend.logic.optimize - INFO - Optimising region ML40F
--- Logging error ---
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
psycopg2.OperationalError: SSL error: wrong version number
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib64/python3.6/concurrent/futures/process.py", line 175, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/code/mit_backend/logic/optimize.py", line 241, in optimize_first_pass_run
config = get_csv_config(user_identity)
File "/code/mit_backend/modules/v1/__init__.py", line 623, in get_csv_config
res = CSVConfig.query.filter(CSVConfig.user_id == user_identity).first()
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3429, in first
ret = list(self[0:1])
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3203, in __getitem__
return list(res)
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
distilled_params,
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL error: wrong version number
[SQL: SELECT csv_config.user_id AS csv_config_user_id, csv_config."csvHeaders" AS "csv_config_csvHeaders", csv_config."deliveryName" AS "csv_config_deliveryName", csv_config.id AS csv_config_id, csv_config.line1 AS csv_config_line1, csv_config.line2 AS csv_config_line2, csv_config.quantity AS csv_config_quantity, csv_config."routeNumber" AS "csv_config_routeNumber", csv_config.suburb AS csv_config_suburb, csv_config.weight AS csv_config_weight
FROM csv_config
WHERE csv_config.user_id = %(user_id_1)s
LIMIT %(param_1)s]
[parameters: {'user_id_1': 'sftp', 'param_1': 1}]
(Background on this error at: http://sqlalche.me/e/13/e3q8)
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/code/mit_backend/logic/optimize.py", line 300, in first_pass_run
future_df_result = future.result()
File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL error: wrong version number
[SQL: SELECT csv_config.user_id AS csv_config_user_id, csv_config."csvHeaders" AS "csv_config_csvHeaders", csv_config."deliveryName" AS "csv_config_deliveryName", csv_config.id AS csv_config_id, csv_config.line1 AS csv_config_line1, csv_config.line2 AS csv_config_line2, csv_config.quantity AS csv_config_quantity, csv_config."routeNumber" AS "csv_config_routeNumber", csv_config.suburb AS csv_config_suburb, csv_config.weight AS csv_config_weight
FROM csv_config
WHERE csv_config.user_id = %(user_id_1)s
LIMIT %(param_1)s]
[parameters: {'user_id_1': 'sftp', 'param_1': 1}]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib64/python3.6/logging/__init__.py", line 994, in emit
msg = self.format(record)
File "/usr/lib64/python3.6/logging/__init__.py", line 840, in format
return fmt.format(record)
File "/usr/lib64/python3.6/logging/__init__.py", line 577, in format
record.message = record.getMessage()
File "/usr/lib64/python3.6/logging/__init__.py", line 338, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib64/python3.6/socketserver.py", line 654, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib64/python3.6/socketserver.py", line 364, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib64/python3.6/socketserver.py", line 724, in __init__
self.handle()
File "/usr/local/lib/python3.6/site-packages/werkzeug/serving.py", line 345, in handle
BaseHTTPRequestHandler.handle(self)
File "/usr/lib64/python3.6/http/server.py", line 418, in handle
self.handle_one_request()
File "/usr/local/lib/python3.6/site-packages/werkzeug/serving.py", line 379, in handle_one_request
return self.run_wsgi()
File "/usr/local/lib/python3.6/site-packages/werkzeug/serving.py", line 323, in run_wsgi
execute(self.server.app)
File "/usr/local/lib/python3.6/site-packages/werkzeug/serving.py", line 312, in execute
application_iter = app(environ, start_response)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2464, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.6/site-packages/flask_restx/api.py", line 375, in wrapper
resp = resource(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask/views.py", line 89, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask_restx/resource.py", line 44, in dispatch_request
resp = meth(*args, **kwargs)
File "/code/mit_backend/modules/v1/controllers/manifest.py", line 161, in post
files, df, sftp_date_latest = do_optimize(addresses, user_identity)
File "/code/mit_backend/logic/optimize.py", line 38, in do_optimize
df = first_pass_run(df, user_identity)
File "/code/mit_backend/logic/optimize.py", line 307, in first_pass_run
LOG.error("Exception is ", e.__cause__)
Message: 'Exception is '
Arguments: (_RemoteTraceback('\n"""\nTraceback (most recent call last):\n File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context\n cursor, statement, parameters, context\n File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute\n cursor.execute(statement, parameters)\npsycopg2.OperationalError: SSL error: decryption failed or bad record mac\n\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/usr/lib64/python3.6/concurrent/futures/process.py", line 175, in _process_worker\n r = call_item.fn(*call_item.args, **call_item.kwargs)\n File "/code/mit_backend/logic/optimize.py", line 241, in optimize_first_pass_run\n config = get_csv_config(user_identity)\n File "/code/mit_backend/modules/v1/__init__.py", line 623, in get_csv_config\n res = CSVConfig.query.filter(CSVConfig.user_id == user_identity).first()\n File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3429, in first\n ret = list(self[0:1])\n File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3203, in __getitem__\n return list(res)\n File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__\n return self._execute_and_instances(context)\n File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances\n result = conn.execute(querycontext.statement, self._params)\n File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1011, in execute\n return meth(self, multiparams, params)\n File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection\n return connection._execute_clauseelement(self, multiparams, params)\n File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement\n distilled_params,\n File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context\n e, statement, parameters, cursor, context\n File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception\n sqlalchemy_exception, with_traceback=exc_info[2], from_=e\n File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_\n raise exception\n File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context\n cursor, statement, parameters, context\n File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute\n cursor.execute(statement, parameters)\nsqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL error: decryption failed or bad record mac\n\n[SQL: SELECT csv_config.user_id AS csv_config_user_id, csv_config."csvHeaders" AS "csv_config_csvHeaders", csv_config."deliveryName" AS "csv_config_deliveryName", csv_config.id AS csv_config_id, csv_config.line1 AS csv_config_line1, csv_config.line2 AS csv_config_line2, csv_config.quantity AS csv_config_quantity, csv_config."routeNumber" AS "csv_config_routeNumber", csv_config.suburb AS csv_config_suburb, csv_config.weight AS csv_config_weight \nFROM csv_config \nWHERE csv_config.user_id = %(user_id_1)s \n LIMIT %(param_1)s]\n[parameters: {\'user_id_1\': \'sftp\', \'param_1\': 1}]\n(Background on this error at: http://sqlalche.me/e/13/e3q8)\n"""',),)
The error happens inside a futures ProcessPoolExecutor:
def first_pass_run(df, user_identity):
LOG.info("Inside first pass run code")
first_json = True
df_final = None
df_grouped = df.groupby('Route Number')
auth_claim = get_authorization_claims_from_header()
with concurrent.futures.ProcessPoolExecutor(max_workers=8) as executor:
_futures = []
for region_name, _df in df_grouped:
_futures.append(
executor.submit(
optimize_first_pass_run, region_name, _df, user_identity, auth_claim
))
for future in concurrent.futures.as_completed(_futures):
try:
future_df_result = future.result()
if first_json:
df_final = future_df_result
first_json = False
else:
df_final = df_final.append(future_df_result, ignore_index=True)
except Exception as e:
LOG.error("Exception is ", e.__cause__)
I start my SQLAlchemy engine in a pessimistic fashion:
pg_db = SQLAlchemy(app)
engine = create_engine(app.config['SQLALCHEMY_DATABASE_URI'], pool_pre_ping=True)
The code is not running on WSGI or gunicorn, it's just as simple as:
if __name__ == '__main__':
LOG.info('running environment: %s', os.environ.get('ENV', 'production'))
app.config['DEBUG'] = os.environ.get('ENV') == 'development'
app.run(host='0.0.0.0', port=5001, debug=False, threaded=True)
Any clues why this might be happening? Thank you.
I haven't been able to find a proper fix for this, but I implemented the following things that although the exception is still being thrown, the code retries and ends up succeeding:
Removing the connection pool
engine = create_engine(app.config['SQLALCHEMY_DATABASE_URI'], poolclass=NullPool)
Event listeners
#event.listens_for(engine, "connect")
def connect(dbapi_connection, connection_record):
connection_record.info['pid'] = os.getpid()
#event.listens_for(engine, "checkout")
def checkout(dbapi_connection, connection_record, connection_proxy):
pid = os.getpid()
if connection_record.info['pid'] != pid:
connection_record.connection = connection_proxy.connection = None
raise exc.DisconnectionError(
"Connection record belongs to pid %s, "
"attempting to check out in pid %s" %
(connection_record.info['pid'], pid)
)
but, as per the documentation, the two definitions above relate to an optimistic way of handling the connection and using Pools
Reconnecting before initializing the Process Pool
engine.dispose()
with concurrent.futures.ProcessPoolExecutor(max_workers=8) as executor:
_futures = []
for region_name, _df in df_grouped:
_futures.append(
executor.submit(
optimize_first_pass_run, region_name, _df, user_identity, auth_claim, True
))
for future in concurrent.futures.as_completed(_futures):
future_df_result = future.result()
Adding retries when trying to insert objects in the database
def add_new_record(row):
attempts = 0
while attempts <= 5:
attempts = attempts + 1
try:
pg_db.session.add(row)
pg_db.session.commit()
return True
except (SQLAlchemyError, psycopg2.OperationalError, sqlalchemy.exc.OperationalError) as e:
if attempts < 5:
LOG.warn("Attempt failed. Trying rollback")
pg_db.session.rollback()
LOG.warn(e.__cause__)
LOG.warn(e.__str__())
time.sleep(5)
else:
LOG.error("Maximum number of retries reached. Raising an error")
raise e
return False
I am trying to push the excel file by using to_sql function and using data frame after transpose the dataframe...
and this is my code looks like
import pandas as pd
import os
import sqlalchemy
# MySQL Connection
MYSQL_USER = 'xxxxx'
MYSQL_PASSWORD = 'xxxxxxxx'
MYSQL_HOST_IP = '127.0.0.1'
MYSQL_PORT = 3306
MYSQL_DATABASE = 'xlsx_test_db'
# connect db
engine = sqlalchemy.create_engine('mysql+mysqlconnector://' + MYSQL_USER + ':' + MYSQL_PASSWORD + '#' + MYSQL_HOST_IP + ':' + str(
MYSQL_PORT) + '/' + MYSQL_DATABASE, echo=False)
engine.connect()
mydir = (os.getcwd()).replace('\\', '/') + '/'
raw_lte = pd.read_excel(r'' + mydir + 'MNM_Rotterdam_5_Daily_Details-20191216081027.xlsx', sheet_name='raw_4G')
dft = raw_lte.T
dft.columns = dft.iloc[0]
dft = dft.iloc[1:]
# reading and insert one file at a time
for file in os.listdir('.'):
# only process excels files
file_basename, extension = file.split('.')
if extension == 'xlsx':
dft.to_sql(file_basename.lower(), con=engine, if_exists='replace')
and this the error
Traceback
Traceback (most recent call last):
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1193, in _execute_context
context)
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\sqlalchemy\engine\default.py", line 507, in do_execute
cursor.execute(statement, parameters)
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\mysql\connector\cursor.py", line 551, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\mysql\connector\connection.py", line 490, in cmd_query
result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\mysql\connector\connection.py", line 395, in _handle_result
raise errors.get_exception(packet)
mysql.connector.errors.ProgrammingError: 1170 (42000): BLOB/TEXT column 'index' used in key specification without a key length
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:/Users/DELL/PycharmProjects/automateDB/swap.py", line 36, in <module>
dft.to_sql(file_basename.lower(), con=engine, if_exists='replace')
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\pandas\core\generic.py", line 2532, in to_sql
dtype=dtype, method=method)
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\pandas\io\sql.py", line 460, in to_sql
chunksize=chunksize, dtype=dtype, method=method)
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\pandas\io\sql.py", line 1173, in to_sql
table.create()
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\pandas\io\sql.py", line 585, in create
self._execute_create()
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\pandas\io\sql.py", line 569, in _execute_create
self.table.create()
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\sqlalchemy\sql\schema.py", line 778, in create
checkfirst=checkfirst)
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1940, in _run_visitor
conn._run_visitor(visitorcallable, element, **kwargs)
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1549, in _run_visitor
**kwargs).traverse_single(element)
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\sqlalchemy\sql\visitors.py", line 121, in traverse_single
return meth(obj, **kw)
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\sqlalchemy\sql\ddl.py", line 796, in visit_table
self.traverse_single(index)
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\sqlalchemy\sql\visitors.py", line 121, in traverse_single
return meth(obj, **kw)
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\sqlalchemy\sql\ddl.py", line 823, in visit_index
self.connection.execute(CreateIndex(index))
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\sqlalchemy\engine\base.py", line 948, in execute
return meth(self, multiparams, params)
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\sqlalchemy\sql\ddl.py", line 68, in _execute_on_connection
return connection._execute_ddl(self, multiparams, params)
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1009, in _execute_ddl
compiled
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1200, in _execute_context
context)
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1413, in _handle_dbapi_exception
exc_info
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\sqlalchemy\util\compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\sqlalchemy\util\compat.py", line 186, in reraise
raise value.with_traceback(tb)
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1193, in _execute_context
context)
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\sqlalchemy\engine\default.py", line 507, in do_execute
cursor.execute(statement, parameters)
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\mysql\connector\cursor.py", line 551, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\mysql\connector\connection.py", line 490, in cmd_query
result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
File "C:\Users\DELL\PycharmProjects\MyALLRefProf\venv\lib\site-packages\mysql\connector\connection.py", line 395, in _handle_result
raise errors.get_exception(packet)
sqlalchemy.exc.ProgrammingError: (mysql.connector.errors.ProgrammingError) 1170 (42000): BLOB/TEXT column 'index' used in key specification without a key length [SQL: 'CREATE INDEX `ix_mnm_rotterdam_5_daily_details-20191216081027_index` ON `mnm_rotterdam_5_daily_details-20191216081027` (`index`)'] (Background on this error at: http://sqlalche.me/e/f405)
I think there's a problem with the excel format, but I don't know to solve this issue
Note:
I tried to use this
raw_4G.to_sql(file_basename.lower(), con=engine, if_exists='replace')
insted of
dft.to_sql(file_basename.lower(), con=engine, if_exists='replace')
It works but it gives me error in the run time with this
Traceback (most recent call last):
File "C:/Users/DELL/PycharmProjects/automateDB/swap.py", line 34, in <module>
file_basename, extension = file.split('.')
ValueError: not enough values to unpack (expected 2, got 1)
I am trying to execute a query and return the results to Excel. The query takes in a string of years as input parameters. I am calling it in Python like this:
def flatten(l):
for el in l:
try:
yield from flatten(el)
except TypeError:
yield el
my_list = (previous_year_1,previous_year,current_year)
sql = 'select year,sum(sales)/case when sum(t_count)=0 then 1 else sum(t_count) as tx_sales from t_sales where year in ({1})'+ 'group by year' + 'order by year'
sql = sql.format ('?',','.join('?' * len(my_list)))
params = tuple(flatten(member_list))
ind_data = pd.read_sql(sql,engine,params)
The query itself, after fixing the end clause, works perfectly when run through SSMS. Just not through the Python code. The error I'm getting is:
Traceback (most recent call last):
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1139, in _execute_context
context)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\default.py", line 450, in do_execute
cursor.execute(statement, parameters)
pyodbc.Error: ('07002', '[07002] [Microsoft][SQL Server Native Client 11.0]COUNT field incorrect or syntax error (0) (SQLExecDirectW)')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "c:\pbp_proj\pbp_proj.py", line 61, in pull_metrics
ind_data = pd.read_sql_query(sql, engine, params)
File "C:\Anaconda3\lib\site-packages\pandas\io\sql.py", line 411, in read_sql_query
parse_dates=parse_dates, chunksize=chunksize)
File "C:\Anaconda3\lib\site-packages\pandas\io\sql.py", line 1128, in read_query
result = self.execute(*args)
File "C:\Anaconda3\lib\site-packages\pandas\io\sql.py", line 1022, in execute
return self.engine.execute(*args, **kwargs)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1989, in execute
return connection.execute(statement, *multiparams, **params)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 906, in execute
return self._execute_text(object, multiparams, params)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1054, in _execute_text
statement, parameters
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1146, in _execute_context
context)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1341, in _handle_dbapi_exception
exc_info
File "C:\Anaconda3\lib\site-packages\sqlalchemy\util\compat.py", line 188, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=exc_value)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\util\compat.py", line 181, in reraise
raise value.with_traceback(tb)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1139, in _execute_context
context)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\default.py", line 450, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.DBAPIError: (pyodbc.Error) ('07002', '[07002] [Microsoft][SQL Server Native Client 11.0]COUNT field incorrect or syntax error (0) (SQLExecDirectW)')
How can I resolve this error?
As #MYGz has already mentioned there is a missing space before order by.
Beside that there is a missing space before group by and the most important one - your CASE ... statement should be "closed" with END.
That said try the following SQL:
sql = 'select year,sum(sales)/(case when sum(t_count)=0 then 1 else sum(t_count) end)' \
+' as tx_sales from t_sales where year in ({1})'+' group by year order by year'
You can use your SQL pattern directly using .format() - there is no need to overwrite it:
params = tuple(flatten(member_list))
ind_data = pd.read_sql(sql.format('?',','.join('?' * len(params))), engine, params)
You have missed a space in your sql string between year and order by.
Try this:
sql = 'select year,sum(sales)/case when sum(t_count)=0 then 1 else sum(t_count) as tx_sales from t_sales where year in ({1}) '+ 'group by year ' + 'order by year '
Resolved this. A bit of hack, but works. I first changed it to using pyodbc instead of sqlalchemy.
so my query string became:
sql = 'select year,sum(sales)/case when sum(t_count)=0 then 1 else sum(t_count) end as tx_sales from t_sales where year in (?,?,?) '+ ' group by year' + ' order by year'
ind_data = pd.read_sql(sql, conn, params=member_list)
summary = ind_data.transpose()
I then had to add another AND clause with another parameter. For this I created:
cur_params = (member_list)
cur_params.append(var_premium)
then passsed cur_params to ind_data.
ind_data = pd.read_sql(sql, conn, params=cur_params)
both sets return data correctly now.
Thank you all for reading my post and for all the suggestions.
I have a database of names and prices that gets updated daily when I run it through a batch of code and update a second database all of the names until it gets to 'aapl' at which point it throws a 1064 error which looks like this
-----------------------------------
Traceback (most recent call last):
File "testrun.PY", line 45, in <module>
t.Push.find_all(conn, cursor)
File "c:\tradetools.py", line 198, in find_all
Push.find_streak(conn, cursor, name)
File "c:\tradetools.py", line 189, in find_strea
k
.format(c, name))
File "C:\AppData\Local\Programs\Python\Python35\lib\site-packages\p
ymysql\cursors.py", line 166, in execute
result = self._query(query)
File "C:\AppData\Local\Programs\Python\Python35\lib\site-packages\p
ymysql\cursors.py", line 322, in _query
conn.query(q)
File "C:\AppData\Local\Programs\Python\Python35\lib\site-packages\p
ymysql\connections.py", line 837, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "C:\AppData\Local\Programs\Python\Python35\lib\site-packages\p
ymysql\connections.py", line 1021, in _read_query_result
result.read()
File "C:\AppData\Local\Programs\Python\Python35\lib\site-packages\p
ymysql\connections.py", line 1304, in read
first_packet = self.connection._read_packet()
File "C:\AppData\Local\Programs\Python\Python35\lib\site-packages\p
ymysql\connections.py", line 983, in _read_packet
packet.check_error()
File "C:\AppData\Local\Programs\Python\Python35\lib\site-packages\p
ymysql\connections.py", line 395, in check_error
err.raise_mysql_exception(self._data)
File "C:\AppData\Local\Programs\Python\Python35\lib\site-packages\p
ymysql\err.py", line 102, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.ProgrammingError: (1064, "42000You have an error in your SQL syntax;
check the manual that corresponds to your MySQL server version for the right sy
ntax to use near 'AAPL''' at line 1")
c:\>
The code that its running through looks like this, why does the update add more commas to the name appl?
def find_streak(conn, cursor, name):
print(name)
cursor.execute("SELECT * FROM `trade_data`.`import_data`"
" WHERE name =%s;", name)
logs = cursor.fetchall()
cursor.execute("INSERT IGNORE INTO `trade_data`.`analysis`(`name`) "
"VALUES (%s) ON DUPLICATE KEY UPDATE "
"ndays=0;", name)
conn.commit()
logs = [list(x) for x in logs]
logs.sort()
....
cursor.execute ("UPDATE `trade_data`.`analysis` "
"SET `ndays` = {0} WHERE name='{1}'"
.format(c, name))
conn.commit()
its pulling from a table that looks like this
date|name|price|
and entering into a table that is
date|name|result