Develop Reference

SQLServer: query runs on console but not on SQLAlchemy

I'm trying to run a very expensive SQL Server query via Python + SQLAlchemy. It runs just fine on sql server console but it errors out when called via sqlalchemy.
Test run looks like this:
Run query on SQL Server console.
Wait about 15 minutes for it to finish.
Query runs just fine and returns ~50,000 rows.
When running the same query using Python + SQLAlchemy, it looks like this:
Run query.
Wait a long time.
Code errors out and throws a misleading error stating that the query did not return any rows.
I am positive that this error message cannot possibly be right, because I have tested the same query on console and it runs just fine and returns A LOT of rows. Does anyone know what is really happening here?
Query looks like this:
USE DB_NAME;
DROP TABLE IF EXISTS #TB_1;
CREATE TABLE #TB_1 (FIELD_1 BIGINT, FIELD_2 BIGINT);
INSERT INTO #TB_1 VALUES (1, 5), (2, 6), (3, 7);
--------------------------------------------------
DROP TABLE IF EXISTS #TB_2;
SELECT * INTO #TB_2 FROM (
SELECT DISTINCT FIELD_1, FIELD_2
FROM dbo.PRIMARY_TABLE_1 (NOLOCK)
WHERE FIELD_1 IN (SELECT * FROM #TB_1)
) AS TB_2;
--------------------------------------------------
SELECT FIELD_1, FIELD_2 FROM #TB_1
UNION ALL
SELECT FIELD_1, FIELD_2 FROM #TB_2
Code looks like this:
from sqlalchemy.engine import create_engine
engine = create_engine(SQLServer_URI)
with engine.connect() as connection:
connection.execute(huge_query).fetchall()
Here's the Stack Trace:
Traceback (most recent call last):
...
File "path-to-project/src/etl/ETL.py", line 48, in extract
raw_data = connection.execute(query).fetchall()
File "path-to-project\venv\lib\site-packages\sqlalchemy\engine\result.py", line 984, in fetchall
return self._allrows()
File "path-to-project\venv\lib\site-packages\sqlalchemy\engine\result.py", line 398, in _allrows
make_row = self._row_getter
File "path-to-project\venv\lib\site-packages\sqlalchemy\util\langhelpers.py", line 1160, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
File "path-to-project\venv\lib\site-packages\sqlalchemy\engine\result.py", line 319, in _row_getter
keymap = metadata._keymap
File "path-to-project\venv\lib\site-packages\sqlalchemy\engine\cursor.py", line 1197, in _keymap
self._we_dont_return_rows()
File "path-to-project\venv\lib\site-packages\sqlalchemy\engine\cursor.py", line 1178, in _we_dont_return_rows
util.raise_(
File "path-to-project\venv\lib\site-packages\sqlalchemy\util\compat.py", line 211, in raise_
raise exception
sqlalchemy.exc.ResourceClosedError: This result object does not return rows. It has been closed automatically.

Yep adding "SET NOCOUNT ON;" as per Gord Thompson's comment, to the front of my query solved it for me. I got the same error running a stored procedure containing multiple queries. So this will do the trick:
data = pd.read_sql("SET NOCOUNT ON; Your SQL Query.....
This answer https://stackoverflow.com/a/55597613/11692538 helped solve it and explain it for me. Basically sqlalchemy (or pyodbc) reads any messages sent in the execution of a query as the result, hence why you get the error "name": "ResourceClosedError", "message": "This result object does not return rows. It has been closed automatically." So setting NOCOUNT prevents sql server sending back count messages during the execution of the query(s).

Dremio ODBC with Python

Getting below error while running this code in Python, If anyone could advise me on this that would be appreciated. Thanks
dataframe = pandas.read_sql(sql,cnxn)
DatabaseError: Execution failed on sql 'SELECT * FROM train_data': ('HY000', "[HY000] [Dremio][Connector] (1040) Dremio failed to execute the query: SELECT * FROM train_data\n[30038]Query execution error. Details:[ \nVALIDATION ERROR: Table 'train_data' not found\n\nSQL Query SELECT * FROM train_data\nstartLine 1\nstartColumn 15\nendLine 1\nendColumn 24\n\n[Error Id: 24c7de0e-6e23-44c6-8cb6-b0a110bbd2fd on user:31010]\n\n (org.apache.calcite.runtime.CalciteContextException) From line 1, column 15 to line 1, column 24: ...[see log] (1040) (SQLExecDirectW)")

You only need to provide your Space name, just before your table name.
for example:
SELECT * FROM
SpaceName.train_data
This is a query to fetch data from Dremio Space, Dremio source cannot be used for data ingestion. Dremio Source only be used to establish a connection between database and Dremio.

this is being solved, it says that table does not exist, should give a valid table, in dremio it can be inside a specific space

Python SQLAlchemy: psycopg2.ProgrammingError relation already exists?

I have repeatable tried to create a table MYTABLENAME with SQLAlchemy in Python. I deleted all tables through my SQL client Dbeaver but I am getting an error that the table exists such that
Traceback (most recent call last):
File "/home/hhh/anaconda3/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
context)
File "/home/hhh/anaconda3/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
cursor.execute(statement, parameters)
psycopg2.ProgrammingError: relation "ix_MYTABLENAME_index" already exists
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) relation "ix_MYTABLENAME_index" already exists
[SQL: 'CREATE INDEX "ix_MYTABLENAME_index" ON "MYTABLENAME" (index)']
I succeed in the creation of tables and their insertions with an unique name but the second time I am getting the error despite deleting the tables in Dbeaver.
Small example
from datetime import date
from sqlalchemy import create_engine
import numpy as np
import pandas as pd
def storePandasDF2PSQL(myDF_):
#Store results as Pandas Dataframe to PostgreSQL database.
#
#Example
#df=pd.DataFrame(np.random.randn(8, 4), columns=['A','B','C','D'])
#dbName= date.today().strftime("%Y%m%d")+"_TABLE"
#engine = create_engine('postgresql://hhh:yourPassword#localhost:1234/hhh')
#df.to_sql(dbName, engine)
df = myDF_
dbName = date.today().strftime("%Y%m%d")+"_TABLE"
engine = create_engine('postgresql://hhh:yourPassword#localhost:1234/hhh')
# ERROR: NameError: name 'table' is not defined
#table.declarative_base.metadata.drop_all(engine) #Drop all tables
#TODO: This step is causing errors because the SQLAlchemy thinks the
#TODO: table still exists even though deleted
df.to_sql(dbName, engine)
What is the proper way to clean up the backend such as some hanging index in order to recreate the table with fresh data? In other words, how to solve the error?

The issue might be from sqlalchemy side which believes that there is an index as message of deletion of tables was not notified to the sqlalchemy. There is a sqlalchemy way of deleting the tables
table.declarative_base.metadata.drop_all(engine)
This should keep Sqlalchemy informed about the deletions.

This answer does not address the reusing of the same table names and hence not about cleaning up the SQLAlchemy metadata.
Instead of reusing the table names, add the execution time like this to the end of the tableName
import time
dbName = date.today().strftime("%Y%m%d")+"_TABLE_"+str(time.time())
dbTableName = dbName
so your SQL developmnet environment, such as SQL client locking up the connection or specific tables, does not matter that much. Closing Dbeaver can help while running the Python with SQLAlchemy.

sqlalchemy syntax for using mysql constants

So i am trying to call the mysql function TIMEDATEDIFF with the SECOND constant as the first parameter like so
query = session.query(func.TIME(Log.IncidentDetection).label('detection'), func.TIMESTAMPDIFF(SECOND,Log.IncidentDetection, Log.IncidentClear).label("duration")).all()
print(query)
I have tried it as a string and I get a mysql/mariadb error:
query = session.query(func.TIME(Log.IncidentDetection).label('detection'), func.TIMESTAMPDIFF("SECOND",Log.IncidentDetection, Log.IncidentClear).label("duration")).all()
print(query)
Gives me this
sqlalchemy.exc.ProgrammingError: (mysql.connector.errors.ProgrammingError) 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''SECOND', log.`IncidentDetection`, log.`IncidentClear`) AS duration
FROM log' at line 1 [SQL: 'SELECT TIME(log.`IncidentDetection`) AS detection, TIMESTAMPDIFF(%(TIMESTAMPDIFF_1)s, log.`IncidentDetection`, log.`IncidentClear`) AS duration \nFROM log'] [parameters: {'TIMESTAMPDIFF_1': 'SECOND'}]
I am sure it is something simple, some sort of escape sequence or import that I am missing. I have looked through the sqlalchemy documentation to no avail.

To get sqlalchemy to parse the string exactly into the query I used the _literal_as_text() function
Working solution
from sqlalchemy.sql.expression import func, _literal_as_text
# ...
query = session.query(func.TIME(Log.IncidentDetection).label('detection'), func.TIMESTAMPDIFF(_literal_as_text("SECOND"),Log.IncidentDetection, Log.IncidentClear).label("duration")).all()
print(query)

Python using mysql connector list databases LIKE and then use those databases in order and run query

I'm trying to write a script using pythong and the mysql-connector library. The script should connect to the mysql server do a "SHOW DATABASES LIKE 'pdns_%' and then using the results returned by the query use each database and then run another query while using that database.
Here is the code
import datetime
import mysql.connector
from mysql.connector import errorcode
cnx = mysql.connector.connect (user='user', password='thepassword',
host='mysql.server.com',buffered=True)
cursor = cnx.cursor()
query = ("show databases like 'pdns_%'")
cursor.execute(query)
databases = query
for (databases) in cursor:
cursor.execute("USE %s",(databases[0],))
hitcounts = ("SELECT Monthname(hitdatetime) AS 'Month', Count(hitdatetime) AS 'Hits' WHERE hitdatetime >= Date_add(Last_day(Date_sub(Curdate(), interval 4 month)), interval 1 day) AND hitdatetime < Date_add(Last_day(Date_sub(Curdate(), interval 1 month)), interval 1 day) GROUP BY Monthname(hitdatetime) ORDER BY Month(hitdatetime)")
cursor.execute(hitcounts)
print(hitcounts)
cursor.close()
cnx.close()
When running the script it stops with the following error'd output
Traceback (most recent call last):
File "./mysql-test.py", line 18, in <module>
cursor.execute("USE %s",(databases[0],))
File "/usr/lib/python2.6/site-packages/mysql/connector/cursor.py", line 491, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "/usr/lib/python2.6/site-packages/mysql/connector/connection.py", line 635, in cmd_query
statement))
File "/usr/lib/python2.6/site-packages/mysql/connector/connection.py", line 553, in _handle_result
raise errors.get_exception(packet)
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''pdns_382'' at line 1
Based on the error I'm guessing there is an issue with how its doing the datbase name from the first query. Any pointers in the correct direction would be very helpful as I'm very much a beginner. Thank you very much.

Alas, the two-args form of execute does not support "meta" parameters, such as names of databases, tables, or fields (roughly, think of identifiers you wouldn't quote if writing the query out manually). So, the failing statement:
cursor.execute("USE %s",(databases[0],))
needs to be re-coded as:
cursor.execute("USE %s" % (databases[0],))
i.e, the single arg form of execute, with a string interpolation. Fortunately, this particular case does not expose you to SQL injection risks, since you're only interpolating DB names coming right from the DB engine.