Develop Reference

Pyodbc insert of large number fails int to big to convert when inserting into Sql Server using Oracle cursor

I am trying to do an executemany with an insert into a SQL Server using an Oracle cursor like this:
sqlservercursor.executemany("INSERT INTO tablename (col1,col2...) VALUES (?,?…)",oraclecursor)
This fails with error:OverflowError: int too big to convert
I have diagnosed it to the id columns which are large numbers(NUMBER(25)).
I can reproduce this using:
sqlservercursor.execute('INSERT INTO tablename (Id) VALUES (?)',(90100111000002885322904,))
however this works:
sqlservercursor.execute('INSERT INTO tablename (Id) VALUES (90100111000002885322904)')
Is there a way fix this or must I loop through the cursor and insert manually? This would be slow compared to pyodbc's fast execute as well as having to deal with characters...

I was able to reproduce your issue and work around it using fast_executemany:
cnxn = pyodbc.connect("DSN=mssqlLocal64")
crsr = cnxn.cursor()
crsr.execute("CREATE TABLE ##tablename (Id decimal(25,0))")
data = [(90100111000002885322904,)] # list of tuple(s)
sql = "INSERT INTO ##tablename (Id) VALUES (?)"
crsr.fast_executemany = True
crsr.executemany(sql, data)

"ProgrammingError: No results. Previous SQL was not a query." while trying to truncate a table

I'm trying to delete all the entries from a table but are not able to do it.
Does not matter if it is TRUNCATE, or DELETE keyword. The same error occurs
import pyodbc
conn = pyodbc.connect(
r'Driver={SQL Server};'
r'Server=' + ip + '\SQLEXPRESS;'
r'Database=...;'
r'UID=...;'
r'PWD=...;', timeout=5)
cursor = conn.cursor()
data = cursor.execute("TRUNCATE TABLE table_name")
pyodbc.ProgrammingError: No results. Previous SQL was not a query.
Setting autocommit to True does not work. Parametrizing it also does not work. The connection is right because SELECT clause works well and returns the right value. With truncating and deleting it does not work at all. The DDBB is still intact.
When excecuting from the pycharm's Python Console i get the folowwing error whenever i try to access the data object (f.e. print(data.fetchval()):
Traceback (most recent call last):
File "", line 1, in
pyodbc.ProgrammingError: No results. Previous SQL was not a query.
I've read before i might have to do with how the DDBB table is indexed and its private key, but i'm not able to explain it.

I was hoping on getting the number of rows affected.
When we execute a single SQL statement via Cursor.execute, the server can return one of three things:
zero or more rows of data in a result set (for a SELECT statement), or
an integer row count (for DML statements like UPDATE, DELETE, etc.), or
an error.
We retrieve information from a result set via the pyodbc methods .fetchall(), .fetchone(), .fetchval(), etc.. We retrieve row counts using the cursor's rowcount attribute.
crsr = cnxn.cursor()
crsr.execute("DROP TABLE IF EXISTS so64124053")
crsr.execute("CREATE TABLE so64124053 (id int primary key, txt varchar(10))")
crsr.execute("INSERT INTO so64124053 (id, txt) VALUES (1, 'foo')")
print(crsr.rowcount) # 1
print(crsr.execute("SELECT COUNT(*) AS n FROM so64124053").fetchval()) # 1
crsr.execute("INSERT INTO so64124053 (id, txt) VALUES (2, 'bar')")
print(crsr.rowcount) # 1
print(crsr.execute("SELECT COUNT(*) AS n FROM so64124053").fetchval()) # 2
Note that TRUNCATE is a special case because it doesn't bother counting the rows it removes from the table; it just returns a row count of -1 …
crsr.execute("TRUNCATE TABLE so64124053")
print(crsr.rowcount) # -1
… however the rows are indeed removed
print(crsr.execute("SELECT COUNT(*) AS n FROM so64124053").fetchval()) # 0

Improve performance SQL query in Python Cx_Oracle

I actually use Cx_Oracle library in Python to work with my database Oracle.
import cx_Oracle as Cx
# Parameters for server connexion
dsn_tns = Cx.makedsn(_ip, _port, service_name=_service_name)
# Connexion with Oracle Database
db = Cx.connect(_user, _password, dsn_tns)
# Obtain a cursor for make SQL query
cursor = db.cursor()
One of my query write in an INSERT of a Python dataframe into my Oracle target table among some conditions.
query = INSERT INTO ORA_TABLE(ID1, ID2)
SELECT :1, :2
FROM DUAL
WHERE (:1 != 'NF' AND :1 NOT IN (SELECT ID1 FROM ORA_TABLE))
OR (:1 = 'NF' AND :2 NOT IN (SELECT ID2 FROM ORA_TABLE))
The goal of this query is to write only rows who respect conditions into the WHERE.
Actually ,this query works well when my Oracle target table have few rows. But, if my target Oracle table have more than 100 000 rows, it's very slow because I read through all the table in WHERE condition.
Is there a way to improve performance of this query with join or something else ?
End of code :
# SQL query incoming
cursor.prepare(query)
# Launch query with Python dataset
cursor.executemany(None, _py_table.values.tolist())
# Commit changes into Oracle database
db.commit()
# Close the cursor
cursor.close()
# Close the server connexion
db.close()

Here is a possible solution that could help: The sql that you have has an OR condition and only one part of this condition will be true for a given value. So I would divide it in two parts by checking the following in the code and constructing two inserts instead of one and at any point of time, only one would execute:
IF :1 != 'NF' then use the following insert:
INSERT INTO ORA_TABLE (ID1, ID2)
SELECT :1, :2
FROM DUAL
WHERE (:1 NOT IN (SELECT ID1
FROM ORA_TABLE));
and IF :1 = 'NF' then use the following insert:
INSERT INTO ORA_TABLE (ID1, ID2)
SELECT :1, :2
FROM DUAL
WHERE (:2 NOT IN (SELECT ID2
FROM ORA_TABLE));
So you check in code what is the value of :1 and depending on that use the two simplified insert. Please check if this is functionally the same as original query and verify if it improves the response time.

Assuming Pandas, consider exporting your data as a table to be used as staging for final migration where you run your subquery only once and not for every row of data set. In Pandas, you would need to interface with sqlalchemy to run the to_sql export operation. Note: this assumes your connected user has such DROP TABLE and CREATE TABLE privileges.
Also, consider using EXISTS subquery to combine both IN subqueries. Below subquery attempts to run opposite of your logic for exclusion.
import sqlalchemy
...
engine = sqlalchemy.create_engine("oracle+cx_oracle://user:password#dsn")
# EXPORT DATA -ALWAYS REPLACING
pandas_df.to_sql('myTempTable', con=engine, if_exists='replace')
# RUN TRANSACTION
with engine.begin() as cn:
sql = """INSERT INTO ORA_TABLE (ID1, ID2)
SELECT t.ID1, t.ID2
FROM myTempTable t
WHERE EXISTS
(
SELECT 1 FROM ORA_TABLE sub
WHERE (t.ID1 != 'NF' AND t.ID1 = sub.ID1)
OR (t.ID1 = 'NF' AND t.ID2 = sub.ID2)
)
"""
cn.execute(sql)

Python & SQL via pyodbc - executing queries via loop with dynamic where clause

I'm trying to generate & execute SQL statements via pyodbc. I expect multiple SQL statements, all of which start with the same SELECT & FROM but have a different value in the WHERE. The value in my WHERE clause is derived from looping through a table - each distinct value the SQL script finds in the table, I need Python to generate another SQL statement with this value as the WHERE clause.
I'm almost there with this, I'm just struggling to get pyodbc to put my query strings in formats that SQL likes. My code so far:
import pyodbc
cn = pyodbc.connect(connection info)
cursor = cn.cursor()
result = cursor.execute('SELECT distinct searchterm_name FROM table1')
for row in result:
sql = str("SELECT * from table2 WHERE table1.searchterm_name = {c}".format(c=row)),
#print sql
This code generates an output like this, where "name here" is based on the value found in table1.
('SELECT * from ifb_person WHERE searchterm_name = (u\'name here\', )',)
I just need to remove all the crap surrounding the query & where clause so it looks like this. Then I can pass it into another cursor.execute()
SELECT * from ifb_person WHERE searchterm_name = 'name here'
EDIT
for row in result:
cursor.execute("insert into test (searchterm_name) SELECT searchterm_name FROM ifb_person WHERE searchterm_name = ?",
(row[0],))
This query fails with the error pyodbc.ProgrammingError: No results. Previous SQL was not a query.
Basically what I am trying to do is get Python to generate a fresh SQL statement for every result it finds in table1. The second query is running searches against the table ifb_person and inserting the results to a table "test". I want to run separate SQL statements for every result found in table1

pyodbc allows us to iterate over a Cursor object to return the rows, during which time the Cursor object is still "in use", so we cannot use the same Cursor object to perform other operations. For example, this code will fail:
crsr = cnxn.cursor()
result = crsr.execute("SELECT ...") # result is just a reference to the crsr object
for row in result:
# we are actually iterating over the crsr object
crsr.execute("INSERT ...") # this clobbers the previous crsr object ...
# ... so the next iteration of the for loop fails with " Previous SQL was not a query."
We can work around that by using fetchall() to retrieve all the rows into result ...
result = crsr.execute("SELECT ...").fetchall()
# result is now a list of pyodbc.Row objects and the crsr object is no longer "in use"
... or use a different Cursor object in the loop
crsr_select = cnxn.cursor()
crsr_insert = cnxn.cursor()
crsr_select.execute("SELECT ...")
for row in crsr_select:
crsr_insert.execute("INSERT ...")

Database Primary Key is incrementing, but no new rows are being added

I am writing my first python script, and I am trying to connect it to a mysql db to insert rows.
import MySQLdb
db = MySQLdb.connect("localhost","root","xxx","pytest" )
cursor = db.cursor()
cursor.execute("INSERT INTO `first_table` (`name`) VALUES ('boop') ")
When I check the mysql db via phpmyadmin, it contains no rows, however if the auto incrementing ID was 5 and then I run the script 2 times, when I insert a new row it inserts it as id= 8 so the script has been incrementing the primary key but not inserting the rows?
The script reports no mysql errors, so I'm a bit lost here.

In yuor case please use
import MySQLdb
db = MySQLdb.connect("localhost","root","jimmypq79","pytest" )
cursor = db.cursor()
cursor.execute("INSERT INTO `first_table` (`name`) VALUES ('boop') ")
db.commit()
Please put this in top of the code like this--
db = MySQLdb.connect("localhost","root","jimmypq79","pytest" )
db.autocommit(True)
check here

You can use
cursor.autocommit(True)
in the beginning of the code for automatically committing the changes .

you use,
db.commit()
after insert query