I'm attempting to bulk insert a csv into a table in SQL server. The catch is, the data doesn't match the columns of the destination table. The destination table has several audit columns that are not found in the source file. The solution I found for this is to insert into a view instead. The code is pretty simple:
from sqlalchemy import create_engine
engine = create_engine('mssql+pyodbc://[DNS]')
conn = engine.connect()
sql = "BULK INSERT [table view] FROM '[source file path]' WITH (FIELDTERMINATOR = ',',ROWTERMINATOR = '\n')"
conn.execute(sql)
conn.close()
When I run the SQL statement inside of SSMS it works perfectly. When I try to execute it from inside a Python script, the script runs but no data winds up in the table. What am I missing?
Update: It turns out bulk inserting into a normal table doesn't work either.
Before closing the connection, you need to call commit() or the SQL actions will be rolled back on connection close.
conn.commit()
conn.close()
It turns out that instead of using SQL Alchemy, I had to use pypyodbc. Not sure why this worked and the other way didn't. Example code found here:How to Speed up with Bulk Insert to MS Server from Python with Pyodbc from CSV
This works for me after checking sqlalchemy transactions refeference. I don't explicitly set conn.commit() as
The block managed by each .begin() method has the behavior such that the transaction is committed when the block completes.
with engine.begin() as conn:
conn.execute(sql_bulk_insert)
Related
I am going to insert data into a table which is in Oracle using by python. It did not give me any error but also it did not insert anything into my table in Oracle SQL Developer. Also I wrote a query to get select rom my table in python environment it gave me output.
My CREATE TABLE code:
cursor = conn.cursor()
cursor.execute("CREATE TABLE INSERT_TEST(FISRT_COLUMN VARCHAR2(30),SECOND_COLUMN NUMBER(10,2))")
My INSERT code:
cursor = conn.cursor()
data = [
('Mina', 20),
('Minoo', 32),
('Sara', 22),
('Ehssan', 25),
('Ladan', 55)
]
for row in data:
cursor.execute("""
insert into INSERT_TEST (FISRT_COLUMN, SECOND_COLUMN)
values (:1, :2)""", row)
My SELECT query:
re = cursor.execute("select * from INSERT_TEST")
for row in re:
print(row)
My output in python environment:
('Mina', 20.0) ('Minoo', 32.0) ('Sara', 22.0) ('Ehssan', 25.0) ('Ladan', 55.0)
My output in Oracle SQL Developer is empty!
I think you need a conn.commit() at the end of your program. commit belongs to the connection class, not to the cursor.
I understand it depends on table in oracle SQL developer, a the end of my code I have to write cursor.execute('commit') to show that it end and insert all of the data into my table.
Check the documentation for your database driver.
For some database drivers, the default behaviour is to COMMIT any uncommitted data when you close the transaction and the connection; so, make sure you close the connection when you have finished.
For other database drivers, the default behaviour is to ROLLBACK any uncommitted data when you close the transaction and the connection; in this case, you will need to explicitly COMMIT the data before you close the connection otherwise it will ROLLBACK your changes.
The Oracle database does not let other sessions (even belonging to the same user) see uncommitted data so, if you INSERT from Python but do not COMMIT (either explicitly or, if the driver supports it, implicitly when you close the connection) then you will be able to see it from the Python session where you created the data but it will be hidden when you view it in another session (i.e. from SQL Developer).
I am trying to truncate a table in my Postgres DB using a python script:
conn = get_psql_conn()
cursor = conn.cursor()
cursor.execute("""TRUNCATE table table_name;""")
cursor.close()
conn.close()
Nothing happens.
The script finishes quickly, no error is raised.
The table still has its rows.
I was able to execute other queries with no problem using the same setup.
I did appreciate it if anyone can point out my mistake here!
Thanks
I use a MS SQL express db. I can connect and fetch data. But inserting data does not work:
cursor.execute("insert into [mydb].[dbo].[ConvertToolLog] ([Message]) values('test')")
I get no error but nothing is inserted into the table.
Directly after I fetch the data the inserted row is fetched. But nothing is saved.
In MS SQL Server Management Studio the insertion does work.
You need to commit the data. Each SQL command is in a transaction and the transaction must be committed to write the transaction to the SQL Server so that it can be read by other SQL commands.
Under MS SQL Server Management Studio the default is to allow auto-commit which means each SQL command immediately works and you cannot rollback.
The example is from pyodbc Getting Started document
First opening the database and set up a cursor
import pyodbc
# Specifying the ODBC driver, server name, database, etc. directly
cnxn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER=localhost;DATABASE=testdb;UID=me;PWD=pass')
# Create a cursor from the connection
cursor = cnxn.cursor()
The insert example in the document is then
# Do the insert
cursor.execute("insert into products(id, name) values ('pyodbc', 'awesome library')")
#commit the transaction
cnxn.commit()
or better using parameters
cursor.execute("insert into products(id, name) values (?, ?)", 'pyodbc', 'awesome library')
cnxn.commit()
As the document says
Note the calls to cnxn.commit(). You must call commit or your changes will be lost! When the connection is closed, any pending changes will be rolled back. This makes error recovery very easy, but you must remember to call commit.
Same issue here, slight modification to answers above. I struggled with this for a while, realized the SQL was running but not committing. My environment is Python 3.8.2, MSSQL Server Express 15.0.2070.41 (2019). I had to use cursor.commit(), and it worked as expected.
import pyodbc
# cnxn info is in a text file so I can change it in one place
def cnxn():
f=open("sql_conn.csv")
sql_split=f.read().split(",")
server=sql_split[0]
database=sql_split[1]
uid=sql_split[2]
pwd=sql_split[3]
return pyodbc.connect('DRIVER={ODBC Driver 17 for SQL
Server};SERVER=%s;DATABASE=%s;UID=%s;PWD=%s' % (server,database,uid,pwd))
# sql statement is passed by .py script
def f_sql_insert(sql):
cursor = cnxn().cursor()
cursor.execute(sql)
cursor.commit()
cnxn().close()
I'm connecting mysql on my Kivy application.
import mysql.connector
con = mysql.connector.Connect(host='XXX', port=XXX, user='XXX', password='XXX', database='XXX')
cur = con.cursor()
db = cur.execute("""select SELECT SQL_NO_CACHE * from abc""")
data = cur.fetchall()
print (data)
After inserting or deleting on table abc from another connection; i call the same query on python; but data is not updating.
I add the query "SET SESSION query_cache_type = OFF;" before select query, but it didn't work. Someone said "select NOW() ..." query is not cachable but it didn't work again. What should I do?
I solved this by adding the code after fetchall()
con.commit()
Calling the same select query without doing a commit, won't update the results.
The solution is to use:
Once:
con.autocommit(True)
Or, after each select query:
con.commit()
With this option, there will be a commit after each select query.
Otherwise, subsequent selects will render the same result.
This error seems to be Bug #42197 related to Query cache and auto-commit in MySQL. The status is won't fix!
In a few months, this should be irrelevant because MySQL 8.0 is dropping Query Cache.
I encounterd the same problem that has been solved and used the above method.
conn.commit()
and I found that different DBMS has different behavior,not all DBMS exist in the connection cache
try this,
conn.autocommit(True);
this will auto commit after each of you select query.
The MySQL query cache is flushed when tables are modified, so it wouldn't have that effect. It's impossible to say without seeing the rest of your code, but it's most likely that your INSERT / DELETE query is failing to run.
I use a MS SQL express db. I can connect and fetch data. But inserting data does not work:
cursor.execute("insert into [mydb].[dbo].[ConvertToolLog] ([Message]) values('test')")
I get no error but nothing is inserted into the table.
Directly after I fetch the data the inserted row is fetched. But nothing is saved.
In MS SQL Server Management Studio the insertion does work.
You need to commit the data. Each SQL command is in a transaction and the transaction must be committed to write the transaction to the SQL Server so that it can be read by other SQL commands.
Under MS SQL Server Management Studio the default is to allow auto-commit which means each SQL command immediately works and you cannot rollback.
The example is from pyodbc Getting Started document
First opening the database and set up a cursor
import pyodbc
# Specifying the ODBC driver, server name, database, etc. directly
cnxn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER=localhost;DATABASE=testdb;UID=me;PWD=pass')
# Create a cursor from the connection
cursor = cnxn.cursor()
The insert example in the document is then
# Do the insert
cursor.execute("insert into products(id, name) values ('pyodbc', 'awesome library')")
#commit the transaction
cnxn.commit()
or better using parameters
cursor.execute("insert into products(id, name) values (?, ?)", 'pyodbc', 'awesome library')
cnxn.commit()
As the document says
Note the calls to cnxn.commit(). You must call commit or your changes will be lost! When the connection is closed, any pending changes will be rolled back. This makes error recovery very easy, but you must remember to call commit.
Same issue here, slight modification to answers above. I struggled with this for a while, realized the SQL was running but not committing. My environment is Python 3.8.2, MSSQL Server Express 15.0.2070.41 (2019). I had to use cursor.commit(), and it worked as expected.
import pyodbc
# cnxn info is in a text file so I can change it in one place
def cnxn():
f=open("sql_conn.csv")
sql_split=f.read().split(",")
server=sql_split[0]
database=sql_split[1]
uid=sql_split[2]
pwd=sql_split[3]
return pyodbc.connect('DRIVER={ODBC Driver 17 for SQL
Server};SERVER=%s;DATABASE=%s;UID=%s;PWD=%s' % (server,database,uid,pwd))
# sql statement is passed by .py script
def f_sql_insert(sql):
cursor = cnxn().cursor()
cursor.execute(sql)
cursor.commit()
cnxn().close()