I have data in MySQL table which I want to copy to a PostgreSQL table.
Everything works except when the MySQL contains a string with " and/or '
For example:
The data in MySQL:
When I run my code I get:
ProgrammingError: ERROR: syntax error at or near "t"
(the t of the can't)
This is my code:
postgre = pg.connect(dbname=DB,user=USR,passwd=PASSWD,host=HOST, port=PORT)
crs = db_remote.cursor(MySQLdb.cursors.DictCursor)
crs.execute ("""select post_id, post_excerpt from tabl""")
data = crs.fetchall ()
for row in data :
postgre.query("""INSERT INTO importfrommysql(id,message)
VALUES ('%s','%s')"""%(row["post_id"],row["post_excerpt"]))
the connection pg.connect is from PygreSQL package.
What can I do? Is it possible to get the text as it is? or the only solution is to drop all " / ' before the insert?
Use the Psycopg cursor.execute parameter passing:
import psycopg2
conn = psycopg2.connect(database='DB')
cursor = conn.cursor()
for row in data :
cursor.execute ("""
INSERT INTO importfrommysql (id,message)
VALUES (%s,%s)
""",
(row["post_id"],row["post_excerpt"])
)
It will escape and quote as necessary.
Related
fairly new to SQL in general. I'm currently trying to bolster my general understanding of how to pass commands via cursor.execute(). I'm currently trying to grab a column from a table and rename it to something different.
import mysql.connector
user = 'root'
pw = 'test!*'
host = 'localhost'
db = 'test1'
conn = mysql.connector.connect(user=user, password=pw, host=host, database=db)
cursor = conn.cursor(prepared=True)
new_name = 'Company Name'
query = f'SELECT company_name AS {new_name} from company_directory'
cursor.execute(query)
fetch = cursor.fetchall()
I've also tried it like this:
query = 'SELECT company_name AS %s from company_directory'
cursor.execute(query, ('Company Name'),)
fetch = cursor.fetchall()
but that returns the following error:
stmt = self._cmysql.stmt_prepare(statement)
_mysql_connector.MySQLInterfaceError: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '? from company_directory' at line 1
I'm using python and mySQL. I keep reading about database injection and not using string concatenation but every time I try to use %s I get an error similar to the one below where. I've tried switching to ? syntax but i get the same error.
If someone could ELI5 what the difference is and what exactly database injection is and if what I'm doing in the first attempt qualifies as string concatenation that I should be trying to avoid.
Thank you so much!
If a column name or alias contains spaces, you need to put it in backticks.
query = f'SELECT company_name AS `{new_name}` from company_directory'
You can't use a placeholder for identifiers like table and column names or aliases, only where expressions are allowed.
You can't make a query parameter in place of a column alias. The rules for column aliases are the same as column identifiers, and they must be fixed in the query before you pass the query string.
So you could do this:
query = f"SELECT company_name AS `{'Company Name'}` from company_directory'
cursor.execute(query)
I am trying to create a temporary table from a pandas df and then use it in a sql statement
import snowflake.connector
from snowflake.connector.pandas_tools import write_pandas
with snowflake.connector.connect(
account='snoflakewebsite',
user='username',
authenticator='externalbrowser',
database='db',
schema='schema'
) as con:
success, nchunks, nrows, _ = write_pandas(
conn=con,
df=df,
table_name='temp_table',
auto_create_table = True,
table_type='temporary',
overwrite = True,
database='db',
schema='schema'
)
cur = con.cursor()
cur.execute('select * from temp_table')
The error I get:
ProgrammingError: 002003 (42S02): SQL compilation error:
Object 'TEMP_TABLE' does not exist or not authorized.
write_pandas() creates a table using the letter case exactly how it is passed in table_name=, while the query submitted in cur.execute() passes the entire string with the query to Snowflake SQL, and Snowflake SQL capitalizes the object names unless they are written in double quotes.
Therefore, either you create a table using capital letters table_name='TEMP_TABLE',
or you query it using double quotes:
cur.execute('select * from "temp_table"')
In this case, you will get your table created in small letters, and you always need to add double quotes to refer to its name.
I'm currently trying to query a deltadna database. Their Direct SQL Access guide states that any PostgreSQL ODBC compliant tools should be able to connect without issue. Using the guide, I set up an ODBC data source in windows
I have tried adding Set nocount on, changed various formats for the connection string, changed the table name to be (account).(system).(tablename), all to no avail. The simple query works in Excel and I have cross referenced with how Excel formats everything as well, so it is all the more strange that I get the no query problem.
import pyodbc
conn_str = 'DSN=name'
query1 = 'select eventName from table_name limit 5'
conn = pyodbc.connect(conn_str)
conn.setdecoding(pyodbc.SQL_CHAR,encoding='utf-8')
query1_cursor = conn.cursor().execute(query1)
row = query1_cursor.fetchone()
print(row)
Result is ProgrammingError: No results. Previous SQL was not a query.
Try it like this:
import pyodbc
conn_str = 'DSN=name'
query1 = 'select eventName from table_name limit 5'
conn = pyodbc.connect(conn_str)
conn.setdecoding(pyodbc.SQL_CHAR,encoding='utf-8')
query1_cursor = conn.cursor()
query1_cursor.execute(query1)
row = query1_cursor.fetchone()
print(row)
You can't do the cursor declaration and execution in the same row. Since then your query1_cursor variable will point to a cursor object which hasn't executed any query.
I am trying to load a big list of sql queries into a table in Vertica using PYODBC. Here's my code:
tablename = DVXTEMP.my_table_name
sql = my_sql_query.strip().strip(';')
samplesize = 1000
createstring = 'CREATE TABLE %s AS %s \n limit %s;' %(tablename, sql, samplesize)
cursor.execute(createstring)
when I print createstring and run it in Toad, it works fine. when I try to execute it in pyodbc, it gives me the following error:
'Syntax error at or near "DVXTEMP" at character 1\n (4856) (SQLExecDirectW)'
We are using Vertica Analytic Database v7.1.2-6
Any ideas what might be causing this?
Thanks
1) did you import pyodbc?
2) did you define "cursor" from "pyodbc.connect"?
import pyodbc
DB = '[string for dbfile]'
DRV = '[string of which driver you are going to use]'
con = pyodbc.connect('DRIVER={};DBQ={}'.format(DRV,DB))
cursor = con.cursor()
##build SQL code and execute as you have done
Try SQL commands after you can connect without an error.
3) I use pyodbc for mdb files (MS Access) and some of my queries will not run unless I put single quotes outside double quotes on table/field names.
mytbl_1 = "mytbl"
SQL = 'SELECT * FROM ' + mytbl_1
print SQL
print result -> SELECT * FROM mytbl
(this fails)
mytbl_2 = '"mytbl"' #single quotes outside of double quote
SQL = 'SELECT * FROM ' + mytbl_2
print SQL
print result -> SELECT * FROM "mytbl"
(this string gets passed w/o error works for me with MDB files)
I have a DDL object (create_function_foo) that contains a create function statement. In first line of it I put DROP FUNCTION IF EXISTS foo; but engine.execute(create_function_foo) returns:
sqlalchemy.exc.InterfaceError: (InterfaceError) Use multi=True when executing multiple statements
I put multi=True as parameter for create_engine, engine.execute_options and engine.execute but it doesn't work.
NOTE: engine if my instance of create_engine
NOTE: I'm using python 3.2 + mysql.connector 1.0.12 + sqlalchemy 0.8.2
create_function_foo = DDL("""\
DROP FUNCTION IF EXISTS foo;
CREATE FUNCTION `foo`(
SID INT
) RETURNS double
READS SQL DATA
BEGIN
...
END
""")
Where I should put it?
multi=True is a requirement for MySql connector. You can not set this flag passing it to SQLAlchemy methods. Do this:
conn = session.connection().connection
cursor = conn.cursor() # get mysql db-api cursor
cursor.execute(sql, multi=True)
More info here: http://www.mail-archive.com/sqlalchemy#googlegroups.com/msg30129.html
Yeah... This seems like a bummer to me. I don't want to use the ORM so the accepted answer didn't work for me.
I did this instead:
with open('sql_statements_file.sql') as sql_file:
for statement in sql_file.read().split(';'):
if len(statement.strip()) > 0:
connection.execute(statement + ';')
And then this failed for a CREATE function.... YMMV.
There are some cases where SQLAlchemy does not provide a generic way at accessing some DBAPI functions, such as as dealing with multiple result sets. In these cases, you should deal with the raw DBAPI connection directly.
From SQLAlchemy documentation:
connection = engine.raw_connection()
try:
cursor = connection.cursor()
cursor.execute("select * from table1; select * from table2")
results_one = cursor.fetchall()
cursor.nextset()
results_two = cursor.fetchall()
cursor.close()
finally:
connection.close()
You can also do the same using mysql connector as seen here:
operation = 'SELECT 1; INSERT INTO t1 VALUES (); SELECT 2'
for result in cursor.execute(operation, multi=True):
if result.with_rows:
print("Rows produced by statement '{}':".format(
result.statement))
print(result.fetchall())
else:
print("Number of rows affected by statement '{}': {}".format(
result.statement, result.rowcount))