I am trying to load a big list of sql queries into a table in Vertica using PYODBC. Here's my code:
tablename = DVXTEMP.my_table_name
sql = my_sql_query.strip().strip(';')
samplesize = 1000
createstring = 'CREATE TABLE %s AS %s \n limit %s;' %(tablename, sql, samplesize)
cursor.execute(createstring)
when I print createstring and run it in Toad, it works fine. when I try to execute it in pyodbc, it gives me the following error:
'Syntax error at or near "DVXTEMP" at character 1\n (4856) (SQLExecDirectW)'
We are using Vertica Analytic Database v7.1.2-6
Any ideas what might be causing this?
Thanks
1) did you import pyodbc?
2) did you define "cursor" from "pyodbc.connect"?
import pyodbc
DB = '[string for dbfile]'
DRV = '[string of which driver you are going to use]'
con = pyodbc.connect('DRIVER={};DBQ={}'.format(DRV,DB))
cursor = con.cursor()
##build SQL code and execute as you have done
Try SQL commands after you can connect without an error.
3) I use pyodbc for mdb files (MS Access) and some of my queries will not run unless I put single quotes outside double quotes on table/field names.
mytbl_1 = "mytbl"
SQL = 'SELECT * FROM ' + mytbl_1
print SQL
print result -> SELECT * FROM mytbl
(this fails)
mytbl_2 = '"mytbl"' #single quotes outside of double quote
SQL = 'SELECT * FROM ' + mytbl_2
print SQL
print result -> SELECT * FROM "mytbl"
(this string gets passed w/o error works for me with MDB files)
Related
I am trying to create a temporary table from a pandas df and then use it in a sql statement
import snowflake.connector
from snowflake.connector.pandas_tools import write_pandas
with snowflake.connector.connect(
account='snoflakewebsite',
user='username',
authenticator='externalbrowser',
database='db',
schema='schema'
) as con:
success, nchunks, nrows, _ = write_pandas(
conn=con,
df=df,
table_name='temp_table',
auto_create_table = True,
table_type='temporary',
overwrite = True,
database='db',
schema='schema'
)
cur = con.cursor()
cur.execute('select * from temp_table')
The error I get:
ProgrammingError: 002003 (42S02): SQL compilation error:
Object 'TEMP_TABLE' does not exist or not authorized.
write_pandas() creates a table using the letter case exactly how it is passed in table_name=, while the query submitted in cur.execute() passes the entire string with the query to Snowflake SQL, and Snowflake SQL capitalizes the object names unless they are written in double quotes.
Therefore, either you create a table using capital letters table_name='TEMP_TABLE',
or you query it using double quotes:
cur.execute('select * from "temp_table"')
In this case, you will get your table created in small letters, and you always need to add double quotes to refer to its name.
I'm currently trying to query a deltadna database. Their Direct SQL Access guide states that any PostgreSQL ODBC compliant tools should be able to connect without issue. Using the guide, I set up an ODBC data source in windows
I have tried adding Set nocount on, changed various formats for the connection string, changed the table name to be (account).(system).(tablename), all to no avail. The simple query works in Excel and I have cross referenced with how Excel formats everything as well, so it is all the more strange that I get the no query problem.
import pyodbc
conn_str = 'DSN=name'
query1 = 'select eventName from table_name limit 5'
conn = pyodbc.connect(conn_str)
conn.setdecoding(pyodbc.SQL_CHAR,encoding='utf-8')
query1_cursor = conn.cursor().execute(query1)
row = query1_cursor.fetchone()
print(row)
Result is ProgrammingError: No results. Previous SQL was not a query.
Try it like this:
import pyodbc
conn_str = 'DSN=name'
query1 = 'select eventName from table_name limit 5'
conn = pyodbc.connect(conn_str)
conn.setdecoding(pyodbc.SQL_CHAR,encoding='utf-8')
query1_cursor = conn.cursor()
query1_cursor.execute(query1)
row = query1_cursor.fetchone()
print(row)
You can't do the cursor declaration and execution in the same row. Since then your query1_cursor variable will point to a cursor object which hasn't executed any query.
I am trying to execute .sql file from Python3.
Below is the python code I am trying
import time
userdate=time.strftime("%m_%d_%H_%M%S")
import pypyodbc as pyodbc
db_host='hostname\DBTEST'
db_name='dbname'
conn='Driver={SQL Server};Server=' + db_host + ';Database=' +db_name +
';Trusted_Connection=yes;'
db=pyodbc.connect(conn)
cursor=db.cursor()
file=open('C:\\abc\\xyz.sql','r')
line=file.read()
sql_cmd=line.split('\n')
for x in sql_cmd:
cursor.execute(x)
Below is the xyz.sql script
DECLARE #XML XML;
DECLARE #FileName VARCHAR(1000);
DECLARE #Id UNIQUEIDENTIFIER
SELECT #Id = NEWID()
SELECT #FileName = 'ggg.xml'
SELECT #XML = '<Model>
....xml tags here...
....
</Model>'
IF EXISTS (SELECT * FROM tablename CM WHERE CM.columnname = 'test') BEGIN
UPDATE CM
SET CM.pn = '01-00001',
CM.rev= '06',
CM.Model = #XML,
CM.ModifiedOn = GETUTCDATE()
FROM cm.tablename CM
WHERE CM.columnname= 'test'
PRINT 'Updated ' + #FileName
END ELSE BEGIN
INSERT INTO cm.tablename(cmID, MN, CMType, Description, PN, Rev, CM,
RowStatus, ModifiedBy, ModifiedOn)
SELECT #Id, 'test123', 'abc.1', '', '01-00011', '01', #XML, 'A',
'74E8A3E0-E5CA-4563-BD49-12DFD210ED92', GETUTCDATE()
PRINT 'Inserted ' + #FileName
END
I get below error when I run the python code.
pypyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][ODBC SQL
Server Driver][SQL Server]Must declare the scalar variable "#Id".')
DECLARE #XML XML;
DECLARE #FileName VARCHAR(1000);
DECLARE #Id UNIQUEIDENTIFIER
SELECT #Id = NEWID()
Process finished with exit code 1
Note: If I run the sql query from M/S SQL Management studio (sql server 2016), it runs successfully.
any help on this would be appreciated.
The key is to NOT execute the script one command at a time, I instead use the simpler approach of passing the entire query to cursor.execute as one unedited script.
The great thing about doing it this way is you can fully develop/debug the query in MS Sql Server and then just copy that procedure into a file (making the simple adjustments for passing arguments of course).
As a (python 3.x, with pyodbc) example, I use:
SQL_QueryName = SQL_Folder + AllAreasReportName + ".sql"
Textfile = open( SQL_QueryName, 'r' )
SQL_COMMANDS = Textfile.read()
cursor.execute( SQL_COMMANDS, ParameterList )
The same approach should work with pypyodbc.
BTW, if the query must appear within the python procedure, put the entire query inside a (triply quoted) string and pass that string to cursor.execute.
I'm trying to read the Table names from a database into a list using Pandas.read_sql.
I have tried different SQL queries found online:
cnxn = pyodbc.connect('DRIVER={Microsoft Access Driver (*.mdb)};DBQ=' + str(self.file_selected)+';Uid=Admin;Pwd=; ')
# sql = "SELECT * FROM SYS.TABLES" # tried this - also an error
sql = "SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = 'BASE TABLE' AND TABLE_SCHEMA='database_name.MDB'"
self.TableNames = pd.io.sql.read_sql(sql, cnxn)
cnxn.close()
but I get an error that it can not find the file database_name.INFORMATION_SCHEMA.TABLES
what should I use for the sql query?
In MS Access, you can retrieve metadata on a database using its system table, MSysObjects. Below is a DML call to retrieve all table names:
SELECT MSysObjects.Name
FROM MsysObjects
WHERE ((MSysObjects.Type)=1)
ORDER BY MSysObjects.Name;
However, by default this will not work with external ODBC calls such as you do in Python as permission is not allowed. To resolve, consider two routes:
Grant Permission (for Admin user)
Inside the MSAccess.exe GUI, open database and run VBA subroutine (in standalone module) which only needs to be run once:
Public Sub GrantMSysPermission()
Dim strSQL As String
strSQL = "GRANT SELECT ON MSysObjects TO Admin;"
CurrentProject.Connection.Execute strSQL
End Sub
Once done run above query in pandas read_sql call.
Saved Table
Inside MS Access.exe GUI program, run below make-table query:
SELECT MSysObjects.Name
INTO DBTables
FROM MsysObjects
WHERE ((MSysObjects.Type)=1)
ORDER BY MSysObjects.Name;
Then in Python pandas, refer to new table:
cnxn = pyodbc.connect('DRIVER={{Microsoft Access Driver (*.mdb)}};DBQ=' + \
'{};Uid=Admin;Pwd=;'.format(str(self.file_selected)))
sql = "SELECT * DBTables"
self.TableNames = pd.io.sql.read_sql(sql, cnxn)
cnxn.close()
I have data in MySQL table which I want to copy to a PostgreSQL table.
Everything works except when the MySQL contains a string with " and/or '
For example:
The data in MySQL:
When I run my code I get:
ProgrammingError: ERROR: syntax error at or near "t"
(the t of the can't)
This is my code:
postgre = pg.connect(dbname=DB,user=USR,passwd=PASSWD,host=HOST, port=PORT)
crs = db_remote.cursor(MySQLdb.cursors.DictCursor)
crs.execute ("""select post_id, post_excerpt from tabl""")
data = crs.fetchall ()
for row in data :
postgre.query("""INSERT INTO importfrommysql(id,message)
VALUES ('%s','%s')"""%(row["post_id"],row["post_excerpt"]))
the connection pg.connect is from PygreSQL package.
What can I do? Is it possible to get the text as it is? or the only solution is to drop all " / ' before the insert?
Use the Psycopg cursor.execute parameter passing:
import psycopg2
conn = psycopg2.connect(database='DB')
cursor = conn.cursor()
for row in data :
cursor.execute ("""
INSERT INTO importfrommysql (id,message)
VALUES (%s,%s)
""",
(row["post_id"],row["post_excerpt"])
)
It will escape and quote as necessary.