I have the following dataframe in pandas
need to insert all value into a datawarehouse with chinese characters but chinese characters are instered as junk (?????) (百å¨è‹±åšï¼ˆèˆŸå±±ï¼‰å•¤é…’有é™å…¬å¸
) like above one
The insert query is prepared dynamically.
I need help on how to handle the following scenerio:
Read file as UTF-8 and writte into a datawarehouse using pyodbc connection using character set UTF-8.
df=pd.read_csv(filename,dtype='str',encoding='UTF-8')
cnxn = database_connect() ##Connect to database##
cnxn.setencoding(ctype=pyodbc.SQL_CHAR, encoding='UTF-8')
cnxn.autocommit = True
cursor = cnxn.cursor()
for y in range(len(df)):
inst='insert into '+tablename+' values ('
for x in range(len(clm)):
if str(df.iloc[y,x])=='nan':
df.iloc[y,x]=''
if x!=len(clm)-1:
inst_val=inst_val+"'"+str(df.iloc[y,x]).strip().replace("'",'')+"'"+","
else:
inst_val=inst_val+"'"+str(df.iloc[y,x]).strip().replace("'",'')+"'"+")"
inst=inst+inst_val #########prepare insert statment from values inside in-memory data###########
inst_val=''
print("Inserting value into table")
try:
cursor.execute(inst) ##########Execute insert statement##############
print("1 row inserted")
except Exception as e:
print (inst)
print (e)
same like value should inserted into sql datawarehouse
You are using dynamic SQL to construct string literals containing Chinese characters, but you are creating them as
insert into tablename values ('你好')
when SQL Server expects Unicode string literals to be of the form
insert into tablename values (N'你好')
You would be better off to use a proper parameterized query to avoid such issues:
sql = "insert into tablename values (?)"
params = ('你好',)
cursor.execute(sql, params)
Related
I have tried to query all elements from a list into an insertion query, I tried by making the list into a list of tuples and directly by adding the elements from the list. But it did not work, and I don't know the best practice for this as I am no SQL shark. I have below added the two different outputs I have before I do the query. I don't know which is easier to work with for this, but the code example is for the list of elements.
Output
['testuser', 'AskeMeyer']
and
[('testuser',), ('AskeMeyer',)]
Code to query
try:
conn = psycopg2.connect(host=ENDPOINT, port=PORT, database=DBNAME, user=USER, password=PASS, sslmode='require', sslrootcert="SSLCERTIFICATE")
cur = conn.cursor()
var_string = ', '.join(map(str, res))
sql = 'INSERT INTO users_from_group(name) VALUES %s;' % var_string)
cur.execute(sql)
error from above
Database connection failed due to syntax error at or near ")"
You need to use brackets and quotes in your insert statement, for example:
var_string = ', '.join([f"(\"{name}\")" for name in ['testuser', 'AskeMeyer']])
Also, there is a bracket at the end of your sql statement which causes the syntax error:
sql = 'INSERT INTO users_from_group(name) VALUES %s;' % var_string)
Should be
sql = 'INSERT INTO users_from_group(name) VALUES %s;' % var_string
it's not the proper way to store lists.
you should save it as json :
json.dumps(['testuser', 'AskeMeyer'] )
then save it
Try this:
var_string = '(' + ','.join(map(str, res)) + ')'
sql = 'INSERT INTO users_from_group(name) VALUES %s;' % var_string
I am using the below code to delete a row from sqlite table.
def deleteFromTable(item):
conn = sqlite3.connect("lite.db")
cur = conn.cursor()
cur.execute("DELETE FROM store WHERE item=?", (item,))
conn.commit()
conn.close()
Why do i need to use comma after item (item,) while passing the argument?
('String') evaluates into string, but ('string',) evaluates into tuple. that's why you need comma.
I have the following code
engine = create_engine('database info',convert_unicode=True)
result = engine.execute(" select title from table")
line=(result.fetchall())
print(line)
Some of the results have "\xa0" inserted in them
Is there a way to return my query so that I don't have to try to fix it after the fact?
I have to connect to an Oracle database and see if a table exists. While I can get a list of the tables, I'm having trouble seeing if the table I'm looking for is in the list. Some tables have associated table which I'll have to join on, some do not, thus I have to check.
What is in my list: ('NYSDOH_CI_EI_HOSPITAL',)
sql = "SELECT table_name FROM all_tables"
cur.execute(sql)
searchstr = 'NYSDOH_CI_EI_HOSPITAL'
p = re.compile(searchstr)
#create data array to load in SQL results in.
ciDataSet = []
cxRows = cur.fetchall()
for i in cxRows:
#print i # list of tables
if p.match(str(i)):
print i
It doesn't find it, even if I use a wildcard.
fetchall() returns a list of tuples.
So when you do
for i in cxRows:
'i' is of type tuple. In your case, this tuple will have only single value. You can access it using i[0] and match it with p.
Currently you are converting a tuple to string so regular expression will not match.
Corrected code:
sql = "SELECT table_name FROM all_tables"
cur.execute(sql)
searchstr = 'NYSDOH_CI_EI_HOSPITAL'
p = re.compile(searchstr)
#create data array to load in SQL results in.
ciDataSet = []
cxRows = cur.fetchall()
for i in cxRows:
#print i # list of tables
if p.match(str(i[0])):
print i
To improve on the syntax of #vaichidrewar, you could simplify the fetch loop to:
for tabname, in cur:
if p.match(str(tabname)):
print(tabname)
But it's going to be more efficient to do the reg exp matching in the query:
sql = "select table_name from all_tables where regexp_like(table_name, :tn, 'i')"
searchstr = 'EMP'
cur.execute(sql, (searchstr,))
for tabname, in cur:
print(tabname)
The 'i' option does a case-insensitive match. You can adjust the regexp as you like.
I ran into problems, while trying to insert to database:
ur_psql.execute("""insert into smth(data, filedate, filedby)"""
""" values('%s', NOW(), %s)""" % (data, userid))
where data=""" "5.09,448,1418112000"; d="scan'208" """(a string containing double and single quotes)
Any ideas how to insert such string to db ?
Thanks
You can read about it at: http://initd.org/psycopg/docs/usage.html#the-problem-with-the-query-parameters
Simply do not use quotes in SQL and instead of % string Python operator use 2nd parameter of execute() which is data you want to pass to SQL query:
sql = "insert into smth (data, filedate, filedby) values (%s, NOW(), %s)"
ur_psql.execute(sql, (data, userid))