Develop Reference

How to INSERT query a list of elements in python

I have tried to query all elements from a list into an insertion query, I tried by making the list into a list of tuples and directly by adding the elements from the list. But it did not work, and I don't know the best practice for this as I am no SQL shark. I have below added the two different outputs I have before I do the query. I don't know which is easier to work with for this, but the code example is for the list of elements.
Output
['testuser', 'AskeMeyer']
and
[('testuser',), ('AskeMeyer',)]
Code to query
try:
conn = psycopg2.connect(host=ENDPOINT, port=PORT, database=DBNAME, user=USER, password=PASS, sslmode='require', sslrootcert="SSLCERTIFICATE")
cur = conn.cursor()
var_string = ', '.join(map(str, res))
sql = 'INSERT INTO users_from_group(name) VALUES %s;' % var_string)
cur.execute(sql)
error from above
Database connection failed due to syntax error at or near ")"

You need to use brackets and quotes in your insert statement, for example:
var_string = ', '.join([f"(\"{name}\")" for name in ['testuser', 'AskeMeyer']])
Also, there is a bracket at the end of your sql statement which causes the syntax error:
sql = 'INSERT INTO users_from_group(name) VALUES %s;' % var_string)
Should be
sql = 'INSERT INTO users_from_group(name) VALUES %s;' % var_string

it's not the proper way to store lists.
you should save it as json :
json.dumps(['testuser', 'AskeMeyer'] )
then save it

Try this:
var_string = '(' + ','.join(map(str, res)) + ')'
sql = 'INSERT INTO users_from_group(name) VALUES %s;' % var_string

How to get rid of a comma?

import sqlite3
conn = sqlite3.connect('contacten.db')
c = conn.cursor()
#c.execute("""CREATE TABLE mail (
# mail text
# )""")
#c.execute("INSERT INTO mail VALUES ('test#gmail.com')")
conn.commit()
c.execute("SELECT * FROM mail")
print(c.fetchall())
conn.commit()
conn.close()
This is the code I've made but as a result, I get:
[('test#gmail.com',), ('test1#gmail.com',), ('test2#gmail.com',)]
But in this array I have a comma too much. ,), like this. does anyone of you know how to get rid of this extra comma?

The commas are there for a good reason, your result is a list of tuples; this is a consequence of how sqlite represents the result set, the data itself doesn't contain commas:
result = c.fetchall()
print(result)
=> [('test#gmail.com',), ('test1#gmail.com',), ('test2#gmail.com',)]
That's because each row can have more than one field. In your case you only have one field, but Python can't just remove the comma, because if we did you'd end up with a list of elements between brackets, not a list of tuples (see here to understand why).
Of course, if you're certain that the result will only have one field per row, you can simply get rid of the tuples by extracting the one (and only) field from each row at the moment of printing the result:
result = c.fetchall()
print([f[0] for f in result])
=> ['test#gmail.com', 'test1#gmail.com', 'test2#gmail.com']

Using python zip:
>>> emails = [('test#gmail.com',), ('test1#gmail.com',), ('test2#gmail.com',)]
>>> emails, = zip(*emails)
>>> type(emails)
<type 'tuple'>
>>> emails
('test#gmail.com', 'test1#gmail.com', 'test2#gmail.com')
>>> list(emails)
['test#gmail.com', 'test1#gmail.com', 'test2#gmail.com']

import sqlite3
conn = sqlite3.connect('contacten.db')
c = conn.cursor()
#c.execute("""CREATE TABLE mail (
# mail text
# )""")
#c.execute("INSERT INTO mail VALUES ('test#gmail.com')")
#c.execute("INSERT INTO mail VALUES ('test1#gmail.com')")
#c.execute("INSERT INTO mail VALUES ('test2#gmail.com')")
conn.commit()
c.execute("SELECT * FROM mail")
#print(c.fetchall())
for item in c.fetchall():
print item[0]
conn.commit()
conn.close()

Chinese character insert issue

I have the following dataframe in pandas
need to insert all value into a datawarehouse with chinese characters but chinese characters are instered as junk (?????) (百å¨è‹±åšï¼ˆèˆŸå±±ï¼‰å•¤é…’有é™å…¬å¸
) like above one
The insert query is prepared dynamically.
I need help on how to handle the following scenerio:
Read file as UTF-8 and writte into a datawarehouse using pyodbc connection using character set UTF-8.
df=pd.read_csv(filename,dtype='str',encoding='UTF-8')
cnxn = database_connect() ##Connect to database##
cnxn.setencoding(ctype=pyodbc.SQL_CHAR, encoding='UTF-8')
cnxn.autocommit = True
cursor = cnxn.cursor()
for y in range(len(df)):
inst='insert into '+tablename+' values ('
for x in range(len(clm)):
if str(df.iloc[y,x])=='nan':
df.iloc[y,x]=''
if x!=len(clm)-1:
inst_val=inst_val+"'"+str(df.iloc[y,x]).strip().replace("'",'')+"'"+","
else:
inst_val=inst_val+"'"+str(df.iloc[y,x]).strip().replace("'",'')+"'"+")"
inst=inst+inst_val #########prepare insert statment from values inside in-memory data###########
inst_val=''
print("Inserting value into table")
try:
cursor.execute(inst) ##########Execute insert statement##############
print("1 row inserted")
except Exception as e:
print (inst)
print (e)
same like value should inserted into sql datawarehouse

You are using dynamic SQL to construct string literals containing Chinese characters, but you are creating them as
insert into tablename values ('你好')
when SQL Server expects Unicode string literals to be of the form
insert into tablename values (N'你好')
You would be better off to use a proper parameterized query to avoid such issues:
sql = "insert into tablename values (?)"
params = ('你好',)
cursor.execute(sql, params)

Using Python see if Table exists

I have to connect to an Oracle database and see if a table exists. While I can get a list of the tables, I'm having trouble seeing if the table I'm looking for is in the list. Some tables have associated table which I'll have to join on, some do not, thus I have to check.
What is in my list: ('NYSDOH_CI_EI_HOSPITAL',)
sql = "SELECT table_name FROM all_tables"
cur.execute(sql)
searchstr = 'NYSDOH_CI_EI_HOSPITAL'
p = re.compile(searchstr)
#create data array to load in SQL results in.
ciDataSet = []
cxRows = cur.fetchall()
for i in cxRows:
#print i # list of tables
if p.match(str(i)):
print i
It doesn't find it, even if I use a wildcard.

fetchall() returns a list of tuples.
So when you do
for i in cxRows:
'i' is of type tuple. In your case, this tuple will have only single value. You can access it using i[0] and match it with p.
Currently you are converting a tuple to string so regular expression will not match.
Corrected code:
sql = "SELECT table_name FROM all_tables"
cur.execute(sql)
searchstr = 'NYSDOH_CI_EI_HOSPITAL'
p = re.compile(searchstr)
#create data array to load in SQL results in.
ciDataSet = []
cxRows = cur.fetchall()
for i in cxRows:
#print i # list of tables
if p.match(str(i[0])):
print i

To improve on the syntax of #vaichidrewar, you could simplify the fetch loop to:
for tabname, in cur:
if p.match(str(tabname)):
print(tabname)
But it's going to be more efficient to do the reg exp matching in the query:
sql = "select table_name from all_tables where regexp_like(table_name, :tn, 'i')"
searchstr = 'EMP'
cur.execute(sql, (searchstr,))
for tabname, in cur:
print(tabname)
The 'i' option does a case-insensitive match. You can adjust the regexp as you like.

This code is writing only one row in DB?

This code is writing only one row in DB
I find no error in this code . . .
But why is this not inserting more than the first row ?
def transremovechars():
cur.execute('drop table transforms')
char_cfg = config.get('Transform_Variables', 'Chars_to_be_removed') #Reads all the special chars to be removed from specialchars.txt#
cur.execute('select * from originallist')
for row in cur: #Applies transformation to remove chars for each row in a loop#
company = row[0]
for specialchars in char_cfg:
company = company.replace(specialchars, '')
cur.execute('Insert into transforms (Transresult1) values (\'' + company + '\')')
con.commit()

You forgot the cur.fetchall():
def transremovechars():
cur.execute('drop table transforms')
char_cfg = config.get('Transform_Variables', 'Chars_to_be_removed') #Reads all the special chars to be removed from specialchars.txt#
cur.execute('select * from originallist')
for row in cur.fetchall(): #Applies transformation to remove chars for each row in a loop#
company = row[0]
for specialchars in char_cfg:
company = company.replace(specialchars, '')
cur.execute('Insert into transforms (Transresult1) values (\'' + company + '\')')
con.commit()

You seem to drop your table transforms before working with it. Are you sure you want that? Or maybe have you forgotten to show the code which creates it again?
Your select * might e overkill if you only use the 1st column. Maybe you want to name that field in the SELECT.
Besides, you should replace your INSERT line with
cur.execute('Insert into transforms (Transresult1) values (?)', company)
Iterating over the cursor should be fine, however. Maybe you could insert some print statements into your for loop...

Comments to the effect that you should cur.fetchall() and iterate over that will work and be OK. The real fault in your code is that once you use cur to insert, it is a "new thing" and the original generator is reset (cur has the next()) method.
You can use cur without doing fetchall as you wanted, just create a second cursor ins_cur = con.curson() and use both. Many more advanced effects can be accomplished by iterating or using multiple cursors open on one connection.
And yes please use the correct variable binding for your dbapi module.