Develop Reference

SQL query format

I have a list of string that I need to pass to an sql query.
listofinput = []
for i in input:
listofinput.append(i)
if(len(listofinput)>1):
listofinput = format(tuple(listofinput))
sql_query = f"""SELECT * FROM countries
where
name in {listofinput};
"""
This works when I have a list, but in case of just one value it fails.
as listofinput = ['USA'] for one value
but listofinput ('USA', 'Germany') for multiple
also I need to do this for thousands of input, what is the best optimized way to achieve the same. name in my table countries is an indexed column

You can just convert to tuple and then if the second last character is a coma, remove it.
listofinput = format(tuple(input))
if listofinput[-2] == ",":
listofinput = f"{listofinput[:-2]})"
sql_query = f"""SELECT * FROM countries
where name in {listofinput};"""

Change if(len(listofinput)>1): to if(len(listofinput)>=1):
This might work.

Remove condition if(len(listofinput)>1) .
Because if you don't convert to tuple your query should be like this:
... where name in ['USA']
or
... where name in []
and in [...] not acceptable in SQL, only in (...) is acceptable.
You can remove format() too:
listofinput = tuple(listofinput)
Final Code:
listofinput = []
for i in input:
listofinput.append(i)
listofinput = tuple(listofinput)
sql_query = f"""SELECT * FROM countries
WHERE
name IN {listofinput};
"""

Yes the tuple with one element will required a ","
To circumvent your problem, maybe you can use string instead by just changing your code to the below:
listofinput = []
for i in input:
listofinput.append(i)
if(len(listofinput)>1):
listofinput = format(tuple(listofinput))
else:
listofinput='('+listofinput[0]+')'

How to use multiple variables to be sent to Impala for SQL statements in Python [duplicate]

This question already has answers here:
imploding a list for use in a python MySQLDB IN clause
(8 answers)
Closed 1 year ago.
I have a python list, say l
l = [1,5,8]
I want to write a sql query to get the data for all the elements of the list, say
select name from students where id = |IN THE LIST l|
How do I accomplish this?

Answers so far have been templating the values into a plain SQL string. That's absolutely fine for integers, but if we wanted to do it for strings we get the escaping issue.
Here's a variant using a parameterised query that would work for both:
placeholder= '?' # For SQLite. See DBAPI paramstyle.
placeholders= ', '.join(placeholder for unused in l)
query= 'SELECT name FROM students WHERE id IN (%s)' % placeholders
cursor.execute(query, l)

Easiest way is to turn the list to tuple first
t = tuple(l)
query = "select name from studens where id IN {}".format(t)

Dont complicate it, Solution for this is simple.
l = [1,5,8]
l = tuple(l)
params = {'l': l}
cursor.execute('SELECT * FROM table where id in %(l)s',params)
I hope this helped !!!

The SQL you want is
select name from studens where id in (1, 5, 8)
If you want to construct this from the python you could use
l = [1, 5, 8]
sql_query = 'select name from studens where id in (' + ','.join(map(str, l)) + ')'
The map function will transform the list into a list of strings that can be glued together by commas using the str.join method.
Alternatively:
l = [1, 5, 8]
sql_query = 'select name from studens where id in (' + ','.join((str(n) for n in l)) + ')'
if you prefer generator expressions to the map function.
UPDATE: S. Lott mentions in the comments that the Python SQLite bindings don't support sequences. In that case, you might want
select name from studens where id = 1 or id = 5 or id = 8
Generated by
sql_query = 'select name from studens where ' + ' or '.join(('id = ' + str(n) for n in l))

string.join the list values separated by commas, and use the format operator to form a query string.
myquery = "select name from studens where id in (%s)" % ",".join(map(str,mylist))
(Thanks, blair-conrad)

I like bobince's answer:
placeholder= '?' # For SQLite. See DBAPI paramstyle.
placeholders= ', '.join(placeholder for unused in l)
query= 'SELECT name FROM students WHERE id IN (%s)' % placeholders
cursor.execute(query, l)
But I noticed this:
placeholders= ', '.join(placeholder for unused in l)
Can be replaced with:
placeholders= ', '.join(placeholder*len(l))
I find this more direct if less clever and less general. Here l is required to have a length (i.e. refer to an object that defines a __len__ method), which shouldn't be a problem. But placeholder must also be a single character. To support a multi-character placeholder use:
placeholders= ', '.join([placeholder]*len(l))

If you're using PostgreSQL with the Psycopg2 library you can let its tuple adaption do all the escaping and string interpolation for you, e.g:
ids = [1,2,3]
cur.execute(
"SELECT * FROM foo WHERE id IN %s",
[tuple(ids)])
i.e. just make sure that you're passing the IN parameter as a tuple. if it's a list you can use the = ANY array syntax:
cur.execute(
"SELECT * FROM foo WHERE id = ANY (%s)",
[list(ids)])
note that these both will get turned into the same query plan so you should just use whichever is easier. e.g. if your list comes in a tuple use the former, if they're stored in a list use the latter.

Solution for #umounted answer, because that broke with a one-element tuple, since (1,) is not valid SQL.:
>>> random_ids = [1234,123,54,56,57,58,78,91]
>>> cursor.execute("create table test (id)")
>>> for item in random_ids:
cursor.execute("insert into test values (%d)" % item)
>>> sublist = [56,57,58]
>>> cursor.execute("select id from test where id in %s" % str(tuple(sublist)).replace(',)',')'))
>>> a = cursor.fetchall()
>>> a
[(56,), (57,), (58,)]
Other solution for sql string:
cursor.execute("select id from test where id in (%s)" % ('"'+'", "'.join(l)+'"'))

Just use inline if operation with tuple function:
query = "Select * from hr_employee WHERE id in " % tuple(employee_ids) if len(employee_ids) != 1 else "("+ str(employee_ids[0]) + ")"

To run a select from where field is in list of strings (instead of int), as per this question use repr(tuple(map(str, l))). Full example:
l = ['a','b','c']
sql = f'''
select name
from students
where id in {repr(tuple(map(str, l)))}
'''
print(sql)
Returns:
select name from students where id in ('a', 'b', 'c')
For a list of dates in Oracle, this worked
l = ['2020-11-24', '2020-12-28']
dates_str = ','.join([f'DATE {repr(s)}' for s in l])
dates_str = f'({dates_str})'
sql_cmd = f'''
select *
from students
where
and date in {dates_str}
'''
Returns:
select * from students where and date in (DATE '2020-11-24',DATE '2020-12-28')
If you need to get the list of dates from a pandas df, it's df['date'].dt.strftime('%Y-%m-%d').unique()
And since I often needed it too, adding columns from a list
# single list
f'select {','.join(l)}'
# multi list in different tables
sql_cmd = f'''
select {','.join(f't1.{s}' for s in l1)},
{','.join(f't1.{s}' for s in l2)},
{','.join(f't2.{s}' for s in l3)}
'''

placeholders= ', '.join("'{"+str(i)+"}'" for i in range(len(l)))
query="select name from students where id (%s)"%placeholders
query=query.format(*l)
cursor.execute(query)
This should solve your problem.

a simpler solution:
lst = [1,2,3,a,b,c]
query = f"""SELECT * FROM table WHERE IN {str(lst)[1:-1}"""

l = [1] # or [1,2,3]
query = "SELECT * FROM table WHERE id IN :l"
params = {'l' : tuple(l)}
cursor.execute(query, params)
The :var notation seems simpler. (Python 3.7)

For example, if you want the sql query:
select name from studens where id in (1, 5, 8)
What about:
my_list = [1, 5, 8]
cur.execute("select name from studens where id in %s" % repr(my_list).replace('[','(').replace(']',')') )

This uses parameter substitution and takes care of the single value list case:
l = [1,5,8]
get_operator = lambda x: '=' if len(x) == 1 else 'IN'
get_value = lambda x: int(x[0]) if len(x) == 1 else x
query = 'SELECT * FROM table where id ' + get_operator(l) + ' %s'
cursor.execute(query, (get_value(l),))

This Will Work If Number of Values in List equals to 1 or greater than 1
t = str(tuple(l))
if t[-2] == ',':
t= t.replace(t[-2],"")
query = "select name from studens where id IN {}".format(t)

Parameter binding for IN clause with pymssql [duplicate]

This question already has answers here:
imploding a list for use in a python MySQLDB IN clause
(8 answers)
Closed 1 year ago.
I have a python list, say l
l = [1,5,8]
I want to write a sql query to get the data for all the elements of the list, say
select name from students where id = |IN THE LIST l|
How do I accomplish this?

Answers so far have been templating the values into a plain SQL string. That's absolutely fine for integers, but if we wanted to do it for strings we get the escaping issue.
Here's a variant using a parameterised query that would work for both:
placeholder= '?' # For SQLite. See DBAPI paramstyle.
placeholders= ', '.join(placeholder for unused in l)
query= 'SELECT name FROM students WHERE id IN (%s)' % placeholders
cursor.execute(query, l)

Easiest way is to turn the list to tuple first
t = tuple(l)
query = "select name from studens where id IN {}".format(t)

Dont complicate it, Solution for this is simple.
l = [1,5,8]
l = tuple(l)
params = {'l': l}
cursor.execute('SELECT * FROM table where id in %(l)s',params)
I hope this helped !!!

The SQL you want is
select name from studens where id in (1, 5, 8)
If you want to construct this from the python you could use
l = [1, 5, 8]
sql_query = 'select name from studens where id in (' + ','.join(map(str, l)) + ')'
The map function will transform the list into a list of strings that can be glued together by commas using the str.join method.
Alternatively:
l = [1, 5, 8]
sql_query = 'select name from studens where id in (' + ','.join((str(n) for n in l)) + ')'
if you prefer generator expressions to the map function.
UPDATE: S. Lott mentions in the comments that the Python SQLite bindings don't support sequences. In that case, you might want
select name from studens where id = 1 or id = 5 or id = 8
Generated by
sql_query = 'select name from studens where ' + ' or '.join(('id = ' + str(n) for n in l))

string.join the list values separated by commas, and use the format operator to form a query string.
myquery = "select name from studens where id in (%s)" % ",".join(map(str,mylist))
(Thanks, blair-conrad)

I like bobince's answer:
placeholder= '?' # For SQLite. See DBAPI paramstyle.
placeholders= ', '.join(placeholder for unused in l)
query= 'SELECT name FROM students WHERE id IN (%s)' % placeholders
cursor.execute(query, l)
But I noticed this:
placeholders= ', '.join(placeholder for unused in l)
Can be replaced with:
placeholders= ', '.join(placeholder*len(l))
I find this more direct if less clever and less general. Here l is required to have a length (i.e. refer to an object that defines a __len__ method), which shouldn't be a problem. But placeholder must also be a single character. To support a multi-character placeholder use:
placeholders= ', '.join([placeholder]*len(l))

If you're using PostgreSQL with the Psycopg2 library you can let its tuple adaption do all the escaping and string interpolation for you, e.g:
ids = [1,2,3]
cur.execute(
"SELECT * FROM foo WHERE id IN %s",
[tuple(ids)])
i.e. just make sure that you're passing the IN parameter as a tuple. if it's a list you can use the = ANY array syntax:
cur.execute(
"SELECT * FROM foo WHERE id = ANY (%s)",
[list(ids)])
note that these both will get turned into the same query plan so you should just use whichever is easier. e.g. if your list comes in a tuple use the former, if they're stored in a list use the latter.

Solution for #umounted answer, because that broke with a one-element tuple, since (1,) is not valid SQL.:
>>> random_ids = [1234,123,54,56,57,58,78,91]
>>> cursor.execute("create table test (id)")
>>> for item in random_ids:
cursor.execute("insert into test values (%d)" % item)
>>> sublist = [56,57,58]
>>> cursor.execute("select id from test where id in %s" % str(tuple(sublist)).replace(',)',')'))
>>> a = cursor.fetchall()
>>> a
[(56,), (57,), (58,)]
Other solution for sql string:
cursor.execute("select id from test where id in (%s)" % ('"'+'", "'.join(l)+'"'))

Just use inline if operation with tuple function:
query = "Select * from hr_employee WHERE id in " % tuple(employee_ids) if len(employee_ids) != 1 else "("+ str(employee_ids[0]) + ")"

To run a select from where field is in list of strings (instead of int), as per this question use repr(tuple(map(str, l))). Full example:
l = ['a','b','c']
sql = f'''
select name
from students
where id in {repr(tuple(map(str, l)))}
'''
print(sql)
Returns:
select name from students where id in ('a', 'b', 'c')
For a list of dates in Oracle, this worked
l = ['2020-11-24', '2020-12-28']
dates_str = ','.join([f'DATE {repr(s)}' for s in l])
dates_str = f'({dates_str})'
sql_cmd = f'''
select *
from students
where
and date in {dates_str}
'''
Returns:
select * from students where and date in (DATE '2020-11-24',DATE '2020-12-28')
If you need to get the list of dates from a pandas df, it's df['date'].dt.strftime('%Y-%m-%d').unique()
And since I often needed it too, adding columns from a list
# single list
f'select {','.join(l)}'
# multi list in different tables
sql_cmd = f'''
select {','.join(f't1.{s}' for s in l1)},
{','.join(f't1.{s}' for s in l2)},
{','.join(f't2.{s}' for s in l3)}
'''

placeholders= ', '.join("'{"+str(i)+"}'" for i in range(len(l)))
query="select name from students where id (%s)"%placeholders
query=query.format(*l)
cursor.execute(query)
This should solve your problem.

a simpler solution:
lst = [1,2,3,a,b,c]
query = f"""SELECT * FROM table WHERE IN {str(lst)[1:-1}"""

l = [1] # or [1,2,3]
query = "SELECT * FROM table WHERE id IN :l"
params = {'l' : tuple(l)}
cursor.execute(query, params)
The :var notation seems simpler. (Python 3.7)

For example, if you want the sql query:
select name from studens where id in (1, 5, 8)
What about:
my_list = [1, 5, 8]
cur.execute("select name from studens where id in %s" % repr(my_list).replace('[','(').replace(']',')') )

This uses parameter substitution and takes care of the single value list case:
l = [1,5,8]
get_operator = lambda x: '=' if len(x) == 1 else 'IN'
get_value = lambda x: int(x[0]) if len(x) == 1 else x
query = 'SELECT * FROM table where id ' + get_operator(l) + ' %s'
cursor.execute(query, (get_value(l),))

This Will Work If Number of Values in List equals to 1 or greater than 1
t = str(tuple(l))
if t[-2] == ',':
t= t.replace(t[-2],"")
query = "select name from studens where id IN {}".format(t)

Using Python see if Table exists

I have to connect to an Oracle database and see if a table exists. While I can get a list of the tables, I'm having trouble seeing if the table I'm looking for is in the list. Some tables have associated table which I'll have to join on, some do not, thus I have to check.
What is in my list: ('NYSDOH_CI_EI_HOSPITAL',)
sql = "SELECT table_name FROM all_tables"
cur.execute(sql)
searchstr = 'NYSDOH_CI_EI_HOSPITAL'
p = re.compile(searchstr)
#create data array to load in SQL results in.
ciDataSet = []
cxRows = cur.fetchall()
for i in cxRows:
#print i # list of tables
if p.match(str(i)):
print i
It doesn't find it, even if I use a wildcard.

fetchall() returns a list of tuples.
So when you do
for i in cxRows:
'i' is of type tuple. In your case, this tuple will have only single value. You can access it using i[0] and match it with p.
Currently you are converting a tuple to string so regular expression will not match.
Corrected code:
sql = "SELECT table_name FROM all_tables"
cur.execute(sql)
searchstr = 'NYSDOH_CI_EI_HOSPITAL'
p = re.compile(searchstr)
#create data array to load in SQL results in.
ciDataSet = []
cxRows = cur.fetchall()
for i in cxRows:
#print i # list of tables
if p.match(str(i[0])):
print i

To improve on the syntax of #vaichidrewar, you could simplify the fetch loop to:
for tabname, in cur:
if p.match(str(tabname)):
print(tabname)
But it's going to be more efficient to do the reg exp matching in the query:
sql = "select table_name from all_tables where regexp_like(table_name, :tn, 'i')"
searchstr = 'EMP'
cur.execute(sql, (searchstr,))
for tabname, in cur:
print(tabname)
The 'i' option does a case-insensitive match. You can adjust the regexp as you like.

remove last comma from dictionary/list

I am trying to python to generate a script that generates unload command in redshift. I not an expert Python programmer. I need to where I can generate all columns for the unload list. If the column is of specific name, I need to replace with a function. The challenge I am facing is it appending "," to last item in the dictionary. Is there a way I can avoid the last comma? Any help would be appreciated.
import psycopg2 from psycopg2.extras
import RealDictCursor
try:
conn = psycopg2.connect("dbname='test' port='5439' user='scott' host='something.redshift.amazonaws.com' password='tiger'");
except:
print "Unable to connect to the database"
conn.cursor_factory = RealDictCursor
cur = conn.cursor()
conn.set_isolation_level( psycopg2.extensions.ISOLATION_LEVEL_AUTOCOMMIT )
try:
cur.execute("SELECT column from pg_table_def where schema_name='myschema' and table_name ='tab1' " );
except:
print "Unable to execute select statement from the database!"
result = cur.fetchall()
print "unload mychema.tab1 (select "
for row in result:
for key,value in row.items():
print "%s,"%(value)
print ") AWS Credentials here on..."
conn.close()

Use the join function on the list of values in each row:
print ",".join(row.values())
Briefly, the join function is called on a string which we can think of as the "glue", and takes a list of "pieces" as its argument. The result is a string of "pieces" "held together" by the "glue". Example:
>>> glue = "+"
>>> pieces = ["a", "b", "c"]
>>> glue.join(pieces)
"a+b+c"
(since row.values() returns a list, you don't actually need the comprehension, so it's even simpler than I wrote it at first)

Infact, this worked better.
columns = []
for row in result:
if (row['column_name'] == 'mycol1') or (row['column_name'] == 'mycol2') :
columns.append("func_sha1(" + row['column_name'] + "||" + salt +")")
else:
columns.append(row['column_name'])
print selstr + ",".join(columns)+" ) TO s3://"
Thanks for your help, Jon