Develop Reference

Pass a Python Variable containing multiple ID Numbers into external BigQuery Script

I have created a python class, and one of my methods is meant to take in either a single ID number or a list of ID numbers. The function will then use the ID numbers to query from a table in BigQuery using a .sql script. Currently, the function works fine for a single ID number using the following:
def state_data(self, state, id_number):
if state == 'NY':
sql_script = self.sql_scripts['get_data_ny']
else:
sql_script = self.sql_scripts['get_data_rest']
sql_script = sql_script.replace('##id_number##', id_number)
I'm having issues with passing in multiple ID numbers at once. There are 3 different ways that I've tried without success:
The above method, passing in the multiple ID numbers as a tuple to use with WHERE ID_NUM IN('##id_number##'). This doesn't work, as when the .sql script gets called, a syntax error is returned, as parentheses and quotes are automatically added. For example, the SQL statement attempts to run as WHERE ID_NUM IN('('123', '124')'). This would run fine without one of the two sets of parentheses and quotes, but no matter what I try to pass in, they always get added.
The second technique I have tried is to create a table, populate it with the passed in ID numbers, and then join with the larger table in BQ. It goes as follows:
CREATE OR REPLACE TABLE ID_Numbers
(
ID_Number STRING
);
INSERT INTO ID_Numbers (ID_Number)
VALUES ('##id_number##');
-- rest of script is a simple left join of the above created table with the BQ table containing the data for each ID
This again works fine for single ID numbers, but passing in multiple VALUES (in this case ID Numbers) would require a ('##id_number##') per unique ID. One thing that I have not yet attempted - to assign a variable to each unique ID and pass each one in as a new VALUE. I am not sure if this technique will work.
The third technique I've tried is to include the full SQL query in the function, rather than calling a .sql script. The list of ID numbers get passed in as tuple, and the query goes as follows:
id_nums = tuple(id_number)
query = ("""SELECT * FROM `data_table`
WHERE ID_NUM IN{}""").format(id_nums)
This technique also does not work, as I get the following error:
AttributeError: 'QueryJob' object has no attribute 'format'.
I've attempted to look into this error but I cannot find anything that helps me out effectively.
Finally, I'll note that none of the posts asking the same or similar questions have solved my issues so far.
I am looking for any and all advice for a way that I can successfully pass a variable containing multiple ID numbers into my function that ultimately calls and runs a BQ query.

You should be able to use *args to get the id_numbers as a sequence and f-strings and str.join() to build the SQL query:
class MyClass:
def state_data(self, state, *id_numbers):
print(f"{state=}")
query = f"""
SELECT * FROM `data_table`
WHERE ID_NUM IN ({", ".join(str(id_number) for id_number in id_numbers)})
"""
print(query)
my_class = MyClass()
my_class.state_data("some state", 123)
my_class.state_data("some more state", 123, 124)
On my machine, this prints:
➜ sql python main.py
state='some state'
SELECT * FROM `data_table`
WHERE ID_NUM IN (123)
state='some more state'
SELECT * FROM `data_table`
WHERE ID_NUM IN (123, 124)

Python Formatting SQL WHERE clause

I'm having this function that communicates via pymysql to an SQL database stored to my localhost. I know there are similar posts about formatting an SQL section especially this one but could anyone suggest a solution?
Always getting TypeError: can't concat tuple to bytes. I suppose it's sth with the WHERE clause.
def likeMovement(pID):
print("Give a rating for the movement with #id:%s" %pID)
rate=input("Give from 0-5: ")
userID=str(1)
print(rate,type(rate))
print(pID,type(pID))
print(userID,type(userID))
cursor=con.cursor()
sqlquery='''UDPATE likesartmovement SET likesartmovement.rating=%s WHERE
likesartmovement.artisticID=? AND likesartmovement.userID=?''' % (rate,),
(pID,userID)
cursor.execute(sqlquery)
TypeError: not all arguments converted during string formatting
Thanks in advance!

The problem is that you're storing (pID,userID) as part of a tuple stored in sqlquery, instead of passing them as the arguments to execute:
sqlquery='''UDPATE likesartmovement SET likesartmovement.rating=%s WHERE
likesartmovement.artisticID=? AND likesartmovement.userID=?''' % (rate,)
cursor.execute(sqlquery, (pID,userID))
It may be clearer to see why these are different if you take a simpler example:
s = 'abc'
spam(s, 2)
s = 'abc', 2
spam(s)
Obviously those two don't do the same thing.
While we're at it:
You have to spell UPDATE right.
You usually want to use query parameters for SET clauses for exactly the same reasons you want to for WHERE clauses.
You don't need to include the table name in single-table operations, and you're not allowed to include the table name in SET clauses in single-table updates.
So:
sqlquery='''UDPATE likesartmovement SET rating=? WHERE
artisticID=? AND userID=?'''
cursor.execute(sqlquery, (rating, pID, userID))

Using variable combining LIKE and %s% in SQLite and Python

Using:
Python3
SQLite
TKinter
I am currently trying to create a function to search for a keyword in a database, but as soon as I try to combine it with TKinter, it seems to fail.
Here are the relevant lines:
(I tried it in a lot of different ways, those 3 lines below seem to work with variables, but not with the input from TKinter, so I thought they might actually work, if I edit them a little.
The problem I got is, that I'm not experienced in TKinter and SQLite yet and worked with those 2 for about 3 days yet.
def searcher(column):
#Getting input from user (TKinter)
keyword = tk.Entry(self)
keyword.pack()
#Assigning the input to a variable
kword = keyword.get()
c.execute("SELECT * FROM my_lib WHERE {kappa} LIKE {%goal%}".format(kappa=column, goal=kword))
#c.execute("SELECT * FROM my_lib WHERE "+column+"=?", (kword,))
#c.execute("SELECT * FROM my_lib WHERE {} LIKE '%kword%'".format(column))
I want to check if any of the data CONTAINS the keyword, so basically:
k_word in column_data
and not
column_data == k_word
My question is:
Is there a way to take the user input (by TKinter) and search in the database (SQLite) and check, if any data in the database contains the keyword.

The SQLite docs explain that you can use ? as a placeholder in the query string, which allows you so substitute in a tuple of values. They also advise against ever assembling a full query using variables with Python's string operations (explained below):
c.execute("SELECT * FROM my_lib WHERE ? LIKE ?", (column, '%'+kword+'%'))
You can see above that I concatenated the % with kword, which will get substituted into the second ?. This also is secure, meaning it will protect against SQL Injection attacks if needed.
Docs: https://docs.python.org/2/library/sqlite3.html
Related posts: Escaping chars in Python and sqlite

Try:
kword = "%"+kword+"%"
c.execute("SELECT * FROM my_lib WHERE kappa LIKE '%s'" % kword)

After trying over and over again, I got the actual solution. It seems like I can't just add the '%' to the variable like a string, but rather:
c.execute("SELECT * FROM my_lib WHERE {} LIKE '%{}%'".format(column, kword))

To convert from Python arrays to PostgreSQL quickly?

This is a follow-up question to: How to cast to int array in PostgreSQL?
I am thinking how to convert Python's datatype of array-array of signed integer into to int of PostgreSQL quickly:
import numpy as np; # use any data format of Python here
event = np.array([[1,2],[3,4]]);
where [] should be replaced by {} and surrounded by ' if manually.
In PostgreSQL, the following is accepted as the syntax of the datatype
...
FOR EACH ROW EXECUTE PROCEDURE insaft_function('{{1,2},{3,4}}');
#JohnMee's suggestion
str(event).replace('[','{').replace(']','}').replace('\n ',',')
#ErwinBrandstetter's suggestion
Stick to signed integers because it is supported by SQL standard.
Map to int, so just in PostgreSQL side:
TG_ARGV::int[]
I want to stick to this Erwin's suggestion.
Test run of simpler version of #ErwinBrandstetter's answer
I have to simplify his answer to keep it enough focused here by removing the table-name from function so just keeping the trigger for one initial table measurements:
CREATE OR REPLACE FUNCTION f_create_my_trigger(_arg0 text)
RETURNS void AS
$func$
BEGIN
EXECUTE format($$
DROP TRIGGER IF EXISTS insaft_ids ON measurements;
CREATE TRIGGER insaft_ids
AFTER INSERT ON measurements
FOR EACH ROW EXECUTE PROCEDURE insaft_function(%1$L)$$
, _arg0
);
END
$func$ LANGUAGE plpgsql;
And I run:
sudo -u postgres psql detector -c "SELECT f_create_my_trigger('[[1,2],[3,4]]');"
But get empty output:
f_create_my_trigger
---------------------
(1 row)
How can you map to int for PostgreSQL 9.4 in Python?

Setup
You want to create triggers (repeatedly?) using the same trigger function like outlined in my related answer on dba.SE. You need to pass values to the trigger function to create multiple rows with multiple column values, hence the two-dimensional array. (But we can work with any clearly defined string!)
The only way to pass values to a PL/pgSQL trigger function (other than column values of the triggering row) are text parameters, which are accessible inside the function as 0-based array of text in the special array variable TG_ARGV[]. You can pass a variable number of parameters, but we discussed a single string literal representing your 2-dimenstional array earlier.
Input comes from a 2-dimensional Python array with signed integer numbers, that fits into the Postgres type integer. Use the Postgres type bigint to cover unsigned integer numbers, as commented.
The text representation in Python looks like this:
[[1,2],[3,4]]
Syntax for a Postgres array literal:
{{1,2},{3,4}}
And you want to automate the process.
Full automation
You can concatenate the string for the CREATE TRIGGER statement in your client or you can persist the logic in a server-side function and just pass parameters.
Demonstrating an example function taking a table name and the string that's passed to the trigger function. The trigger function insaft_function() is defined in your previous question on dba.SE.
CREATE OR REPLACE FUNCTION f_create_my_trigger(_tbl regclass, _arg0 text)
RETURNS void
LANGUAGE plpgsql AS
$func$
BEGIN
EXECUTE format($$
DROP TRIGGER IF EXISTS insaft_%1$s_ids ON %1$s;
CREATE TRIGGER insaft_%1$s_ids
AFTER INSERT ON %1$s
FOR EACH ROW EXECUTE PROCEDURE insaft_function(%2$L)$$
, _tbl
, translate(_arg0, '[]', '{}')
);
END
$func$;
Call:
SELECT f_create_my_trigger('measurements', '[[1,2],[3,4]]');
Or:
SELECT f_create_my_trigger('some_other_table', '{{5,6},{7,8}}');
db<>fiddle here
Old sqlfiddle
Now you can pass either [[1,2],[3,4]] (with square brackets) or {{1,2},{3,4}} (with curly braces). Both work the same. translate(_arg0, '[]', '{}' transforms the first into the second form.
This function drops a trigger of the same name if it exists, before creating the new one. You may want to drop or keep this line:
DROP TRIGGER IF EXISTS insaft_%1$s_ids ON %1$s;
This runs with the privileges of the calling DB role. You could make it run with superuser (or any other) privileges if need be. See:
Is there a way to disable updates/deletes but still allow triggers to perform them?
There are many ways to achieve this. It depends on exact requirements.
Explaining format()
format() and the data type regclass help to safely concatenate the DDL command and make SQL injection impossible. See:
Table name as a PostgreSQL function parameter
The first argument is the "format string", followed by arguments to be embedded in the string. I use dollar-quoting, which is not strictly necessary for the example, but generally a good idea for concatenating long strings containing single-quotes: $$DROP TRIGGER ... $$
format() is modeled along the C function sprintf. %1$s is a format specifier of the format() function. It means that the first (1$) argument after the format string is inserted as unquoted string (%s), hence: %1$s. The first argument to format is _tbl in the example - the regclass parameter is rendered as legal identifier automatically, double-quoted if necessary, so format() does not have to do more. Hence just %s, not %I (identifier). Read the linked answer above for details.
The other format specifier in use is %2$L: Second argument as quoted string literal.
If you are new to format(), play with these simple examples to understand:
SELECT format('input -->|%s|<-- here', '[1,2]')
, format('input -->|%s|<-- here', translate('[1,2]', '[]', '{}'))
, format('input -->|%L|<-- here', translate('[1,2]', '[]', '{}'))
, format('input -->|%I|<-- here', translate('[1,2]', '[]', '{}'));
And read the manual.

Use of SQL - IN in python

I have a list of ids in python. For example:
x = [1,2,3,4,5,6]
And i want to select a list of records in my (mysql ) data-base under the condition that the ids of these records are in x. something like below:
SELECT * FROM mytable WHERE id IN x
but I don't know who I can do this in python. I have seen some examples using %s in their sql string. However this does not work when the variable is a list. does anyone know how I can do this?
Thanks

Try something like this:
'(%s)' % ','.join(map(str,x))
This will give you a string that you could use to send to MySql as a valid IN clause:
(1,2,3,4,5,6)

Well, if all of those are known to be numbers of good standing, then you can simply call
"SELECT * FROM mytable WHERE ID IN ({0})".format(','.join(x))
If you know that they are numbers but any of them might have been from the user, then I might use:
"SELECT * FROM mytable WHERE ID IN ({0})".format(','.join(list(map(int,x))))
format will perform the replacement at the appropriate index. join is used so that you don't have the []. list converts everything to a list, map applies a function to a list/tuple/iterable. In Python 3, however, map returns a generator, which isn't what you need. You need a list. So, list(map(x,y)) will return the list form of map(x,y).