Develop Reference

Python MySQL connector SELECT statement issue

When I'm trying to format an SQL statement this way:
cursor.execute('SELECT (%s) FROM table WHERE id = 12345', (column,))
it doesn't work properly. Instead of returning values from the specified column, it just returns the name of the column. Any way to fix that?
It only works with formatted string, but according to reviews it is not the safest approach

You can't bind column names like that -you're binding a string literal with that name.
If you want to dynamically set the column names, you'll have to resort to string manipulation of some sort, e.g.:
cursor.execute(f'SELECT {column} FROM table WHERE id = 12345')
Of course if column is created by user-input, you'll have to sanitize it somehow.

How do I upsert all the rows from one table into another table using Postgres?

I am working in Python, using Pandas to pull data from a TSV, convert it to a data frame, then syncing that data frame to a temp table in postgres using df.to_sql. That process works great.
However, once that table exists, I want to move all the rows from that table to the permanent table. The two tables will always be identical. The permanent table has a unique index, so if the id already exists it should update the row instead.
Here is my attempt to upsert all rows from one table to another:
WITH moved_rows AS (
DELETE FROM my_table_tmp a
RETURNING a.*
)
INSERT INTO my_table
SELECT * FROM moved_rows
ON CONFLICT ("unique_id") DO
UPDATE SET
Field_A = excluded.Field_A,
Field_B = excluded.Field_B,
Field_C = excluded.Field_C
Unfortunately, when I run this, I get this error:
psycopg2.errors.UndefinedColumn: column excluded.field_a does not exist
LINE 10: Field_A = excluded.Field_A,
^
HINT: Perhaps you meant to reference the column "excluded.Field_A".
But in fact, that column does exist. What am I missing here? I've tried removing Field_A from the set and then I get the same error with Field_B.

Answering my own question here - the issue is that Postgres ignores capitalization unless it's quoted.
This was not clear in the example I posted because I obscured the naming of the fields I was working with. I've updated them now to show the issue.
In order to fix this you need to wrap your field names with double quotes. E.g. "Field_A" = excluded."Field_A"

Select Query: Error Code 1292 - Truncated incorrect DOUBLE value

I'm attempting to retrieve all the data from a column in mysql by having the user input which table and column the data is through the mysqlconnector library in python. When I ran the query through python no data would show up and then when I ran it through Phpmyadmin I would get these errors:
Warning: #1292 Truncated incorrect DOUBLE value: 'Matisse'
Warning: #1292 Truncated incorrect DOUBLE value: 'Picasso'
Warning: #1292 Truncated incorrect DOUBLE value: 'van Gogh'
Warning: #1292 Truncated incorrect DOUBLE value: 'Deli'
I found the query only works for columns that are integer based and does not work for date-time or varchar columns (The L_Name one from which the query doesn't work is varchar(25).
Here is the query:
SELECT * FROM `artist` WHERE L_Name
After the query is run and throws those errors, the query changes to this by itself:
SELECT * FROM `artist` WHERE 1
This new query returns the whole table and all of its columns and rows but of course all I want is for it to simply return the single column.
EDIT: To clarify, the point of running the SELECT * FROM `artist` WHERE L_Name
query is to bring up the whole list of values in that column for that table. This is just one case and there's many other cases like if the user wanted to search up a specific record from the art_show table and then look at all the values in the column of the gallery location.

I don't think its the error thats the problem since you did varchar, maybe check your python code?

Figured out the solution. My Python had an issue where it would only print the first value due to how I set up the print statement. Fixed it by changing the query on the 3rd line and also changing the print statement from print(rec[0]) to print(rec)
def show_entries(table, column):
print(f"Here's the records in the table {table} and the column {column}.")
mycursor.execute(f"SELECT {column} FROM {table}")
myresult = mycursor.fetchall()
for rec in myresult:
print(rec)

sqlite3 prints without explicit command

I'm trying to get a rowid if a data row exists. What I have now is
row_id = self.dbc.cursor.execute("SELECT ROWID FROM Names where unq_id=?",(namesrow['unq_id'],)).fetchall()[0][0]
where namesrow is a dictionary of column names with corresponding data to fill into the table. The problem is this prints 'unq_id' when runs and I'm not sure how to get rid of it.
I'm using sqlite3 and python. Any help's appreciated!

quoting the sqlite documentation:
With one exception noted below, if a rowid table has a primary key
that consists of a single column and the declared type of that column
is "INTEGER" in any mixture of upper and lower case, then the column
becomes an alias for the rowid.
So if your unq_id is the integer primary key in this table, then rowid and unq_id will be the same field.

Wildcards in column name for MySQL

I am trying to select multiple columns, but not all of the columns, from the database. All of the columns I want to select are going to start with "word".
So in pseudocode I'd like to do this:
SELECT "word%" from searchterms where onstate = 1;
More or less. I am not finding any documentation on how to do this - is it possible in MySQL? Basically, I am trying to store a list of words in a single row, with an identifier, and I want to associate all of the words with that identifier when I pull the records. All of the words are going to be joined as a string and passed to another function in an array/dictionary with their identifier.
I am trying to make as FEW database calls as possible to keep speedy code.
Ok, here's another question for you guys:
There are going to be a variable number of columns with the name "word" in them. Would it be faster to do a separate database call for each row, with a generated Python query per row, or would it be faster to simply SELECT *, and only use the columns I needed? Is it possible to say SELECT * NOT XYZ?

No, SQL doesn't provide you with any syntax to do such a select.
What you can do is ask MySQL for a list of column names first, then generate the SQL query from that information.
SELECT column_name
FROM information_schema.columns
WHERE table_name = 'your_table'
AND column_name LIKE 'word%'
let's you select the column names. Then you can do, in Python:
"SELECT * FROM your_table WHERE " + ' '.join(['%s = 1' % name for name in columns])
Instead of using string concatenation, I would recommend using SQLAlchemy instead to do the SQL generating for you.
However, if all you are doing is limit the number of columns there is no need to do a dynamic query like this at all. The hard work for the database is selecting the rows; it makes little difference to send you 5 columns out of 10, or all 10.
In that case just use a "SELECT * FROM ..." and use Python to pick out the columns from the result set.

No, you cannot dynamically produce the list of columns to be selected. It will have to be hardcoded in your final query.
Your current query would produce a result set with one column and the value of that column would be the string "word%" in all rows that satisfy the condition.

You can generate the list of column names first by using
SHOW COLUMNS IN tblname LIKE "word%"
Then loop through the cursor and generate SQL statement uses all the columns from the query above.
"SELECT {0} FROM searchterms WHERE onstate = 1".format(', '.join(columns))

This could be helpful: MySQL wildcard in select
In conclusion it is not possible in MySQL directly.
What you could do as a dirty workaround is get all the column names from the table with an initial query (http://dev.mysql.com/doc/refman/5.0/en/show-columns.html) and then compare in python if the name matches your pattern. Afterwards you could do the MySQL select statement with the found column names like this:
SELECT word1, word2, word3 from searchterms where onstate = 1;