I am trying to select multiple columns, but not all of the columns, from the database. All of the columns I want to select are going to start with "word".
So in pseudocode I'd like to do this:
SELECT "word%" from searchterms where onstate = 1;
More or less. I am not finding any documentation on how to do this - is it possible in MySQL? Basically, I am trying to store a list of words in a single row, with an identifier, and I want to associate all of the words with that identifier when I pull the records. All of the words are going to be joined as a string and passed to another function in an array/dictionary with their identifier.
I am trying to make as FEW database calls as possible to keep speedy code.
Ok, here's another question for you guys:
There are going to be a variable number of columns with the name "word" in them. Would it be faster to do a separate database call for each row, with a generated Python query per row, or would it be faster to simply SELECT *, and only use the columns I needed? Is it possible to say SELECT * NOT XYZ?
No, SQL doesn't provide you with any syntax to do such a select.
What you can do is ask MySQL for a list of column names first, then generate the SQL query from that information.
SELECT column_name
FROM information_schema.columns
WHERE table_name = 'your_table'
AND column_name LIKE 'word%'
let's you select the column names. Then you can do, in Python:
"SELECT * FROM your_table WHERE " + ' '.join(['%s = 1' % name for name in columns])
Instead of using string concatenation, I would recommend using SQLAlchemy instead to do the SQL generating for you.
However, if all you are doing is limit the number of columns there is no need to do a dynamic query like this at all. The hard work for the database is selecting the rows; it makes little difference to send you 5 columns out of 10, or all 10.
In that case just use a "SELECT * FROM ..." and use Python to pick out the columns from the result set.
No, you cannot dynamically produce the list of columns to be selected. It will have to be hardcoded in your final query.
Your current query would produce a result set with one column and the value of that column would be the string "word%" in all rows that satisfy the condition.
You can generate the list of column names first by using
SHOW COLUMNS IN tblname LIKE "word%"
Then loop through the cursor and generate SQL statement uses all the columns from the query above.
"SELECT {0} FROM searchterms WHERE onstate = 1".format(', '.join(columns))
This could be helpful: MySQL wildcard in select
In conclusion it is not possible in MySQL directly.
What you could do as a dirty workaround is get all the column names from the table with an initial query (http://dev.mysql.com/doc/refman/5.0/en/show-columns.html) and then compare in python if the name matches your pattern. Afterwards you could do the MySQL select statement with the found column names like this:
SELECT word1, word2, word3 from searchterms where onstate = 1;
Related
When I'm trying to format an SQL statement this way:
cursor.execute('SELECT (%s) FROM table WHERE id = 12345', (column,))
it doesn't work properly. Instead of returning values from the specified column, it just returns the name of the column. Any way to fix that?
It only works with formatted string, but according to reviews it is not the safest approach
You can't bind column names like that -you're binding a string literal with that name.
If you want to dynamically set the column names, you'll have to resort to string manipulation of some sort, e.g.:
cursor.execute(f'SELECT {column} FROM table WHERE id = 12345')
Of course if column is created by user-input, you'll have to sanitize it somehow.
I have been searching for quite some time but did not succeed to figure out how to select the id column from a table where either of the given other columns is not null.
I tried tied a nested query like:
SELECT id, name FROM spam_table WHERE (SELECT c.name FROM pragma_table_info('spam_table') c WHERE c.name LIKE '%ham%' OR c.name LIKE '%eggs%') IS NOT NULL
Is there any way that the inner PRAGMA returns the corresponding column names to be used for the outer query. And how assure the outer query is been put together using OR
Cheers.
Is there any way that the inner PRAGMA returns the corresponding
column names to be used for the outer query.
No. There is no "dynamic" column names (or table names) in sqlite.
One way to do it in python:
execute the pragma_table_info select
fetch the results
iterate the results and create the desired sql string
execute the created sql string
Thanks #DinoCoderSaurus for pointing out that there is no dynamic column names
The code I am using need some more pythonic style but in fact it I am running
for a in spam_table[0]: # Tables Header from the pragma_table_info(spam_table)
for i in eggs: # Search terms given by the UI
if i in a:
spam_eggs.append(self.spam_table[0].index(a))
Now I know which columns to check to extract the id
I am trying to create an SQL statement in Python that adds a column with a specific name to a table, but I am getting the error you see in the tittle.
I'm currently using this code.
columnname = 'note'
cur.execute("ALTER TABLE my_table_name ADD COLUMN %s MEDIUMTEXT", (columnname,))
That should create a column in my_table_name named note but it gives me this error:
MySQLdb._exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''note' MEDIUMTEXT' at line 1")
I know I should use normal string interpolation. But I want to keep adding columns like, column1 then column2 then column3 if the other ones are already storing data.
You are trying to use SQL "bind variables", also known as parameter substitution, for a column name. You Can't Do That™, You must specify column names (and table names) directly in your SQL statement.
Use normal string interpolation to construct your statements. Try something like this:
columnname = 'note'
cur.execute(f"ALTER TABLE my_table_name ADD COLUMN {columname} MEDIUMTEXT")
Or if you're using old-timey Python 2, use this:
columnname = 'note'
cur.execute("ALTER TABLE my_table_name ADD COLUMN " + columname + " MEDIUMTEXT")
You can, obviously, do this in a loop of some kind.
(Careful: adding columns to tables with many rows can be time consuming. And, adding columns to get around a UNIQUE index restriction is a very strange way to design a table. Instead, maybe, use a column without a UNIQUE index.)
If your columname values come from an end user, you must validate them before you use them. Otherwise, cybercreeps.
I have a DB with ID/Topic/Definition columns. When a select query is made, with possibly hundreds of parameters, I would like the fetchall call to also return the topic of any non-existent rows with a default text (i.e. "Not Found").
I realize this could be done in a loop, but that would query the DB every cycle and have a significant performance hit. With the parameters joined by "OR" in a single select statement the search is nearly instantaneous.
Is there a way to get a return of the query (topic) with default text for non-existent rows in SQLite?
Table Structure (named "dictionary")
ID|Topic|Definition
1|wd1|def1
2|wd3|def3
Sample Query
SELECT Topic,Definition FROM dictionary WHERE Topic = "wd1" or Topic = "wd2" or topic = "wd3"'
Desired Return
[(wd1, def1), (wd2, "Not Found"), (wd3, def3)]
To get data like wd2 out of a query, such data must be in the database in the first place.
You could put it into a temporary table, or use a common table expression.
To include rows without a match, use an outer join:
WITH IDs(ID) AS ( VALUES ('wd1'), ('wd2'), ('wd3') )
SELECT Topic,
IFNULL(Definition, 'Not Found') AS Definition
FROM IDs
LEFT JOIN dictionary USING (ID);
This question already has answers here:
imploding a list for use in a python MySQLDB IN clause
(8 answers)
Closed 1 year ago.
I want to insert a list in my database but I can't.
Here is an example of what I need:
variable_1 = "HELLO"
variable_2 = "ADIOS"
list = [variable_1,variable_2]
INSERT INTO table VALUES ('%s') % list
Can something like this be done? Can I insert a list as a value?
When I try it, an error says that is because of an error in MySQL syntax
The answer to your original question is: No, you can't insert a list like that.
However, with some tweaking, you could make that code work by using %r and passing in a tuple:
variable_1 = "HELLO"
variable_2 = "ADIOS"
varlist = [variable_1, variable_2]
print "INSERT INTO table VALUES %r;" % (tuple(varlist),)
Unfortunately, that style of variable insertion leaves your code vulnerable to SQL injection attacks.
Instead, we recommend using Python's DB API and building a customized query string with multiple question marks for the data to be inserted:
variable_1 = "HELLO"
variable_2 = "ADIOS"
varlist = [variable_1,variable_2]
var_string = ', '.join('?' * len(varlist))
query_string = 'INSERT INTO table VALUES (%s);' % var_string
cursor.execute(query_string, varlist)
The example at the beginning of the SQLite3 docs shows how to pass arguments using the question marks and it explains why they are necessary (essentially, it assures correct quoting of your variables).
Your question is not clear.
Do you want to insert the list as a comma-delimited text string into a single column in the database? Or do you want to insert each element into a separate column? Either is possible, but the technique is different.
Insert comma-delimited list into one column:
conn.execute('INSERT INTO table (ColName) VALUES (?);', [','.join(list)])
Insert into separate columns:
params = ['?' for item in list]
sql = 'INSERT INTO table (Col1, Col2. . .) VALUES (%s);' % ','.join(params)
conn.execute(sql, list)
both assuming you have established a connection name conn.
A few other suggestions:
Try to avoid INSERT statements that do not list the names and order of the columns you're inserting into. That kind of statement leads to very fragile code; it breaks if you add, delete, or move columns around in your table.
If you're inserting a comma-separted list into a single-field, that generally violates principals of database design and you should use a separate table with one value per record.
If you're inserting into separate fields and they have names like Word1 and Word2, that is likewise an indication that you should be using a separate table instead.
Never use direct string substitution to create SQL statements. It will break if one of the values is, for example o'clock. It also opens you to attacks by people using SQL injection techniques.
You can use json.dumps to convert a list to json and write the json to db.
For example:
insert table example_table(column_name) values(json.dumps(your_list))