So I'm attempting to write my own sqlite3 to postgresql migration script, which takes all the tables from one database to another.
I'm currently seeing the following syntax error:
ERROR: syntax error at or near "'name'"
LINE 1: UPDATE django_site SET ('name', 'domain')=('127.0.0.1:8080',...
^
It's not quite liking ('name', 'domain') ... I'm perhaps thinking there's something subtle I'm missing...
('name', 'domain') is generated from the following line of code:
col = tuple([desc[0] for desc in self.cur_sql3.description])
i.e., taking all of the column names as a generated list then converting to tuple type.
And the SQL query is currently being built as such:
cur_psql.execute("UPDATE {0} SET {1}={2} WHERE id={3}".format(table[0], col[1:], row[1:], col[0]))
table[0] is the table name, col[1:] is everything but the primary id key, row[1:] are all the row values minus the primary key, and col[0] is the primary key value, e.g., 1,2,3,4 or 5 etc
Any obvious tips to avoid this issue?
You'll need to split this into separate SQL statements or you'll need to do some kind of looping over your column value pairs. I would need more information to write the corrected Python code, but your SQL should look something like this:
UPDATE table SET col = val WHERE id = id_val
You're doing something like:
UPDATE table SET ('col1', 'col2') = ('val1', 'val2')
If you want to update multiple columns in one statement, you'd do this:
UPDATE table SET col1 = val1, col2 = val2 WHERE id = id_val
Keep in mind you'll need to escape your values if they're character columns (but not if they're numeric), like so:
UPDATE table SET col1 = 'character value', col2 = 123 WHERE id = 1
Ultimately, this should be accomplished with bind variables rather than string formatting, as string formatting can expose your code to SQL injection attacks (might not be a concern if this is strictly a utility script that you're running yourself but it's a good habit to get into regardless). You can find the documentation for bind variables in psycopg2 here: http://initd.org/psycopg/docs/usage.html#query-parameters
Following my advice above would result in something like this:
set_clauses = ",".join(["{} = %s".format(col) for col in col[1:]])
query = "update {} set {} where id = {}".format(table[0], set_clauses, row[0])
cur_psql.execute(query, row[1:])
Related
Let's say mytable has 5 columns, id being the first one.
Is it possible to do an UPDATE without hardcoding the column names?
UPDATE mytable VALUES(4, "hello", 31.12, 4141.12, "gjhg") WHERE id = 4
I haven't found it in most tutorials.
Desired use case with sqlite3:
row = (4, "hello", 31.12, 4141.12, "gjhg")
c.execute('UPDATE mytable VALUES(?) WHERE id = ?', row, row[0])
As far as I know you cannot do that. According to https://sqlite.org/lang_update.html
However there are other ways of writing the query:
https://stackoverflow.com/a/13482298/7791653
You could generate a query yourself.
First you select all the column names and do a for loop where you build a String like "column1, column1, column1" and add that to the appropriate place of the query. All you have to do then is something like
"UPDATE users
SET (" + generatedQueryPart = ")
= ('value1', 'value2', 'value3')
WHERE some_condition ";
Hope this gave you some more information.
I am new to python and pyodbc
I try to print the first a row from a table from a progress openedge database. (Windows 7) Here is the code block that is not running:
cursor.execute("select my-nr, my-dt-my from mytable")
row = cursor.fetchone()
print(row.my-nr, row.my-dt-my)
This gives errors undefined name: 'nr'
undefined name 'dt'
undefined name 'my'
I guess it has something to do with the minus - symbols behind the dot . in print(row.my-nr, row.my-dt-my)
It was easy to print out the table names and column names from the database earlier but for some reason printing out rows is harder.
Any ideas how to get the rows printed?
pyodbc allows us to reference values in a pyodbc.Row object using the form row.column_name provided that the column names are legal Python identifiers. So, for example, we can do something like
row = crsr.fetchone()
print(row.city)
to print the value of the "city" column. Unfortunately, my-nr is not a legal Python identifier so if we try to print the value of the "my-nr" column using ...
row = crsr.fetchone()
print(row.my-nr) # error
... Python parses that as "row.my minus nr" where row.my would be interpreted as a column in the Row object and nr would be interpreted as a Python variable.
To work around the issue we can grab a list of the column names, merge those names with the row values into a dictionary, and then refer to the values in the dictionary:
crsr.execute(sql)
col_names = [x[0] for x in crsr.description]
row = crsr.fetchone()
row_as_dict = dict(zip(col_names, row))
print(row_as_dict['my-nr']) # no error
The most simple solution I can think of is this. First, columns containing hyphens need to be quoted in OpenEdge (see here). Second, you can alias the columns so they can be referenced as valid Python attributes. You'll need to do something like this:
cursor.execute('select "my-nr" as mynr, "my-dt-my" as mydtmy from mytable')
row = cursor.fetchone()
print(row.mynr, row.mydtmy)
Good luck!
I beleive that you need to change the variable names of the database, make sure they don't contain any '-' characters.
Variables can not contain characters reserved by python. For example you have to avoid hyphens(-), exclamation marks (!), colons (:) and so on.
According to this answer it seems like underscore (_) is the only character allowed in variable names.
I have a query where I join multiple tables with similar column names. To disambiguate them, I want to suffix the table name to the column name like: <column_name>_<table_name>. There are hundreds of columns in each table, so I would like to do it programmatically.
Is there a way to do something like?
sa.select([
table1.c.suffix('_1'),
table2.c.suffix('_2')]).
select_from(table1.join(table2, table1.c.id == table2.c.id))
You want to use the label keyword:
sa.select([
table1.c.column_name.label('_1'),
table2.c.column_name.label('_2')]).
select_from(table1.join(table2, table1.c.id == table2.c.id))
This will allow you to have the same column name from different tables.
If you have a table that is dynamic, or tons of columns, your best bet will be to do something like this:
pseudo code:
select * from information_schema.columns where table_name = 'my_table"
get the results from a query
return_columns = []
counter = 0
for r in results:
counter += 1
return_columns.append("`table_name`.`" + r.column_name + "` as col_{}".format(counter))
Creating dynamic sql will require you to do a bit of building out. I do this in my application all the time. Except I don't use information schema. I have a table which has my column names in it.
This should lead you in the right direction.
I have a database table with multiple fields which I am querying and pulling out all data which meets certain parameters. I am using psycopg2 for python with the following syntax:
cur.execute("SELECT * FROM failed_inserts where insertid='%s' AND site_failure=True"%import_id)
failed_sites= cur.fetchall()
This returns the correct values as a list with the data's integrity and order maintained. However I want to query the list returned somewhere else in my application and I only have this list of values, i.e. it is not a dictionary with the fields as the keys for these values. Rather than having to do
desiredValue = failed_sites[13] //where 13 is an arbitrary number of the index for desiredValue
I want to be able to query by the field name like:
desiredValue = failed_sites[fieldName] //where fieldName is the name of the field I am looking for
Is there a simple way and efficient way to do this?
Thank you!
cursor.description will give your the column information (http://www.python.org/dev/peps/pep-0249/#cursor-objects). You can get the column names from it and use them to create a dictionary.
cursor.execute('SELECT ...')
columns = []
for column in cursor.description:
columns.append(column[0].lower())
failed_sites = {}
for row in cursor:
for i in range(len(row)):
failed_sites[columns[i]] = row[i]
if isinstance(row[i], basestring):
failed_sites[columns[i]] = row[i].strip()
The "Dictionary-like cursor", part of psycopg2.extras, seems what you're looking for.
Sqlite table structure:
id sno
1 100
2 200
3 300
4 400
conn=sqlite3.connect('test.sqlite')
c=conn.cursor()
c.execute("select * from abc")
mysel=c.execute("select * from abc where [id] = 1 ")
out put is:
1 100
its not printing id and sno i.e the First line of the table
how i can print First Line of table along with any kind of selection
please help
ID and sno are not data, they are part of your table structure (the column names).
If you want to get the names of the columns you need to do something like
connection = sqllite3.connect('test.sqlite')
cursor = connection.execute('select * from abc')
names = list(map(lambda x: x[0], cursor.description))
There isn't really a 'first line' containing the column names, that's just something the command line client prints out by default to help you read the returned records.
A dbapi2 conforming cursor has an attribute description, which is a list of tuples containing information about the data returned by the last query. The first element of each tuple will be the name of the column, so to print the column names, you can do something similar to:
c.execute("select * from abc")
print(tuple(d[0] for d in c.description))
for row in c:
print(row)
This will just print a tuple representation of the names and the records.
If you want to obtain details on the table you can use the following statement
PRAGMA table_info('[your table name]')
This will return a list of tuple with each tuple containing informations about a column
You will still have to add it to the data collected using the SELECT statement
When you write ... WHERE id = 1, you get only that particular record.
If you want to also get the first record, you have to tell SQLite that you want it:
SELECT id, sno FROM abc WHERE id = 'id'
UNION ALL
SELECT id, sno FROM abc WHERE id = 1
And when you already know what this particular subquery returns, you do not even need to bother with searching the table (and thus do not need to actually store the column names in the table):
SELECT 'id', 'sno'
UNION ALL
SELECT id, sno FROM abc WHERE id = 1