Develop Reference

Combine two SQL lite databases with Python

I have the following code in python to update db where the first column is "id" INTEGER PRIMARY KEY AUTOINCREMENT UNIQUE:
con = lite.connect('test_score.db')
with con:
cur = con.cursor()
cur.execute("INSERT INTO scores VALUES (NULL,?,?,?)", (first,last,score))
item = cur.fetchone()
on.commit()
cur.close()
con.close()
I get table "scores" with following data:
1,Adam,Smith,68
2,John,Snow,76
3,Jim,Green,88
Two different users (userA and userB) copy test_score.db and code to their computer and use it separately.
I get back two db test_score.db but now with different content:
user A test_score.db :
1,Adam,Smith,68
2,John,Snow,76
3,Jim,Green,88
4,Jim,Green,91
5,Tom,Hanks,15
user A test_score.db :
1,Adam,Smith,68
2,John,Snow,76
3,Jim,Green,88
4,Chris,Prat,99
5,Tom,Hanks,09
6,Tom,Hanks,15
I was trying to use
insert into AuditRecords select * from toMerge.AuditRecords;
to combine two db into one but failed as the first column is a unique id. Two db have now the same ids but with different or the same data and merging is failing.
I would like to find unique rows in both db (all values different ignoring id) and merge results to one full db.
Result should be something like this:
1,Adam,Smith,68
2,John,Snow,76
3,Jim,Green,88
4,Jim,Green,91
5,Tom,Hanks,15
6,Chris,Prat,99
7,Tom,Hanks,09
I can extract each value one by one and compare but want to avoid it as I might have longer rows in the future with more columns.
Sorry if it is obvious and easy questions, I'm still learning. I tried to find the answer but failed, please point me to answer if it already exists somewhere else. Thank you very much for your help.

You need to define the approach to resolve duplicated rows. Will consider the max score? The min? The first one?
Considering the table AuditRecords has all the lines of both User A and B, you can use GROUP BY to deduplicate rows and use an aggregation function to resolve the score:
insert into
AuditRecords
select
id,
first_name,
last_name,
max(score) as score
from
toMerge.AuditRecords
group by
id,
first_name,
last_name;

For this requirement you should have defined a UNIQUE constraint for the combination of the columns first, last and score:
CREATE TABLE AuditRecords(
id INTEGER PRIMARY KEY AUTOINCREMENT,
first TEXT,
last TEXT,
score INTEGER,
UNIQUE(first, last, score)
);
Now you can use INSERT OR IGNORE to merge the tables:
INSERT OR IGNORE INTO AuditRecords(first, last, score)
SELECT first, last, score
FROM toMerge.AuditRecords;
Note that you must explicitly define the list of the columns that will receive the values and in this list the id is missing because its value will be autoincremented by each insertion.
Another way to do it without defining the UNIQUE constraint is to use EXCEPT:
INSERT INTO AuditRecords(first, last, score)
SELECT first, last, score FROM toMerge.AuditRecords
EXCEPT
SELECT first, last, score FROM AuditRecords

SQL syntax error with psycopg2 (python)

So I'm attempting to write my own sqlite3 to postgresql migration script, which takes all the tables from one database to another.
I'm currently seeing the following syntax error:
ERROR: syntax error at or near "'name'"
LINE 1: UPDATE django_site SET ('name', 'domain')=('127.0.0.1:8080',...
^
It's not quite liking ('name', 'domain') ... I'm perhaps thinking there's something subtle I'm missing...
('name', 'domain') is generated from the following line of code:
col = tuple([desc[0] for desc in self.cur_sql3.description])
i.e., taking all of the column names as a generated list then converting to tuple type.
And the SQL query is currently being built as such:
cur_psql.execute("UPDATE {0} SET {1}={2} WHERE id={3}".format(table[0], col[1:], row[1:], col[0]))
table[0] is the table name, col[1:] is everything but the primary id key, row[1:] are all the row values minus the primary key, and col[0] is the primary key value, e.g., 1,2,3,4 or 5 etc
Any obvious tips to avoid this issue?

You'll need to split this into separate SQL statements or you'll need to do some kind of looping over your column value pairs. I would need more information to write the corrected Python code, but your SQL should look something like this:
UPDATE table SET col = val WHERE id = id_val
You're doing something like:
UPDATE table SET ('col1', 'col2') = ('val1', 'val2')
If you want to update multiple columns in one statement, you'd do this:
UPDATE table SET col1 = val1, col2 = val2 WHERE id = id_val
Keep in mind you'll need to escape your values if they're character columns (but not if they're numeric), like so:
UPDATE table SET col1 = 'character value', col2 = 123 WHERE id = 1
Ultimately, this should be accomplished with bind variables rather than string formatting, as string formatting can expose your code to SQL injection attacks (might not be a concern if this is strictly a utility script that you're running yourself but it's a good habit to get into regardless). You can find the documentation for bind variables in psycopg2 here: http://initd.org/psycopg/docs/usage.html#query-parameters
Following my advice above would result in something like this:
set_clauses = ",".join(["{} = %s".format(col) for col in col[1:]])
query = "update {} set {} where id = {}".format(table[0], set_clauses, row[0])
cur_psql.execute(query, row[1:])

How to concatenate ORDER BY in sql

I am working on a GUI gird and sql. I have two GUI buttons that can be clicked depending on what order the user wants the information. The order can be by Employee Last_name or First_name, but not both. I am not sure how to use that. I am suppose to use concatenation, but am not sure how to.
Below is what I tried to do:
def sort_employees(self, column):
try:
cursor = db.cursor()
display="SELECT * FROM company ORDER BY '%" + column + "%' "
cursor.execute(display)
entry = cursor.fetchall()
self.display_rows(entry)
Also, the code works fine if I only have on entry:
display="SELECT * FROM company ORDER BY Last_name"

Not sure why you have % in your query string, it's possible you're confusing it with the %s syntax for string formatting.
display = "SELECT * FROM company ORDER BY '%" + column + "%' "
It seems what you want is more like this:
display = "SELECT * FROM company ORDER BY " + column
Or, as I prefer:
display = 'SELECT * FROM company ORDER BY {column}'.format(column=column)
Of course be careful creating queries like this, you're exposed to SQL security vulnerabilities.
It's better to use a parametrised query instead of string interpolation/concatenation, but I don't know which database interface you're using, but it's easy to find that by searching the docs.

In SQL, the ORDER BY clause takes, as arguments, a list of column names:
--correct
ORDER BY firstname, lastname, age
It can also take function outputs:
--correct, sort names beginning with a Z first
ORDER BY CASE WHEN first name LIKE 'Z%' THEN 1 ELSE 2 END, firstname
In some db, putting an integer ordinal on will sort by that column, numbered from the left, starting with 1:
--correct, sort by 3rd column then first
ORDER BY 3,1
It does not take a list of strings that happen to contain column names:
--incorrect - not necessarily a syntax error but will not sort by any named column
ORDER BY 'firstname', 'lastname', 'age'
Nor does it take a string of csv column names:
--incorrect - again not necessarily a syntax error but won't sort on any of the named columns
ORDER BY 'firstname, lastname, age'
Your code falls into the latter categories: you're turning the column name into a string. This is wrong. The "not working sql" and the "working sql" are very different. Print the result of he concatenation to screen and look at them if you're having a hard time seeing it from the code

Elegant way to update multiple values for Sqlite3 Python

I have an Sqlite3 database called MYTABLE like this:
My objective is to update the values of COUNT column by simply adding the existing value with the new value.
There will be two inputs that I will recieve:
Firstly, a list of IDs that I need to update for COUNT column.
For example: ['1','3','2','5']
And Secondly, the number of count to be added to the every IDs in
the list above.
So far, the best I can come up with is:
#my input1, the list of IDs that need updating
id_list = ['1','2','5','3']
#my input2, the value to be added to the existing count value
new_count = 3
#empty list to store the original count values before updating
original_counts = []
#iterate through the input1 and retrieve the original count values
for item in id_list :
cursor = conn.execute("SELECT COUNT from MYTABLE where ID=?",[item])
for row in cursor:
original_counts.append(row[0])
#iterate through the input1 and update the count values
for i in range(len(id_list )):
conn.execute("UPDATE MYTABLE set COUNT = ? where ID=?",[original_counts[i]+new_count ,mylist[i])
Is there better/more elegent and more efficient way to achieve what I want?
UPDATE 1:
I have tried this based on N Reed's answer(not exactly the same) like this and it worked!
for item in mylist:
conn.execute("UPDATE MYTABLE set VAL=VAL+? where ID=?",[new_count,item])
Take Away for me is we can update a value in sqlite3 based on it's current value(which I didn't know)

You want to create a query that looks like this:
UPDATE MYTABLE set COUNT = COUNT + 5 where ID in (1, 2, 3, 4)
I don't know python that well, but you probably want code in python something like:
conn.execute("UPDATE MYTABLE set COUNT = COUNT + ? where ID in (?)", new_count , ",".join(mylist))
Keep in mind there is a limit to the number of items you can have in the Id list with sqllite (I think it is something like 1000)
Also be very careful about sql injection when you are creating queries this way. You probably will want to make sure that all the items in mylist have already been escaped somewhere else.
I also recommend against having a column called 'count' as it is a keyword in sql.

Python Sqlite, not able to print first line

Sqlite table structure:
id sno
1 100
2 200
3 300
4 400
conn=sqlite3.connect('test.sqlite')
c=conn.cursor()
c.execute("select * from abc")
mysel=c.execute("select * from abc where [id] = 1 ")
out put is:
1 100
its not printing id and sno i.e the First line of the table
how i can print First Line of table along with any kind of selection
please help

ID and sno are not data, they are part of your table structure (the column names).
If you want to get the names of the columns you need to do something like
connection = sqllite3.connect('test.sqlite')
cursor = connection.execute('select * from abc')
names = list(map(lambda x: x[0], cursor.description))

There isn't really a 'first line' containing the column names, that's just something the command line client prints out by default to help you read the returned records.
A dbapi2 conforming cursor has an attribute description, which is a list of tuples containing information about the data returned by the last query. The first element of each tuple will be the name of the column, so to print the column names, you can do something similar to:
c.execute("select * from abc")
print(tuple(d[0] for d in c.description))
for row in c:
print(row)
This will just print a tuple representation of the names and the records.

If you want to obtain details on the table you can use the following statement
PRAGMA table_info('[your table name]')
This will return a list of tuple with each tuple containing informations about a column
You will still have to add it to the data collected using the SELECT statement

When you write ... WHERE id = 1, you get only that particular record.
If you want to also get the first record, you have to tell SQLite that you want it:
SELECT id, sno FROM abc WHERE id = 'id'
UNION ALL
SELECT id, sno FROM abc WHERE id = 1
And when you already know what this particular subquery returns, you do not even need to bother with searching the table (and thus do not need to actually store the column names in the table):
SELECT 'id', 'sno'
UNION ALL
SELECT id, sno FROM abc WHERE id = 1