I have been trying to mess around with this issue for the past week and have not been able to work around it. I have a PostgreSQL database which keeps track of players playing matches in a tournament. I am working on a function which reports the results of matches. The way it reports the results of matches is by simply updating the database (I'm not worrying about the actual reporting system at the moment).
Here is the function which does the reporting:
def reportMatch(winner, loser):
"""Records the outcome of a single match between two players.
Args:
winner: the id number of the player who won
loser: the id number of the player who lost
"""
connection = connect()
cursor = connection.cursor()
match_played = 1
insert_statement = "INSERT INTO matches (winner_id, loser_id) VALUES (%s, %s)"
cursor.execute(insert_statement, (winner, loser))
cursor.execute("INSERT INTO players (match_count) VALUES (%s) SELECT players.id FROM players where (id = winner)" (match_played,)) # here is where I have my issue at runtime
connection.commit()
cursor.execute("INSERT INTO players(match_count) SELECT (id) FROM players VALUES (%s, %s)",(match_played, loser,))
connection.commit()
connection.close
The line which is I have commented out above is where I get an error. To be more precise and pinpoint it out: cursor.execute("INSERT INTO players(match_count) VALUES (%s) SELECT (id) FROM players WHERE (id = %s)",(match_played, winner,))
The error given is the following:
File "/vagrant/tournament/tournament.py", line 103, in reportMatch
cursor.execute(insert_statement_two, (match_played, winner))
psycopg2.ProgrammingError: syntax error at or near "SELECT"
LINE 1: INSERT INTO players(match_count) VALUES (1) SELECT (id) FROM...
If it helps, here is my schema:
CREATE TABLE players (
id serial PRIMARY KEY,
name varchar(50),
match_count int DEFAULT 0,
wins int DEFAULT 0,
losses int DEFAULT 0,
bye_count int
);
CREATE TABLE matches (
winner_id serial references players(id),
loser_id serial references players(id),
did_battle BOOLEAN DEFAULT FALSE,
match_id serial PRIMARY KEY
);
I have some experience with MySQL databases, but am fairly new to PostgreSQL. I spent a lot of time poking around with guides and tutorials online but haven't had the best of luck. Any help would be appreciated!
You can't both specify VALUES and use SELECT in an INSERT statement. It's one or the other.
See the definition of INSERT in the Postgres doc for more details.
You also do not need to specify the id value, as serial is a special sequence.
That line is also lacking a comma after the INSERT string.
In order to specify particular values, you can narrow down the SELECT with a WHERE, and/or leverage RETURNING from the earlier INSERTs, but the specifics of that will depend on exactly how you want to link them together. (I'm not quite sure exactly what you're going for in terms of linking the tables together from the code above.)
I suspect using RETURNING to get 2 IDs from players that were INSERTed and using those in turn in the INSERT into matches is along the lines of what you're looking to do.
Related
Instead of using a JSON file to store data, I've decided I wanted to use a database instead. Here is how the data currently looks inside of the JSON file:
{"userID": ["reason 1", "reason 2", "reason 3"]}
I made it so that after a certain amount of time a reason is removed. For example, "reason 2" will be removed after 12 hours of it being added. However, I realised that if I terminate the process and then run it again the reason would just stay there until I manually remove it.
I've decided to use sqlite3 to make a database and have a discord.py task loop to remove it for me. How can I replicate the dictionary inside the database? Here is what I'm thinking at the moment:
c = sqlite3.connect('file_name.db')
cursor = c.cursor()
cursor.execute("""CREATE TABLE table_name (
userID text,
reason blob
)"""
Try the following table to store the reasons:
CREATE TABLE reasons (
reason_id PRIMARY KEY,
user_id,
reason,
is_visible,
created_at
)
Then the reasons could be soft deleted for every user by running:
UPDATE reasons
SET is_visible = 0
WHERE created_at + 3600 < CAST( strftime('%s', 'now') AS INT )
The example shows hiding reasons after 1 hour (3600 seconds).
The reasons can be hard deleted later by running the following query:
DELETE reasons
WHERE is_visible = 0
The soft delete comes in handy for verification and getting data back in case of a future bug in the software.
Simply use nested loops to insert each reason into a row of the table.
insert_query = """INSERT INTO table_name (userID, reason) VALUES (?, ?)"""
for user, reasons in json_data.items():
for reason in reasons:
cursor.execute(insert_query, (user, reason))
I am running this code in a method that is called in a for loop and passing to it 'text_body':
*connecting to db*
cursor.execute('CREATE TABLE IF NOT EXISTS temp_data (temp_text_body text, id integer)')
cursor.execute('INSERT INTO temp_data(temp_text_body, id) VALUES (?, 1) ON CONFLICT (id) DO UPDATE '
'SET temp_text_body = temp_text_body || (?)', (text_body, text_body))
*commiting connection*
My goal is to, on first time this is called to create a table and fill it with 'text_body', then on a second call add to the first 'text_body' second 'text_body' and so on.
I tried many combinations of this code, I was also trying to do it with UPDATE (and it was fine but UPDATE needs some data to actually start working).
When my code comes to this places it's just stop running, no error no anything. Could some please let me know where I made a mistake?
You cannot set 'ON CONFLICT' on a field which is not a primary key or which does not have a 'unique constraint'. Here's the error message I get when running your code:
sqlite3.OperationalError:
ON CONFLICT clause does not match any PRIMARY KEY or UNIQUE constraint
This in essence is because you cannot have a conflict if there is no Primary Key/ Unique constraint on that field.
Under the assumption that you want id to be a unique Primary Key, I've created a working snippet:
import sqlite3
cursor = sqlite3.connect('stack_test.db')
cursor.execute('''CREATE TABLE IF NOT EXISTS temp_data
(temp_text_body TEXT,
id INTEGER PRIMARY KEY)''')
cursor.execute('''
INSERT INTO
temp_data(temp_text_body, id)
VALUES
(?, 1)
ON CONFLICT (id) DO UPDATE
SET
temp_text_body = temp_text_body || (?)''',
("foo", "bar"))
cursor.commit()
I've also made some stylistic changes to how you write your query statements which I hope will made them a little easier for you to debug!
Best of luck with the SQLite'ing, feel free to ask if any part of my answer is unclear :)
I have a MySQL server running on a remote host. The connection to the host is fairly slow and it affects the performance of the Python code I am using. I find that using the executemany() function makes a big improvement over using an loop to insert many rows. My challenge is that for each row I insert into one table, I need to insert several rows in another table. My sample below does not contain much data, but my production data could be thousands of rows.
I know that this subject has been asked about many times in many places, but I don't see any kind of definitive answer, so I'm asking here...
Is there a way to get a list of auto generated keys that were created using an executemany() call?
If not, can I use last_insert_id() and assume that the auto generated keys will be in sequence?
Looking at the sample code below, is there a simpler or better way do accomplish this task?
What if my cars dictionary were empty? No rows would be inserted so what would the last_insert_id() return?
My tables...
Table: makes
pkey bigint autoincrement primary_key
make varchar(255) not_null
Table: models
pkey bigint autoincrement primary_key
make_key bigint not null
model varchar(255) not_null
...and the code...
...
cars = {"Ford": ["F150", "Fusion", "Taurus"],
"Chevrolet": ["Malibu", "Camaro", "Vega"],
"Chrysler": ["300", "200"],
"Toyota": ["Prius", "Corolla"]}
# Fill makes table with car makes
sql_data = list(cars.keys())
sql = "INSERT INTO makes (make) VALUES (%s)"
cursor.executemany(sql, sql_data)
rows_added = len(sqldata)
# Find the primary key for the first row that was just added
sql = "SELECT LAST_INSERT_ID()"
cursor.execute(sql)
rows = cursor.fetchall()
first_key = rows[0][0]
# Fill the models table with the car models, linked to their make
this_key = first_key
sql_data = []
for car in cars:
for model in cars[car]:
sql_data.append((this_key, car))
this_key += 1
sql = "INSERT INTO models (make_key, model) VALUES (%s, %s)"
cursor.executemany(sql, sql_data)
cursor.execute("COMMIT")
...
I have, more than once, measured about 10x speedup when batching inserts.
If you are inserting 1 row in table A, then 100 rows in table B, don't worry about the speed of the 1 row; worry about the speed of the 100.
Yes, it is clumsy to get the ids generated by an insert. I have found no straightforward way like LAST_INSERT_ID, but that works only for a single-row insert.
So, I have developed the following to do a batch of "normalization" inserts. This is where you a have a table that maps strings to ids (where the string is likely to show up repeatedly). It takes 2 steps: First a batch insert of the "new" strings. Then fetch all the needed ids and copy them into the other table. The details are laid out here: http://mysql.rjweb.org/doc.php/staging_table#normalization
(Sorry, I am not fluent in python or the hundred other ways to talk to MySQL, so I can't give you python code.)
Your use case example is "normalization"; I recommend doing it outside the main transaction. Note that my code takes care of multiple connections, avoiding 'burning' ids, etc.
When you have subcategories ("make" + "model" or "city" + "state" + "country"), I recommend a single normalization table, not one for each.
In your example, pkey could be a 2-byte SMALLINT UNSIGNED (limit 64K) instead of a bulky 8-byte BIGINT.
I am trying to insert a new row into my MySQL table. I had this code working when I randomly generated the IDs, however when I changed the table properties so that the IDs are now auto increment fields, the code no longer works as the Ids are no longer known to the function.
Code Snippet:
def register():
#Insert Into Usertable
userInsertSQL="""INSERT INTO users (username,password,stakeholder) VALUES (%s,%s,%s)"""
mycursor.execute(userInsertSQL,(inputedUsername.get(),inputedPassword.get(),inputedStakeholder.get()))
mydb.commit()
# Determine which table to place new user into
if inputedStakeholder.get() == "Employee":
employeeInsertSQL="""INSERT INTO employee (firstname,secondname,gender,userID) VALUES(%s,%s,%s,%s)"""
mycursor.execute(employeeInsertSQL,(inputedFirstName.get(),inputedSecondName.get(),inputedGender.get(),userID))
Is there a method of attaining the userID produced in the first Insert statement without SELECTING the last column in the users table?
You may try using the LAST_INSERT_ID() function provided by MySQL:
if inputedStakeholder.get() == "Employee":
employeeInsertSQL = """INSERT INTO employee (firstname,secondname,gender,userID)
SELECT %s, %s, %s, LAST_INSERT_ID()"""
mycursor.execute(inputedFirstName.get(), inputedSecondName.get(),
inputedGender.get())
From the MySQL documentation, you will find that LAST_INSERT_ID() returns the last generated auto increment value in a table. So long as you are just using a single connection for your Python script, and no other threads are sharing that connection, the above approach should work.
I'm using PyGreSQL to access my DB. In the use-case I'm currently working on; I am trying to insert a record into a table and return the last rowid... aka the value that the DB created for my ID field:
create table job_runners (
id SERIAL PRIMARY KEY,
hostname varchar(100) not null,
is_available boolean default FALSE
);
sql = "insert into job_runners (hostname) values ('localhost')"
When I used the db.insert(), which made the most sense, I received an "AttributeError". And when I tried db.query(sql) I get nothing but an OID.
Q: Using PyGreSQL what is the best way to insert records and return the value of the ID field without doing any additional reads or queries?
INSERT INTO job_runners
(hostname,is_available) VALUES ('localhost',true)
RETURNING id
That said, I have no idea about pygresql, but by what you've already written, I guess it's db.query() that you want to use here.
The documentation in PyGreSQL says that if you call dbconn.query() with and insert/update statement that it will return the OID. It goes on to say something about lists of OIDs when there are multiple rows involved.
First of all; I found that the OID features did not work. I suppose knowing the version numbers of the libs and tools would have helped, however, I was not trying to return the OID.
Finally; by appending "returning id", as suggested by #hacker, pygresql simply did the right thing and returned a record-set with the ID in the resulting dictionary (see code below).
sql = "insert into job_runners (hostname) values ('localhost') returning id"
rv = dbconn.query(sql)
id = rv.dictresult()[0]['id']
Assuming you have a cursor object cur:
cur.execute("INSERT INTO job_runners (hostname) VALUES (%(hostname)s) RETURNING id",
{'hostname': 'localhost'})
id = cur.fetchone()[0]
This ensures PyGreSQL correctly escapes the input string, preventing SQL injection.