EDIT: At this point, I found the errant typo that was responsible, and my question has become "How did the typo that I made cause the error that I received" and "How might I have better debugged this in the future?"
I've setup a database script for SQLite (through pysqlite) as follows:
DROP TABLE IF EXISTS LandTerritory;
CREATE TABLE LandTerritory (
name varchar(50) PRIMARY KEY NOT NULL UNIQUE,
hasSC boolean NOT NULL DEFAULT 0
);
I'm expecting this to always run without error. However, if I run this script twice, (using the sqlite.Connection.executescript method) I get this error:
OperationalError:table LandTerritory already exists
Trying to debug this myself, I run DROP TABLE LandTerritory on its own and get:
sqlite3.OperationalError: no such table: main.LandTerrito
I'm guessing this has something to do with the "main." part, but I'm not sure what.
EDIT:
Okay PRAGMA foreign_keys=ON is definitely involved here, too. When I create my connection, I turned on foreign_keys. If I don't turn that on, I don't seem to get this error.
And I should have mentioned that there's more to the script, but I had assumed the error was occurring in these first 2 statements. The rest of the script just does the same, drop table, define table. A few of the tables have foreign key references to LandTerritory.
Is there a way to get something like line number information about the sqlite errors? That would be really helpful.
EDIT 2:
Okay, here's another table in the script that references the first.
DROP TABLE IF EXISTS LandAdjacent;
CREATE TABLE LandAdjacent (
tname1 varchar(50) NOT NULL,
tname2 varchar(50) NOT NULL,
PRIMARY KEY (tname1, tname2),
/* Foreign keys */
FOREIGN KEY (tname1)
REFERENCES LandTerrito
ON DELETE CASCADE
ON UPDATE CASCADE,
FOREIGN KEY (tname2)
REFERENCES LandTerritory(name)
ON DELETE CASCADE
ON UPDATE CASCADE
);
Looking at this, I found were the "LandTerrito" came from, somehow a few characters got cut off. I'm guessing fixing this may fix my problem.
But I'm really confused how a broken line in this table led to the script running correctly the first time, and then giving me an error related to a different table when I run it the second time, and how foreign keys played into this.
I guess, to reiterate from above, is there a better way to debug this sort of thing?
The source of the error is your typo
REFERENCES LandTerrito
in line 8 of your script. This leads to the "missing" table LandTerrito in the CREATE TABLE LandAdjacent statement.
If you run your two CREATE TABLE statements Sqlite wont complain. But if you have PRAGMA foreign_keys=ON; and try to run an INSERT or DELETE statement on the table LandAdjacent, you'll get the error no such table: main.LandTerrito.
Because of the foreign key constraints DROP TABLE on LandTerritory however will result in a DELETE on the table LandAdjacent, which triggers the error.
The following things will avoid the error
set PRAGMA foreign_keys=ON; before you drop the table (tested) or
add a dummy table LandTerrito (tested) or
drop LandAdjacent first, then LandTerritory (tested) or
dont use ON DELETE CASCADE (not tested)
and of course correcting the original typo.
Put a "GO" (or whatever equivalent is used in SQLlite) to terminate a batch between the drop table statement and the create statement
Related
Right off the bat I want to say, due to my position I cannot paste the full code. So I will do what I can to symbolize the code and get straight to the point.
Programmed in: Python.
Simply put I am getting a Duplicate Key Error. I have looked into other questions that have been raised about this and to my knowledge I am following the suggestion those answers have provided.
Table Structure Snippet:
CREATE TABLE `BAR_TABLE` (
`timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`foo` tinytext,
`bar` tinytext,
`uuid` varchar(25) NOT NULL,
PRIMARY KEY (`uuid`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
In this case 'uuid' is the PRIMARY as there already a dataset this is taking in, that uuid is always unique(Unless it already exists then it's updating potential info)
SQL Snippet
INSERT INTO BAR_TABLE (
foo,
bar,
uuid
) VALUES (
%(foo)s,
%(bar)s,
%(uuid)s
) ON DUPLICATE KEY UPDATE
foo = VALUES(foo),
bar = VALUES(bar)
So the reason the SQL looks like this is because I am using executemany(). As well as the data that comes in is a Python Dictionary. So this allows it to assign all the values in the dictionary data to the SQL statement. Then all that gets shifted into the DB using executemany().
The item it is throwing the duplicate entry is actually in the table. I managed to get this to run a couple times, and then at some point during its testing it hit this error, and hasn't moved past it.
Obviously I am miss understanding something. Am I miss-understanding how the PRIMARY KEY works? Or what a KEY is using the ON DUPLICATE KEY UPDATE?
I have 2 tables (contracts and contract_items) where the latter has a foreign key set to the first one.
When using SQLAlchemy for inserting new data from a list into my postgre database I'm basically doing the following:
for row in list:
# Get contract and item from row
...
session.add(contract)
session.add(contract_item)
# Do some select statements (which will raise an auto-flush)
...
session.commit()
Now... this works for maybe 2-3 runs, sometimes more, sometimes less. Then the part where an auto-flush is executed will end in an exception telling me that contract_item could not be inserted because it has an foreign key to contract and the contract row does not exist yet.
Is the order in which I pass the data to the add-function of the session not the order in which the data will be flushed? I actually hoped SQLAlchemy would find the right order in which to flush statements on it's own based on the dependencies. It should be clear that the contract_item row should not be inserted before the contract row, when contract_item has a foreign key to contract set. Yet the order seems to be random.
I then tried to flush the contract manually before adding contract_item:
for row in list:
# Getting contract and item from row
...
session.add(contract)
session.flush() # Flushing manually
session.add(contract_item)
# Do some select statements (which will raise an auto-flush)
...
session.commit()
This worked without any problems and the rows got inserted into the database.
Is there any way to set the order in which statements will be flushed for the session? Does SQLAlchemy really not care about dependencies such as foreign keys or am I making a mistake when adding the data? I'd rather not manage the flushs manually if somehow possible.
Is there a way to make SQLAlchemy get the order right?
Had the same problem. What solved it in my case is creating a biderictional relationship - you need to make relationship from contracts to contract_items, as described HERE
UPD: actually you can do it simplier: just add relationship from contract_items table to contract table and that should do the thing.
The way session handles related objects is defined by cascades. Use "save-update" cascade on a relationship (that is enabled by default) to automatically add related objects, so that you only have to use one add call. The documentation I linked contains code example.
I just had an issue with Django and PostgreSQL that I don't understand.
I have a simple model, defined such as:
class MyModel(models.Model):
my_field = models.IntegerField()
my_other_field = models.TextField()
In my view, i have something similar to:
my_object = MyModel(my_field=1, my_other_field='blah')
my_object.save()
Everything was working fine, until this morning. I got this error:
IntegrityError at /my_url/
duplicate key value violates unique constraint "my_model_pkey"
DETAIL: Key (id)=(3) already exists.
CONTEXT: Remote SQL command: INSERT INTO public.my_model(id, my_field, my_other_field) VALUES ($1, $2, $3) RETURNING id
I had this error once, I know it is related to the way PostgreSQL syncs the sequential table associated with my model with the id column. I has to run this function in PostgreSQL until the id returned was greater than the biggest value of the id.
select nextval('my_model_id_seq'::regclass);
My question is: Why did this happen in the first place? And how to prevent it in the future ?
By the way, that's the only way I insert data into the table, I've never inserted data manually.
I hope the question is clear enough
I think the question is not "why is my sequence getting messed up" - rather it is "why is Django trying to supply a value for the id column when inserting a row, instead of allowing the database to insert the next value in the sequence".
The Django documentation describes the algorithm it uses to decide whether it should be doing an UPDATE or an INSERT when you call save().
This algorithm involves checking if the 'id' field of the object is already set to some value. If it is not, then it does an INSERT (presumably not specifying a value for the 'id' field). If it is set, then it first tries to do an UPDATE; if that does not result in an updated record, then it will do an INSERT (this time presumably it would specify a value for the 'id' field).
As pointed out in Erwin's answer, the error message which you seeing indicates it is trying to insert a row while specifying the value for the 'id' field.
I note that it appears this algorithm has changed in version 1.6 of Django. Previously it used a SELECT first to see if a record existed, then an UPDATE if it did or an INSERT if it did not. If your problem has started occurring since upgrading, then that could be a cause. The documentation notes:
There are some rare cases where the database doesn’t report that a row
was updated even if the database contains a row for the object’s
primary key value. An example is the PostgreSQL ON UPDATE trigger
which returns NULL. In such cases it is possible to revert to the old
algorithm by setting the select_on_save option to True.
If this were happening for you, then it would explain your symptoms: the error would actually be occurring when trying to update a value in the database, and django would erroneously think that the row did not exist and then try to create it.
You could check for this by setting 'select_on_save' to true to revert to the old behavior.
Another possible reason for this would be if your code inadvertently set the 'id' attribute on an object to some value, and then called save(). This could cause various problems, depending on whether the value already existed in the database or not. In particular, it might result in creating a row which has an 'id' value which is ahead of the current range of the sequence associated with the column, so that later on you would get errors trying to insert into the row.
Another possible reason could be using the 'force_insert' argument to save(), on a row which had previously loaded from the database (so that it was actually an existing row you should be updating).
The root of the problem lies here (SQL command from your error message):
INSERT INTO public.my_model(id, my_field, my_other_field)
VALUES ($1, $2, $3)
RETURNING id
Since your id column seems to be a serial type, do not insert values manually. Let the default draw from the sequence automatically. Should be:
INSERT INTO public.my_model(my_field, my_other_field)
VALUES ($1, $2)
RETURNING id;
That's the whole point of adding RETURNING id to begin with: to return the newly generated id. If you pass in a value yourself, you wouldn't need to have it returned.
Fix
If the sequence got out of sync somehow, because manual entries conflict with the numbers from nextval(), run this query once:
SELECT setval('my_model_id_seq', max(id)) FROM my_model;
This sets the sequence to the current maximum. Next call is next number, no off-by-one error.
I'm working with sqlite3 on python 2.7 and I am facing a problem with a many-to-many relationship. I have a table from which I am fetching its primary key like this
current.execute("SELECT ExtensionID FROM tblExtensionLookup where ExtensionName = ?",[ext])
and then i am fetching another primary key from another table
current.execute("SELECT HostID FROM tblHostLookup where HostName = ?",[host])
now what i am doing is i have a third table with these two keys as foreign keys and i inserted them like this
current.execute("INSERT INTO tblExtensionHistory VALUES(?,?)",[Hid,Eid])
The problem is i don't know why but the last insertion is not working it keeps giving errors. Now what i have tried is:
First I thought it was because I have an autoincrement primary id for the last mapping table which I didn't provide, but isn't it supposed to consider itself as it's auto incremented? However I went ahead and tried adding Null,None,0 but nothing works.
Secondly I thought maybe because i'm not getting the values from tables above so I tried printing it out and it shows so it works.
Any suggestions what I am doing wrong here?
EDIT :
When i don't provide primary key i get error as
The table has three columns but you provided only two values
and when i do provide them as None,Null or 0 it says
Parameter 0 is not supported probably because of unsupported type
I tried implementing the #abarnet way but still keeps saying parameter 0 not supported
connection = sqlite3.connect('WebInfrastructureScan.db')
with connection:
current = connection.cursor()
current.execute("SELECT ExtensionID FROM tblExtensionLookup where ExtensionName = ?",[ext])
Eid = current.fetchone()
print Eid
current.execute("SELECT HostID FROM tblHostLookup where HostName = ?",[host])
Hid = current.fetchone()
print Hid
current.execute("INSERT INTO tblExtensionHistory(HostID,ExtensionID) VALUES(?,?)",[Hid,Eid])
EDIT 2 :
The database schema is :
table 1:
CREATE TABLE tblHostLookup (
HostID INTEGER PRIMARY KEY AUTOINCREMENT,
HostName TEXT);
table2:
CREATE TABLE tblExtensionLookup (
ExtensionID INTEGER PRIMARY KEY AUTOINCREMENT,
ExtensionName TEXT);
table3:
CREATE TABLE tblExtensionHistory (
ExtensionHistoryID INTEGER PRIMARY KEY AUTOINCREMENT,
HostID INTEGER,
FOREIGN KEY(HostID) REFERENCES tblHostLookup(HostID),
ExtensionID INTEGER,
FOREIGN KEY(ExtensionID) REFERENCES tblExtensionLookup(ExtensionID));
It's hard to be sure without full details, but I think I can guess the problem.
If you use the INSERT statement without column names, the values must exactly match the columns as given in the schema. You can't skip over any of them.*
The right way to fix this is to just use the column names in your INSERT statement. Something like:
current.execute("INSERT INTO tblExtensionHistory (HostID, ExtensionID) VALUES (?,?)",
[Hid, Eid])
Now you can skip any columns you want (as long as they're autoincrement, nullable, or otherwise skippable, of course), or provide them in any order you want.
For your second problem, you're trying to pass in rows as if they were single values. You can't do that. From your code:
Eid = current.fetchone()
This will return something like:
[3]
And then you try to bind that to the ExtensionID column, which gives you an error.
In the future, you may want to try to write and debug the SQL statements in the sqlite3 command-line tool and/or your favorite GUI database manager (there's a simple extension that runs in for Firefox if you don't want anything fancy) and get them right, before you try getting the Python right.
* This is not true with all databases. For example, in MSJET/Access, you must skip over autoincrement columns. See the SQLite documentation for how SQLite interprets INSERT with no column names, or similar documentation for other databases.
I am using sqlite with python. When i insert into table A i need to feed it an ID from table B. So what i wanted to do is insert default data into B, grab the id (which is auto increment) and use it in table A. Whats the best way receive the key from the table i just inserted into?
As Christian said, sqlite3_last_insert_rowid() is what you want... but that's the C level API, and you're using the Python DB-API bindings for SQLite.
It looks like the cursor method lastrowid will do what you want (search for 'lastrowid' in the documentation for more information). Insert your row with cursor.execute( ... ), then do something like lastid = cursor.lastrowid to check the last ID inserted.
That you say you need "an" ID worries me, though... it doesn't matter which ID you have? Unless you are using the data just inserted into B for something, in which case you need that row ID, your database structure is seriously screwed up if you just need any old row ID for table B.
Check out sqlite3_last_insert_rowid() -- it's probably what you're looking for:
Each entry in an SQLite table has a
unique 64-bit signed integer key
called the "rowid". The rowid is
always available as an undeclared
column named ROWID, OID, or _ROWID_ as
long as those names are not also used
by explicitly declared columns. If the
table has a column of type INTEGER
PRIMARY KEY then that column is
another alias for the rowid.
This routine returns the rowid of the
most recent successful INSERT into the
database from the database connection
in the first argument. If no
successful INSERTs have ever occurred
on that database connection, zero is
returned.
Hope it helps! (More info on ROWID is available here and here.)
Simply use:
SELECT last_insert_rowid();
However, if you have multiple connections writing to the database, you might not get back the key that you expect.