Background:
The application I am currently developing is in transition from SQLite3 to PostgreSQL. All the data has been successfully migrated, using the .dump from the current database, changing all the tables of the type
CREATE TABLE foo (
id INTEGER NOT NULL,
bar INTEGER,
...
PRIMARY KEY (id),
FOREIGN KEY(bar) REFERENCES foobar (id),
...
);
to
CREATE TABLE foo (
id SERIAL NOT NULL,
bar INTEGER,
...
PRIMARY KEY (id),
FOREIGN KEY(bar) REFERENCES foobar (id) DEFERRABLE,
...
);
and SET CONSTRAINTS ALL DEFERRED;.
Since I am using SQLAlchemy I was expecting things to work smoothly from then on, after of course changing the engine. But the problem seems to be with the autoincrement of the primary key to a unique value on INSERT.
The table, say foo, I am currently having trouble with has 7500+ rows but the sequence foo_id_seq's current value is set on 5(because I have tried the inserts five times now all of which have failed).
Question:
So now my question is that without explicitly supplying the id, in the INSERT statement, how can I make Postgres automatically assign a unique value to the id field if foo? Or more specifically, have the sequence return a unique value for it?
Sugar:
Achieve all that through the SQLAlchemy interface.
Environment details:
Python 2.6
SQLAlchemy 8.2
PostgreSQL 9.2
psycopg2 - 2.5.1 (dt dec pq3 ext)
PS: If anybody finds a more appropriate title for this question please edit it.
Your PRIMARY KEY should be defined to use a SEQUENCE as a DEFAULT, either via the SERIAL convenience pseudo-type:
CREATE TABLE blah (
id serial primary key,
...
);
or an explicit SEQUENCE:
CREATE SEQUENCE blah_id_seq;
CREATE TABLE blah (
id integer primary key default nextval('blah_id_seq'),
...
);
ALTER SEQUENCE blah_id_seq OWNED BY blah.id;
This is discussed in the SQLAlchemy documentation.
You can add this to an existing table:
CREATE SEQUENCE blah_id_seq OWNED BY blah.id;
ALTER TABLE blah ALTER COLUMN id SET DEFAULT nextval('blah_id_seq');
if you prefer to restore a dump then add sequences manually.
If there's existing data you've loaded directly into the tables with COPY or similar, you need to set the sequence starting point:
SELECT setval('blah_id_seq', max(id)+1) FROM blah;
I'd say the issue is likely to be to do with your developing in SQLite, then doing a dump and restoring that dump to PostgreSQL. SQLAlchemy expects to create the schema its self with the appropriate defaults and sequences.
What I recommend you do instead is to get SQLAlchemy to create a new, empty database. Dump the data for each table from the SQLite DB to CSV, then COPY that data into the PostgreSQL tables. Finally, update the sequences with setval so they generate the appropriate values.
One way or the other, you will need to make sure that the appropriate sequences are created. You can do it by SERIAL pseudo-column types, or by manual SEQUENCE creation and DEFAULT setting, but you must do it. Otherwise there's no way to assign a generated ID to the table in an efficient, concurrency-safe way.
Use
alter sequence foo_id_seq restart with 7600
should give you 7601 next time you call the sequence.
http://www.postgresql.org/docs/current/static/sql-altersequence.html
And then subsequent values. Just make sure that you restart it with a value > the last id.
Related
I have an existing SQL Server (2012) DB with many tables having a primary key of Numeric(9, 0) type. For all intents and purposes they are integers.
The ORM mapping (generated using sqlacodegen) looks like:
class SomeTable(Base):
__tablename__ = 'SOME_TABLE'
__table_args__ = {'schema': 'dbo'}
SOME_TABLE_ID = Column(Numeric(9, 0), primary_key=True)
some_more_fields_here = XXX
Sample code to insert data:
some_table = SomeTable(_not_specifying_SOME_TABLE_ID_explicityly_)
session.add(some_table)
session.flush() # <---BOOM, FlushError here
When I try to insert data into such tables, my app crashes on session.flush() with the following error:
sqlalchemy.orm.exc.FlushError: Instance SomeTable ... has a NULL identity key. If this is an auto-generated value, check that the database table allows generation of new primary key values, and that the mapped Column object is configured to expect these generated values. Ensure also that this flush() is not occurring at an inappropriate time, such aswithin a load() event.
If I replace Numeric with BigInteger then everything works fine. I did some digging and the query generated with Numeric is like this:
INSERT INTO dbo.[SOME_TABLE] (_columns_without_SOME_TABLE_ID)
VALUES (...)
seems like a valid query from SQL point of view, but Sqlalchemy raises the above exception there
The query generated using BigInteger is as follows:
INSERT INTO dbo.[SOME_TABLE] (_columns_without_SOME_TABLE_ID)
OUTPUT inserted.[SOME_TABLE_ID]
VALUES (...)
I also found this peace of documentation about autoincrement property. And sure enough, it explains the behavior I observe, i.e. autoincrement only works with integers.
So my question is whether there is some kind of workaround to make autoincrement work with Numeric columns without converting them to BigInteger?
My system configuration is - Centos 7 64 bit, Python 3.5.2, Sqlalchemy 1.1.4, pymssql 2.2.0, SQL Server 2012.
Is it possible to include many-to-many relationships when running a Postgres COPY command? If so, can you give me an example?
For example:
CREATE TABLE "lap" (
"id" serial NOT NULL PRIMARY KEY,
"Lap_number" integer,
"Lap_time" interval,
)
;
CREATE TABLE "datasinglerace_Laps" (
"id" serial NOT NULL PRIMARY KEY,
"datasinglerace_id" integer NOT NULL,
"lap_id" integer NOT NULL REFERENCES "lap" ("id") DEFERRABLE INITIALLY DEFERRED,
UNIQUE ("datasinglerace_id", "lap_id")
)
;
CREATE TABLE "datasinglerace" (
"id" serial NOT NULL PRIMARY KEY,
"Notes" text,
)
;
ALTER TABLE "datasinglerace_Laps" ADD CONSTRAINT "datasinglerace_id_refs_id_620382df" FOREIGN KEY ("datasinglerace_id")
REFERENCES "datasinglerace" ("id") DEFERRABLE INITIALLY DEFERRED;
The lap objects are already in the db. For the COPY file, I'd like to put the info for the datasinglerace id's and a list of the lap object's id's I want to attach. There will be a variable number of lap objects I want to attach.
This SQL was created with the Django framework. I want to keep this in the Django framework so I don't want to change the SQL. Importing the data has been really slow, so I'm working on improving the import data speed.
You can use COPY to improve the speed for importaing batches of data once off - I wouldn't use it normally - have you isolated the bottle neck? What are you copying from? You would need something like CSV files with the same structure as the tables..
COPY copies directly to PostgreSQL it has nothing to do with Django so you will need interact with Postgres using pgsql or whatever tool you use. Your command will look like this:
COPY datasinglerace FROM datasinglerace.csv;
COPY datasinglerace_Laps FROM datasinglerace_Laps.csv;
COPY has many options see the documentation.
Note anything that is referenced by something else needs to be added first otherwise you would need to relax (delete then add back) your reference constraints. In this case you would need to COPY datasinglerace_Laps last so the references it has already exist
I have a table (on which I have no control) that I must copy. The target schema can be the same as the original one, so all indexes and constraints have to be defined without a name, implicitly.
I'm using Python 3.4.3 with SQLAlchemy 1.0.8 and cx_oracle 5.2.
The table is like this:
CREATE TABLE "MY_TABLE"
( "ITEMID" NUMBER(*,0) NOT NULL ENABLE,
"LABEL" NVARCHAR2(80) NOT NULL ENABLE,
"FIRSTCHILDID" NUMBER(*,0) NOT NULL ENABLE,
"LASTCHILDID" NUMBER(*,0) NOT NULL ENABLE,
"DEFAULTPARENTID" NUMBER(*,0) NOT NULL ENABLE,
"PICTUREID" NUMBER(6,0) NOT NULL ENABLE,
"SECURITYID" NUMBER(*,0) NOT NULL ENABLE,
PRIMARY KEY ("ITEMID")
UNIQUE ("LABEL"));
The code I'm using is at https://gist.github.com/toyg/9fb541ff3dbc8c175329 but the core of it is this (smeta and dmeta are source and target Metadata, bound):
table = Table(table_name, smeta, autoload=True)
target_name = prefix + str(table.name)
target_table = table.tometadata(dmeta, name=target_name)
for constraint in target_table.constraints:
constraint.name = None
target_table.metadata.create_all(dengine)
It fails with this error:
sqlalchemy.exc.DatabaseError: (cx_Oracle.DatabaseError)
ORA-00955: name is already used by an existing object
[SQL: b'CREATE UNIQUE INDEX sys_c009016 ON "TMP_MY_TABLE" (label)']
This is because SQLAlchemy is trying to create the Unique index after creating the table, when it's already too late: CREATE INDEX requires a name, so SA uses the same name as the existing one, and it fails.
I tried setting the index name to None before creation, to give SA a hint, but that results in errors because it expects a string there at all times.
Is there any way to tell SA to just append the bloody UNIQUE clause to the table DDL right away?
"UNIQUE INDEX" means that the Index construct is used. Its DDL is not emitted within the CREATE TABLE. It sounds like you are looking for a UniqueConstraint construct. It seems likely that in this case, Oracle returns reflected information about what you first created as a UniqueConstraint object as an Index object with unique=True (these constructs are "different", but on many backends they are synonymous and/or mixed and matched and sometimes even mirrored, it's totally confusing).
at the end of the day if you want the UNIQUE keyword as an inline constraint you need to use the UniqueConstraint object, and you'd need to remove this Index from the table - you might be able to get away with table.indexes.remove(index). The Index object wouldn't be in table.constraints. You probably want to do your "copy" of the table in a more programmatic way rather than using tometadata(). Look perhaps into using the inspection interface directly and just build the Table you want from that.
How can I instruct Django to call nextval on a sequence for a given model's field?
I realize that I can make a trigger in the DB:
CREATE TRIGGER foo_trg
BEFORE INSERT ON foo FOR EACH ROW
BEGIN
SELECT foo_id_seq.NEXTVAL INTO :new.foo_id FROM dual;
END;
However I'm curious if Django can do it via configuration like MySQL and autoincrement.
I didn't see anything specified in the Django Oracle notes.
Oracle 12c has introduced two new features which relates to mysql autoincrement.
Default values using sequence
CREATE SEQUENCE seq;
CREATE TABLE t (
x number default seq.nextval, /*-- no need for a trigger to autoincrement */.
y varchar2(10)
);
http://docs.oracle.com/database/121/SQLRF/pseudocolumns002.htm#SQLRF50946
Identity columns
CREATE TABLE t (
x number generated by default on null as identity,
y varchar2(10)
);
http://docs.oracle.com/database/121/SQLRF/statements_7002.htm#SQLRF01402
I'm working with sqlite3 on python 2.7 and I am facing a problem with a many-to-many relationship. I have a table from which I am fetching its primary key like this
current.execute("SELECT ExtensionID FROM tblExtensionLookup where ExtensionName = ?",[ext])
and then i am fetching another primary key from another table
current.execute("SELECT HostID FROM tblHostLookup where HostName = ?",[host])
now what i am doing is i have a third table with these two keys as foreign keys and i inserted them like this
current.execute("INSERT INTO tblExtensionHistory VALUES(?,?)",[Hid,Eid])
The problem is i don't know why but the last insertion is not working it keeps giving errors. Now what i have tried is:
First I thought it was because I have an autoincrement primary id for the last mapping table which I didn't provide, but isn't it supposed to consider itself as it's auto incremented? However I went ahead and tried adding Null,None,0 but nothing works.
Secondly I thought maybe because i'm not getting the values from tables above so I tried printing it out and it shows so it works.
Any suggestions what I am doing wrong here?
EDIT :
When i don't provide primary key i get error as
The table has three columns but you provided only two values
and when i do provide them as None,Null or 0 it says
Parameter 0 is not supported probably because of unsupported type
I tried implementing the #abarnet way but still keeps saying parameter 0 not supported
connection = sqlite3.connect('WebInfrastructureScan.db')
with connection:
current = connection.cursor()
current.execute("SELECT ExtensionID FROM tblExtensionLookup where ExtensionName = ?",[ext])
Eid = current.fetchone()
print Eid
current.execute("SELECT HostID FROM tblHostLookup where HostName = ?",[host])
Hid = current.fetchone()
print Hid
current.execute("INSERT INTO tblExtensionHistory(HostID,ExtensionID) VALUES(?,?)",[Hid,Eid])
EDIT 2 :
The database schema is :
table 1:
CREATE TABLE tblHostLookup (
HostID INTEGER PRIMARY KEY AUTOINCREMENT,
HostName TEXT);
table2:
CREATE TABLE tblExtensionLookup (
ExtensionID INTEGER PRIMARY KEY AUTOINCREMENT,
ExtensionName TEXT);
table3:
CREATE TABLE tblExtensionHistory (
ExtensionHistoryID INTEGER PRIMARY KEY AUTOINCREMENT,
HostID INTEGER,
FOREIGN KEY(HostID) REFERENCES tblHostLookup(HostID),
ExtensionID INTEGER,
FOREIGN KEY(ExtensionID) REFERENCES tblExtensionLookup(ExtensionID));
It's hard to be sure without full details, but I think I can guess the problem.
If you use the INSERT statement without column names, the values must exactly match the columns as given in the schema. You can't skip over any of them.*
The right way to fix this is to just use the column names in your INSERT statement. Something like:
current.execute("INSERT INTO tblExtensionHistory (HostID, ExtensionID) VALUES (?,?)",
[Hid, Eid])
Now you can skip any columns you want (as long as they're autoincrement, nullable, or otherwise skippable, of course), or provide them in any order you want.
For your second problem, you're trying to pass in rows as if they were single values. You can't do that. From your code:
Eid = current.fetchone()
This will return something like:
[3]
And then you try to bind that to the ExtensionID column, which gives you an error.
In the future, you may want to try to write and debug the SQL statements in the sqlite3 command-line tool and/or your favorite GUI database manager (there's a simple extension that runs in for Firefox if you don't want anything fancy) and get them right, before you try getting the Python right.
* This is not true with all databases. For example, in MSJET/Access, you must skip over autoincrement columns. See the SQLite documentation for how SQLite interprets INSERT with no column names, or similar documentation for other databases.