Develop Reference

Unable to insert a row in SQL Server table using Python SQLAlchemy (PK not set as IDENTITY) [duplicate]

This question already has answers here:
Prevent SQLAlchemy from automatically setting IDENTITY_INSERT
(4 answers)
Closed last year.
Have this Python Flask SQLAlchemy app that fetch data from a third party SQL Server database.
There is a table with to columns that I need to insert rows:
TABLE [dbo].[TableName](
[Id] [bigint] NOT NULL,
[Desc] [varchar](150) NOT NULL,
CONSTRAINT [PK_Id] PRIMARY KEY CLUSTERED ...
The primary key is not set as IDENTITY
Using SQLAlchemy ORM, if I try to add a new row without an explicit value for Id field, I have this error:
sqlalchemy.exc.IntegrityError: (pyodbc.IntegrityError) ('23000', "[23000] ...
The column not allow Null values* (translated text)
If I explicit an Id value, another error occurs:
sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('42000', '[42000] ...*
It is not possible to find the object "dbo.TableName", because it not exists or you don't have permissions (translated text)
This error is followed by the sentence:
[SQL: SET IDENTITY_INSERT dbo.[TableName] ON]
I'm supposing SQLAlchemy is trying to execute this command, but as Id is not set as IDENTITY, there's no need for that.
Using SQL Server Management Studio, with the same user of pyodbc connection, I'm able to insert new records, choosing whatever value for Id.
I would appreciate any hint.

Your INSERT will fail because a value must be defined for the primary key column of a table, either explicitly in your INSERT or implicitly by way of an IDENTITY property.
This requirement is due to the nature of primary keys and cannot be subverted. Further, you are unable to insert a NULL because the table definition explicitly disallows NULLs in that column.
You must provide a value in your INSERT statement explicitly due to the combination of design factors present.
Based on the documentation (https://docs-sqlalchemy.readthedocs.io/ko/latest/dialects/mssql.html#:~:text=The%20SQLAlchemy%20dialect%20will%20detect%20when%20an%20INSERT,OFF%20subsequent%20to%20the%20execution.%20Given%20this%20example%3A), it appears that SqlAlchemy may be assuming that column is an IDENTITY and is attempting to toggle IDENTITY_INSERT to on. As it is not an identity column, it is encountering an exception.
In your table metadata, check that you have autoincrement=False set for the Id column.
Edit to add: According to comments in an answer on a related question (Prevent SQLAlchemy from automatically setting IDENTITY_INSERT), it appears that SqlAlchemy assumes all integer-valued primary keys to be identity, auto-incrementing as well - meaning that you need to explicitly override that assumption as described here.

Stop values from being entered into database python sqlite

How would I stop sqlite3 from adding the same exact values into a table if it is the exact same but otherwise add it? I'm totaly new to sqlite and don't know how to do this.

When you create the table, specify a unique constraint:
create table foo ( name varchar, id integer, unique ( name, id) );

You should define your table as #Robᵩ answered.
If you don't want, however, change an existing table definition - in SQLite you are very limited in ALTER TABLE, you can create a unique index:
CREATE UNIQUE INDEX foo_idx ON foo (name, id);
Note you are not allowed to create this index until you remove all duplicates.

Sqlite insert not working with python

I'm working with sqlite3 on python 2.7 and I am facing a problem with a many-to-many relationship. I have a table from which I am fetching its primary key like this
current.execute("SELECT ExtensionID FROM tblExtensionLookup where ExtensionName = ?",[ext])
and then i am fetching another primary key from another table
current.execute("SELECT HostID FROM tblHostLookup where HostName = ?",[host])
now what i am doing is i have a third table with these two keys as foreign keys and i inserted them like this
current.execute("INSERT INTO tblExtensionHistory VALUES(?,?)",[Hid,Eid])
The problem is i don't know why but the last insertion is not working it keeps giving errors. Now what i have tried is:
First I thought it was because I have an autoincrement primary id for the last mapping table which I didn't provide, but isn't it supposed to consider itself as it's auto incremented? However I went ahead and tried adding Null,None,0 but nothing works.
Secondly I thought maybe because i'm not getting the values from tables above so I tried printing it out and it shows so it works.
Any suggestions what I am doing wrong here?
EDIT :
When i don't provide primary key i get error as
The table has three columns but you provided only two values
and when i do provide them as None,Null or 0 it says
Parameter 0 is not supported probably because of unsupported type
I tried implementing the #abarnet way but still keeps saying parameter 0 not supported
connection = sqlite3.connect('WebInfrastructureScan.db')
with connection:
current = connection.cursor()
current.execute("SELECT ExtensionID FROM tblExtensionLookup where ExtensionName = ?",[ext])
Eid = current.fetchone()
print Eid
current.execute("SELECT HostID FROM tblHostLookup where HostName = ?",[host])
Hid = current.fetchone()
print Hid
current.execute("INSERT INTO tblExtensionHistory(HostID,ExtensionID) VALUES(?,?)",[Hid,Eid])
EDIT 2 :
The database schema is :
table 1:
CREATE TABLE tblHostLookup (
HostID INTEGER PRIMARY KEY AUTOINCREMENT,
HostName TEXT);
table2:
CREATE TABLE tblExtensionLookup (
ExtensionID INTEGER PRIMARY KEY AUTOINCREMENT,
ExtensionName TEXT);
table3:
CREATE TABLE tblExtensionHistory (
ExtensionHistoryID INTEGER PRIMARY KEY AUTOINCREMENT,
HostID INTEGER,
FOREIGN KEY(HostID) REFERENCES tblHostLookup(HostID),
ExtensionID INTEGER,
FOREIGN KEY(ExtensionID) REFERENCES tblExtensionLookup(ExtensionID));

It's hard to be sure without full details, but I think I can guess the problem.
If you use the INSERT statement without column names, the values must exactly match the columns as given in the schema. You can't skip over any of them.*
The right way to fix this is to just use the column names in your INSERT statement. Something like:
current.execute("INSERT INTO tblExtensionHistory (HostID, ExtensionID) VALUES (?,?)",
[Hid, Eid])
Now you can skip any columns you want (as long as they're autoincrement, nullable, or otherwise skippable, of course), or provide them in any order you want.
For your second problem, you're trying to pass in rows as if they were single values. You can't do that. From your code:
Eid = current.fetchone()
This will return something like:
[3]
And then you try to bind that to the ExtensionID column, which gives you an error.
In the future, you may want to try to write and debug the SQL statements in the sqlite3 command-line tool and/or your favorite GUI database manager (there's a simple extension that runs in for Firefox if you don't want anything fancy) and get them right, before you try getting the Python right.
* This is not true with all databases. For example, in MSJET/Access, you must skip over autoincrement columns. See the SQLite documentation for how SQLite interprets INSERT with no column names, or similar documentation for other databases.

Set SQLAlchemy to use PostgreSQL SERIAL for identity generation

Background:
The application I am currently developing is in transition from SQLite3 to PostgreSQL. All the data has been successfully migrated, using the .dump from the current database, changing all the tables of the type
CREATE TABLE foo (
id INTEGER NOT NULL,
bar INTEGER,
...
PRIMARY KEY (id),
FOREIGN KEY(bar) REFERENCES foobar (id),
...
);
to
CREATE TABLE foo (
id SERIAL NOT NULL,
bar INTEGER,
...
PRIMARY KEY (id),
FOREIGN KEY(bar) REFERENCES foobar (id) DEFERRABLE,
...
);
and SET CONSTRAINTS ALL DEFERRED;.
Since I am using SQLAlchemy I was expecting things to work smoothly from then on, after of course changing the engine. But the problem seems to be with the autoincrement of the primary key to a unique value on INSERT.
The table, say foo, I am currently having trouble with has 7500+ rows but the sequence foo_id_seq's current value is set on 5(because I have tried the inserts five times now all of which have failed).
Question:
So now my question is that without explicitly supplying the id, in the INSERT statement, how can I make Postgres automatically assign a unique value to the id field if foo? Or more specifically, have the sequence return a unique value for it?
Sugar:
Achieve all that through the SQLAlchemy interface.
Environment details:
Python 2.6
SQLAlchemy 8.2
PostgreSQL 9.2
psycopg2 - 2.5.1 (dt dec pq3 ext)
PS: If anybody finds a more appropriate title for this question please edit it.

Your PRIMARY KEY should be defined to use a SEQUENCE as a DEFAULT, either via the SERIAL convenience pseudo-type:
CREATE TABLE blah (
id serial primary key,
...
);
or an explicit SEQUENCE:
CREATE SEQUENCE blah_id_seq;
CREATE TABLE blah (
id integer primary key default nextval('blah_id_seq'),
...
);
ALTER SEQUENCE blah_id_seq OWNED BY blah.id;
This is discussed in the SQLAlchemy documentation.
You can add this to an existing table:
CREATE SEQUENCE blah_id_seq OWNED BY blah.id;
ALTER TABLE blah ALTER COLUMN id SET DEFAULT nextval('blah_id_seq');
if you prefer to restore a dump then add sequences manually.
If there's existing data you've loaded directly into the tables with COPY or similar, you need to set the sequence starting point:
SELECT setval('blah_id_seq', max(id)+1) FROM blah;
I'd say the issue is likely to be to do with your developing in SQLite, then doing a dump and restoring that dump to PostgreSQL. SQLAlchemy expects to create the schema its self with the appropriate defaults and sequences.
What I recommend you do instead is to get SQLAlchemy to create a new, empty database. Dump the data for each table from the SQLite DB to CSV, then COPY that data into the PostgreSQL tables. Finally, update the sequences with setval so they generate the appropriate values.
One way or the other, you will need to make sure that the appropriate sequences are created. You can do it by SERIAL pseudo-column types, or by manual SEQUENCE creation and DEFAULT setting, but you must do it. Otherwise there's no way to assign a generated ID to the table in an efficient, concurrency-safe way.

Use
alter sequence foo_id_seq restart with 7600
should give you 7601 next time you call the sequence.
http://www.postgresql.org/docs/current/static/sql-altersequence.html
And then subsequent values. Just make sure that you restart it with a value > the last id.

creating blank field and receving the INTEGER PRIMARY KEY with sqlite, python

I am using sqlite with python. When i insert into table A i need to feed it an ID from table B. So what i wanted to do is insert default data into B, grab the id (which is auto increment) and use it in table A. Whats the best way receive the key from the table i just inserted into?

As Christian said, sqlite3_last_insert_rowid() is what you want... but that's the C level API, and you're using the Python DB-API bindings for SQLite.
It looks like the cursor method lastrowid will do what you want (search for 'lastrowid' in the documentation for more information). Insert your row with cursor.execute( ... ), then do something like lastid = cursor.lastrowid to check the last ID inserted.
That you say you need "an" ID worries me, though... it doesn't matter which ID you have? Unless you are using the data just inserted into B for something, in which case you need that row ID, your database structure is seriously screwed up if you just need any old row ID for table B.

Check out sqlite3_last_insert_rowid() -- it's probably what you're looking for:
Each entry in an SQLite table has a
unique 64-bit signed integer key
called the "rowid". The rowid is
always available as an undeclared
column named ROWID, OID, or _ROWID_ as
long as those names are not also used
by explicitly declared columns. If the
table has a column of type INTEGER
PRIMARY KEY then that column is
another alias for the rowid.
This routine returns the rowid of the
most recent successful INSERT into the
database from the database connection
in the first argument. If no
successful INSERTs have ever occurred
on that database connection, zero is
returned.
Hope it helps! (More info on ROWID is available here and here.)

Simply use:
SELECT last_insert_rowid();
However, if you have multiple connections writing to the database, you might not get back the key that you expect.