Develop Reference

Unable to insert a row in SQL Server table using Python SQLAlchemy (PK not set as IDENTITY) [duplicate]

This question already has answers here:
Prevent SQLAlchemy from automatically setting IDENTITY_INSERT
(4 answers)
Closed last year.
Have this Python Flask SQLAlchemy app that fetch data from a third party SQL Server database.
There is a table with to columns that I need to insert rows:
TABLE [dbo].[TableName](
[Id] [bigint] NOT NULL,
[Desc] [varchar](150) NOT NULL,
CONSTRAINT [PK_Id] PRIMARY KEY CLUSTERED ...
The primary key is not set as IDENTITY
Using SQLAlchemy ORM, if I try to add a new row without an explicit value for Id field, I have this error:
sqlalchemy.exc.IntegrityError: (pyodbc.IntegrityError) ('23000', "[23000] ...
The column not allow Null values* (translated text)
If I explicit an Id value, another error occurs:
sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('42000', '[42000] ...*
It is not possible to find the object "dbo.TableName", because it not exists or you don't have permissions (translated text)
This error is followed by the sentence:
[SQL: SET IDENTITY_INSERT dbo.[TableName] ON]
I'm supposing SQLAlchemy is trying to execute this command, but as Id is not set as IDENTITY, there's no need for that.
Using SQL Server Management Studio, with the same user of pyodbc connection, I'm able to insert new records, choosing whatever value for Id.
I would appreciate any hint.

Your INSERT will fail because a value must be defined for the primary key column of a table, either explicitly in your INSERT or implicitly by way of an IDENTITY property.
This requirement is due to the nature of primary keys and cannot be subverted. Further, you are unable to insert a NULL because the table definition explicitly disallows NULLs in that column.
You must provide a value in your INSERT statement explicitly due to the combination of design factors present.
Based on the documentation (https://docs-sqlalchemy.readthedocs.io/ko/latest/dialects/mssql.html#:~:text=The%20SQLAlchemy%20dialect%20will%20detect%20when%20an%20INSERT,OFF%20subsequent%20to%20the%20execution.%20Given%20this%20example%3A), it appears that SqlAlchemy may be assuming that column is an IDENTITY and is attempting to toggle IDENTITY_INSERT to on. As it is not an identity column, it is encountering an exception.
In your table metadata, check that you have autoincrement=False set for the Id column.
Edit to add: According to comments in an answer on a related question (Prevent SQLAlchemy from automatically setting IDENTITY_INSERT), it appears that SqlAlchemy assumes all integer-valued primary keys to be identity, auto-incrementing as well - meaning that you need to explicitly override that assumption as described here.

Syntax error while executing this query with python in sqlite databse

I have a table name globalData, in my sqlite database, with column 'index', 'rank_d30'. While executing this query with python, i receive syntax error near ON ...
cur.execute('INSERT INTO globalData (`index`, rank_d30) VALUES(0, 9) ON CONFLICT(`index`) DO UPDATE SET rank_d30 = VALUES(rank_d30)')
How it can be corrected?

SQLite does not recognize the VALUES() syntax in the DO UPDATE clause of the query, as MySQL does in the INSERT ... ON DUPLICATE KEY syntax. To refer to the value that was initially given for insert, you must use pseudo-table excluded instead.
Consider:
INSERT INTO globalData (`index`, rank_d30)
VALUES(0, 9)
ON CONFLICT(`index`) DO UPDATE SET rank_d30 = EXCLUDED.rank_d30
Note that, for this to work, you need a unique or primary key constraint on column index.

Sqlite insert not working with python

I'm working with sqlite3 on python 2.7 and I am facing a problem with a many-to-many relationship. I have a table from which I am fetching its primary key like this
current.execute("SELECT ExtensionID FROM tblExtensionLookup where ExtensionName = ?",[ext])
and then i am fetching another primary key from another table
current.execute("SELECT HostID FROM tblHostLookup where HostName = ?",[host])
now what i am doing is i have a third table with these two keys as foreign keys and i inserted them like this
current.execute("INSERT INTO tblExtensionHistory VALUES(?,?)",[Hid,Eid])
The problem is i don't know why but the last insertion is not working it keeps giving errors. Now what i have tried is:
First I thought it was because I have an autoincrement primary id for the last mapping table which I didn't provide, but isn't it supposed to consider itself as it's auto incremented? However I went ahead and tried adding Null,None,0 but nothing works.
Secondly I thought maybe because i'm not getting the values from tables above so I tried printing it out and it shows so it works.
Any suggestions what I am doing wrong here?
EDIT :
When i don't provide primary key i get error as
The table has three columns but you provided only two values
and when i do provide them as None,Null or 0 it says
Parameter 0 is not supported probably because of unsupported type
I tried implementing the #abarnet way but still keeps saying parameter 0 not supported
connection = sqlite3.connect('WebInfrastructureScan.db')
with connection:
current = connection.cursor()
current.execute("SELECT ExtensionID FROM tblExtensionLookup where ExtensionName = ?",[ext])
Eid = current.fetchone()
print Eid
current.execute("SELECT HostID FROM tblHostLookup where HostName = ?",[host])
Hid = current.fetchone()
print Hid
current.execute("INSERT INTO tblExtensionHistory(HostID,ExtensionID) VALUES(?,?)",[Hid,Eid])
EDIT 2 :
The database schema is :
table 1:
CREATE TABLE tblHostLookup (
HostID INTEGER PRIMARY KEY AUTOINCREMENT,
HostName TEXT);
table2:
CREATE TABLE tblExtensionLookup (
ExtensionID INTEGER PRIMARY KEY AUTOINCREMENT,
ExtensionName TEXT);
table3:
CREATE TABLE tblExtensionHistory (
ExtensionHistoryID INTEGER PRIMARY KEY AUTOINCREMENT,
HostID INTEGER,
FOREIGN KEY(HostID) REFERENCES tblHostLookup(HostID),
ExtensionID INTEGER,
FOREIGN KEY(ExtensionID) REFERENCES tblExtensionLookup(ExtensionID));

It's hard to be sure without full details, but I think I can guess the problem.
If you use the INSERT statement without column names, the values must exactly match the columns as given in the schema. You can't skip over any of them.*
The right way to fix this is to just use the column names in your INSERT statement. Something like:
current.execute("INSERT INTO tblExtensionHistory (HostID, ExtensionID) VALUES (?,?)",
[Hid, Eid])
Now you can skip any columns you want (as long as they're autoincrement, nullable, or otherwise skippable, of course), or provide them in any order you want.
For your second problem, you're trying to pass in rows as if they were single values. You can't do that. From your code:
Eid = current.fetchone()
This will return something like:
[3]
And then you try to bind that to the ExtensionID column, which gives you an error.
In the future, you may want to try to write and debug the SQL statements in the sqlite3 command-line tool and/or your favorite GUI database manager (there's a simple extension that runs in for Firefox if you don't want anything fancy) and get them right, before you try getting the Python right.
* This is not true with all databases. For example, in MSJET/Access, you must skip over autoincrement columns. See the SQLite documentation for how SQLite interprets INSERT with no column names, or similar documentation for other databases.

Adding primary key to existing MySQL table in alembic

I am trying to add an 'id' primary key column to an already existing MySQL table using alembic. I tried the following...
op.add_column('mytable', sa.Column('id', sa.Integer(), nullable=False))
op.alter_column('mytable', 'id', autoincrement=True, existing_type=sa.Integer(), existing_server_default=False, existing_nullable=False)
but got the following error
sqlalchemy.exc.OperationalError: (OperationalError) (1075, 'Incorrect table definition; there can be only one auto column and it must be defined as a key') 'ALTER TABLE mytable CHANGE id id INTEGER NOT NULL AUTO_INCREMENT' ()
looks like the sql statement generated by alembic did not add PRIMARY KEY at the end of the alter statement. Could I have missed some settings?
Thanks in advance!

I spent some time digging through the alembic source code, and this doesn't seem to be supported. You can specify primary keys when creating a table, but not when adding columns. In fact, it specifically checks and won't let you (link to source):
# from alembic.operations.toimpl.add_column, line 132
for constraint in t.constraints:
if not isinstance(constraint, sa_schema.PrimaryKeyConstraint):
operations.impl.add_constraint(constraint)
I looked around, and adding a primary key to an existing table may result in unspecified behavior - primary keys aren't supposed to be null, so your engine may or may not create primary keys for existing rows. See this SO discussion for more info: Insert auto increment primary key to existing table
I'd just run the alter query directly, and create primary keys if you need to.
op.execute("ALTER TABLE mytable ADD id INT PRIMARY KEY AUTO_INCREMENT;")
If you really need cross-engine compatibility, the big hammer would be to (1) create a new table identical to the old one with a primary key, (2) migrate all your data, (3)delete the old table and (4) rename the new table.
Hope that helps.

You have to remove the primary key that is in the table and then create a new one that includes all columns that you want as the primary key.
eg. In psql use \d <table name> to define the schema, then check the primary key constraint.
Indexes:
"enrollments_pkey" PRIMARY KEY, btree (se_crs_id, se_std_id)
then use this information in alembic
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.add_column('enrollments', sa.Column(
'se_semester', sa.String(length=30), nullable=False))
op.drop_constraint('enrollments_pkey', 'enrollments', type_='primary')
op.create_primary_key('enrollments_pkey', 'enrollments', [
'se_std_id', 'se_crs_id', 'se_semester'])
The results after running \d enrollments should be updated to
Indexes:
"enrollments_pkey" PRIMARY KEY, btree (se_std_id, se_crs_id, se_semester)
This solution worked fine for me.

Issue with SQLite DROP TABLE statement

EDIT: At this point, I found the errant typo that was responsible, and my question has become "How did the typo that I made cause the error that I received" and "How might I have better debugged this in the future?"
I've setup a database script for SQLite (through pysqlite) as follows:
DROP TABLE IF EXISTS LandTerritory;
CREATE TABLE LandTerritory (
name varchar(50) PRIMARY KEY NOT NULL UNIQUE,
hasSC boolean NOT NULL DEFAULT 0
);
I'm expecting this to always run without error. However, if I run this script twice, (using the sqlite.Connection.executescript method) I get this error:
OperationalError:table LandTerritory already exists
Trying to debug this myself, I run DROP TABLE LandTerritory on its own and get:
sqlite3.OperationalError: no such table: main.LandTerrito
I'm guessing this has something to do with the "main." part, but I'm not sure what.
EDIT:
Okay PRAGMA foreign_keys=ON is definitely involved here, too. When I create my connection, I turned on foreign_keys. If I don't turn that on, I don't seem to get this error.
And I should have mentioned that there's more to the script, but I had assumed the error was occurring in these first 2 statements. The rest of the script just does the same, drop table, define table. A few of the tables have foreign key references to LandTerritory.
Is there a way to get something like line number information about the sqlite errors? That would be really helpful.
EDIT 2:
Okay, here's another table in the script that references the first.
DROP TABLE IF EXISTS LandAdjacent;
CREATE TABLE LandAdjacent (
tname1 varchar(50) NOT NULL,
tname2 varchar(50) NOT NULL,
PRIMARY KEY (tname1, tname2),
/* Foreign keys */
FOREIGN KEY (tname1)
REFERENCES LandTerrito
ON DELETE CASCADE
ON UPDATE CASCADE,
FOREIGN KEY (tname2)
REFERENCES LandTerritory(name)
ON DELETE CASCADE
ON UPDATE CASCADE
);
Looking at this, I found were the "LandTerrito" came from, somehow a few characters got cut off. I'm guessing fixing this may fix my problem.
But I'm really confused how a broken line in this table led to the script running correctly the first time, and then giving me an error related to a different table when I run it the second time, and how foreign keys played into this.
I guess, to reiterate from above, is there a better way to debug this sort of thing?

The source of the error is your typo
REFERENCES LandTerrito
in line 8 of your script. This leads to the "missing" table LandTerrito in the CREATE TABLE LandAdjacent statement.
If you run your two CREATE TABLE statements Sqlite wont complain. But if you have PRAGMA foreign_keys=ON; and try to run an INSERT or DELETE statement on the table LandAdjacent, you'll get the error no such table: main.LandTerrito.
Because of the foreign key constraints DROP TABLE on LandTerritory however will result in a DELETE on the table LandAdjacent, which triggers the error.
The following things will avoid the error
set PRAGMA foreign_keys=ON; before you drop the table (tested) or
add a dummy table LandTerrito (tested) or
drop LandAdjacent first, then LandTerritory (tested) or
dont use ON DELETE CASCADE (not tested)
and of course correcting the original typo.

Put a "GO" (or whatever equivalent is used in SQLlite) to terminate a batch between the drop table statement and the create statement