We're trying to migrate a large table (bigger than local memory space) from a postgresql database to a local sqlite3 database using sqlalchemy, but we're getting a syntax error when trying to create the table in the sqlite3 database. Reading through the other posts, the recommended fixes don't seem to apply to this error. Is there some usage detail we're missing here?
# reflect table from remote postgresql database
read_engine = create_engine('postgresql://dbusername:dbuserpassword#server/dbname')
meta = MetaData()
raw = Table('db_table', meta, autoload=True, autoload_with=read_engine)
# write table to local sqlite3 database
write_engine = create_engine('sqlite:///db_local.sqlite3')
table.create(write_engine)
This is the error we get:
OperationalError: (sqlite3.OperationalError) near "(": syntax error
[SQL:
CREATE TABLE db_table (
id INTEGER DEFAULT nextval('db_table_id_seq'::regclass) NOT NULL,
created_at TIMESTAMP,
updated_at TIMESTAMP,
data TEXT,
device_key TEXT,
format TEXT,
uuid TEXT,
CONSTRAINT db_table_pkey PRIMARY KEY (id)
)
]
I think the issue is that SQLite doesn't support DEFAULT nextval(...) -- nextval() is a Postgres-specific function that isn't part of the SQLite database.
You'll need to re-write the CREATE TABLE statement to use AUTOINCREMENT, I believe.
Disclosure: I am an EnterpriseDB (EDB) employee
Related
I'm not able to connect to temporary tables created on an SQL server using SQLAlchemy.
I connect to the server:
engine = create_engine(URL, poolclass=StaticPool)
I fill a temporary table with data from a pandas dataframe:
df_tmp.to_sql('#table_test', con=engine)
The table exists on the server:
res = engine.execute('SELECT * FROM tempdb..#table_test')
print(res)
which returns a list of tuples of my data. But then when I try to make an SQLAlchemy table it fails with a NoSuchTableError:
from sqlalchemy import create_engine, MetaData, Table
metadata = MetaData(engine)
metadata.create_all()
table = Table('#table_test', metadata, autoload=True, autoload_with=engine)
I also tried this, which gives the same error:
table = Table('tempdb..#table_test', metadata, autoload=True, autoload_with=engine)
And I also tried creating a blank table with an SQL command, which gives the same error when I try to read it with SQLAlchemy:
engine.execute('CREATE TABLE #table_test (id_number INT, name TEXT)')
Does SQLAlchemy support temporary tables? If so what is going wrong here? I'd like to have the temporary table as an sqlalchemy.schema.Table object if possible, as then it fits with all my other code.
(re: comments to the question)
Actually, it is a limitation of the current mechanism by which SQLAlchemy's mssql dialect checks for the existence of a table. It queries INFORMATION_SCHEMA.TABLES for the current catalog (database), and #temp tables do not appear in that view. They do appear — after a fashion, and in a not-particularly-helpful way — if we USE tempdb and then query INFORMATION_SCHEMA.TABLES from there.
For now, I have created a GitHub issue here to see if we can improve on this.
Update 2020-09-01
The changes for the above GitHub issue have been merged into SQLAlchemy's master branch and will be included in version 1.4. If you want to take advantage of this feature before 1.4 is officially released you can install SQLAlchemy via
pip install git+https://github.com/sqlalchemy/sqlalchemy.git
Could you please help me figure out what I am doing wrong?
I am trying to insert a new player into the players table.
this is the python code:
def registerPlayer(name):
code for connecting to db and cursor
c.execute("INSERT INTO players(player_name) VALUES({name});".format(name=name))
code for committing to db and closing the connection
here is my table schema:
CREATE TABLE players(
player_id serial PRIMARY KEY,
player_name varchar(50) NOT NULL
);
below is error:
psycopg2.ProgrammingError: syntax error at or near "Nalaar" LINE 1:
INSERT INTO players(player_name) VALUES(Chandra Nalaar);
You should never use string formatting for placing values into sql query. Instead, you should use %s and pass the name in the vars parameter. The reason behind doing it this way is because it helps you convert the parameters into the appropriate data types.
Btw, having a ; at the end in the sql string is redundant when called by cursor.execute, since it does it for you automatically.
c.execute("INSERT INTO players(player_name) VALUES(%(name)s)", {"name":name})
See this page for further details:
http://initd.org/psycopg/docs/usage.html#query-parameters
I have an existing SQL Server (2012) DB with many tables having a primary key of Numeric(9, 0) type. For all intents and purposes they are integers.
The ORM mapping (generated using sqlacodegen) looks like:
class SomeTable(Base):
__tablename__ = 'SOME_TABLE'
__table_args__ = {'schema': 'dbo'}
SOME_TABLE_ID = Column(Numeric(9, 0), primary_key=True)
some_more_fields_here = XXX
Sample code to insert data:
some_table = SomeTable(_not_specifying_SOME_TABLE_ID_explicityly_)
session.add(some_table)
session.flush() # <---BOOM, FlushError here
When I try to insert data into such tables, my app crashes on session.flush() with the following error:
sqlalchemy.orm.exc.FlushError: Instance SomeTable ... has a NULL identity key. If this is an auto-generated value, check that the database table allows generation of new primary key values, and that the mapped Column object is configured to expect these generated values. Ensure also that this flush() is not occurring at an inappropriate time, such aswithin a load() event.
If I replace Numeric with BigInteger then everything works fine. I did some digging and the query generated with Numeric is like this:
INSERT INTO dbo.[SOME_TABLE] (_columns_without_SOME_TABLE_ID)
VALUES (...)
seems like a valid query from SQL point of view, but Sqlalchemy raises the above exception there
The query generated using BigInteger is as follows:
INSERT INTO dbo.[SOME_TABLE] (_columns_without_SOME_TABLE_ID)
OUTPUT inserted.[SOME_TABLE_ID]
VALUES (...)
I also found this peace of documentation about autoincrement property. And sure enough, it explains the behavior I observe, i.e. autoincrement only works with integers.
So my question is whether there is some kind of workaround to make autoincrement work with Numeric columns without converting them to BigInteger?
My system configuration is - Centos 7 64 bit, Python 3.5.2, Sqlalchemy 1.1.4, pymssql 2.2.0, SQL Server 2012.
I am doing an INSERT into a postgres DB, which has an auto increment field called id.
How do I get the value of id after doing the insert in Python? There are good references on this site to this for MySQL databases (i.e. using cursor.lastrowid, or connection.insert_id()) but these don't seem to work with my postgres DB.
The id field is using a sequence to auto increment. So the id field is not null, default, nextval('table_id_seq')
Thanks
Pls try that:
cursor.execute("INSERT INTO .... RETURNING
id_of_new_row = cursor.fetchone()[0]
I'm having a little trouble querying my sqlite DB. Has anyone seen this before? It may very well be that I am doing something silly
Here we have the schema:
BEGIN;
CREATE TABLE "homepage_user" (
"username" varchar(20) NOT NULL PRIMARY KEY,
"first_name" varchar(20) NOT NULL,
"last_name" varchar(20) NOT NULL
)
;
CREATE TABLE "homepage_oncall" (
"id" integer NOT NULL PRIMARY KEY,
"user_id" varchar(20) NOT NULL REFERENCES "homepage_user" ("username"),
"start_date" datetime,
"end_date" datetime
)
;
COMMIT;
And then when I query that data from django's interactive shell:
...
>>> cursor.execute('SELECT user_id from homepage_oncall').fetchall()
[(u'amencke',), (u'jdoe',), (u'jbloggs',)]
However - any attempt to query 'homepage_oncall.user_id' will be met with a stern rebuke:
>>> query = cursor.execute('SELECT first_name, last_name FROM homepage_user WHERE homepage_user.username = homepage_oncall.user_id AND NOW() BETWEEN homepage_oncall.start_date AND homepage_oncall.end_date')
...
DatabaseError: no such column: homepage_oncall.user_id
I had a look around on here and it seems this issue surfaces after editing a schema after it was created - but (after failing to solve the problem with 'manage.py migrate' I actually deleted and recreated the entire database to rule that out - there was nothing in it anyway
Thanks,
Arthur
Your problem is in your SQL. You SELECT from one table and try to apply a WHERE condition on another table without performing an implicit or explicit JOIN.
Also Django provides a pretty nice ORM, so I would recommend sticking to it instead of running raw queries against the database especially if you are not well versed with writing SQL queries