Develop Reference

How to convert sqlalchemy dialect specific column type to another dialect?

I would like to read table's metadata from one DB and create it another DB using Python & SQL Alchemy. It works perfectly if source and destination DBs are of the same type/dialect. But when I'm trying to replicate a table from MySQL DB to Oracle DB i'm experiencing problems with incompatible data types. For example in MySQL there are columns of SMALLINT / TINYINT / BIGINT / DATETIME data types, that don't exist in Oracle DB.
When I'm trying to convert one of those types using .dialect_impl(ora_dialect) i'm getting an TypeError: 'memoized_property' object is not subscriptable error.
Here is a small example:
from sqlalchemy.dialects.mysql import BIGINT
from sqlalchemy.dialects.oracle import dialect as ora_dialect
i_mysql = BIGINT(20)
i_ora = i_mysql.dialect_impl(ora_dialect)
i'm getting the following error:
i_ora = i_mysql.dialect_impl(ora_dialect)
Traceback (most recent call last):
File "/opt/anaconda3/envs/umm/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3418, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-135-998536c3e513>", line 1, in <module>
i_ora = i_mysql.dialect_impl(ora_dialect)
File "/opt/anaconda3/envs/umm/lib/python3.8/site-packages/sqlalchemy/sql/type_api.py", line 459, in dialect_impl
return dialect._type_memos[self]["impl"]
TypeError: 'memoized_property' object is not subscriptable
What would be the right way to perform such a type conversion from one sqlalchemy dialect to another one?
PS here is the code that is used in order to get table's metadata:
meta = MetaData(bind=engine)
meta.reflect(views=True, resolve_fks=False)
tab = meta.tables[table_name]
I've seen the parameter meta.reflect(**dialect_kwargs), but I couldn't find any usage example...

define_table does not create table in database

I am running define_tables in the recommended way:
db = DAL('postgres://user:XXXX#localhost:5432/mydb', migrate_enabled=False, auto_import=False, lazy_tables=True)
db.define_table('auth_user',
Field('email', unique=True),
Field('password', length=512, type='password', readable=False, label='Password'),
...)
This gets executed without errors, but no table is created in the database. Whenever I try to insert a new user:
relation "auth_user" does not exist
What can be going on? Once the tables are created (manually, for example), the application works fine. I am using a postgres backend. This happens no matter what value I give to lazy_tables
EDIT
This is the full test script:
from gluon import DAL
from gluon import Field
db = DAL('postgres://user:pass#localhost:5432/mydb', migrate_enabled=False)
db.define_table(
'auth_user',
Field('email', type='string', unique=True),
Field('password', type='password'),
Field('registration_key', type='string', length=512, writable=False, readable=False, default=''),
Field('reset_password_key', type='string', length=512, writable=False, readable=False, default=''),
Field('registration_id', type='string', length=512, writable=False, readable=False, default=''),
)
db.commit()
print db.tables
db.auth_user.insert(email='g#b.c')
And I get the following output:
['auth_user']
Traceback (most recent call last):
File "xxx.py", line 19, in <module>
db.auth_user.insert(email='g#b.c')
File "/tmp/web2py/gluon/dal.py", line 9293, in insert
ret = self._db._adapter.insert(self, self._listify(fields))
File "/tmp/web2py/gluon/dal.py", line 1361, in insert
raise e
psycopg2.ProgrammingError: relation "auth_user" does not exist
LINE 1: INSERT INTO auth_user(reset_password_key,registration_id,reg...
The table is somehow "created" (in memory?), but it is not really in the postgres database. What does this mean?

Simply remove migrate_enabled=False, which turns off migrations and therefore prevents the creation or modification of database tables. There is also no need to explicitly set auto_import=False as that is already the default.
If the above doesn't help, it is possible that web2py did successfully create such a table previously and it was removed without web2py knowing about it. If the application's /databases folder includes a file with a name like *_auth_user.table, delete that file and try again.
If that's not the issue, check the /databases/sql.log file and confirm that web2py attempted to create the table. Most likely, something in your system configuration is preventing the table from being created.
UPDATE: From your edit, it appears you are using the DAL outside of a web2py application. Because you have not specified the folder argument to the DAL() constructor, it will save the *.table migration files in the current working directory, and it will not create a sql.log file. In this case, it is best to create a separate folder for the migration and log files:
DAL('postgres://user:pass#localhost:5432/mydb', folder='/path/to/folder')
In that case, it will save all of the *.table migration files and the sql.log file in the specified folder.

Does SQLAlchemy support H2DB?

Does SQLAlchemy support H2 db? I'm using pyramid and would like to connect to H2 db database. If using postgres dialect, I'm getting error like the following:
File "/Users/homecamera/gotocamera/hcadmin/env/lib/python2.7/site-packages/SQLAlchemy-0.7.3-py2.7-macosx-10.4-x86_64.egg/sqlalchemy/dialects/postgresql/base.py", line 871, in initialize
super(PGDialect, self).initialize(connection)
File "/Users/homecamera/gotocamera/hcadmin/env/lib/python2.7/site-packages/SQLAlchemy-0.7.3-py2.7-macosx-10.4-x86_64.egg/sqlalchemy/engine/default.py", line 181, in initialize
self.get_isolation_level(connection.connection)
File "/Users/homecamera/gotocamera/hcadmin/env/lib/python2.7/site-packages/SQLAlchemy-0.7.3-py2.7-macosx-10.4-x86_64.egg/sqlalchemy/dialects/postgresql/base.py", line 910, in get_isolation_level
cursor.execute('show transaction isolation level')
ProgrammingError: Syntax error in SQL statement "SELECT "; expected "TOP, LIMIT, DISTINCT, ALL, *, NOT, EXISTS"; SQL statement:
show transaction isolation level [42001-140]
DETAIL: org.h2.jdbc.JdbcSQLException: Syntax error in SQL statement "SELECT "; expected "TOP, LIMIT, DISTINCT, ALL, *, NOT, EXISTS"; SQL statement:
show transaction isolation level [42001-140]

AFAIK there's no official support for either HSQLDB dialects or native H2 dialect.
Using Postgres dialect with H2 (without using HSQLDB) would definitely result in error you're getting.
You might have better luck trying sqlalchemy-jython and using H2 dialect.

Just in case anyone stumbles over this again, I tried to get this running (as an alternative to sqlite) and it only partially works, and the only driver that works is pg8000.
With the server running using:
nohup java -cp /opt/h2/bin/h2*.jar org.h2.tools.Server -pg -pgAllowOthers -pgPort 5435 -baseDir /opt/h2-data &
This code works in sqlalchemy:
from sqlalchemy import create_engine
engine = create_engine('postgresql+pg8000://sa:sa#localhost:5435/main')
engine.execute("SELECT 1")
However this code throws an exception:
from sqlalchemy_utils import create_database
create_database('postgresql+pg8000://sa:sa#localhost:5435/main')
Exception:
sqlalchemy.exc.ProgrammingError: (pg8000.core.ProgrammingError) ('ERROR',
'HY000', 'General error: "java.lang.IllegalStateException: output binary
format is undefined" [50000-196]', 'org.h2.jdbc.JdbcSQLException: General
error: "java.lang.IllegalStateException: output binary format is undefined"
[50000-196]') [SQL: 'select version()']

How do you save models in Elixir

I've got a database created using Django models which I'm now accessing using SQLAlchemy and Elixir. The querying works and I can pull items out of the database perfectly happily but when I edit them and try to save them it throws the following exception:
>>> p = Problem.query.first()
>>> p
<Problem('Test Problem', 'This is a test problem so the database has something in it', 'SBMT')>
>>> p.name
u'Test Problem'
>>> p.name = "Test_Problem"
>>> p.save()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/site-packages/elixir/entity.py", line 1116, in save
return self._global_session.save(self, *args, **kwargs)
AttributeError: 'Session' object has no attribute 'save'
What am I doing wrong? Have I missed a crucial part of the set up that is preventing me from saving things to the database or is it a problem with my versions of elixir and SQLAlchemy?
I've already run setup_all() and the metadata.bind is all set, hence I can query the database.

I'm no Elixir expert, but from the documentation it looks like it uses something called a global session.
To save the changes to the database, you do a session.commit(), there's no save() method like in Django.

SQLAlchemy IntegrityError

I'm having a problem using SQLAlchemy with PySide(PyQt). I'm trying to pop-up a QtGui.QDialog, but when I do this SQLAlchemy throws an exception:
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\preo\preodb\dbviewandmodel.py", line 32, in rowCount
return len(self.rows())
File "C:\Python27\lib\site-packages\preo\preodb\dbviewandmodel.py", line 30, in rows
return self.tableobj.query.all()
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\orm\query.py", line 1579, in all
return list(self)
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\orm\query.py", line 1688, in __iter__
self.session._autoflush()
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\orm\session.py", line 862, in _autoflush
self.flush()
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\orm\session.py", line 1388, in flush
self._flush(objects)
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\orm\session.py", line 1469, in _flush
flush_context.execute()
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\orm\unitofwork.py", line 302, in execute
rec.execute(self)
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\orm\unitofwork.py", line 446, in execute
uow
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\orm\mapper.py", line 1878, in _save_obj
execute(statement, params)
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\engine\base.py", line 1191, in execute
params)
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\engine\base.py", line 1271, in _execute_clauseelement
return self.__execute_context(context)
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\engine\base.py", line 1302, in __execute_context
context.parameters[0], context=context)
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\engine\base.py", line 1401, in _cursor_execute
context)
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\engine\base.py", line 1394, in _cursor_execute
context)
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\engine\default.py", line 299, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (IntegrityError) ('23000', "[23000] [Microsoft][ODBC
SQL Server Driver][SQL Server]Violation of UNIQUE KEY
constraint 'UQ__users__F3DBC5720DAF0CB0'. Cannot insert duplicate key in
object 'dbo.users'. (2627) (SQLExecDirectW); [01000] [Microsoft][ODBC SQL Server
Driver][SQL Server]The statement has been terminated. (3621)") u'INSERT INTO users
(username, fullname, email, passwordmd5) OUTPUT inserted.id VALUES (?, ?, ?, ?)'
(None, None, None, None)
This is particularly troubling because I have no code, anywhere, that even attempts to insert records into SQL; I am only ever attempting to query data from the database. In fact, my DB model is read-only with respect to what PySide/PyQt are doing (i.e., I'm using a QtGui.QTableView model/view and there is no insertRows function in that model).
I have no idea what's going on or how to solve it - again, I have no code to modify SQL records at all, but still SQLAlchemy attempts to be inserting blank records into one of my SQL tables. All I can see, in the background, is the QTableView data model is querying the database A LOT. It just seems that when I popup this QDialog (which does have some code in it to query some table column) this error is thrown. Oddly, this isn't consistent, sometime the popup appears first before the exception, sometimes the popup appears after the exception. Under normal circumstances, the QTableView data model works great, just not when I popup this dialog (and ironically, the popup isn't using any QTableView at all, just standard widgets like QLineEdit, QTextEdit, etc.)
If it helps, I'm using Python 2.7 with SQLAlchemy 0.6.6 (also with Elixir 0.7.1), and PySide 1.0.0 (and PyQt4 4.8.3). I'm on Windows 7 using SQL 2008 R2 (Express). And yes, I've tried rebooting the PC, but the problem still occurs after a reboot. I'm reluctant to post more code because I have a lot of it in this particular project and I can't nail down the problem anything specific.
I'm hoping someone might know of oddities in SQLAlchemy and/or PyQt that might be related to this. I'm also hoping I can continue using SQLAlchemy as I have a large data model built; I'm reluctant, at this point, to abandon this and use PyQt's SQL features.

I've managed to make this problem go away, but it's still not really clear to me why SQLAlchemy was trying to insert rows in my database - that really bothers me, but it's not happening anymore.
At any rate, what was, I think, happening, was related to my SQLAlchemy data model and the way I was accessing it, here is a snippet of that model:
from elixir import *
metadata.bind = 'mssql+pyodbc://username:password/dbname'
metadata.bind.echo = False
class Users(Entity):
using_options(tablename = 'users')
username = Field(String(50), unique=True)
fullname = Field(String(255))
email = Field(String(255))
passwordmd5 = Field(String(32))
def __repr__(self):
return "<Users ({})({})({})>".format(self.username, self.fullname, self.email)
def prettyname(self):
return {'username':'User Name', 'fullname':'Full Name', 'email':'Email Address', 'passwordmd5':'$hidden$'}
In my code I needed a way of getting 'pretty' label names for a GUI without having to hard code this in a GUI (I've been trying to create a dynamic way of building GUI forms). So, I added the 'prettyname' method to my data model to give me some application specific metadata in that data model. All I'm doing is returning a dictionary of items.
I had a secondary problem in that sometimes I needed to get this data from the class instance for Users and sometimes for a query result for Users (for example, Users.get_by(id=1)). As it turned out, retrieving this data had to be done in two ways. In the class instances I had to get the value this way:
prettyname = Users().prettyname()['username']
But when I was using query results it was:
prettyname = queryresult.prettyname()['username']
SQLAlchemy seems to have a real problem when I was using the former method (the class instance method) - as this was being used everytime I was seeing the crash. When I was using the latter instance I was never seeing a crash. Still, I needed access to that metadata in the class instance.
The fix, or should I say what turned out to fix this came from another Stackoverflow article (thank you everyone at Stackoverflow - I'd be nothing without you). I changed the structure of the dbmodel:
class Users(Entity):
using_options(tablename = 'users')
username = Field(String(50), unique=True, info={'prettyname':'User Name'})
fullname = Field(String(255), info={'prettyname':'Full Name'})
email = Field(String(255), info={'prettyname':'Email Address'})
passwordmd5 = Field(String(32), info={'hidden':True})
def __repr__(self):
return "<Users ({})({})({})>".format(self.username, self.fullname, self.email)
This allows me to use a common method of introspection to get the dictionary data in the info argument, regardless if I'm looking at a class instance, or a query result. In this case I use the '.table' method of either the class or query result, then get the column that I need (.c), then use the .info method of that column to return the dictionary.
Whatever the case, now SQLAlchemy no longer tries to arbitrarily insert rows in the database anymore.