Develop Reference

can't use pony orm on sqlite3 blob fields

Just trying some basic exercises with pony ORM (and python3.5, sqlite3).
I just want to print a select query of some data I have without further processing to start with. Pony orm does not seem to like that at all....
The sqlite db dump
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE sums (t text, path BLOB, name BLOB, sum text, primary key (path,name));
INSERT INTO "sums" VALUES('directory','','','');
INSERT INTO "sums" VALUES('file','','sums-backup-f.db','6859b35f9f026317c5df48932f9f2a91');
INSERT INTO "sums" VALUES('file','','md5-tree.py','c7af81d4aad9d00e88db7af950c264c2');
INSERT INTO "sums" VALUES('file','','test.db','a403e9b46e54d6ece851881a895b1953');
INSERT INTO "sums" VALUES('file','','sirius-alexa.db','22a20434cec550a83c675acd849002fa');
INSERT INTO "sums" VALUES('file','','sums-reseau-y.db','1021614f692b5d7bdeef2a45b6b1af5b');
INSERT INTO "sums" VALUES('file','','.md5-tree.py.swp','1c3c195b679e99ef18b3d46044f6e6c5');
INSERT INTO "sums" VALUES('file','','compare-md5.py','cfb4a5b3c7c4e62346aa5e1affef210a');
INSERT INTO "sums" VALUES('file','','charles.local.db','9c50689e8185e5a79fd9077c14636405');
COMMIT;
Here is the code I try to run on python3.5 interactive shell:
from pony.orm import *
db = Database()
class File(db.Entity) :
_table_ = 'sums'
t = Required(str)
path = Required(bytes)
name = Required(bytes)
sum = Required(str)
PrimaryKey(path,name)
db.bind('sqlite','/some/edited/path/test.db')
db.generate_mapping()
File.select().show()
And it fails like this :
Traceback (most recent call last):
File "/usr/lib/python3.5/site-packages/pony/orm/core.py", line 5149, in _fetch
try: result = cache.query_results[query_key]
KeyError: (('f', 0, ()), (<pony.orm.ormtypes.SetType object at 0x7fd2d2701708>,), False, None, None, None, False, False, False, ())
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 2, in show
File "/usr/lib/python3.5/site-packages/pony/utils/utils.py", line 75, in cut_traceback
raise exc # Set "pony.options.CUT_TRACEBACK = False" to see full traceback
File "/usr/lib/python3.5/site-packages/pony/utils/utils.py", line 60, in cut_traceback
try: return func(*args, **kwargs)
File "/usr/lib/python3.5/site-packages/pony/orm/core.py", line 5256, in show
query._fetch().show(width)
File "/usr/lib/python3.5/site-packages/pony/orm/core.py", line 5155, in _fetch
used_attrs=translator.get_used_attrs())
File "/usr/lib/python3.5/site-packages/pony/orm/core.py", line 3859, in _fetch_objects
real_entity_subclass, pkval, avdict = entity._parse_row_(row, attr_offsets)
File "/usr/lib/python3.5/site-packages/pony/orm/core.py", line 3889, in _parse_row_
avdict[attr] = attr.parse_value(row, offsets)
File "/usr/lib/python3.5/site-packages/pony/orm/core.py", line 1922, in parse_value
val = attr.validate(row[offset], None, attr.entity, from_db=True)
File "/usr/lib/python3.5/site-packages/pony/orm/core.py", line 2218, in validate
val = Attribute.validate(attr, val, obj, entity, from_db)
File "/usr/lib/python3.5/site-packages/pony/orm/core.py", line 1894, in validate
if from_db: return converter.sql2py(val)
File "/usr/lib/python3.5/site-packages/pony/orm/dbapiprovider.py", line 619, in sql2py
if not isinstance(val, buffer): val = buffer(val)
TypeError: string argument without an encoding
Am I using this wrong, or is this a bug ? I don't mind go filing a bug, but it's the first time I'm using this orm, so I thought it might be better to check first ...

SQLite has a (mis)feature, which allows a column to store an arbitrary value disregarding the column type. Instead of rigid data type, each SQLite column has an affinity, while each value has a storage class which can be different within the same column. For example, you can store text value inside an integer column, and vice versa. See Datatypes In SQLite Version 3 for more information.
The reason for the error is that the table contains values of "wrong" type in its BLOB columns. Correct SQLite binary literal looks like x'abcdef'. The INSERT commands that you use insert UTF8 strings instead.
This problem was somewhat fixed in the latest version of Pony which you can take from GitHub. Now if Pony receives a string value from a BLOB column it just keep that value without throwing an exception.
If you populate the table with Pony, it will writes BLOB data as a correct binary values, so it can read them later without any problem.

In Flask-migrate ValueError: invalid interpolation syntax in connection string at position 15

I am using flask migrate to for database creation & migration in flask with flask-sqlalchemy.
Everything was working fine until I changed my database user password contains '#' then it stopped working so, I updated my code based on
Writing a connection string when password contains special characters
It working for application but not for flask-migration, Its showing error while migrating
i.e on python manage.py db migrate
ValueError: invalid interpolation syntax in u'mysql://user:p%40ssword#localhost/testdb' at position 15
Here password is p#ssword and its escaped by urlquote (see above question link).
Full error stack:
Traceback (most recent call last):
File "manage.py", line 20, in <module>
manager.run()
File "/usr/local/lib/python2.7/dist-packages/flask_script/__init__.py", line 412, in run
result = self.handle(sys.argv[0], sys.argv[1:])
File "/usr/local/lib/python2.7/dist-packages/flask_script/__init__.py", line 383, in handle
res = handle(*args, **config)
File "/usr/local/lib/python2.7/dist-packages/flask_script/commands.py", line 216, in __call__
return self.run(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/flask_migrate/__init__.py", line 177, in migrate
version_path=version_path, rev_id=rev_id)
File "/usr/local/lib/python2.7/dist-packages/alembic/command.py", line 117, in revision
script_directory.run_env()
File "/usr/local/lib/python2.7/dist-packages/alembic/script/base.py", line 407, in run_env
util.load_python_file(self.dir, 'env.py')
File "/usr/local/lib/python2.7/dist-packages/alembic/util/pyfiles.py", line 93, in load_python_file
module = load_module_py(module_id, path)
File "/usr/local/lib/python2.7/dist-packages/alembic/util/compat.py", line 79, in load_module_py
mod = imp.load_source(module_id, path, fp)
File "migrations/env.py", line 22, in <module>
current_app.config.get('SQLALCHEMY_DATABASE_URI'))
File "/usr/local/lib/python2.7/dist-packages/alembic/config.py", line 218, in set_main_option
self.set_section_option(self.config_ini_section, name, value)
File "/usr/local/lib/python2.7/dist-packages/alembic/config.py", line 245, in set_section_option
self.file_config.set(section, name, value)
File "/usr/lib/python2.7/ConfigParser.py", line 752, in set
"position %d" % (value, tmp_value.find('%')))
ValueError: invalid interpolation syntax in u'mysql://user:p%40ssword#localhost/testdb' at position 15
Please help

In the migrations/env.py file, you will find the code that is responsible for this issue.
config.set_main_option('sqlalchemy.url',
current_app.config.get('SQLALCHEMY_DATABASE_URI'))
If there are % signs in the SQLALCHEMY_DATABASE_URI, this will cause an error.
You can solve this by editing the migrations/env.py file, and changing the offending line as follows
db_url_escaped = current_app.config.get('SQLALCHEMY_DATABASE_URI').replace('%', '%%')
config.set_main_option('sqlalchemy.url', db_url_escaped)
Also see the documentation of set_main_option:
Note that this value is passed to ConfigParser.set, which supports variable interpolation using pyformat (e.g. %(some_value)s). A raw percent sign not part of an interpolation symbol must therefore be escaped, e.g. %%. The given value may refer to another value already in the file using the interpolation format.

I have a solution for this issue after experiencing it as well.
There's an issue with '%' (percent signs) in the db connection URI after you urlencode the string.
I tried substituting the percent sign with double percent signs ('%%') which gets me past the interpolation error. However, that resulted in not being able to connect to the database because of an incorrect password.
Solution I'm going with for now is to avoid using '%' in my db password. Not a satisfactory solution, but will do for now. I'll make a note in "alembic"'s github of the issue. Seems using RawConfigParser in their package could help avoid this issue.

You may want to look at http://docs.sqlalchemy.org/en/latest/dialects/mysql.html#mysql-unicode
I was having the same issue with my password and the mysql connector. using the mysql+pymysql connector allowed me to connect in application and in migration scripts.

Django - Error selecting objects: "ProgrammingError: operator does not exist: character varying = integer)

My models.py contains
class Patient(models.Model):
cpf_id = models.CharField(max_length=15, unique=True)
name_txt = models.CharField(max_length=50)
nr_record = models.AutoField(primary_key=True)
The database is postgresql.
When I try to select an object with
>>> Patient.objects.get(nr_record=1)
I get the error
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py", line 151, in get
return self.get_queryset().get(*args, **kwargs)
.
.
.
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
ProgrammingError: operator does not exist: character varying = integer
LINE 1: ...d" FROM "quiz_patient" WHERE "quiz_patient"."nr_record" = 1
^
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
I'm newbie in Django-python. But until I know models.Autofield is integer type.
Searched web, but despite posts with same error message, didn't find nothing like my problem.

The problem is most certainly with your database. Maybe you had a previous version of the model where nr_record was not an integer?
Try removing the nr_record column in the database and adding it again.

Web2py Query Legacy Database

I have a legacy database called my_legacy_db which is separate from the normal db.
my_legacy_db
users
- email
- username
- name
So cliff, your first part would work to generate field names and put everything in a dict to build the query's. The problem is when I do this query:
db().select(my_legacy_db.users)
I get this error:
In [20] : db().select(my_legacy_db.users)
Traceback (most recent call last):
File "/opt/web-apps/web2py/gluon/contrib/shell.py", line 233, in run
exec compiled in statement_module.__dict__
File "<string>", line 1, in <module>
File "/opt/web-apps/web2py/gluon/dal.py", line 7578, in select
return adapter.select(self.query,fields,attributes)
File "/opt/web-apps/web2py/gluon/dal.py", line 1307, in select
sql = self._select(query, fields, attributes)
File "/opt/web-apps/web2py/gluon/dal.py", line 1196, in _select
raise SyntaxError, 'Set: no tables selected'
SyntaxError: Set: no tables selected
In [21] : print (flickr_db.users)
users
In [22] : print flickr_db
<DAL {'_migrate_enabled': True, '_lastsql': "SET sql_mode='NO_BACKSLASH_ESCAPES';", '_db_codec': 'UTF-8', '_timings': [('SET FOREIGN_KEY_CHECKS=1;', 0.0002460479736328125), ("SET sql_mode='NO_BACKSLASH_ESCAPES';", 0.00025606155395507812)], '_fake_migrate': False, '_dbname': 'mysql', '_request_tenant': 'request_tenant', '_adapter': <gluon.dal.MySQLAdapter object at 0x91375ac>, '_tables': ['users'], '_pending_references': {}, '_fake_migrate_all': False, 'check_reserved': None, '_uri': 'mysql://CENSORED', 'users': <Table 'username': <gluon.dal.Field object at 0x9137b6c>, '_db': <DAL {...}>, 'cycled': <gluon.dal.Field object at 0x94d0b8c>, 'id': <gluon.dal.Field object at 0x95054ac>, 'ALL': <gluon.dal.SQLALL object at 0x969a7ac>, '_sequence_name': 'users_sequence', 'name': <gluon.dal.Field object at 0x9137ecc>, '_referenced_by': [], '_singular': 'Users', '_common_filter': None, '_id': <gluon.dal.Field object at 0x95054ac>}>, '_referee_name': '%(table)s', '_migrate': True, '_pool_size': 0, '_common_fields': [], '_uri_hash': 'dfb3272fc537e3339819a1549180722e'}>
Am I doing something wrong here? Is the legacy db not built in /databases right? Thanks in advance for any help.
UPDATE: I tried as anthony suggested in the model shell:
In [3] : db(my_legacy_db.users).select()
Traceback (most recent call last):
File "/opt/web-apps/web2py/gluon/contrib/shell.py", line 233, in run
exec compiled in statement_module.__dict__
File "<string>", line 1, in <module>
File "/opt/web-apps/web2py/gluon/dal.py", line 7577, in select
fields = adapter.expand_all(fields, adapter.tables(self.query))
File "/opt/web-apps/web2py/gluon/dal.py", line 1172, in expand_all
for field in self.db[table]:
File "/opt/web-apps/web2py/gluon/dal.py", line 6337, in __getitem__
return dict.__getitem__(self, str(key))
KeyError: 'users'
Now I know that users is defined in my_legacy_db, and all syntax is correct. Is this an error that is there because the db files aren't generating correctly? Or am I still doing something wrong with the select syntax?

If "users" is the name of a table and you want to select all records and all fields, you would do:
db(my_legacy_db.users).select()
The query goes inside db(), not inside select() (select() is where you list the fields you want returned, or leave it empty if you want all fields). Note, in the above line, my_legacy_db.users is not actually a query but just a table -- that's a shortcut to tell web2py you want all records in the table.
You could also do:
db().select(my_legacy_db.users.ALL)
That indicates you want all fields, and by excluding the query, it assumes you want all records in the table.
See the book for more details.

changing django default model settings

I'm just starting with the django creating your own app tutorial (creating a Poll) I'm deviating slightly as I'm wanting to create an app using my own database model that already exists.
And in the tutorial it says
Table names are automatically
generated by combining the name of
the app (polls) and the lowercase
name of the model -- poll and choice.
(You can override this behavior.)
Primary keys (IDs) are added
automatically. (You can override
this, too.)
By convention, Django appends
"_id" to the foreign key field
name. Yes, you can override this,
as well.
But I can't see where it mentions how you can override this behaviour? I've defined my model as so
from django.db import models
# Create your models here.
class Channels(models.Model):
channelid = models.IntegerField()
channelid.primary_key = True
channelname = models.CharField(max_length=50)
Now when I go in to the shell this is what I get
>>> from tvlistings.models import *
>>> Channels.objects.all()
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "C:\Python26\lib\site-packages\django\db\models\query.py", line 67, in __
repr__
data = list(self[:REPR_OUTPUT_SIZE + 1])
File "C:\Python26\lib\site-packages\django\db\models\query.py", line 82, in __
len__
self._result_cache.extend(list(self._iter))
File "C:\Python26\lib\site-packages\django\db\models\query.py", line 271, in i
terator
for row in compiler.results_iter():
File "C:\Python26\lib\site-packages\django\db\models\sql\compiler.py", line 67
7, in results_iter
for rows in self.execute_sql(MULTI):
File "C:\Python26\lib\site-packages\django\db\models\sql\compiler.py", line 73
2, in execute_sql
cursor.execute(sql, params)
File "C:\Python26\lib\site-packages\django\db\backends\util.py", line 15, in e
xecute
return self.cursor.execute(sql, params)
File "C:\Python26\lib\site-packages\django\db\backends\mysql\base.py", line 86
, in execute
return self.cursor.execute(query, args)
File "C:\Python26\lib\site-packages\MySQLdb\cursors.py", line 166, in execute
self.errorhandler(self, exc, value)
File "C:\Python26\lib\site-packages\MySQLdb\connections.py", line 35, in defau
lterrorhandler
raise errorclass, errorvalue
DatabaseError: (1146, "Table 'tvlistings.tvlistings_channels' doesn't exist")
Obviously it can't find the table tvlistings_channels as it's actually called channels. So how do I change the default?

You can override Model behavior in Django largely through the use of an inner Meta class
db_table allows you to rename the table name
specifying another field as the primary key will have it use that rather than a surrogate key (not in the Meta class, just in the model itself)

You should try and work your way all through the tutorial before you try and customise things. All these things are covered in the actual documentation, but it's best to have a basic understanding of things first before diving into that.
FWIW, here are the docs on defining your own primary key and specifying a table name. But really, do the tutorial as written first.