I have a python code which queries psql and returns a batch of results using cursor.fetchall().
It throws an exception and fails the process if a casting fails, due to bad data in the DB.
I get this exception:
File "/usr/local/lib/python2.7/site-packages/psycopg2cffi/_impl/cursor.py", line 377, in fetchall
return [self._build_row() for _ in xrange(size)]
File "/usr/local/lib/python2.7/site-packages/psycopg2cffi/_impl/cursor.py", line 891, in _build_row
self._casts[i], val, length, self)
File "/usr/local/lib/python2.7/site-packages/psycopg2cffi/_impl/typecasts.py", line 71, in typecast
return caster.cast(value, cursor, length)
File "/usr/local/lib/python2.7/site-packages/psycopg2cffi/_impl/typecasts.py", line 39, in cast
return self.caster(value, length, cursor)
File "/usr/local/lib/python2.7/site-packages/psycopg2cffi/_impl/typecasts.py", line 311, in parse_date
raise DataError("bad datetime: '%s'" % bytes_to_ascii(value))
DataError: bad datetime: '32014-03-03'
Is there a way to tell the caster to ignore this error and parse this as a string instead of failing the entire batch?
You can "hack" the parser of psycopg2cffi to return DATE objects as strings instead:
If you look in the code you can see the registration of the DATE parser, so you can replace the serializer of DATE in your code.
import psycopg2cffi
psycopg2cffi._impl.typecasts._default_type('DATE', [1082],
psycopg2cffi._impl.typecasts.parse_string)
Of course this can be done, for every type.
change your psql query to cast and get the date column as string
e.g. select date_column_name:: to_char from table_name.
Related
Just trying some basic exercises with pony ORM (and python3.5, sqlite3).
I just want to print a select query of some data I have without further processing to start with. Pony orm does not seem to like that at all....
The sqlite db dump
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE sums (t text, path BLOB, name BLOB, sum text, primary key (path,name));
INSERT INTO "sums" VALUES('directory','','','');
INSERT INTO "sums" VALUES('file','','sums-backup-f.db','6859b35f9f026317c5df48932f9f2a91');
INSERT INTO "sums" VALUES('file','','md5-tree.py','c7af81d4aad9d00e88db7af950c264c2');
INSERT INTO "sums" VALUES('file','','test.db','a403e9b46e54d6ece851881a895b1953');
INSERT INTO "sums" VALUES('file','','sirius-alexa.db','22a20434cec550a83c675acd849002fa');
INSERT INTO "sums" VALUES('file','','sums-reseau-y.db','1021614f692b5d7bdeef2a45b6b1af5b');
INSERT INTO "sums" VALUES('file','','.md5-tree.py.swp','1c3c195b679e99ef18b3d46044f6e6c5');
INSERT INTO "sums" VALUES('file','','compare-md5.py','cfb4a5b3c7c4e62346aa5e1affef210a');
INSERT INTO "sums" VALUES('file','','charles.local.db','9c50689e8185e5a79fd9077c14636405');
COMMIT;
Here is the code I try to run on python3.5 interactive shell:
from pony.orm import *
db = Database()
class File(db.Entity) :
_table_ = 'sums'
t = Required(str)
path = Required(bytes)
name = Required(bytes)
sum = Required(str)
PrimaryKey(path,name)
db.bind('sqlite','/some/edited/path/test.db')
db.generate_mapping()
File.select().show()
And it fails like this :
Traceback (most recent call last):
File "/usr/lib/python3.5/site-packages/pony/orm/core.py", line 5149, in _fetch
try: result = cache.query_results[query_key]
KeyError: (('f', 0, ()), (<pony.orm.ormtypes.SetType object at 0x7fd2d2701708>,), False, None, None, None, False, False, False, ())
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 2, in show
File "/usr/lib/python3.5/site-packages/pony/utils/utils.py", line 75, in cut_traceback
raise exc # Set "pony.options.CUT_TRACEBACK = False" to see full traceback
File "/usr/lib/python3.5/site-packages/pony/utils/utils.py", line 60, in cut_traceback
try: return func(*args, **kwargs)
File "/usr/lib/python3.5/site-packages/pony/orm/core.py", line 5256, in show
query._fetch().show(width)
File "/usr/lib/python3.5/site-packages/pony/orm/core.py", line 5155, in _fetch
used_attrs=translator.get_used_attrs())
File "/usr/lib/python3.5/site-packages/pony/orm/core.py", line 3859, in _fetch_objects
real_entity_subclass, pkval, avdict = entity._parse_row_(row, attr_offsets)
File "/usr/lib/python3.5/site-packages/pony/orm/core.py", line 3889, in _parse_row_
avdict[attr] = attr.parse_value(row, offsets)
File "/usr/lib/python3.5/site-packages/pony/orm/core.py", line 1922, in parse_value
val = attr.validate(row[offset], None, attr.entity, from_db=True)
File "/usr/lib/python3.5/site-packages/pony/orm/core.py", line 2218, in validate
val = Attribute.validate(attr, val, obj, entity, from_db)
File "/usr/lib/python3.5/site-packages/pony/orm/core.py", line 1894, in validate
if from_db: return converter.sql2py(val)
File "/usr/lib/python3.5/site-packages/pony/orm/dbapiprovider.py", line 619, in sql2py
if not isinstance(val, buffer): val = buffer(val)
TypeError: string argument without an encoding
Am I using this wrong, or is this a bug ? I don't mind go filing a bug, but it's the first time I'm using this orm, so I thought it might be better to check first ...
SQLite has a (mis)feature, which allows a column to store an arbitrary value disregarding the column type. Instead of rigid data type, each SQLite column has an affinity, while each value has a storage class which can be different within the same column. For example, you can store text value inside an integer column, and vice versa. See Datatypes In SQLite Version 3 for more information.
The reason for the error is that the table contains values of "wrong" type in its BLOB columns. Correct SQLite binary literal looks like x'abcdef'. The INSERT commands that you use insert UTF8 strings instead.
This problem was somewhat fixed in the latest version of Pony which you can take from GitHub. Now if Pony receives a string value from a BLOB column it just keep that value without throwing an exception.
If you populate the table with Pony, it will writes BLOB data as a correct binary values, so it can read them later without any problem.
I am using flask migrate to for database creation & migration in flask with flask-sqlalchemy.
Everything was working fine until I changed my database user password contains '#' then it stopped working so, I updated my code based on
Writing a connection string when password contains special characters
It working for application but not for flask-migration, Its showing error while migrating
i.e on python manage.py db migrate
ValueError: invalid interpolation syntax in u'mysql://user:p%40ssword#localhost/testdb' at position 15
Here password is p#ssword and its escaped by urlquote (see above question link).
Full error stack:
Traceback (most recent call last):
File "manage.py", line 20, in <module>
manager.run()
File "/usr/local/lib/python2.7/dist-packages/flask_script/__init__.py", line 412, in run
result = self.handle(sys.argv[0], sys.argv[1:])
File "/usr/local/lib/python2.7/dist-packages/flask_script/__init__.py", line 383, in handle
res = handle(*args, **config)
File "/usr/local/lib/python2.7/dist-packages/flask_script/commands.py", line 216, in __call__
return self.run(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/flask_migrate/__init__.py", line 177, in migrate
version_path=version_path, rev_id=rev_id)
File "/usr/local/lib/python2.7/dist-packages/alembic/command.py", line 117, in revision
script_directory.run_env()
File "/usr/local/lib/python2.7/dist-packages/alembic/script/base.py", line 407, in run_env
util.load_python_file(self.dir, 'env.py')
File "/usr/local/lib/python2.7/dist-packages/alembic/util/pyfiles.py", line 93, in load_python_file
module = load_module_py(module_id, path)
File "/usr/local/lib/python2.7/dist-packages/alembic/util/compat.py", line 79, in load_module_py
mod = imp.load_source(module_id, path, fp)
File "migrations/env.py", line 22, in <module>
current_app.config.get('SQLALCHEMY_DATABASE_URI'))
File "/usr/local/lib/python2.7/dist-packages/alembic/config.py", line 218, in set_main_option
self.set_section_option(self.config_ini_section, name, value)
File "/usr/local/lib/python2.7/dist-packages/alembic/config.py", line 245, in set_section_option
self.file_config.set(section, name, value)
File "/usr/lib/python2.7/ConfigParser.py", line 752, in set
"position %d" % (value, tmp_value.find('%')))
ValueError: invalid interpolation syntax in u'mysql://user:p%40ssword#localhost/testdb' at position 15
Please help
In the migrations/env.py file, you will find the code that is responsible for this issue.
config.set_main_option('sqlalchemy.url',
current_app.config.get('SQLALCHEMY_DATABASE_URI'))
If there are % signs in the SQLALCHEMY_DATABASE_URI, this will cause an error.
You can solve this by editing the migrations/env.py file, and changing the offending line as follows
db_url_escaped = current_app.config.get('SQLALCHEMY_DATABASE_URI').replace('%', '%%')
config.set_main_option('sqlalchemy.url', db_url_escaped)
Also see the documentation of set_main_option:
Note that this value is passed to ConfigParser.set, which supports variable interpolation using pyformat (e.g. %(some_value)s). A raw percent sign not part of an interpolation symbol must therefore be escaped, e.g. %%. The given value may refer to another value already in the file using the interpolation format.
I have a solution for this issue after experiencing it as well.
There's an issue with '%' (percent signs) in the db connection URI after you urlencode the string.
I tried substituting the percent sign with double percent signs ('%%') which gets me past the interpolation error. However, that resulted in not being able to connect to the database because of an incorrect password.
Solution I'm going with for now is to avoid using '%' in my db password. Not a satisfactory solution, but will do for now. I'll make a note in "alembic"'s github of the issue. Seems using RawConfigParser in their package could help avoid this issue.
You may want to look at http://docs.sqlalchemy.org/en/latest/dialects/mysql.html#mysql-unicode
I was having the same issue with my password and the mysql connector. using the mysql+pymysql connector allowed me to connect in application and in migration scripts.
I'm not sure even how to ask the question, as it seems it would require quite a lot of code to get into the details. Rather than show the code, I will discuss the behavior when I run.
I am using requests to grab information from an online database. When I run a for loop to go through all of my entries, I get an error like the one below on one of the first 20 entries (usually the first, but not always). The entries in the list are all alike (just different ID numbers). I am using sleep() to ensure that I do not go beyond my rate limit (I have tried increasing sleep to ridiculous wait times, but still get the error). What really surprises me is that it works some, and then gets stuck.... what could cause that?
Also, the code was working before, then I made a large number of edits to other code in the same file, but I didn't think I edited anything related to this.
Traceback (most recent call last):
File "C:/Users/Mark/PycharmProjects/Riot_API_Challenger_Stats/Main.py", line 233, in <module>
main()
File "C:/Users/Mark/PycharmProjects/Riot_API_Challenger_Stats/Main.py", line 212, in main
match_histories=get_match_histories(challenger_Ids+master_Ids)
File "C:/Users/Mark/PycharmProjects/Riot_API_Challenger_Stats/Main.py", line 62, in get_match_histories
match_histories[summoner_Ids[i]]=api.get_match_history_data(summoner_Ids[i])
File "C:\Users\Mark\PycharmProjects\Riot_API_Challenger_Stats\RiotAPI.py", line 52, in get_match_history_data
return self._request(api_url)
File "C:\Users\Mark\PycharmProjects\Riot_API_Challenger_Stats\RiotAPI.py", line 25, in _request
return response.json()
File "C:\Users\Mark\Anaconda3\lib\site-packages\requests\models.py", line 819, in json
return json.loads(self.text, **kwargs)
File "C:\Users\Mark\Anaconda3\lib\json\__init__.py", line 318, in loads
return _default_decoder.decode(s)
File "C:\Users\Mark\Anaconda3\lib\json\decoder.py", line 343, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\Mark\Anaconda3\lib\json\decoder.py", line 361, in raw_decode
raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)
Here are lines 10-25 of RiotAPI
def _request(self, api_url, params={}):
args = {'api_key':self.api_key}
for key, value in params.items():
if key not in args:
args[key] = value
#requests.get accesses the URL
response = requests.get(
Consts.URL['base'].format(
proxy=self.region,
region=self.region,
url=api_url
),
params=args
)
print(response.url)
return response.json()
Here is the response:
{"matches":[{"matchId":1878534497,"region":"NA","platformId":"NA1","matchMode":"CLASSIC","matchType":"MATCHED_GAME","matchCreation":1436223958539,"matchDuration":2097,"queueType":"RANKED_SOLO_5x5","mapId":11,"season":"SEASON2015","matchVersion":"5.12.0.348","participants":[{"teamId":200,"spell1Id":4,"spell2Id":7,"championId":15,"highestAchievedSeasonTier":"UNRANKED","timeline":{"creepsPerMinDeltas":{"zeroToTen":5.699999999999999,"tenToTwenty":6.9,"twentyToThirty":7.1},"xpPerMinDeltas":{"zeroToTen":358.5,"tenToTwenty":350.0,"twentyToThirty":364.20000000000005},"goldPerMinDeltas":{"zeroToTen":365.3,"tenToTwenty":337.5,"twentyToThirty":287.5},"csDiffPerMinDeltas":{"zeroToTen":-0.7,"tenToTwenty":-1.7000000000000004,"twentyToThirty":1.0999999999999999},"xpDiffPerMinDeltas":{"zeroToTen":-0.9000000000000057,"tenToTwenty":-114.75,"twentyToThirty":-121.19999999999999},"damageTakenPerMinDeltas":{"zeroToTen":480.5,"tenToTwenty":565.3,"twentyToThirty":1258.6},"damageTakenDiffPerMinDeltas":{"zeroToTen":-147.49999999999994,"tenToTwenty":-134.69999999999996,"twentyToThirty":15.0},"role":"DUO_CARRY","lane":"BOTTOM"},"masteries":[{"masteryId":4112,"rank":4},{"masteryId":4114,"rank":1},{"masteryId":4122,"rank":3},{"masteryId":4124,"rank":1},{"masteryId":4132,"rank":1},{"masteryId":4134,"rank":3},{"masteryId":4142,"rank":2},{"masteryId":4144,"rank":1},{"masteryId":4151,"rank":1},{"masteryId":4152,"rank":3},{"masteryId":4162,"rank":1},{"masteryId":4211,"rank":2},{"masteryId":4212,"rank":2},{"masteryId":4221,"rank":1},{"masteryId":4222,"rank":3},{"masteryId":4232,"rank":1}],"stats":{"winner":false,"champLevel":14,"item0":3031,"item1":0,"item2":3142,"item3":3035,"item4":1053,"item5":3250,"item6":3342,"kills":4,"doubleKills":1,"tripleKills":0,"quadraKills":0,"pentaKills":0,"unrealKills":0,"largestKillingSpree":3,"deaths":12,"assists":5,"totalDamageDealt":184710,"totalDamageDealtToChampions":27477,"totalDamageTaken":30740,"largestCriticalStrike":684,"totalHeal":2952,"minionsKilled":237,"neutralMinionsKilled":1,"neutralMinionsKilledTeamJungle":1,"neutralMinionsKilledEnemyJungle":0,"goldEarned":12074,"goldSpent":12065,"combatPlayerScore":0.....etc.}}]}]}
It'd be helpful if you can include a snippet of your code to see how you've actually used it.
However, from the exception, I think you're trying to do a
return response.json()
at line 25 of C:\Users\Mark\PycharmProjects\Riot_API_Challenger_Stats\RiotAPI.py
but the response is not in JSON format.
You can see the raw response with
print response.text
to see the string version of the response and check that the string is in JSON format.
It would be nice if you could post the actual code. But if you are unable to do so due to confidential reasons we can surmise some information from the stack trace.
You are using some HTTP protocol (SOAP/ResT) API to get a number( or series of numbers) in JSON format. One of these ID numbers has a character that is not expected or the JSON is invalid itself. Try and print the JSON request you receive before you pass it to see which one fails. Then create a unit test and run it try to analyze it with breakpoints.
Could be some sort of hyphenated or foreign character based on the database.
I am accessing a database using SQLAlchemy. When I try to filter the table using a bunch of public and private keys I get an Attribute error saying 'int' object has no attribute 'date'.
Sometimes, I am able to filter the results once and when the filter is called again, it crashes giving me the same error. Is this the problem of SQLAlchemy or PyDev?
Below is the snippet of my filter.
randomize_query(session('test').query(tableName).filter(tableName.field1 == criteria, tableName.field2 == 2).order_by(desc(tableName.field3))).first()
The full traceback is as below
File "C:\Python27\lib\site-packages\sqlalchemy\orm\query.py", line 2145, in first
ret = list(self[0:1])
File "C:\Python27\lib\site-packages\sqlalchemy\orm\query.py", line 2012, in __getitem__
return list(res)
File "C:\Python27\lib\site-packages\sqlalchemy\orm\loading.py", line 72, in instances
rows = [process[0](row, None) for row in fetch]
File "C:\Python27\lib\site-packages\sqlalchemy\orm\loading.py", line 447, in _instance
populate_state(state, dict_, row, isnew, only_load_props)
File "C:\Python27\lib\site-packages\sqlalchemy\orm\loading.py", line 301, in populate_state
populator(state, dict_, row)
File "C:\Python27\lib\site-packages\sqlalchemy\orm\strategies.py", line 150, in fetch_col
dict_[key] = row[col]
File "C:\Python27\lib\site-packages\sqlalchemy\engine\result.py", line 89, in __getitem__
return processor(self._row[index])
File "C:\Python27\lib\site-packages\sqlalchemy\dialects\oracle\cx_oracle.py", line 250, in process
return value.date()
AttributeError: 'int' object has no attribute 'date'
The exception is thrown when the result set is loaded and SQLAlchemy wants to populate the result objects. One column is qualified as a Date type, but the Oracle result set is giving you an integer instead.
The cx_Oracle library will normally convert Oracle-supplied native DATE column value into Python datetime.datetime object. However, this is not happening for all your rows here.
You'll need to narrow down what row or rows have a column that is not being translated to a datetime object. Find a pattern in the filters that include or exclude these rows and narrow it down so you can inspect the database rows by hand in a different client.
I'm having a weird error with some Google App Engine code I'm writing.
My program contains some code like this:
import datetime
...
class Action(db.Model):
visibleDate = db.DateTimeProperty()
...
getActionQuery = Action.gql("WHERE user = :user AND __key__ = :key", user = user, key = self.request.get("key"))
theAction = getActionQuery.get()
....
theAction.visibleDate = datetime.datetime.strptime(self.request.get("visibleDate"), "%Y/%m/%d")
Yet this produces the following error:
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/webapp/__init__.py", line 509, in __call__
handler.post(*groups)
File "/Users/redbird/Developer/betterdo-it/main.py", line 132, in post
theAction.visibleDate = datetime.datetime.strptime(self.request.get("visibleDate"), "%Y/%m/%d"),
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/db/__init__.py", line 472, in __set__
value = self.validate(value)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/db/__init__.py", line 2308, in validate
(self.name, self.data_type.__name__))
BadValueError: Property visibleDate must be a datetime
Any ideas on why this is happening? I've tested it and I know that my time is coming in, is getting converted correctly, but then hits this error.
I think there's something you missed in your traceback.
I'm seeing:datetime.datetime.strptime(self.request.get("visibleDate"), "%Y/%m/%d"),
Notice the comma at the end of the line.
That comma makes that line return a tuple with your date inside it. I'm assuming you accidentally added the comma, so just remove it and you should be correctly assigning a datetime.
To review:
from datetime import datetime
a = (datetime(2000,1,1),)
assert isinstance(a, tuple)
a = (datetime(2000,1,1))
assert isinstance(a, datetime)