When I run my application, which finds links, checks if they exist in the database, and adds them to the database, I get an error.
Traceback (most recent call last):
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1536, in __call__
rv = self.handle_exception(request, response, e)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1530, in __call__
rv = self.router.dispatch(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~nothing-but-net/1.360769209191920043/main.py", line 460, in get
blchrlinks(True, a)
File "/base/data/home/apps/s~nothing-but-net/1.360769209191920043/main.py", line 271, in blchrlinks
if Articles.by_name(title):
File "/base/data/home/apps/s~nothing-but-net/1.360769209191920043/main.py", line 498, in by_name
u = Articles.all().filter("name =", name).get()
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 2099, in get
results = self.run(limit=1, **kwargs)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 2063, in run
iterator = raw_query.Run(**kwargs)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/datastore.py", line 1622, in Run
itr = Iterator(self.GetBatcher(config=config))
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/datastore.py", line 1601, in GetBatcher
return self.GetQuery().run(_GetConnection(), query_options)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/datastore.py", line 1490, in GetQuery
filter_predicate=self.GetFilterPredicate(),
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/datastore.py", line 1534, in GetFilterPredicate
property_filters.append(datastore_query.make_filter(name, op, values))
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 107, in make_filter
properties = datastore_types.ToPropertyPb(name, values)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/datastore_types.py", line 1745, in ToPropertyPb
pbvalue = pack_prop(name, v, pb.mutable_value())
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/datastore_types.py", line 1556, in PackString
pbvalue.set_stringvalue(unicode(value).encode('utf-8'))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 57: ordinal not in range(128)
My code for by_name is:
def by_name(cls, name):
u = Articles.all().filter("name =", name).get()
return u
As the last line of the stack trace shows, it's trying to convert the value to unicode first and then encoding it as utf-8. However, the (implicit) conversion is using ascii, which is not enough for your string. You could try converting it to unicode yourself, using the right encoding, before passing it to filter. Example:
u = Articles.all().filter("name =", name.decode('utf-8')).get()
(remember that you need to provide the correct encoding; if name is not a UTF-8 string, but a Cp1252, ISO-Latin or something else, you need to specify that in the decode call)
Related
I'm using ArtUshak's wiki_tool_python scripts codes to dump images from an Vietnamese wiki at Fandom.com, in this case: https://lhmn.fandom.com/vi.
At the moment I decided to dump the images, using predefined text file from a previous code.
Because most of the files on that wiki have at least, an Vietnamese character on it's name, I have to use --confine-encoding UTF-8 option to properly get all of the files, using list-images --confine-encoding UTF-8 https://lhmn.fandom.com/vi. After that, using the result I got from using list-images, I copies all the files list on to a text file, calling it images.txt.
download-images, using the previous file created, I get it to run the command to dump the images: download-images images.txt .\data\download
(The download-images command do not accept --confine-encoding UTF-8 option)
Then this happen and the script stop:
Traceback (most recent call last):
File "C:\Users\ADMIN\Downloads\wiki_tool\wiki_tool_python-master\wiki_tool_python\wikitool.py", line 1198, in <module>
cli(obj={})
File "C:\Users\ADMIN\AppData\Local\pypoetry\Cache\virtualenvs\wiki-tool-python-UGT00s18-py3.11\Lib\site-packages\click\core.py", line 829, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ADMIN\AppData\Local\pypoetry\Cache\virtualenvs\wiki-tool-python-UGT00s18-py3.11\Lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "C:\Users\ADMIN\AppData\Local\pypoetry\Cache\virtualenvs\wiki-tool-python-UGT00s18-py3.11\Lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ADMIN\AppData\Local\pypoetry\Cache\virtualenvs\wiki-tool-python-UGT00s18-py3.11\Lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ADMIN\AppData\Local\pypoetry\Cache\virtualenvs\wiki-tool-python-UGT00s18-py3.11\Lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ADMIN\AppData\Local\pypoetry\Cache\virtualenvs\wiki-tool-python-UGT00s18-py3.11\Lib\site-packages\click\decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ADMIN\Downloads\wiki_tool\wiki_tool_python-master\wiki_tool_python\wikitool.py", line 652, in download_images
with click.progressbar(list(read_image_list(list_file))) as bar:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ADMIN\Downloads\wiki_tool\wiki_tool_python-master\wiki_tool_python\wikitool.py", line 37, in read_image_list
url_line = next(file_iterator)
^^^^^^^^^^^^^^^^^^^
File "F:\VMN - TC21TH\PYTHON\Lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 4914: character maps to <undefined>
It supposed to get all the files from the previously defined list and download them to the path .\data\download.
I am having some kind of encoding error when using SQLAlchemy with mysql+mysqlconnector.
I found a lot of similar questions answered on StackOverflow but none of them seem to work for me. Think I have tried everything. I have tried setting charset=utf8 in the connection string.
I have tried setting the character set using SQL.
src_conn.execute("SET CHARACTER SET utf8mb4")
This is the initial error I got:
ExceptionTraceback (most recent call last):
File "transfer_binlog.py", line 73, in <module>
events = events.fetchall()
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/result.py", line 1216, in fetchall
e, None, None, self.cursor, self.context
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1478, in _handle_dbapi_exception
util.reraise(*exc_info)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/result.py", line 1211, in fetchall
l = self.process_rows(self._fetchall_impl())
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/result.py", line 1161, in _fetchall_impl
return self.cursor.fetchall()
File "/usr/local/lib/python3.7/site-packages/mysql/connector/cursor.py", line 990, in fetchall
row, self.description))
File "/usr/local/lib/python3.7/site-packages/mysql/connector/conversion.py", line 407, in row_to_python
result[i] = self._cache_field_types[field_type](row[i], field)
File "/usr/local/lib/python3.7/site-packages/mysql/connector/conversion.py", line 567, in _STRING_to_python
return value.decode(self.charset)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 55: invalid continuation byte
And here is an excerpt of the code:
src_connect = create_engine('mysql+mysqlconnector://user:pw#host/core?charset=utf8', isolation_level="READ UNCOMMITTED")
src_conn = src_connect.connect()
events = src_conn.execute("SHOW BINLOG EVENTS")
x = events.fetchall()
Is there anything I can have missed or is sqlalchemy or mysql+mysqlconnector driver buggy?
Anya ideas are appreciated
EDIT: According to #GordThompsons advice I switched to mysql+pymysql.
There error is now (quite similar):
ExceptionTraceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 581, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python3.7/site-packages/pymysql/cursors.py", line 170, in execute
result = self._query(query)
File "/usr/local/lib/python3.7/site-packages/pymysql/cursors.py", line 328, in _query
conn.query(q)
File "/usr/local/lib/python3.7/site-packages/pymysql/connections.py", line 517, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/usr/local/lib/python3.7/site-packages/pymysql/connections.py", line 732, in _read_query_result
result.read()
File "/usr/local/lib/python3.7/site-packages/pymysql/connections.py", line 1082, in read
self._read_result_packet(first_packet)
File "/usr/local/lib/python3.7/site-packages/pymysql/connections.py", line 1152, in _read_result_packet
self._read_rowdata_packet()
File "/usr/local/lib/python3.7/site-packages/pymysql/connections.py", line 1190, in _read_rowdata_packet
rows.append(self._read_row_from_packet(packet))
File "/usr/local/lib/python3.7/site-packages/pymysql/connections.py", line 1206, in _read_row_from_packet
data = data.decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 55: invalid continuation byte
EDIT 2
#GordThompson it sais
'collation_connection','utf8mb4_general_ci'
'collation_database','utf8_general_ci'
'collation_server','utf8_general_ci'
I also tried SHOW VARIABLES LIKE 'char%'; which give me
'character_set_client','utf8mb4'
'character_set_connection','utf8mb4'
'character_set_database','utf8'
'character_set_filesystem','binary'
'character_set_results','utf8mb4'
'character_set_server','latin1'
'character_set_system','utf8'
'character_sets_dir','/rdsdbbin/mysql-5.6.44.R1/share/charsets/'
It is strange that character_set_server shows latin1, maybe that is the issue since I guess SHOW BINLOG EVENTS is a server feature.
But then I changed it like this: SET SESSION character_set_server=utf8; but the problem remains.
And just to be clear I runned the above statements in MySQLWorkbench and not from python.
I'm trying to get some data from one web page and write these data into xlsx file. Everything seems good but the Encoding error probably raises if it tries to write it into xlsx file during CLOSING the file.
ERROR:
Traceback (most recent call last):
File "C:/Users/Milano/PycharmProjects/distrelec/crawler.py", line 429, in <module>
temp_file_to_xlsx()
File "C:/Users/Milano/PycharmProjects/distrelec/crawler.py", line 119, in temp_file_to_xlsx
wb.close()
File "C:\Python27\lib\site-packages\xlsxwriter\workbook.py", line 295, in close
self._store_workbook()
File "C:\Python27\lib\site-packages\xlsxwriter\workbook.py", line 518, in _store_workbook
xml_files = packager._create_package()
File "C:\Python27\lib\site-packages\xlsxwriter\packager.py", line 134, in _create_package
self._write_workbook_file()
File "C:\Python27\lib\site-packages\xlsxwriter\packager.py", line 174, in _write_workbook_file
workbook._assemble_xml_file()
File "C:\Python27\lib\site-packages\xlsxwriter\workbook.py", line 464, in _assemble_xml_file
self._write_sheets()
File "C:\Python27\lib\site-packages\xlsxwriter\workbook.py", line 1455, in _write_sheets
self._write_sheet(worksheet.name, id_num, worksheet.hidden)
File "C:\Python27\lib\site-packages\xlsxwriter\workbook.py", line 1472, in _write_sheet
self._xml_empty_tag('sheet', attributes)
File "C:\Python27\lib\site-packages\xlsxwriter\xmlwriter.py", line 80, in _xml_empty_tag
self.fh.write("<%s/>" % tag)
File "C:\Python27\lib\codecs.py", line 694, in write
return self.writer.write(data)
File "C:\Python27\lib\codecs.py", line 357, in write
data, consumed = self.encode(object, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 23: ordinal not in range(128)
To find out where is the problem I've edited codecs module:
def write(self, object):
""" Writes the object's contents encoded to self.stream.
"""
try:
data, consumed = self.encode(object, self.errors)
self.stream.write(data)
except:
print object
print repr(object)
raise Exception
The output is:
<sheet name="Android PC–APC" sheetId="42" r:id="rId42"/>
'<sheet name="Android PC\xe2\x80\x93APC" sheetId="42" r:id="rId42"/>'
temp_file_to_xlsx()
File "C:/Users/Milano/PycharmProjects/distrelec/crawler.py", line 119, in temp_file_to_xlsx
wb.close()
File "C:\Python27\lib\site-packages\xlsxwriter\workbook.py", line 295, in close
self._store_workbook()
File "C:\Python27\lib\site-packages\xlsxwriter\workbook.py", line 518, in _store_workbook
xml_files = packager._create_package()
File "C:\Python27\lib\site-packages\xlsxwriter\packager.py", line 134, in _create_package
self._write_workbook_file()
File "C:\Python27\lib\site-packages\xlsxwriter\packager.py", line 174, in _write_workbook_file
workbook._assemble_xml_file()
File "C:\Python27\lib\site-packages\xlsxwriter\workbook.py", line 464, in _assemble_xml_file
self._write_sheets()
File "C:\Python27\lib\site-packages\xlsxwriter\workbook.py", line 1455, in _write_sheets
self._write_sheet(worksheet.name, id_num, worksheet.hidden)
File "C:\Python27\lib\site-packages\xlsxwriter\workbook.py", line 1472, in _write_sheet
self._xml_empty_tag('sheet', attributes)
File "C:\Python27\lib\site-packages\xlsxwriter\xmlwriter.py", line 80, in _xml_empty_tag
self.fh.write("<%s/>" % tag)
File "C:\Python27\lib\codecs.py", line 699, in write
return self.writer.write(data)
File "C:\Python27\lib\codecs.py", line 363, in write
raise Exception
Exception
What should I do with that please?
You have to decode your input data with the correct encoding, which seems to be 'utf-8'.
You may want to look at this:
Example: Simple Unicode with Python 2
My application experiences over quota issues and I would like to handle such cases properly in my code. The limit is reached just for Datastore Read Operations, but I get TypeError: ConjunctionNode() requires at least one node exception when try to read the data from the memcache - entries = memcache.get('mykey').
Why this exception happens?
Exception details:
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 266, in Handle
result = handler(dict(self._environ), self._StartResponse)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1519, in __call__
response = self._internal_error(e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
rv = self.handle_exception(request, response, e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
return handler.dispatch()
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~myapp/1.373233284460557570/myapp.py", line 595, in get
entries = memcache.get('mykey')
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 559, in get
results = rpc.get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 612, in get_result
return self.__get_result_hook(self)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 624, in __get_hook
self._do_unpickle)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 271, in _decode_value
return do_unpickle(value)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 401, in _do_unpickle
return unpickler.load()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py", line 550, in __new__
raise TypeError('ConjunctionNode() requires at least one node.')
TypeError: ConjunctionNode() requires at least one node.
Upd. here is what I save to memcache:
entries = MyModel.query()
entries = entries.fetch(keys_only=True)
entries = random.sample(entries, 10)
entries = [list_key.get() for list_key in entries]
memcache.set('mykey', entries, 60*60*24)
Upd2. Online memcache viewer shows the following value stored (just first part is shown below):
Type: Object
..cgoogle.appengine.ext.ndb.query.Query.q.).q.}q.(U._Query__projectionq.NU._Query__filtersq.cgoogle.appengine.ext.ndb.query.ConjunctionNode.q.).q.}q.U._ConjunctionNode__nodesq.]q.(cgoogle.appengine.ext.ndb.query.FilterNode.q.).q.}q.(U._Filt
I believe that this happens simply because you are trying to put a pickled Query object into memcache, but Query objects are not pickle-able.
What I do instead in this case is to convert the Query to a list prior to putting it into the cache. If your code doesn't care about the distinction (that is, if it's not calling .filter() or something like that) then it'll probably work fine for you too.
when uploading a file with web.py, there's a exception " SystemError: error return without exception set" raised.
here's traceback
...
File "../web/template.py", line 882, in __call__
return BaseTemplate.__call__(self, *a, **kw)
File "../web/template.py", line 809, in __call__
return self.t(*a, **kw)
File "", line 193, in __template__
File "../web/webapi.py", line 276, in input
out = rawinput(_method)
File "../web/webapi.py", line 249, in rawinput
a = cgi.FieldStorage(fp=fp, environ=e, keep_blank_values=1)
File "../python2.7/cgi.py", line 508, in __init__
self.read_multi(environ, keep_blank_values, strict_parsing)
File "../python2.7/cgi.py", line 632, in read_multi
environ, keep_blank_values, strict_parsing)
File "../python2.7/cgi.py", line 510, in __init__
self.read_single()
File "../python2.7/cgi.py", line 647, in read_single
self.read_lines()
File "../python2.7/cgi.py", line 669, in read_lines
self.read_lines_to_outerboundary()
File "../python2.7/cgi.py", line 697, in read_lines_to_outerboundary
line = self.fp.readline(1
"""
def POST(self):
x = web.input(myfile= {})
return x.myfile.file.read()
Not sure, but I'd switch to WSGI anyway, its faster and easy to use. Do you get that error when running the built in webserver?