I'm completely new to Flask. I have some code that will copy a file over to a virtual machine using the pysphere library. This works fine on its own, but when I try using a Flask app, I get:
UnicodeDecodeError: 'ascii' codec can't decode byte 0x90 in position 2: ordinal not in range(128)
At first I figured it is just because the web form is passing something it doesn't like. However, I decided to hard code the values and it still fails. Here is the code:
#app.route('/begin_install', methods=['POST'])
def begin_install():
source_installer_path = app.root_path + '/installers'
installer_file = str(request.form['installer'])
option_file_path = app.root_path + '/installers/options'
option_file = 'testing.options'
vmserver.start_install( request.form['vm'],
source_installer_path,
installer_file,
option_file_path,
option_file)
return render_template('results.html')
Then, in my pysphere related file:
def start_install(self, vmpath, installer_path, installer_file, options_path, options_file):
vm.revert_to_named_snapshot('python_install')
vm.power_on()
while vm.get_tools_status() != 'RUNNING':
sleep(3)
vm.login_in_guest(self.guest_user, self.guest_password)
vm.send_file('C:\\folder\\filetosend.exe', 'c:\\installer\\filename.exe')
Everything up to the "vm.send_file" works perfectly fine. If I call the same code from a non-Flask app, it also works perfectly fine. I'm very confused about why I'm getting errors from Flask when this part of the code is all pysphere.
EDIT: Here is the traceback
Traceback (most recent call last):
File "C:\Users\username\Flask\lib\site-packages\flask\app.py", line 1701, in __call__
return self.wsgi_app(environ, start_response)
File "C:\Users\username\Flask\lib\site-packages\flask\app.py", line 1689, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "C:\Users\username\Flask\lib\site-packages\flask\app.py", line 1687, in wsgi_app
response = self.full_dispatch_request()
File "C:\Users\username\Flask\lib\site-packages\flask\app.py", line 1360, in full_dispatch_request
rv = self.handle_user_exception(e)
File "C:\Users\username\Flask\lib\site-packages\flask\app.py", line 1358, in full_dispatch_request
rv = self.dispatch_request()
File "C:\Users\username\Flask\lib\site-packages\flask\app.py", line 1344, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "C:\python\PycharmProjects\installtest\installtest.py", line 51, in begin_install
option_file)
File "C:\python\PycharmProjects\installtest\testmachines.py", line 56, in start_install
'c:\\installer\\filename.exe')
File "C:\Users\username\Flask\lib\site-packages\pysphere\vi_virtual_machine.py", line 1282, in send_file
resp = opener.open(request)
File "C:\Python27\Lib\urllib2.py", line 400, in open
response = self._open(req, data)
File "C:\Python27\Lib\urllib2.py", line 418, in _open
'_open', req)
File "C:\Python27\Lib\urllib2.py", line 378, in _call_chain
result = func(*args)
File "C:\Python27\Lib\urllib2.py", line 1215, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "C:\Python27\Lib\urllib2.py", line 1174, in do_open
h.request(req.get_method(), req.get_selector(), req.data, headers)
File "C:\Python27\Lib\httplib.py", line 958, in request
self._send_request(method, url, body, headers)
File "C:\Python27\Lib\httplib.py", line 992, in _send_request
self.endheaders(body)
File "C:\Python27\Lib\httplib.py", line 954, in endheaders
self._send_output(message_body)
File "C:\Python27\Lib\httplib.py", line 812, in _send_output
msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0x90 in position 2: ordinal not in range(128)
As you can see in the traceback pysphere uses urllib2 which is in turn using httplib.
As far as I know, httplib is concatenating the whole http request without further checking for unicode strings. And python returns always unicode if one part is unicode.
I think your pysphere gets somehow poluted with unicodes either in the request or the opener in the send_file method:
File "C:\Users\username\Flask\lib\site-packages\pysphere\vi_virtual_machine.py", line 1282, in send_file
resp = opener.open(request)
you should check how you are creating resp. configuring your vm instance and probably in detail which header is unicode and how is that affecting the request or the opener instantiation.
Related
I was trying to run the ar cut paste using the source everything works fine but when the Image is not getting pasted in photoshop the image is taken and saved but its not getting pasted in photoshop. and when i checked the logs in the server cmd prompt it had this line at last "UnicodeError: encoding with 'idna' codec failed (UnicodeError: label empty or too long)"
UnicodeError: encoding with 'idna' codec failed (UnicodeError: label empty or too long)
I was trying to run the ar cut paste using the source everything works fine but when the Image is not getting pasted in photoshop the image is taken and saved but its not getting pasted in photoshop. and when i checked the logs in the server cmd prompt it had this line at last "UnicodeError: encoding with 'idna' codec failed (UnicodeError: label empty or too long)"
This is the complete LOG(Please comment if any other resource is required for diagnosing)
`
Traceback (most recent call last):
File "C:\Users\Lenovo\Documents\ar-cutpaste-master\server\venv\lib\site-packages\flask\app.py", line 2095, in __call__
return self.wsgi_app(environ, start_response)
File "C:\Users\Lenovo\Documents\ar-cutpaste-master\server\venv\lib\site-packages\flask\app.py", line 2080, in wsgi_app
response = self.handle_exception(e)
File "C:\Users\Lenovo\Documents\ar-cutpaste-master\server\venv\lib\site-packages\flask_cors\extension.py", line 165, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "C:\Users\Lenovo\Documents\ar-cutpaste-master\server\venv\lib\site-packages\flask\app.py", line 2077, in wsgi_app
response = self.full_dispatch_request()
File "C:\Users\Lenovo\Documents\ar-cutpaste-master\server\venv\lib\site-packages\flask\app.py", line 1525, in full_dispatch_request
rv = self.handle_user_exception(e)
File "C:\Users\Lenovo\Documents\ar-cutpaste-master\server\venv\lib\site-packages\flask_cors\extension.py", line 165, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "C:\Users\Lenovo\Documents\ar-cutpaste-master\server\venv\lib\site-packages\flask\app.py", line 1523, in full_dispatch_request
rv = self.dispatch_request()
File "C:\Users\Lenovo\Documents\ar-cutpaste-master\server\venv\lib\site-packages\flask\app.py", line 1509, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "C:\Users\Lenovo\Documents\ar-cutpaste-master\server\src\main.py", line 74, in save
res = requests.post(args.basnet_service_ip, headers=headers, files=files )
File "C:\Users\Lenovo\Documents\ar-cutpaste-master\server\venv\lib\site-packages\requests\api.py", line 119, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Users\Lenovo\Documents\ar-cutpaste-master\server\venv\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\Lenovo\Documents\ar-cutpaste-master\server\venv\lib\site-packages\requests\sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\Lenovo\Documents\ar-cutpaste-master\server\venv\lib\site-packages\requests\sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "C:\Users\Lenovo\Documents\ar-cutpaste-master\server\venv\lib\site-packages\requests\adapters.py", line 449, in send
timeout=timeout
File "C:\Users\Lenovo\Documents\ar-cutpaste-master\server\venv\lib\site-packages\urllib3\connectionpool.py", line 677, in urlopen
chunked=chunked,
File "C:\Users\Lenovo\Documents\ar-cutpaste-master\server\venv\lib\site-packages\urllib3\connectionpool.py", line 392, in _make_request
conn.request(method, url, **httplib_request_kw)
File "c:\users\lenovo\appdata\local\programs\python\python37\lib\http\client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "c:\users\lenovo\appdata\local\programs\python\python37\lib\http\client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "c:\users\lenovo\appdata\local\programs\python\python37\lib\http\client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "c:\users\lenovo\appdata\local\programs\python\python37\lib\http\client.py", line 1016, in _send_output
self.send(msg)
File "c:\users\lenovo\appdata\local\programs\python\python37\lib\http\client.py", line 956, in send
self.connect()
File "C:\Users\Lenovo\Documents\ar-cutpaste-master\server\venv\lib\site-packages\urllib3\connection.py", line 187, in connect
conn = self._new_conn()
File "C:\Users\Lenovo\Documents\ar-cutpaste-master\server\venv\lib\site-packages\urllib3\connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "C:\Users\Lenovo\Documents\ar-cutpaste-master\server\venv\lib\site-packages\urllib3\util\connection.py", line 61, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "c:\users\lenovo\appdata\local\programs\python\python37\lib\socket.py", line 748, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
UnicodeError: encoding with 'idna' codec failed (UnicodeError: label empty or too long)
`
I want to download some files to my server's storage from URL. For example, you have url and want to save it. I've found only examples for downloading from the user's computer and it doesn't work in my case.
I used code below to download files in my local programs
with urllib.request.urlopen(url) as response, open(file_name, 'wb') as out_file:
shutil.copyfileobj(response, out_file)
but when running on server it produces this error:
Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python3.8/site-packages/flask/app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python3.8/site-packages/flask/app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/lib/python3.8/site-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3.8/site-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/DenShlk/mysite/flask_app.py", line 26, in hello_world
save_pic_by_id(id)
File "/home/DenShlk/mysite/flask_app.py", line 20, in save_pic_by_id
with urllib.request.urlopen(pic_url) as response, open(file_name, 'wb') as out_file:
File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.8/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/usr/lib/python3.8/urllib/request.py", line 1360, in https_open
return self.do_open(http.client.HTTPSConnection, req,
File "/usr/lib/python3.8/urllib/request.py", line 1320, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error Tunnel connection failed: 403 Forbidden>
Thanks for any help!
Finally, I've found the real problem. I used pythonanywhere.com and there is the white list of sites that can be accessed from free account.
Note the link listed in the comment is for Python 2.7, but this questions pertains to Python 3.7.
I'm using Python 3.7 and Django. I want to read from a URL that has special characters in its string, but get errors when I try the traditional way ...
>>> url = "https://www.supergaming.com/f/gaming/article/pvmqe/was_browsing_the_steam_app_reviews_and_ಠ_ಠ/"
...
>>> html = urllib2.urlopen(req, 5000).read()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1360, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1317, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1240, in _send_request
self.putrequest(method, url, **skips)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1107, in putrequest
self._output(request.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode character '\u0ca0' in position 69: ordinal not in range(128)
So I tried the solution recommended here -- How to convert a url string to safe characters with python? , but I'm still unable to read the URL
>>> urllib.parse.quote_plus(url)
'https%3A%2F%2Fwww.supergaming.com%2Ff%2Fgaming%2Farticle%2Fpvmqe%2Fwas_browsing_the_steam_app_reviews_and_%E0%B2%A0_%E0%B2%A0%2F'
>>> req = urllib2.Request(urllib.parse.quote_plus(url))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 328, in __init__
self.full_url = url
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 354, in full_url
self._parse()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 383, in _parse
raise ValueError("unknown url type: %r" % self.full_url)
ValueError: unknown url type: 'https%3A%2F%2Fwww.supergaming.com%2Fr%2Fgaming%2Farticle%2Fpvmqe%2Fwas_browsing_the_steam_app_reviews_and_%E0%B2%A0_%E0%B2%A0%2F'
What's the proper way to read from a URL if it contains special characters?
I am trying to upload a zip file directly to S3 using a Python script, but running into some Unicode Decode Errors.
What I do is generate a Pre-Signed S3 Link and then upload data to it. I know the link works fine because the upload works when I use curl to do it like this:
curl -v -H "Content-Type: application/zip" -T /Path/To/Local/File.zip https://MySignedAWSS3Link
However, when I attempt this in Python using the code below, I get an error.
infile2 = open('/Path/To/Local/File.zip', 'rb')
filedata2 = infile2.read()
request2 = urllib2.Request("https://MySignedAWSS3Link",data=filedata2)
request2.add_header('Content-Type', 'application/zip')
request2.get_method = lambda: 'PUT'
url2 = opener.open(request2)
I get the following error/traceback in Python:
> Traceback (most recent call last):
File "putFiles.py", line 44, in <module>
url2 = opener.open(request2)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 404, in open
response = self._open(req, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 422, in _open
'_open', req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1222, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1181, in do_open
h.request(req.get_method(), req.get_selector(), req.data, headers)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 973, in request
self._send_request(method, url, body, headers)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1007, in _send_request
self.endheaders(body)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 969, in endheaders
self._send_output(message_body)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 827, in _send_output
msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xca in position 10: ordinal not in range(128)
What am I doing wrong here?
According to your traceback, some string added to your request is unicode. However, according to your script example all strings are ascii encoded (since you are using Python2.7).
Weird things happen on your end, unless you set Python default encoding to UTF-8 (or OS sets it for you).
I hope, this should work for you (convert all strings into ascii):
infile2 = open('/Path/To/Local/File.zip', 'rb')
filedata2 = infile2.read()
request2 = urllib2.Request("https://MySignedAWSS3Link".encode('utf-8'),data=filedata2)
request2.add_header(str('Content-Type'), str('application/zip'))
request2.get_method = lambda: str('PUT')
url2 = opener.open(request2)
Update: This q/a might help you too
So I have a little script that I would like to use to upload some PDFs to my citation-site-of-choice (citeulike.org)
Thing is its not working. It does this:
so want to upload /Users/willwade/Dropbox/Papers/price_promoting_643127.pdf to 12589610
Traceback (most recent call last):
File "citeuupload.py", line 167, in <module>
cureader.parseUserBibTex()
File "citeuupload.py", line 160, in parseUserBibTex
self.uploadFileToCitation(b['citeulike-article-id'],self.localpapers+fileorfalse)
File "citeuupload.py", line 138, in uploadFileToCitation
resp = self.browser.submit()
File "build/bdist.macosx-10.8-intel/egg/mechanize/_mechanize.py", line 541, in submit
File "build/bdist.macosx-10.8-intel/egg/mechanize/_mechanize.py", line 203, in open
File "build/bdist.macosx-10.8-intel/egg/mechanize/_mechanize.py", line 230, in _mech_open
File "build/bdist.macosx-10.8-intel/egg/mechanize/_opener.py", line 193, in open
File "build/bdist.macosx-10.8-intel/egg/mechanize/_urllib2_fork.py", line 344, in _open
File "build/bdist.macosx-10.8-intel/egg/mechanize/_urllib2_fork.py", line 332, in _call_chain
File "build/bdist.macosx-10.8-intel/egg/mechanize/_urllib2_fork.py", line 1142, in http_open
File "build/bdist.macosx-10.8-intel/egg/mechanize/_urllib2_fork.py", line 1115, in do_open
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 955, in request
self._send_request(method, url, body, headers)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 989, in _send_request
self.endheaders(body)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 951, in endheaders
self._send_output(message_body)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 809, in _send_output
msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 544: ordinal not in range(128)
and the code:
def uploadFileToCitation(self,artid,file):
print 'so want to upload', file, ' to ', artid
self.browser.open('http://www.citeulike.org/user/'+cUser+'/article/'+artid)
self.browser.select_form(name="fileupload_frm")
self.browser.form.add_file(open(file, 'rb'), 'application/pdf', file, name='file')
try:
resp = self.browser.submit()
self.wait_for_api_limit()
except mechanize.HTTPError, e:
print 'error'
print e.getcode()
print resp.read()
exit()
NB: I can see it's reading in the file correctly (and it does exist). Also note that I'm doing this elsewhere
self.browser = mechanize.Browser()
self.browser.set_handle_robots(False)
self.browser.addheaders = [
("User-agent", 'me#me.com citeusyncpy/1.0'),
]
Full code is here
Try to check this similar question.
To clarify, the message is constructed in httplib from the method, URL, headers, etc. If any of these is Unicode, the whole string gets converted to Unicode (I presume this is normal Python behavior). Then if you try to append a UTF-8 string you get the error I described in the original question...
From looks of it's a problem with encoding that proper header can fix.
Also you can check this issue.