Develop Reference

Python app using Requests has missing files and errors after cx_freeze

EDIT:
So I ended up just using Portable Python. My goal was to run a Python script on a Windows Server that doesn't have Python installed, and that's what PP can do. Not really a solution to my original question, but if someone else runs in to the same issue that's one way to solve it.
I've built a python program that uses the requests library. It works fine when I run it normally, but not after freezing it to an executable with cx_freeze.
I've already looked at and tried the answers from these questions:
requests library | frozen app | requests & cx_freeze
So neither adding:
import requests.certs
build_exe_options = {"include_files" : [(requests.certs.where(),'cacert.pem')]}
nor
os.environ["REQUESTS_CA_BUNDLE"] = os.path.join(os.getcwd(), "cacert.pem")
are working for me.
My request looks like this:
def get_response(id = constant_id, token = constant_token, page = None):
url = 'https://scm.commerceinterface.com/api/v2/get_orders'
return requests.get(url, params = {'supplier_id':id, 'token':token, 'page':page }).json()
so I'm not even passing it the verify = parameter and it works as a python program...
Here is the error(s) returned when I run the executable:
File "C:\Python34\lib\site packages\requests\packages\urllib3\util\ssl_.py", line 267, in ssl_wrap_socket
context.load_verify_locations(ca_certs)
FileNotFoundError: [Errno 2] No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python34\lib\site-packages\requests\adapters.py", line 370, in send
timeout=timeout
File "C:\Python34\lib\site
packages\requests\packages\urllib3\connectionpool.p
y", line 539, in urlopen
self._prepare_proxy(conn)
File "C:\Python34\lib\site-
packages\requests\packages\urllib3\connectionpool.p
y", line 727, in _prepare_proxy
conn.connect()
File "C:\Python34\lib\site-
packages\requests\packages\urllib3\connection.py",
line 238, in connect
ssl_version=resolved_ssl_version)
File "C:\Python34\lib\site-packages\requests\packages\urllib3\util\ssl_.py", line 269, in ssl_wrap_socket
raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [Errno 2] No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python34\lib\site-packages\cx_Freeze\initscripts\Console.py", line
27, in <module> exec(code, m.__dict__)
File "main.py", line 49, in <module>
File "main.py", line 18, in get_response
File "C:\Python34\lib\site-packages\requests\api.py", line 69, in get
return request('get', url, params=params, **kwargs)
File "C:\Python34\lib\site-packages\requests\api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "C:\Python34\lib\site-packages\requests\sessions.py", line 465, in
request
resp = self.send(prep, **send_kwargs)
File "C:\Python34\lib\site-packages\requests\sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "C:\Python34\lib\site-packages\requests\adapters.py", line 431, in send
raise SSLError(e, request=request) requests.exceptions.SSLError: [Errno 2] No such file or directory

Python : Calling another localhost server from localhost

def jsonCatch(environ,start_response):
results = requests.get("http://localhost:8055/jsonResponse")
start_response('200 OK', [('Content-Type', 'application/json')])
return results.json()
from wsgiref.simple_server import make_server
httpd = make_server('', 8050, application)
print('Serving on port 8050...')
httpd.serve_forever()
serving on port 8055
def jsonResponse(environ,start_response):
responseData={}
responseData['name']="alex"
responseData['age']="12"
start_response('200 OK',[('Content-Type', 'application/json')])
return json.dumps(responseData)
[ERROR]
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/handlers.py", line 86, in run
self.finish_response()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
self.write(data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/handlers.py", line 204, in write
assert type(data) is StringType,"write() argument must be string"
AssertionError: write() argument must be string
127.0.0.1 - - [07/Apr/2015 09:50:04] "GET /jsonCatch HTTP/1.1" 500 59
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 52274)
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
self.process_request(request, client_address)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 321, in process_request
self.finish_request(request, client_address)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 649, in __init__
self.handle()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/simple_server.py", line 124, in handle
handler.run(self.server.get_app())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/handlers.py", line 92, in run
self.close()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/simple_server.py", line 33, in close
self.status.split(' ',1)[0], self.bytes_sent
AttributeError: 'NoneType' object has no attribute 'split'
Edit: Changed https to http and error on server
I am trying to access a localhost server which returns a json response and is used by another localhost server. Can somebody show me an example of it?

I have changed the above code to
n=json.loads(results.text)
start_response('200 OK', [('Content-Type', 'text/plain')])
return n['name'].encode('utf-8')
and it returns me "alex"

try using http:// instead of https:// on this line: results = requests.get("https://localhost:8055/jsonResponse")

App Engine Backend Instance Timeout (Python)

Currently I am using a dynamic backend instance for my site, but I still got timeout issue. The background of my site is:
GAE used as the frontend which handles user requests.
Requests will be sent to a EC2 rest server (using bottle) to process.
Below is the error information from GAE log. It seems like GAE stopped my backend thread after 60 seconds. I appreciate any suggestions.
ERROR 2014-02-20 14:42:18,365 module.py:660] Request to '/_ah/background' failed
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\devappserver2\module.py", line 637, in _handle_request
environ, wrapped_start_response)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\devappserver2\request_rewriter.py", line 311, in _rewriter_middleware
response_body = iter(application(environ, wrapped_start_response))
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\devappserver2\module.py", line 1524, in _handle_script_request
request_type)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\devappserver2\module.py", line 1476, in _handle_instance_request
request_id, request_type)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\devappserver2\instance.py", line 382, in handle
request_type))
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\devappserver2\http_runtime.py", line 244, in handle
response = connection.getresponse()
File "C:\Python27\Lib\httplib.py", line 1045, in getresponse
response.begin()
File "C:\Python27\Lib\httplib.py", line 409, in begin
version, status, reason = self._read_status()
File "C:\Python27\Lib\httplib.py", line 365, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "C:\Python27\Lib\socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
error: [Errno 10054] An existing connection was forcibly closed by the remote host
app.yaml
- url: /backend.html
script: przm_batchmodel.py
backends.yaml
backends:
- name: mybackend
class: B1
instances: 1
options: dynamic
Output.py
def loop_html(thefile):
####do some thing###
response = urlfetch.fetch(url=my_REST_server_URL, payload=data, method=urlfetch.POST, headers=http_headers, deadline=6000)
return response
class przmBatchOutputPage(webapp.RequestHandler):
def post(self):
form = cgi.FieldStorage()
thefile = form['file-0']
html = background_thread.BackgroundThread(target=loop_html, args=[thefile, generate_batch_jid()])
html.start()
self.response.out.write(html)
app = webapp.WSGIApplication([('/.*', przmBatchOutputPage)], debug=True)

urlfetch DeadlineExceededError

I had a create a REST API using bottle framework, which receives calls from GAE. Once this REST API is invoked, it did some calculations and sent outputs as a zip file to AMAZON S3 server and return the link to GAE. Everything works fine expect the timeout issue. I tried to adjust the deadline of urlfetch to 60 seconds, which did not solve the problem. I appreciate any suggestions.
GAE side:
response = urlfetch.fetch(url=url, payload=data, method=urlfetch.POST, headers=http_headers, deadline=60)
Broser error info.:
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1102, in __call__
return handler.dispatch()
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "D:\Dropbox\ubertool_src\przm5/przm5_output.py", line 22, in post
przm5_obj = przm5_rest_model.przm5(args)
File "D:\Dropbox\ubertool_src\przm5\przm5_rest_model.py", line 351, in __init__
self.convertSoil1, self.convert1to3, self.convert2to3)
File "D:\Dropbox\ubertool_src\przm5\przm5_rest_model.py", line 135, in get_jid
response = urlfetch.fetch(url=url, payload=data, method=urlfetch.POST, headers=http_headers)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\urlfetch.py", line 270, in fetch
return rpc.get_result()
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\apiproxy_stub_map.py", line 612, in get_result
return self.__get_result_hook(self)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\urlfetch.py", line 410, in _get_fetch_result
'Deadline exceeded while waiting for HTTP response from URL: ' + url)
DeadlineExceededError: Deadline exceeded while waiting for HTTP response from URL: http://localhost:7777/my_model
REST server:
#route('/my_model', method='POST')
#auth_basic(check)
def my_model():
#run the model
run_my model()
#zip output files
zout=zipfile.ZipFile("test.zip","w")
for name in os.listdir(src1):
zout.write(name)
zout.close()
##upload file to S3
conn = S3Connection(key, secretkey)
bucket = Bucket(conn, 'przm5')
k=Key(bucket)
name1='PRZM5_'+name_temp+'.zip'
k.key=name1
###All the above steps are fine####
k.set_contents_from_filename('test.zip')
link='https://s3.amazonaws.com/'+my_file_path
return {'ff': ff}
run(host='localhost', port=7777, debug=True)
Errors from the REST server:
127.0.0.1 - - [07/Jan/2014 16:16:36] "POST /my_model HTTP/1.1" 200 1663
Traceback (most recent call last):
File "C:\Python27\Lib\wsgiref\handlers.py", line 86, in run
self.finish_response()
File "C:\Python27\Lib\wsgiref\handlers.py", line 128, in finish_response
self.write(data)
File "C:\Python27\Lib\wsgiref\handlers.py", line 212, in write
self.send_headers()
File "C:\Python27\Lib\wsgiref\handlers.py", line 270, in send_headers
self.send_preamble()
File "C:\Python27\Lib\wsgiref\handlers.py", line 194, in send_preamble
'Date: %s\r\n' % format_date_time(time.time())
File "C:\Python27\Lib\socket.py", line 324, in write
self.flush()
File "C:\Python27\Lib\socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 10053] An established connection was aborted by the software in your host machine
127.0.0.1 - - [07/Jan/2014 16:16:36] "POST /my_model HTTP/1.1" 500 59
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 50953)
Traceback (most recent call last):
File "C:\Python27\Lib\SocketServer.py", line 295, in _handle_request_noblock
self.process_request(request, client_address)
File "C:\Python27\Lib\SocketServer.py", line 321, in process_request
self.finish_request(request, client_address)
File "C:\Python27\Lib\SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Python27\Lib\SocketServer.py", line 651, in __init__
self.finish()
File "C:\Python27\Lib\SocketServer.py", line 710, in finish
self.wfile.close()
File "C:\Python27\Lib\socket.py", line 279, in close
self.flush()
File "C:\Python27\Lib\socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 10053] An established connection was aborted by the software in your host machine

The deadline is a maximum value, once reached it'll fail. And it's failing with
Deadline exceeded while waiting for HTTP response
So you should try to catch that exception and try again.
If the entire operation can't be done in under 60 seconds then there is nothing else to be done, it's a hard limit in GAE that HTTP requests can't exceed 60 seconds.

Combining tornado and zmq ioloops: Connection reset by peer exception

I faced with one annoying (while not critical) problem when I tried to combine Tornado and pyzmq ioloops as described in the pyzmq official documentation.
I have a process running tornado (T) server which accepts REST API requests from clients (C) and proxies them though ZMQ transport to another process (Z) that does real work.
C <-> T <-> Z
If C closes the connection before Z replies to T, Z (tornado) outputs long bunch of exception traces (see at the bottom). Imagine the following example:
import tornado.ioloop
from tornado.web import Application, RequestHandler, asynchronous
from zmq.eventloop import ioloop
import time
def time_consuming_task():
time.sleep(5)
class TestHandler(RequestHandler):
def get(self, arg):
print "Test arg", arg
time_consuming_task()
print "Ok, time to reply"
self.write("Reply")
if __name__ == "__main__":
app = tornado.web.Application(
[
(r"/test/([0-9]+)", TestHandler)
])
ioloop.install()
app.listen(8080)
tornado.ioloop.IOLoop.instance().start()
This example doesn't actually talk to any ZMQ peer, it just attaches pyzmq ioloop to tornado's ioloop. Though, it's enough to illustrate the problem.
From console one run server:
% python example.py
From console two run client and interrupt it before server replies (i.e. during 5 secs):
% curl -is http://localhost:8080/test/1
^C
The output of server is:
Test arg 1
Ok, time to reply
WARNING:root:Read error on 24: [Errno 54] Connection reset by peer
ERROR:root:Uncaught exception GET /test/1 (::1)
HTTPRequest(protocol='http', host='localhost:8080', method='GET', uri='/test/1', version='HTTP/1.1', remote_ip='::1', body='', headers={'Host': 'localhost:8080', 'Accept': '*/*', 'User-Agent': 'curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5'})
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/tornado/web.py", line 1023, in _execute
self.finish()
File "/Library/Python/2.7/site-packages/tornado/web.py", line 701, in finish
self.request.finish()
File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 433, in finish
self.connection.finish()
File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 187, in finish
self._finish_request()
File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 223, in _finish_request
self.stream.read_until(b("\r\n\r\n"), self._header_callback)
File "/Library/Python/2.7/site-packages/tornado/iostream.py", line 153, in read_until
self._try_inline_read()
File "/Library/Python/2.7/site-packages/tornado/iostream.py", line 386, in _try_inline_read
if self._read_to_buffer() == 0:
File "/Library/Python/2.7/site-packages/tornado/iostream.py", line 421, in _read_to_buffer
chunk = self._read_from_socket()
File "/Library/Python/2.7/site-packages/tornado/iostream.py", line 402, in _read_from_socket
chunk = self.socket.recv(self.read_chunk_size)
error: [Errno 54] Connection reset by peer
ERROR:root:Cannot send error response after headers written
ERROR:root:Uncaught exception, closing connection.
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/tornado/iostream.py", line 304, in wrapper
callback(*args)
File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 262, in _on_headers
self.request_callback(self._request)
File "/Library/Python/2.7/site-packages/tornado/web.py", line 1412, in __call__
handler._execute(transforms, *args, **kwargs)
File "/Library/Python/2.7/site-packages/tornado/web.py", line 1025, in _execute
self._handle_request_exception(e)
File "/Library/Python/2.7/site-packages/tornado/web.py", line 1065, in _handle_request_exception
self.send_error(500, exc_info=sys.exc_info())
File "/Library/Python/2.7/site-packages/tornado/web.py", line 720, in send_error
self.finish()
File "/Library/Python/2.7/site-packages/tornado/web.py", line 700, in finish
self.flush(include_footers=True)
File "/Library/Python/2.7/site-packages/tornado/web.py", line 660, in flush
self.request.write(headers + chunk, callback=callback)
File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 429, in write
self.connection.write(chunk, callback=callback)
File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 177, in write
assert self._request, "Request closed"
AssertionError: Request closed
ERROR:root:Exception in callback
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pyzmq-2.2.0-py2.7-macosx-10.7-intel.egg/zmq/eventloop/ioloop.py", line 434, in _run_callback
callback()
File "/Library/Python/2.7/site-packages/tornado/iostream.py", line 304, in wrapper
callback(*args)
File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 262, in _on_headers
self.request_callback(self._request)
File "/Library/Python/2.7/site-packages/tornado/web.py", line 1412, in __call__
handler._execute(transforms, *args, **kwargs)
File "/Library/Python/2.7/site-packages/tornado/web.py", line 1025, in _execute
self._handle_request_exception(e)
File "/Library/Python/2.7/site-packages/tornado/web.py", line 1065, in _handle_request_exception
self.send_error(500, exc_info=sys.exc_info())
File "/Library/Python/2.7/site-packages/tornado/web.py", line 720, in send_error
self.finish()
File "/Library/Python/2.7/site-packages/tornado/web.py", line 700, in finish
self.flush(include_footers=True)
File "/Library/Python/2.7/site-packages/tornado/web.py", line 660, in flush
self.request.write(headers + chunk, callback=callback)
File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 429, in write
self.connection.write(chunk, callback=callback)
File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 177, in write
assert self._request, "Request closed"
AssertionError: Request closed
NOTE: It seems it's pyzmq related problem because disappears after excluding pyzmq ioloop.
Server doesn't die, it can be used by other clients, so the problem is not critical. Though, it's very annoying to find these huge confusing traces in logfiles.
So, are there any well-known methods to solve this problem?
Thanks.

It's not ZMQ problem. Request can be closed by other than timeout reasons. ZMQ only problem here is that they are raising AssertionError, which is common, instead of more specific exception.
If you are sure, that you don't want to have these exceptions in log files - do something like this:
try:
time_consuming_task()
except AssertionError as e:
if e.message == 'Request closed':
logging.info('Bad, annoying client, came to us again!')
else:
raise e