I'm building an asynchronous HTTP request class with urllib2. The code looks like this:
import urllib2, threading, datetime
class ExportHandler(urllib2.HTTPHandler):
def http_response(self, link, actualdate, nowdate ):
link += "&export=csv"
export = urllib2.urlopen( link )
for link, actualdate in commissionSummaryLinks:
o = urllib2.build_opener(ExportHandler())
t = threading.Thread(target=o.open, args=(link, actualdate, datetime.datetime.now().strftime("%Y%m%d%H%M%S")))
t.start()
print "I'm asynchronous!"
t.join()
print "ending threading"
Suffice it to say commissionSummaryLinks IS populated and actualdate is a date time.datetime.strptime() object.
Anyway, I'm receiving an error from all the issued threads that looks like this:
Exception in thread Thread-9:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 808, in __bootstrap_inner
self.run()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 761, in run
self.__target(*self.__args, **self.__kwargs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 402, in open
req = meth(req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1123, in do_request_
'Content-length', '%d' % len(data))
TypeError: object of type 'datetime.date' has no len()
I'm running this on OS X (if it matters). Can anyone tell me what the problem here is?
When you instantiate your thread, you need to provide the arguments to o.open, not arguments to the http_response method of ExportHandler.
In this case, o.open has the following method signature:
open(self, fullurl, data=None, timeout=<object object>) method of urllib2.OpenerDirector instance
My guess is that you should only need to set args=(link,).
If you still need to use those other arguments, you'll probably want to modify the constructor of ExportHandler to take the other arguments you need, and then use them as appropriate in the http_response method. Take a look at the Python Class tutorial for more information on defining a class constructor.
Related
I have some working code, that gets data from a queue, processes it and then emits the data via Flask-socketio to the browser.
This works when there aren't many messages to emit, however when the workload increases it simply can't cope.
Additionally I have tried processing the queue without transmitting and this appears to work correctly.
So what I was hoping to do, is rather than emit every time the queue.get() fires, to simply emit whatever the current dataset is on every ping.
In theory this should mean that even if the queue.get() fires multiple times between ping and pong, the messages being sent should remain constant and won't overload the system.
It means of course, some data will not be sent, but the most up-to date data as of the last ping should be sent, which is sufficient for what I need.
Hopefully that makes sense, so on to the (sort of working) code...
This works when there are not a lot of messages to emit (the socketio sleep needs to be there, otherwise the processing doesn't occur before it tries to emit):
def handle_message(*_args, **_kwargs):
try:
order_books = order_queue.get(block=True, timeout=1)
except Empty:
order_books = None
market_books = market_queue.get()
uo = update_orders(order_books, mkt_runners, profitloss, trading, eo, mb)
update_market_book(market_books, mb, uo, market_catalogues, mkt_runners, profitloss, trading)
socketio.sleep(0.2)
emit('initial_market', {'message': 'initial_market', 'mb': json.dumps(mb), 'ob': json.dumps(eo)})
socketio.sleep(0.2)
emit('my_response2', {'message': 'pong'})
def main():
socketio.run(app, debug=True, port=3000)
market_stream.stop()
order_stream.stop()
if __name__ == '__main__':
main()
This works (but I'm not trying to emit any messages here, this is just the Python script getting from the queue and processing):
while True:
try:
order_books = order_queue.get(block=True, timeout=1)
except Empty:
order_books = None
market_books = market_queue.get()
uo = update_orders(order_books, mkt_runners, profitloss, trading, eo, mb)
update_market_book(market_books, mb, uo, market_catalogues, mkt_runners, profitloss, trading)
print(mb)
(The mb at the end is the current dataset, which is returned by the update_market_book function).
Now, I was hoping, by having the While True at the end, this could simply run and the function would return the latest dataset on every ping, however using the above, the While True only fires when the main function is taken out...which of course stops the socketio section from working.
So, is there a way I can combine both these, to achieve what I am trying to do and/or is there an alternative method I haven't considered that might work?
As always, I appreciate your advice and if the question is not clear, please let me know, so I can clarify any bits.
Many thanks!
Just adding a stack trace as requested:
Traceback (most recent call last):
File "D:\Python37\lib\site-packages\betfairlightweight\endpoints\login.py", line 38, in request
response = session.post(self.url, data=self.data, headers=self.client.login_headers, cert=self.client.cert)
File "D:\Python37\lib\site-packages\requests\api.py", line 116, in post
return request('post', url, data=data, json=json, **kwargs)
File "D:\Python37\lib\site-packages\requests\api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "D:\Python37\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "D:\Python37\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "D:\Python37\lib\site-packages\requests\adapters.py", line 449, in send
timeout=timeout
File "D:\Python37\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
chunked=chunked)
File "D:\Python37\lib\site-packages\urllib3\connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "D:\Python37\lib\site-packages\urllib3\connectionpool.py", line 839, in _validate_conn
conn.connect()
File "D:\Python37\lib\site-packages\urllib3\connection.py", line 332, in connect
cert_reqs=resolve_cert_reqs(self.cert_reqs),
File "D:\Python37\lib\site-packages\urllib3\util\ssl_.py", line 279, in create_urllib3_context
context.options |= options
File "D:\Python37\lib\ssl.py", line 507, in options
super(SSLContext, SSLContext).options.__set__(self, value)
File "D:\Python37\lib\ssl.py", line 507, in options
super(SSLContext, SSLContext).options.__set__(self, value)
File "D:\Python37\lib\ssl.py", line 507, in options
super(SSLContext, SSLContext).options.__set__(self, value)
[Previous line repeated 489 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object
I need a simple reverse proxy for Python 3.
I like Twisted and its simple reverse http proxy (http://twistedmatrix.com/documents/14.0.1/_downloads/reverse-proxy.py) ...
# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.
"""
This example demonstrates how to run a reverse proxy.
Run this example with:
$ python reverse-proxy.py
Then visit http://localhost:8080/ in your web browser.
"""
from twisted.internet import reactor
from twisted.web import proxy, server
site = server.Site(proxy.ReverseProxyResource('www.yahoo.com', 80, ''))
reactor.listenTCP(8080, site)
reactor.run()
.... but it throws error in Python 3.
Unhandled Error
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/twisted/protocols/basic.py", line 571, in dataReceived
why = self.lineReceived(line)
File "/usr/local/lib/python3.4/dist-packages/twisted/web/http.py", line 1752, in lineReceived
self.allContentReceived()
File "/usr/local/lib/python3.4/dist-packages/twisted/web/http.py", line 1845, in allContentReceived
req.requestReceived(command, path, version)
File "/usr/local/lib/python3.4/dist-packages/twisted/web/http.py", line 766, in requestReceived
self.process()
--- <exception caught here> ---
File "/usr/local/lib/python3.4/dist-packages/twisted/web/server.py", line 185, in process
resrc = self.site.getResourceFor(self)
File "/usr/local/lib/python3.4/dist-packages/twisted/web/server.py", line 791, in getResourceFor
return resource.getChildForRequest(self.resource, request)
File "/usr/local/lib/python3.4/dist-packages/twisted/web/resource.py", line 98, in getChildForRequest
resource = resource.getChildWithDefault(pathElement, request)
File "/usr/local/lib/python3.4/dist-packages/twisted/web/resource.py", line 201, in getChildWithDefault
return self.getChild(path, request)
File "/usr/local/lib/python3.4/dist-packages/twisted/web/proxy.py", line 278, in getChild
self.host, self.port, self.path + b'/' + urlquote(path, safe=b"").encode('utf-8'),
builtins.TypeError: Can't convert 'bytes' object to str implicitly
Is there something similar that works in python 3?
Having stumbled into this problem myself, I was disappointed to find no answer here. After some digging, I was able to get it working by changing the path argument in proxy.ReverseProxyResource to bytes rather than str, rendering the following line:
site = server.Site(proxy.ReverseProxyResource("www.yahoo.com", 80, b''))
This is necessary because twisted appends a trailing slash as bytes (ie b'/') internally.
I found this simple package on pypi, it seems to be working well and it is similarly simple.
https://pypi.python.org/pypi/maproxy
I am sending requests and receiving their response objects into a hooked function.
Sometimes, I'm sending a new request out of that same function, which in turns calls the function that sends requests (with a new thread for each request).
That creates the following exception in the Requests module itself:
Exception in thread Thread-103:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/api.py", line 272, in thread_func
r = self.session.post(url, data=params, headers=headers, timeout=60, hooks=dict(response=self.http_callback))
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 425, in post
return self.request('POST', url, data=data, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 383, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 494, in send
if r.history:
AttributeError: 'bool' object has no attribute 'history'
I find that very strange. Do you know what might be happening?
Note: I looked at this question but my problem is different: Problems with hooks using Requests Python package
The line before the exception point reads:
r = dispatch_hook('response', hooks, r, **kwargs)
Your hook returned a boolean, while it should return a response object. A boolean doesn't have a history attribute, while a response object would.
Alternatively, return None if you did not mean to alter the response.
From the event hooks documentation:
If the callback function returns a value, it is assumed that it is to replace the data that was passed in. If the function doesn’t return anything, nothing else is effected.
The value passed in for the response hook is the response object; the replacement value should be a response object too.
Fix your hook. From your traceback we can see that you passed in self.http_callback as the hook, if you were wondering where to look.
I'm running a Flask app and internally using a library written in Node.js, which I access through ZeroRPC (the actual node process is managed by Circus). This works fine on its own; I can unit test with no issues. But when starting the Flask app as a listening process, and calling into a REST api which calls this libary, the program throws an exception when trying to start the process. The code to start the service is as follows:
from circus.watcher import Watcher
from circus.arbiter import ThreadedArbiter
from circus.util import (DEFAULT_ENDPOINT_DEALER, DEFAULT_ENDPOINT_SUB,
DEFAULT_ENDPOINT_MULTICAST)
class Node(object):
{... omitted code that initializes self._arbiter and self._client ...}
def start(self):
if self._arbiter and self._client:
return
port = 'ipc:///tmp/inlinejs_%s' % os.getpid()
args = 'lib/server.js --port %s' % port
watcher = Watcher('node', '/usr/local/bin/node', args,
working_dir=INLINEJS_DIR)
self._arbiter = ThreadedArbiter([watcher], DEFAULT_ENDPOINT_DEALER,
DEFAULT_ENDPOINT_SUB, multicast_endpoint=DEFAULT_ENDPOINT_MULTICAST)
self._arbiter.start()
self._client = zerorpc.Client()
self._client.connect(port)
This function returns, but shortly afterwards in a separate thread, I get this error:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/python/lib/python2.7/site-packages/circus/_patch.py", line 21, in _bootstrap_inner
self.run()
File "/python/lib/python2.7/site-packages/circus/arbiter.py", line 647, in run
return Arbiter.start(self)
File "/python/lib/python2.7/site-packages/circus/util.py", line 319, in _log
return func(self, *args, **kw)
File "/python/lib/python2.7/site-packages/circus/arbiter.py", line 456, in start
self.initialize()
File "/python/lib/python2.7/site-packages/circus/util.py", line 319, in _log
return func(self, *args, **kw)
File "/python/lib/python2.7/site-packages/circus/arbiter.py", line 427, in initialize
self.evpub_socket.bind(self.pubsub_endpoint)
File "socket.pyx", line 432, in zmq.core.socket.Socket.bind (zmq/core/socket.c:4022)
File "checkrc.pxd", line 21, in zmq.core.checkrc._check_rc (zmq/core/socket.c:5838)
ZMQError: Address already in use
I have no idea why this is happening, especially since it doesn't happen in unit tests. Can anyone shed any light?
In general, when you get this type of "Address in use" error, it means that your program is trying to bind on an IP port number but something else got there first.
I am not familiar with this library, but since the error is caused by "evpub_socket.bind", I am going to guess that you have a conflict with the port number specified by the constant DEFAULT_ENDPOINT_SUB. From the circus source code I see these constants:
DEFAULT_ENDPOINT_DEALER = "tcp://127.0.0.1:5555"
DEFAULT_ENDPOINT_SUB = "tcp://127.0.0.1:5556"
DEFAULT_ENDPOINT_STATS = "tcp://127.0.0.1:5557"
Check your system (netstat) and see if any process is listening on ports 5555, 5556, 5557. Or perhaps you are running this program twice and you forgot about the first one.
I have a weird problem with SUDS in python.
I am writing an application, that uses a SOAP service. And for approximately 2 months everything was going fine. All of a sudden, it stopped working. Here is the weird part though.
If I run the unittest for the suds manager I wrote, everything goes well and the test passes like usual. If I run the code in the GUI application it fails, with the following traceback:
ERROR:suds.client:<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:ns0="http://service.skip.fiskistofa.is/" xmlns:ns1="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP-ENV:Header/>
<ns1:Body>
<ns0:getLandings>
<from>2013-10-11</from>
<to>2013-10-01</to>
</ns0:getLandings>
</ns1:Body>
</SOAP-ENV:Envelope>
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 808, in __bootstrap_inner
self.run()
File "/home/fintor/bin/python/aflafrettir/gui/gui.py", line 332, in run
landings = self.manager.get_landings(self.date_from, self.date_to)
File "/home/fintor/bin/python/aflafrettir/landings/manager.py", line 66, in get_landings
return self.call_method('getLandings', date_from, date_to)
File "/home/fintor/bin/python/aflafrettir/landings/manager.py", line 49, in call_method
return getattr(self.client.service, method)(*args)
File "/home/fintor/.virtenvs/aflafrettir/lib/python2.7/site-packages/suds/client.py", line 542, in __call__
return client.invoke(args, kwargs)
File "/home/fintor/.virtenvs/aflafrettir/lib/python2.7/site-packages/suds/client.py", line 602, in invoke
result = self.send(soapenv)
File "/home/fintor/.virtenvs/aflafrettir/lib/python2.7/site-packages/suds/client.py", line 649, in send
result = self.failed(binding, e)
File "/home/fintor/.virtenvs/aflafrettir/lib/python2.7/site-packages/suds/client.py", line 702, in failed
r, p = binding.get_fault(reply)
File "/home/fintor/.virtenvs/aflafrettir/lib/python2.7/site-packages/suds/bindings/binding.py", line 265, in get_fault
raise WebFault(p, faultroot)
WebFault: Server raised fault: 'Couldn't create SOAP message due to exception: XML reader error: com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected character 'C' (code 67) in prolog; expected '<'
at [row,col {unknown-source}]: [1,1]'
The code that fails, is running in a thread within the GUI application(PySide), the relevant code is here:
class Worker(Thread):
signal = AflafrettirSignal()
def __init__(self, queue, manager, date_from, date_to):
super(Worker, self).__init__()
self.queue = queue
self.manager = manager
self.date_from = date_from
self.date_to = date_to
self.daemon = True
def run(self):
landings = self.manager.get_landings(self.date_from, self.date_to)
for l in landings:
tmp = Landings()
tmp.set_variable(l)
tmp.calc_total_catch()
self.queue.put(tmp)
self.signal.work_done.emit(True)
and it's called in the gui like so:
manager = LandingsManager()
manager.set_credentials(self.username, self.password)
manager.get_client()
worker = Worker(self.queue, manager, date_from, date_to)
worker.start()
Finally, here is the code from the unittest module, which is working fine:
manager = LandingsManager()
manager.set_credentials(usern, passw)
manager.get_client()
data = manager.get_landings('2013-10-01', '2013-10-11')
I have verified that the username and password is the same when I set the credentials as well as the date from and date to when I call the get_landings method.
Like I said, it is a weird problem, and I am not sure what causes this to fail in one place, and work in another. I hope that someone can enlighten me regarding the cause.
EDIT:
I figured out what was wrong. The password contained a carriage return, and the SOAP service did not like that.