I am attempting to build a download accelerator for Linux. My program utilizes gevent, os, and urllib2. My program receives a URL and attempts to download the file concurrently. All of my code is valid. My only problem is that urllib2.urlopen.read() is blocking me from running the .read() function concurrently.
This is the exception thats thrown at me.
Traceback (most recent call last):
File "/usr/lib/pymodules/python2.7/gevent/greenlet.py", line 405, in run
result = self._run(*self.args, **self.kwargs)
File "gevent_concurrent_downloader.py", line 94, in childTasklet
_tempRead = handle.read(divisor) # Read/Download part
File "/usr/lib/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
File "/usr/lib/python2.7/httplib.py", line 561, in read
s = self.fp.read(amt)
File "/usr/lib/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
File "/usr/lib/pymodules/python2.7/gevent/socket.py", line 407, in recv
wait_read(sock.fileno(), timeout=self.timeout, event=self._read_event)
File "/usr/lib/pymodules/python2.7/gevent/socket.py", line 153, in wait_read
assert event.arg is None, 'This event is already used by another greenlet: %r' % (event.arg, )
AssertionError: This event is already used by another greenlet: (<Greenlet at 0x2304958: childTasklet(<__main__.NewFile object at 0x22c4390>, 4595517, <addinfourl at 37154616 whose fp = <socket._fileob, 459551, 1)>, timeout('timed out',))
<Greenlet at 0x2304ea8: childTasklet(<__main__.NewFile object at 0x22c4390>,4595517, <addinfourl at 37154616 whose fp = <socket._fileob, 7, -1)failed with AssertionError
My program works by getting the file byte size from the URL by invoking:
urllib2.urlopen(URL).info().get("Content-Length")
and dividing the file size by a divisor and thus breaking the download process into parts. In this example i am breaking the download into 10 parts.
Each greenlet runs a command in this fassion:
urllib2.urlopen(URL).read(offset)
Here's a link to my code hosted on pastie: http://pastie.org/3253705
Thank you for the help!
FYI: I am running on Ubuntu 11.10.
You're trying to read a response to a single request from different greenlets.
If you'd like to download the same file using several concurrent connections then you could use Range http header if the server supports it (you get 206 status instead of 200 for the request with Range header). See HTTPRangeHandler.
the argument to read is a number of bytes, not an offset.
It seems gevent will let you call urllib asynchronously, but not let you access the same resource from multiple greenlets.
Furthermore, since it is using wait_read, the effect will still be a synchronous, sequential read from the file (The complete opposite of what you wanted to achieve).
I'd suggest you might need to go lower than, or use a different library from, urllib2.
Related
The question
I created a package for maintaining HTTP requests - just for fun.
It runs perfectly in normal Python, however, when I try to run it in PyPy, the following error occurs while requesting over HTTPS:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/adam/python/httpy/httpy/__init__.py", line 2019, in request
http_version=http_version
File "/home/adam/python/httpy/httpy/__init__.py", line 1863, in _raw_request
proto.send_request(sock,method,defhdr,data,path,debug)
File "/home/adam/python/httpy/httpy/__init__.py", line 1339, in send_request
return self.sender(*args).send(sock)
File "/home/adam/python/httpy/httpy/__init__.py", line 1677, in send
sock.send(self.body)
File "/opt/pypy3/lib/pypy3.8/ssl.py", line 1173, in send
return self._sslobj.write(data)
File "/opt/pypy3/lib/pypy3.8/_cffi_ssl/_stdssl/__init__.py", line 546, in write
return self._write_with_length(_str_to_ffi_buffer(bytestring), len(bytestring))
File "/opt/pypy3/lib/pypy3.8/_cffi_ssl/_stdssl/__init__.py", line 603, in _write_with_length
raise pyssl_error(self, length)
_cffi_ssl._stdssl.error.SSLEOFError: EOF occurred in violation of protocol
Requests, for example, run well even in PyPy.
Could the issue be something with using ssl.wrap_socket() or it's an error on a deeper level?
System info
Python version: 3.10.5
PyPy version: 3.8.13
OS: x86_64 Linux 5.18.10-arch1-1
OK, I found it out. It was because of sending b'' after request if the request body was empty. I thought that sending empty string would be an empty operation, but obviously it is not - it is treated as an EOF. I am still not sure why this fails only using PyPy. It's maybe because it's older version than Python where the code was tested.
Anyway, if you get this error, make sure you are not accidentially sending b''.
I'm working on interfacing an old Optical Spectrum Analyzer Anritsu MS9710B with a RS232 connection. A year ago, I managed to comunicate with him, send SCPI command using pyvisa and receive data. Today, I execute the exact same code with libraries up to date and I get an error Timeout when I query IDN or anything. The RS232-USB drivers are ok, I manage to open a communication but a query or read fail.
I changed the parameter to "RS232C" on the OSA parameters and my communication parameters are the same between OSA and program. Following advices from the forum, I tried the connection with NI-VISA where I get the same error. I tried to change timeout parameter, write_termination and read_termination but there isn't any change. The manual is very hard to understand termination value, but when it worked I didn't add anything.
I don't know anymore what to do.
Any advices and help would be warmly welcome to fix my problem !
import pyvisa
from pyvisa.constants import StopBits, Parity
rm = pyvisa.ResourceManager()
print(rm.list_resources())
my_instrument = rm.open_resource('ASRL5::INSTR')
my_instrument.baud_rate=9600
my_instrument.data_bits=8
my_instrument.parity=Parity.even
my_instrument.stop_bits=StopBits.one
my_instrument.write('*IDN?')
print(my_instrument.read())```
'''
('ASRL5::INSTR',)
Traceback (most recent call last):
File "PremiereComm.py", line 26, in <module>
print(my_instrument.read())
File "C:\Program Files\Python38\lib\site-packages\pyvisa\resources\messagebased.py", line 486, in read
message = self._read_raw().decode(enco)
File "C:\Program Files\Python38\lib\site-packages\pyvisa\resources\messagebased.py", line 442, in _read_raw
chunk, status = self.visalib.read(self.session, size)
File "C:\Program Files\Python38\lib\site-packages\pyvisa\ctwrapper\functions.py", line 2337, in read
ret = library.viRead(session, buffer, count, byref(return_count))
File "C:\Program Files\Python38\lib\site-packages\pyvisa\ctwrapper\highlevel.py", line 222, in _return_handler
return self.handle_return_value(session, ret_value) # type: ignore
File "C:\Program Files\Python38\lib\site-packages\pyvisa\highlevel.py", line 251, in handle_return_value
raise errors.VisaIOError(rv)
pyvisa.errors.VisaIOError: VI_ERROR_TMO (-1073807339): Timeout expired before operation completed.
>>> '''
I'm trying to connect a POS receipt printer to python and have been using python-escpos to do so. I'm running windows 10.
My code in my file 'print.py' is:
from escpos.printer import Usb
p = Usb(0x0456,0x0808)
p.text('Hello World\n')
but when i run it i get the following error log:
Traceback (most recent call last):
File "C:\Users\Harry\AppData\Local\Programs\Python\Python36-32\lib\site-packages\usb\core.py", line 223, in get_interface_and_endpoint
return self._ep_info[endpoint_address]
KeyError: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "print.py", line 5, in <module>
p.text('Hello World\n')
File "C:\Users\Harry\AppData\Local\Programs\Python\Python36-32\lib\site-packages\escpos\escpos.py", line 437, in text
self._raw(txt.encode())
File "C:\Users\Harry\AppData\Local\Programs\Python\Python36-32\lib\site-packages\escpos\printer.py", line 73, in _raw
self.device.write(self.out_ep, msg, self.timeout)
File "C:\Users\Harry\AppData\Local\Programs\Python\Python36-32\lib\site-packages\usb\core.py", line 940, in write
intf, ep = self._ctx.setup_request(self, endpoint)
File "C:\Users\Harry\AppData\Local\Programs\Python\Python36-32\lib\site-packages\usb\core.py", line 102, in wrapper
return f(self, *args, **kwargs)
File "C:\Users\Harry\AppData\Local\Programs\Python\Python36-32\lib\site-packages\usb\core.py", line 215, in setup_request
intf, ep = self.get_interface_and_endpoint(device, endpoint_address)
File "C:\Users\Harry\AppData\Local\Programs\Python\Python36-32\lib\site-packages\usb\core.py", line 102, in wrapper
return f(self, *args, **kwargs)
File "C:\Users\Harry\AppData\Local\Programs\Python\Python36-32\lib\site-packages\usb\core.py", line 225, in get_interface_and_endpoint
for intf in self.get_active_configuration(device):
File "C:\Users\Harry\AppData\Local\Programs\Python\Python36-32\lib\site-packages\usb\core.py", line 102, in wrapper
return f(self, *args, **kwargs)
File "C:\Users\Harry\AppData\Local\Programs\Python\Python36-32\lib\site-packages\usb\core.py", line 239, in get_active_configuration
bConfigurationValue=self.backend.get_configuration(self.handle)
File "C:\Users\Harry\AppData\Local\Programs\Python\Python36-32\lib\site-packages\usb\backend\libusb0.py", line 510, in get_configuration
100)
File "C:\Users\Harry\AppData\Local\Programs\Python\Python36-32\lib\site-packages\usb\backend\libusb0.py", line 593, in ctrl_transfer
timeout
File "C:\Users\Harry\AppData\Local\Programs\Python\Python36-32\lib\site-packages\usb\backend\libusb0.py", line 431, in _check
raise USBError(errmsg, ret)
usb.core.USBError: [Errno None] b'libusb0-dll:err [control_msg] sending control message failed, win error: A device which does not exist was specified.\r\n\n'
Does anyone know what i need to do to make this work.
I have installed libusb into my System32 and SysWOW64 folders, aswell as installing a libusb-32 driver for my printer with zadig.
The printer i'm using is a 'Hoin 58mm POS Printer'.
I would be grateful if someone could explain to me why i'm having this error and how to solve it!
Firstly, you can check this link for the complete thread.
As suggested by KM4YRI,
I ran into the same No backend available exception when trying to follow the tutorial on a Windows 7 machine. The following worked for me, per one of the StackOverflow replies:
1. Download the latest Windows binary: https://sourceforge.net/projects/libusb/files/libusb-1.0/libusb-1.0.21/libusb-1.0.21.7z/download
2. Unzip using 7zip into a temp dir
3. If on 64-bit Windows, copy `MS64\dll\libusb-1.0.dll` into `C:\windows\system32`. If on 32-bit windows, copy `MS32\dll\libusb-1.0.dll` into `C:\windows\SysWOW64`.
I've verified the above answer on Windows-10 64-bit version.
I had a tight experience with that error using at least 3 different USB Storage Devices. After hours of trials I ended up to a feeling that USB transmission errors are the cause of this error.
The USB bus will return it in a random pattern for a low quality connection as if a device remove has presumably happened.
In my scenario I was bulk writing sectors to a \\.\PhysicalDriveN device and I noticed that after a short period the connection reestablished and the file handle was still valid and going.
To overcome the error I've ended up with the following steps:
Seek To File Position;
Write Sector;
For any 433 error:
250 ms delay;
Seek To file Position, ignoring ERROR_NO_SUCH_DEVICE and ERROR_NOT_READY errors and trying for at least 1000 ms before giving up;
Read Sector, ignoring ERROR_NO_SUCH_DEVICE and ERROR_NOT_READY errors and retrying for up to 1000 ms before giving up;
Specially for my scenario: I've compared input buffer and read buffer and stop if matching;
Repeat the whole procedure up to 5 times.
So the idea here is the following: The device driver returns ERROR_NO_SUCH_DEVICE assuming a device remove scenario, but still in a pending state. As the problem is simply quality of physical connection, the link continues and the bus driver starts a renegotiation, in which case a ERROR_NOT_READY is temporarily returned. When the link is stable and negotiated, communication continues without the need to reopen the device handle.
I would recommend adding a sleep statement of a second or two in between the instantiation of the USB printer, and the attempt to print to it. e.g:
from escpos.printer import Usb
from time import sleep
p = Usb(0x0456,0x0808)
sleep(1)
p.text('Hello World\n')
I've been trying to start profiling my CherryPy webserver, but the documentation is lacking in detail in how this should be set up. I understand that I should be able to use cherrypy.lib.profiler as middleware to mount my initial server. Right now, I have code like the following:
server_app = ServerClass()
cherrypy.tree.mount(server_app, '/', '/path/to/config/file.cfg')
cherrypy.engine.start()
cherrypy.engine.block()
I want to mount the profiling middleware, and it seems that something like the following is required:
from cherrypy.lib import profiler
server_app = ServerClass()
server_cpapp = cherrypy.Application(server_app, '/', '/path/to/config/file.cfg')
server_profile_cpapp = profiler.make_app(server_cpapp, '/home/ken/tmp/cprofile', True)
#cherrypy.tree.mount(server_profile_cpapp)
cherrypy.tree.graft(server_profile_cpapp)
cherrypy.engine.start()
cherrypy.engine.block()
For some reason cherrypy.tree.mount doesn't work, but if I use cherrypy.tree.graft all seems to operate fine (I can make requests to the server as normal)
However, the above code generates a cp_0001.prof file under /home/ken/tmp/cprofile and I am not sure how to interpret it. I have tried using pyprof2calltree to read the data into KCacheGrind, but I get a parsing error. Does what I'm doing seem correct, and if so how do I interpret the output file?
It turns out that the profile files generated by CherryPy can be interpreted using the profiler.py script shipped as part of CherryPy. Simply run profiler.py in the <site-packages>/cherrypy/lib directory as follows:
python profiler.py /directory/containing/prof/files 8080
Then navigate to localhost:8080 in your browser and the profiling results for all .prof files in the target directory will be displayed in a simple text interface.
I would still prefer to be able to export the results into a calltree to profile using KCacheGrind, but this seems to do for basic profiling.
This is documented in the change log for v2.1 of CherryPy when the profiler was introduced (although the other details on that page describing how to set up the profiler has since become deprecated)
I am also trying to get profiling up and running for a cherrypy instance. I used the same code you have in your initial question, which seems to work in that it generates a cp_0001.prof file in the folder.
To answer your question, I am able to open this file in runsnakerun to see the profiling output in a tree view.
The problem I have is that every request I do to the server now fails, with the following output in the log:
[29/May/2013:16:39:32] ENGINE AssertionError('Bad call', ('', 0, 'sleep'), <frame object at 0x08522400>, <frame object at 0x08522030>, <frame object at 0x08156748>, <frame object at 0x06D06F10>)
Traceback (most recent call last):
File "<path>\packages\cherrypy\wsgiserver\wsgiserver2.py", line 1302, in communicate
req.respond()
File "<path>\packages\cherrypy\wsgiserver\wsgiserver2.py", line 831, in respond
self.server.gateway(self).respond()
File "<path>\packages\cherrypy\wsgiserver\wsgiserver2.py", line 2115, in respond
response = self.req.server.wsgi_app(self.env, self.start_response)
File "<path>\packages\cherrypy\_cptree.py", line 290, in __call__
return app(environ, start_response)
File "<path>\packages\cherrypy\lib\profiler.py", line 188, in __call__
return self.profiler.run(gather)
File "<path>\packages\cherrypy\lib\profiler.py", line 147, in run
result = self.profiler.runcall(func, *args)
File "<path>\python\lib\profile.py", line 472, in runcall
return func(*args, **kw)
File "<path>\packages\cherrypy\lib\profiler.py", line 183, in gather
def gather():
File "<path>\python\lib\profile.py", line 246, in trace_dispatch_i
if self.dispatch[event](self, frame, t):
File "<path>\python\lib\profile.py", line 301, in trace_dispatch_call
frame, frame.f_back)
AssertionError: ('Bad call', ('', 0, 'sleep'), <frame object at 0x08522400>, <frame object at 0x08522030>, <frame object at 0x08156748>, <frame object at 0x06D06F10>)
I am using python 2.6.6 and cherrypy 3.2.2
Any suggestions?
Appreciate your helping first, I am new for the python 3.x.
When I try to use Python 3.x to parse the testlink xmlprc server. I got below error, but I can run the code under Python 2.x, any idea?
import xmlrpc.client
server = xmlrpc.client.Server("http://172.16.29.132/SITM/lib/api/xmlrpc.php") //here is my testlink server
print (server.system.listMethods()) //I can print the methods list here
print (server.tl.ping()) // Got error.
Here is the error:
['system.multicall', 'system.listMethods', 'system.getCapabilities', 'tl.repeat', 'tl.sayHello', 'tl.ping', 'tl.setTestMode', 'tl.about', 'tl.checkDevKey', 'tl.doesUserExist', 'tl.deleteExecution', 'tl.getTestSuiteByID', 'tl.getFullPath', 'tl.getTestCase', 'tl.getTestCaseAttachments', 'tl.getFirstLevelTestSuitesForTestProject', 'tl.getTestCaseCustomFieldDesignValue', 'tl.getTestCaseIDByName', 'tl.getTestCasesForTestPlan', 'tl.getTestCasesForTestSuite', 'tl.getTestSuitesForTestSuite', 'tl.getTestSuitesForTestPlan', 'tl.getLastExecutionResult', 'tl.getLatestBuildForTestPlan', 'tl.getBuildsForTestPlan', 'tl.getTotalsForTestPlan', 'tl.getTestPlanPlatforms', 'tl.getProjectTestPlans', 'tl.getTestPlanByName', 'tl.getTestProjectByName', 'tl.getProjects', 'tl.addTestCaseToTestPlan', 'tl.assignRequirements', 'tl.uploadAttachment', 'tl.uploadTestCaseAttachment', 'tl.uploadTestSuiteAttachment', 'tl.uploadTestProjectAttachment', 'tl.uploadRequirementAttachment', 'tl.uploadRequirementSpecificationAttachment', 'tl.uploadExecutionAttachment', 'tl.createTestSuite', 'tl.createTestProject', 'tl.createTestPlan', 'tl.createTestCase', 'tl.createBuild', 'tl.setTestCaseExecutionResult', 'tl.reportTCResult']
Traceback (most recent call last):
File "F:\SQA\Python\Testlink\Test.py", line 5, in <module>
print (server.tl.ping())
File "C:\Python31\lib\xmlrpc\client.py", line 1029, in __call__
return self.__send(self.__name, args)
File "C:\Python31\lib\xmlrpc\client.py", line 1271, in __request
verbose=self.__verbose
File "C:\Python31\lib\xmlrpc\client.py", line 1070, in request
return self.parse_response(resp)
File "C:\Python31\lib\xmlrpc\client.py", line 1164, in parse_response
p.feed(response)
File "C:\Python31\lib\xmlrpc\client.py", line 454, in feed
self._parser.Parse(data, 0)
xml.parsers.expat.ExpatError: junk after document element: line 2, column 0
When I've seen this message before, it happened because the contents of the transported data wasn't escaped for XML transport. The solution was to wrap the data in an XMLRPC Binary object.
In your case, you don't control the server side, so the above isn't a solution for you but it may suggest what the actual problem is.
Also, the Python 2 versus Python 3 difference suggests that there is a text/bytes issue at work.
To help diagnose the issue, set verbose=True so you can see the actual HTTP request/response headers and the XML request/response. That may show you what is at line 2: column 0. You may find that the issue may be with the PHP script not wrapping up binary data in base64 encoding as required by the XMLRPC spec.
Thank you , I find out all the methods list, only 'tl.sayHello', 'tl.ping','tl.about' has this problem, and all of them are pass a string with a PHP automatic loader empty file *.class.php to the parser, other methods are pass a xml file. So I give up to use those methods and the script works fine.