I've been trying to start profiling my CherryPy webserver, but the documentation is lacking in detail in how this should be set up. I understand that I should be able to use cherrypy.lib.profiler as middleware to mount my initial server. Right now, I have code like the following:
server_app = ServerClass()
cherrypy.tree.mount(server_app, '/', '/path/to/config/file.cfg')
cherrypy.engine.start()
cherrypy.engine.block()
I want to mount the profiling middleware, and it seems that something like the following is required:
from cherrypy.lib import profiler
server_app = ServerClass()
server_cpapp = cherrypy.Application(server_app, '/', '/path/to/config/file.cfg')
server_profile_cpapp = profiler.make_app(server_cpapp, '/home/ken/tmp/cprofile', True)
#cherrypy.tree.mount(server_profile_cpapp)
cherrypy.tree.graft(server_profile_cpapp)
cherrypy.engine.start()
cherrypy.engine.block()
For some reason cherrypy.tree.mount doesn't work, but if I use cherrypy.tree.graft all seems to operate fine (I can make requests to the server as normal)
However, the above code generates a cp_0001.prof file under /home/ken/tmp/cprofile and I am not sure how to interpret it. I have tried using pyprof2calltree to read the data into KCacheGrind, but I get a parsing error. Does what I'm doing seem correct, and if so how do I interpret the output file?
It turns out that the profile files generated by CherryPy can be interpreted using the profiler.py script shipped as part of CherryPy. Simply run profiler.py in the <site-packages>/cherrypy/lib directory as follows:
python profiler.py /directory/containing/prof/files 8080
Then navigate to localhost:8080 in your browser and the profiling results for all .prof files in the target directory will be displayed in a simple text interface.
I would still prefer to be able to export the results into a calltree to profile using KCacheGrind, but this seems to do for basic profiling.
This is documented in the change log for v2.1 of CherryPy when the profiler was introduced (although the other details on that page describing how to set up the profiler has since become deprecated)
I am also trying to get profiling up and running for a cherrypy instance. I used the same code you have in your initial question, which seems to work in that it generates a cp_0001.prof file in the folder.
To answer your question, I am able to open this file in runsnakerun to see the profiling output in a tree view.
The problem I have is that every request I do to the server now fails, with the following output in the log:
[29/May/2013:16:39:32] ENGINE AssertionError('Bad call', ('', 0, 'sleep'), <frame object at 0x08522400>, <frame object at 0x08522030>, <frame object at 0x08156748>, <frame object at 0x06D06F10>)
Traceback (most recent call last):
File "<path>\packages\cherrypy\wsgiserver\wsgiserver2.py", line 1302, in communicate
req.respond()
File "<path>\packages\cherrypy\wsgiserver\wsgiserver2.py", line 831, in respond
self.server.gateway(self).respond()
File "<path>\packages\cherrypy\wsgiserver\wsgiserver2.py", line 2115, in respond
response = self.req.server.wsgi_app(self.env, self.start_response)
File "<path>\packages\cherrypy\_cptree.py", line 290, in __call__
return app(environ, start_response)
File "<path>\packages\cherrypy\lib\profiler.py", line 188, in __call__
return self.profiler.run(gather)
File "<path>\packages\cherrypy\lib\profiler.py", line 147, in run
result = self.profiler.runcall(func, *args)
File "<path>\python\lib\profile.py", line 472, in runcall
return func(*args, **kw)
File "<path>\packages\cherrypy\lib\profiler.py", line 183, in gather
def gather():
File "<path>\python\lib\profile.py", line 246, in trace_dispatch_i
if self.dispatch[event](self, frame, t):
File "<path>\python\lib\profile.py", line 301, in trace_dispatch_call
frame, frame.f_back)
AssertionError: ('Bad call', ('', 0, 'sleep'), <frame object at 0x08522400>, <frame object at 0x08522030>, <frame object at 0x08156748>, <frame object at 0x06D06F10>)
I am using python 2.6.6 and cherrypy 3.2.2
Any suggestions?
Related
I don't have much experience working with network related stuff, please excuse me if my question is too lame.
I created an executable file with a python script (using pyinstaller). Basically the script uses Rest services to look up data.
I passed this exe to two of my colleagues and tested in their machines. The first machine passed the test (exe file executed as expected).
But the second machine failed with
File "site-packages\TM1py\Services\TM1Service.py", line 12, in __init__
File "site-packages\TM1py\Services\RESTService.py", line 96, in __init__
File "site-packages\TM1py\Services\RESTService.py", line 169, in _start_session
File "site-packages\requests\sessions.py", line 525, in get
File "site-packages\requests\sessions.py", line 512, in request
File "site-packages\requests\sessions.py", line 622, in send
File "site-packages\requests\adapters.py", line 388, in send
File "site-packages\requests\adapters.py", line 292, in get_connection
requests.exceptions.InvalidProxyURL: Please check proxy URL. It is malformed and could be missing the host.
I could see that there is some issue with the proxy settings. So, when I checked the proxy settings I found something weird. I'm attaching screenshots of the proxy settings. The first screenshot is from the successful machine and the latter from failure.
Please advice me how to resolve this issue, what am I missing here?
Thanks,
I have a few twitterbots that I run on my raspberryPi. I have most functions wrapped in a try / except to ensure that if something errors it doesn't break the program and continues to execute.
I'm also using Python's Streaming library as my source of monitoring for the tags that I want the bot to retweet.
Here is an issue that happens that kills the program although I have the main function wrapped in a try/except:
Unhandled exception in thread started by <function startBot5 at 0x762fbed0>
Traceback (most recent call last):
File "TwitButter.py", line 151, in startBot5
'<botnamehere>'
File "/home/pi/twitter/bots/TwitBot.py", line 49, in __init__
self.startFiltering(trackList)
File "/home/pi/twitter/bots/TwitBot.py", line 54, in startFiltering
self.myStream.filter(track=tList)
File "/usr/local/lib/python3.4/dist-packages/tweepy/streaming.py", line 445, in filter
self._start(async)
File "/usr/local/lib/python3.4/dist-packages/tweepy/streaming.py", line 361, in _start
self._run()
File "/usr/local/lib/python3.4/dist-packages/tweepy/streaming.py", line 294, in _run
raise exception
File "/usr/local/lib/python3.4/dist-packages/tweepy/streaming.py", line 263, in _run
self._read_loop(resp)
File "/usr/local/lib/python3.4/dist-packages/tweepy/streaming.py", line 313, in _read_loop
line = buf.read_line().strip()
AttributeError: 'NoneType' object has no attribute 'strip'
My setup:
I have a parent class TwitButter.py, that creates an object from the TwitBot.py. These objects are the bots, and they are started on their own thread so they can run independently.
I have a function in the TwitBot that runs the startFiltering() function. It is wrapped in a try/except, but my except code is never triggered.
My guess is that the error is occurring within the Streaming library. Maybe that library is poorly coded and breaks on the line that is specified at the bottom of the traceback.
Any help would be awesome, and I wonder if others have experienced this issue?
I can provide extra details if needed.
Thanks!!!
This actually is problem in tweepy that was fixed by github #870 in 2017-04. So, should be resolved by updating your local copy to latest master.
What I did to discover that:
Did a web search to find the tweepy source repo.
Looked at streaming.py for context on the last traceback lines.
Noticed the most recent change to the file was the same problem.
I'll also note that most of the time you get a traceback from deep inside a Python library, the problem comes from the code calling it incorrectly, rather than a bug in the library. But not always. :)
I am trying to use twisted but when i try to run some of the example code provided with the twisted package, it seems to always crash when i use "twistd" instead of "python"
for example, using the example code given with twisted,
if i run to command : twisted -ny echoserv.py
Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/twisted/application/app.py", line 652, in run
runApp(config)
File "/usr/lib/python2.7/site-packages/twisted/scripts/twistd.py", line 23, in runApp
_SomeApplicationRunner(config).run()
File "/usr/lib/python2.7/site-packages/twisted/application/app.py", line 386, in run
self.application = self.createOrGetApplication()
File "/usr/lib/python2.7/site-packages/twisted/application/app.py", line 451, in createOrGetApplication
application = getApplication(self.config, passphrase)
--- ---
File "/usr/lib/python2.7/site-packages/twisted/application/app.py", line 462, in getApplication
application = service.loadApplication(filename, style, passphrase)
File "/usr/lib/python2.7/site-packages/twisted/application/service.py", line 405, in loadApplication
application = sob.loadValueFromFile(filename, 'application', passphrase)
File "/usr/lib/python2.7/site-packages/twisted/persisted/sob.py", line 211, in loadValueFromFile
value = d[variable]
exceptions.KeyError: 'application'
Failed to load application: 'application'
Could not find 'application' in the file. To use 'twistd -y', your .tac
file must create a suitable object (e.g., by calling service.Application())
and store it in a variable named 'application'. twistd loads your .tac file
and scans the global variables for one of this name.
Please read the 'Using Application' HOWTO for details.
I was using Twisted version 11.0.0 but then i tried 12.0.0 but i have the same problem.
The version of python i am using is 2.7.2
Any ideas on what to do would be helpful. I have been trying to deal with this problem for a few days now. thanks!
twistd -y is meant to be used with a python file that contains a variable called application, there is none in this file, so it's not going to work.
You might want to spend some time reading twistd's documentation so you get a clearer idea of its role in the twisted ecosystem.
To use 'twistd -y', your .tac file must create a suitable object (e.g., by calling service.Application()) and store it in a variable named 'application'. twistd loads your .tac file and scans the global variables for one of this name.
Please read the 'Using Application' HOWTO for details.
I am attempting to build a download accelerator for Linux. My program utilizes gevent, os, and urllib2. My program receives a URL and attempts to download the file concurrently. All of my code is valid. My only problem is that urllib2.urlopen.read() is blocking me from running the .read() function concurrently.
This is the exception thats thrown at me.
Traceback (most recent call last):
File "/usr/lib/pymodules/python2.7/gevent/greenlet.py", line 405, in run
result = self._run(*self.args, **self.kwargs)
File "gevent_concurrent_downloader.py", line 94, in childTasklet
_tempRead = handle.read(divisor) # Read/Download part
File "/usr/lib/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
File "/usr/lib/python2.7/httplib.py", line 561, in read
s = self.fp.read(amt)
File "/usr/lib/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
File "/usr/lib/pymodules/python2.7/gevent/socket.py", line 407, in recv
wait_read(sock.fileno(), timeout=self.timeout, event=self._read_event)
File "/usr/lib/pymodules/python2.7/gevent/socket.py", line 153, in wait_read
assert event.arg is None, 'This event is already used by another greenlet: %r' % (event.arg, )
AssertionError: This event is already used by another greenlet: (<Greenlet at 0x2304958: childTasklet(<__main__.NewFile object at 0x22c4390>, 4595517, <addinfourl at 37154616 whose fp = <socket._fileob, 459551, 1)>, timeout('timed out',))
<Greenlet at 0x2304ea8: childTasklet(<__main__.NewFile object at 0x22c4390>,4595517, <addinfourl at 37154616 whose fp = <socket._fileob, 7, -1)failed with AssertionError
My program works by getting the file byte size from the URL by invoking:
urllib2.urlopen(URL).info().get("Content-Length")
and dividing the file size by a divisor and thus breaking the download process into parts. In this example i am breaking the download into 10 parts.
Each greenlet runs a command in this fassion:
urllib2.urlopen(URL).read(offset)
Here's a link to my code hosted on pastie: http://pastie.org/3253705
Thank you for the help!
FYI: I am running on Ubuntu 11.10.
You're trying to read a response to a single request from different greenlets.
If you'd like to download the same file using several concurrent connections then you could use Range http header if the server supports it (you get 206 status instead of 200 for the request with Range header). See HTTPRangeHandler.
the argument to read is a number of bytes, not an offset.
It seems gevent will let you call urllib asynchronously, but not let you access the same resource from multiple greenlets.
Furthermore, since it is using wait_read, the effect will still be a synchronous, sequential read from the file (The complete opposite of what you wanted to achieve).
I'd suggest you might need to go lower than, or use a different library from, urllib2.
Appreciate your helping first, I am new for the python 3.x.
When I try to use Python 3.x to parse the testlink xmlprc server. I got below error, but I can run the code under Python 2.x, any idea?
import xmlrpc.client
server = xmlrpc.client.Server("http://172.16.29.132/SITM/lib/api/xmlrpc.php") //here is my testlink server
print (server.system.listMethods()) //I can print the methods list here
print (server.tl.ping()) // Got error.
Here is the error:
['system.multicall', 'system.listMethods', 'system.getCapabilities', 'tl.repeat', 'tl.sayHello', 'tl.ping', 'tl.setTestMode', 'tl.about', 'tl.checkDevKey', 'tl.doesUserExist', 'tl.deleteExecution', 'tl.getTestSuiteByID', 'tl.getFullPath', 'tl.getTestCase', 'tl.getTestCaseAttachments', 'tl.getFirstLevelTestSuitesForTestProject', 'tl.getTestCaseCustomFieldDesignValue', 'tl.getTestCaseIDByName', 'tl.getTestCasesForTestPlan', 'tl.getTestCasesForTestSuite', 'tl.getTestSuitesForTestSuite', 'tl.getTestSuitesForTestPlan', 'tl.getLastExecutionResult', 'tl.getLatestBuildForTestPlan', 'tl.getBuildsForTestPlan', 'tl.getTotalsForTestPlan', 'tl.getTestPlanPlatforms', 'tl.getProjectTestPlans', 'tl.getTestPlanByName', 'tl.getTestProjectByName', 'tl.getProjects', 'tl.addTestCaseToTestPlan', 'tl.assignRequirements', 'tl.uploadAttachment', 'tl.uploadTestCaseAttachment', 'tl.uploadTestSuiteAttachment', 'tl.uploadTestProjectAttachment', 'tl.uploadRequirementAttachment', 'tl.uploadRequirementSpecificationAttachment', 'tl.uploadExecutionAttachment', 'tl.createTestSuite', 'tl.createTestProject', 'tl.createTestPlan', 'tl.createTestCase', 'tl.createBuild', 'tl.setTestCaseExecutionResult', 'tl.reportTCResult']
Traceback (most recent call last):
File "F:\SQA\Python\Testlink\Test.py", line 5, in <module>
print (server.tl.ping())
File "C:\Python31\lib\xmlrpc\client.py", line 1029, in __call__
return self.__send(self.__name, args)
File "C:\Python31\lib\xmlrpc\client.py", line 1271, in __request
verbose=self.__verbose
File "C:\Python31\lib\xmlrpc\client.py", line 1070, in request
return self.parse_response(resp)
File "C:\Python31\lib\xmlrpc\client.py", line 1164, in parse_response
p.feed(response)
File "C:\Python31\lib\xmlrpc\client.py", line 454, in feed
self._parser.Parse(data, 0)
xml.parsers.expat.ExpatError: junk after document element: line 2, column 0
When I've seen this message before, it happened because the contents of the transported data wasn't escaped for XML transport. The solution was to wrap the data in an XMLRPC Binary object.
In your case, you don't control the server side, so the above isn't a solution for you but it may suggest what the actual problem is.
Also, the Python 2 versus Python 3 difference suggests that there is a text/bytes issue at work.
To help diagnose the issue, set verbose=True so you can see the actual HTTP request/response headers and the XML request/response. That may show you what is at line 2: column 0. You may find that the issue may be with the PHP script not wrapping up binary data in base64 encoding as required by the XMLRPC spec.
Thank you , I find out all the methods list, only 'tl.sayHello', 'tl.ping','tl.about' has this problem, and all of them are pass a string with a PHP automatic loader empty file *.class.php to the parser, other methods are pass a xml file. So I give up to use those methods and the script works fine.