Develop Reference

Tweepy issues with twitter bot and python

I have a few twitterbots that I run on my raspberryPi. I have most functions wrapped in a try / except to ensure that if something errors it doesn't break the program and continues to execute.
I'm also using Python's Streaming library as my source of monitoring for the tags that I want the bot to retweet.
Here is an issue that happens that kills the program although I have the main function wrapped in a try/except:
Unhandled exception in thread started by <function startBot5 at 0x762fbed0>
Traceback (most recent call last):
File "TwitButter.py", line 151, in startBot5
'<botnamehere>'
File "/home/pi/twitter/bots/TwitBot.py", line 49, in __init__
self.startFiltering(trackList)
File "/home/pi/twitter/bots/TwitBot.py", line 54, in startFiltering
self.myStream.filter(track=tList)
File "/usr/local/lib/python3.4/dist-packages/tweepy/streaming.py", line 445, in filter
self._start(async)
File "/usr/local/lib/python3.4/dist-packages/tweepy/streaming.py", line 361, in _start
self._run()
File "/usr/local/lib/python3.4/dist-packages/tweepy/streaming.py", line 294, in _run
raise exception
File "/usr/local/lib/python3.4/dist-packages/tweepy/streaming.py", line 263, in _run
self._read_loop(resp)
File "/usr/local/lib/python3.4/dist-packages/tweepy/streaming.py", line 313, in _read_loop
line = buf.read_line().strip()
AttributeError: 'NoneType' object has no attribute 'strip'
My setup:
I have a parent class TwitButter.py, that creates an object from the TwitBot.py. These objects are the bots, and they are started on their own thread so they can run independently.
I have a function in the TwitBot that runs the startFiltering() function. It is wrapped in a try/except, but my except code is never triggered.
My guess is that the error is occurring within the Streaming library. Maybe that library is poorly coded and breaks on the line that is specified at the bottom of the traceback.
Any help would be awesome, and I wonder if others have experienced this issue?
I can provide extra details if needed.
Thanks!!!

This actually is problem in tweepy that was fixed by github #870 in 2017-04. So, should be resolved by updating your local copy to latest master.
What I did to discover that:
Did a web search to find the tweepy source repo.
Looked at streaming.py for context on the last traceback lines.
Noticed the most recent change to the file was the same problem.
I'll also note that most of the time you get a traceback from deep inside a Python library, the problem comes from the code calling it incorrectly, rather than a bug in the library. But not always. :)

GAE Python Deferred.defer set 'task_retry_limit' not working

I want my deferred task to only try once more after failing.
After reading this related question: Specifying retry limit for tasks queued using GAE deferred library it was obvious that I needed to follow the accepted answer, so I modified my code to look like this:
from google.appengine.ext import deferred
deferred.defer(MyFunction, DATA, _retry_options={'task_retry_limit': 1})
Now I get this error:
File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/ext/deferred/deferred.py", line 269, in defer
return task.add(queue, transactional=transactional)
File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/api/taskqueue/taskqueue.py", line 1143, in add
return self.add_async(queue_name, transactional).get_result()
File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/api/taskqueue/taskqueue.py", line 1139, in add_async
return Queue(queue_name).add_async(self, transactional, rpc)
File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/api/taskqueue/taskqueue.py", line 1889, in add_async
rpc)
File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/api/taskqueue/taskqueue.py", line 2008, in __AddTasks
fill_request(task, request.add_add_request(), transactional)
File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/api/taskqueue/taskqueue.py", line 2093, in __FillAddPushTasksRequest
task.retry_options, task_request.mutable_retry_parameters())
File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/api/taskqueue/taskqueue.py", line 2033, in __FillTaskQueueRetryParameters
if retry_options.min_backoff_seconds is not None:
AttributeError: 'dict' object has no attribute 'min_backoff_seconds'
Obviously im making a silly mistake, I just can't figure out what it is.

You need to pass a TaskRetryOptions instance, not just a dict:
from google.appengine.ext import deferred
from google.appengine.api.taskqueue import TaskRetryOptions
options = TaskRetryOptions(task_retry_limit=1)
deferred.defer(MyFunction, DATA, _retry_options=options)

Profiling CherryPy

I've been trying to start profiling my CherryPy webserver, but the documentation is lacking in detail in how this should be set up. I understand that I should be able to use cherrypy.lib.profiler as middleware to mount my initial server. Right now, I have code like the following:
server_app = ServerClass()
cherrypy.tree.mount(server_app, '/', '/path/to/config/file.cfg')
cherrypy.engine.start()
cherrypy.engine.block()
I want to mount the profiling middleware, and it seems that something like the following is required:
from cherrypy.lib import profiler
server_app = ServerClass()
server_cpapp = cherrypy.Application(server_app, '/', '/path/to/config/file.cfg')
server_profile_cpapp = profiler.make_app(server_cpapp, '/home/ken/tmp/cprofile', True)
#cherrypy.tree.mount(server_profile_cpapp)
cherrypy.tree.graft(server_profile_cpapp)
cherrypy.engine.start()
cherrypy.engine.block()
For some reason cherrypy.tree.mount doesn't work, but if I use cherrypy.tree.graft all seems to operate fine (I can make requests to the server as normal)
However, the above code generates a cp_0001.prof file under /home/ken/tmp/cprofile and I am not sure how to interpret it. I have tried using pyprof2calltree to read the data into KCacheGrind, but I get a parsing error. Does what I'm doing seem correct, and if so how do I interpret the output file?

It turns out that the profile files generated by CherryPy can be interpreted using the profiler.py script shipped as part of CherryPy. Simply run profiler.py in the <site-packages>/cherrypy/lib directory as follows:
python profiler.py /directory/containing/prof/files 8080
Then navigate to localhost:8080 in your browser and the profiling results for all .prof files in the target directory will be displayed in a simple text interface.
I would still prefer to be able to export the results into a calltree to profile using KCacheGrind, but this seems to do for basic profiling.
This is documented in the change log for v2.1 of CherryPy when the profiler was introduced (although the other details on that page describing how to set up the profiler has since become deprecated)

I am also trying to get profiling up and running for a cherrypy instance. I used the same code you have in your initial question, which seems to work in that it generates a cp_0001.prof file in the folder.
To answer your question, I am able to open this file in runsnakerun to see the profiling output in a tree view.
The problem I have is that every request I do to the server now fails, with the following output in the log:
[29/May/2013:16:39:32] ENGINE AssertionError('Bad call', ('', 0, 'sleep'), <frame object at 0x08522400>, <frame object at 0x08522030>, <frame object at 0x08156748>, <frame object at 0x06D06F10>)
Traceback (most recent call last):
File "<path>\packages\cherrypy\wsgiserver\wsgiserver2.py", line 1302, in communicate
req.respond()
File "<path>\packages\cherrypy\wsgiserver\wsgiserver2.py", line 831, in respond
self.server.gateway(self).respond()
File "<path>\packages\cherrypy\wsgiserver\wsgiserver2.py", line 2115, in respond
response = self.req.server.wsgi_app(self.env, self.start_response)
File "<path>\packages\cherrypy\_cptree.py", line 290, in __call__
return app(environ, start_response)
File "<path>\packages\cherrypy\lib\profiler.py", line 188, in __call__
return self.profiler.run(gather)
File "<path>\packages\cherrypy\lib\profiler.py", line 147, in run
result = self.profiler.runcall(func, *args)
File "<path>\python\lib\profile.py", line 472, in runcall
return func(*args, **kw)
File "<path>\packages\cherrypy\lib\profiler.py", line 183, in gather
def gather():
File "<path>\python\lib\profile.py", line 246, in trace_dispatch_i
if self.dispatch[event](self, frame, t):
File "<path>\python\lib\profile.py", line 301, in trace_dispatch_call
frame, frame.f_back)
AssertionError: ('Bad call', ('', 0, 'sleep'), <frame object at 0x08522400>, <frame object at 0x08522030>, <frame object at 0x08156748>, <frame object at 0x06D06F10>)
I am using python 2.6.6 and cherrypy 3.2.2
Any suggestions?

Python 3.x and TestLink xmlprc

Appreciate your helping first, I am new for the python 3.x.
When I try to use Python 3.x to parse the testlink xmlprc server. I got below error, but I can run the code under Python 2.x, any idea?
import xmlrpc.client
server = xmlrpc.client.Server("http://172.16.29.132/SITM/lib/api/xmlrpc.php") //here is my testlink server
print (server.system.listMethods()) //I can print the methods list here
print (server.tl.ping()) // Got error.
Here is the error:
['system.multicall', 'system.listMethods', 'system.getCapabilities', 'tl.repeat', 'tl.sayHello', 'tl.ping', 'tl.setTestMode', 'tl.about', 'tl.checkDevKey', 'tl.doesUserExist', 'tl.deleteExecution', 'tl.getTestSuiteByID', 'tl.getFullPath', 'tl.getTestCase', 'tl.getTestCaseAttachments', 'tl.getFirstLevelTestSuitesForTestProject', 'tl.getTestCaseCustomFieldDesignValue', 'tl.getTestCaseIDByName', 'tl.getTestCasesForTestPlan', 'tl.getTestCasesForTestSuite', 'tl.getTestSuitesForTestSuite', 'tl.getTestSuitesForTestPlan', 'tl.getLastExecutionResult', 'tl.getLatestBuildForTestPlan', 'tl.getBuildsForTestPlan', 'tl.getTotalsForTestPlan', 'tl.getTestPlanPlatforms', 'tl.getProjectTestPlans', 'tl.getTestPlanByName', 'tl.getTestProjectByName', 'tl.getProjects', 'tl.addTestCaseToTestPlan', 'tl.assignRequirements', 'tl.uploadAttachment', 'tl.uploadTestCaseAttachment', 'tl.uploadTestSuiteAttachment', 'tl.uploadTestProjectAttachment', 'tl.uploadRequirementAttachment', 'tl.uploadRequirementSpecificationAttachment', 'tl.uploadExecutionAttachment', 'tl.createTestSuite', 'tl.createTestProject', 'tl.createTestPlan', 'tl.createTestCase', 'tl.createBuild', 'tl.setTestCaseExecutionResult', 'tl.reportTCResult']
Traceback (most recent call last):
File "F:\SQA\Python\Testlink\Test.py", line 5, in <module>
print (server.tl.ping())
File "C:\Python31\lib\xmlrpc\client.py", line 1029, in __call__
return self.__send(self.__name, args)
File "C:\Python31\lib\xmlrpc\client.py", line 1271, in __request
verbose=self.__verbose
File "C:\Python31\lib\xmlrpc\client.py", line 1070, in request
return self.parse_response(resp)
File "C:\Python31\lib\xmlrpc\client.py", line 1164, in parse_response
p.feed(response)
File "C:\Python31\lib\xmlrpc\client.py", line 454, in feed
self._parser.Parse(data, 0)
xml.parsers.expat.ExpatError: junk after document element: line 2, column 0

When I've seen this message before, it happened because the contents of the transported data wasn't escaped for XML transport. The solution was to wrap the data in an XMLRPC Binary object.
In your case, you don't control the server side, so the above isn't a solution for you but it may suggest what the actual problem is.
Also, the Python 2 versus Python 3 difference suggests that there is a text/bytes issue at work.
To help diagnose the issue, set verbose=True so you can see the actual HTTP request/response headers and the XML request/response. That may show you what is at line 2: column 0. You may find that the issue may be with the PHP script not wrapping up binary data in base64 encoding as required by the XMLRPC spec.

Thank you , I find out all the methods list, only 'tl.sayHello', 'tl.ping','tl.about' has this problem, and all of them are pass a string with a PHP automatic loader empty file *.class.php to the parser, other methods are pass a xml file. So I give up to use those methods and the script works fine.

Python 2.7.1 Blogger API problem

I recently started writing a simple client using the Blogger API to do some basic posting I implemented the client in Python and used the example code verbatim from the Blogger Developer's Guide to login, get the blog id, and make a new post. I ran the script and everything went fine until I got to this line:
return blogger_service.Post(entry, '/feeds/%s/posts/default' % blog_id)
I got the error message:
Traceback (most recent call last):
File "cs1121post.py", line 38, in <module>
cs1121post()
File "cs1121post.py", line 33, in cs1121post
return blogger_service.Post(entry, '/feeds/%s/posts/default' % blog_id)
File "/usr/local/lib/python2.7/dist-packages/gdata/service.py", line 1236, in Post
media_source=media_source, converter=converter)
File "/usr/local/lib/python2.7/dist-packages/gdata/service.py", line 1322, in PostOrPut
headers=extra_headers, url_params=url_params)
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 93, in optional_warn_function
return f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/atom/service.py", line 176, in request
content_length = CalculateDataLength(data)
File "/usr/local/lib/python2.7/dist-packages/atom/service.py", line 736, in CalculateDataLength
return len(str(data))
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 377, in __str__
return self.ToString()
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 374, in ToString
return ElementTree.tostring(self._ToElementTree(), encoding=string_encoding)
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 369, in _ToElementTree
self._AddMembersToElementTree(new_tree)
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 331, in _AddMembersToElementTree
member._BecomeChildElement(tree)
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 357, in _BecomeChildElement
self._AddMembersToElementTree(new_child)
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 342, in _AddMembersToElementTree
ExtensionContainer._AddMembersToElementTree(self, tree)
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 224, in _AddMembersToElementTree
tree.text = self.text.decode(MEMBER_STRING_ENCODING)
AttributeError: 'list' object has no attribute 'decode'
By which I'm taking it that ElementTree is at fault here. I installed ElementTree via
sudo python setup.py install
in case it matters. Is there some known incompatibility between ElementTree and Python v2.7.1? Has this happened to anybody else and how did you get it working? If you need any additional information, please reply to the thread. All the source code that is relevant is basically just the example code from the Developers Guide mentioned above. I haven't modified that at all (not even the variable names). Any input is greatly appreciated.

The stacktrace is actually pretty clear about this: You're calling decode() on a list instead of a tree element. Try getting the first element from the list and calling decode() on that:
firsttext = self.text[0].decode(MEMBER_STRING_ENCODING)