I am using the following code to stream direct messages received by my Twitter account -:
from tweepy import Stream
from tweepy import OAuthHandler
from tweepy import API
from tweepy.streaming import StreamListener
# These values are appropriately filled in the code
consumer_key = ""
consumer_secret = ""
access_token = ""
access_token_secret = ""
class StdOutListener( StreamListener ):
def __init__( self ):
self.tweetCount = 0
def on_connect( self ):
print("Connection established!!")
def on_disconnect( self, notice ):
print("Connection lost!! : ", notice)
def on_data( self, status ):
print("Entered on_data()")
print(status, flush = True)
return True
def on_direct_message( self, status ):
print("Entered on_direct_message()")
try:
print(status, flush = True)
return True
except BaseException as e:
print("Failed on_direct_message()", str(e))
def on_error( self, status ):
print(status)
def main():
try:
auth = OAuthHandler(consumer_key, consumer_secret)
auth.secure = True
auth.set_access_token(access_token, access_token_secret)
api = API(auth)
# If the authentication was successful, you should
# see the name of the account print out
print(api.me().name)
stream = Stream(auth, StdOutListener())
stream.userstream()
except BaseException as e:
print("Error in main()", e)
if __name__ == '__main__':
main()
I am able to get my name printed as well as the "Connection established!" message.
But whenever I send a direct message to my own profile from my friend's profile, no method is called.
Although, whenever I tweet something from my profile, it is printed correctly by the program.
Me and my friend both follow each other on Twitter, so there shouldn't be any problems with the direct message permissions.
What is the proper way to do it ?
Also, if it cannot be done in Tweepy at all, I am ready to use any other Python library.
I am using Tweepy 3.3.0 and Python 3.4 on Windows 7.
The code provided in the question is indeed correct.
The problem was that I had forgotten to regenerate the access token and secret after changing my application permissions to "Read, write and direct messages".
Note: The direct messages arrive in the on_data() method rather than the on_direct_message() method.
Another solution is to not override the on_data method. That is how I got your code example to work in my case.
The on_data method of tweepy.streaming.StreamListener object contains all the json data handling and calling of other methods, like on_direct_message, that you and I were expecting.
Here is a working version of the code in 2017
from twitter import *
import os
import oauth
#Create a new Twitter app first: https://apps.twitter.com/app/new
APP_KEY,APP_SECRET = 'H3kQtN5PQgRiA0ocRCCjqjt2P', '51UaJFdEally81B7ZXjGHkDoDKTYy430yd1Cb0St5Hb1BVcDfE'
# OAUTH_TOKEN, OAUTH_TOKEN_SECRET = '149655407-TyUPMYjQ8VyLNY5p7jq0aMy8PjtFtd7zkIpDh3ZA', 'IUVpiDpoVmdO75UaHOTinAv5TOsAQmddttNENh9ofYuWO'
MY_TWITTER_CREDS = os.path.expanduser('my_app_credentials')
if not os.path.exists(MY_TWITTER_CREDS):
oauth_dance("crypto sentiments", APP_KEY, APP_SECRET,
MY_TWITTER_CREDS)
oauth_token, oauth_secret = read_token_file(MY_TWITTER_CREDS)
auth = OAuth(
consumer_key=APP_KEY,
consumer_secret=APP_SECRET,
token=oauth_token,
token_secret=oauth_secret
)
twitter_userstream = TwitterStream(auth=auth, domain='userstream.twitter.com')
for msg in twitter_userstream.user():
if 'direct_message' in msg:
print (msg['direct_message']['text'])
Related
So a follow up on this question, where I still have the same code: to start working with some live data, I switched from sandbox to individual key, and started getting this error with code that did work in sandbox mode:
HTTPError: 401 Client Error: Unauthorized for url: https://apisb.etrade.com/v1/market/optionexpiredate?symbol=NFLX&expiryType=ALL
Here's the authentication code, in case I messed up here:
consumer_key = "[key, double checked]"
consumer_secret = "[secret, double checked]"
def authenticate():
#get the auth UIL
oauth = pyetrade.ETradeOAuth(consumer_key, consumer_secret)
oauth.setToken(consumer_key)
oauth.setTokenSecret(consumer_secret)
print(oauth.get_request_token())
#ask the user to put the verification code in to get tokens
verifier_code = input("Enter verification code: ")
global tokens
tokens = oauth.get_access_token(verifier_code)
print(tokens)
#Setting up the object used for Access Management
global authManager
authManager = pyetrade.authorization.ETradeAccessManager(
consumer_key,
consumer_secret,
tokens['oauth_token'],
tokens['oauth_token_secret']
)
And this is the code that returns the error:
try:
q = market.get_option_expire_date(thisSymbol,resp_format='xml')
expiration_dates = option_expire_dates_from_xml(q)
except:
raise Exception("Rip, get date not working")
I'm not sure why this is happening :( any help would be super appreciated!
PROBLEM SOLVED, SEE SOLUTION AT THE END OF THE POST
I need help to estimate running time for my tweepy program calling Twitter Stream API with location filter.
After I kicked it off, it has run for over 20 minutes, which is longer than what I expected. I am new to Twitter Stream API, and have only worked with REST API for couple of days. It looks to me that REST API will give me 50 tweets in a few seconds, easy. But this Stream request is taking a lot more time. My program hasn't died on me or given any error. So I don't know if there's anything wrong with it. If so, please do point out.
In conclusion, if you think my code is correct, could you provide an estimate for the running time? If you think my code is wrong, could you help me to fix it?
Thank you in advance!
Here's the code:
# Import Tweepy, sys, sleep, credentials.py
import tweepy, sys
from time import sleep
from credentials import *
# Access and authorize our Twitter credentials from credentials.py
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
box = [-86.33,41.63,-86.20,41.74]
class CustomStreamListener(tweepy.StreamListener):
def on_error(self, status_code):
print >> sys.stderr, 'Encountered error with status code:', status_code
return True # Don't kill the stream
def on_timeout(self):
print >> sys.stderr, 'Timeout...'
return True # Don't kill the stream
stream = tweepy.streaming.Stream(auth, CustomStreamListener()).filter(locations=box).items(50)
stream
I tried the method from http://docs.tweepy.org/en/v3.4.0/auth_tutorial.html#auth-tutorial Apparently it is not working for me... Here is my code below. Would you mind giving any input? Let me know if you have some working code. Thanks!
# Import Tweepy, sys, sleep, credentials.py
import tweepy, sys
from time import sleep
from credentials import *
# Access and authorize our Twitter credentials from credentials.py
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
# Assign coordinates to the variable
box = [-74.0,40.73,-73.0,41.73]
import tweepy
#override tweepy.StreamListener to add logic to on_status
class MyStreamListener(tweepy.StreamListener):
def on_status(self, status):
print(status.text)
def on_error(self, status_code):
if status_code == 420:
#returning False in on_data disconnects the stream
return False
myStreamListener = MyStreamListener()
myStream = tweepy.Stream(auth = api.auth, listener=myStreamListener())
myStream.filter(track=['python'], locations=(box), async=True)
Here is the error message:
Traceback (most recent call last):
File "test.py", line 26, in <module>
myStream = tweepy.Stream(auth = api.auth, listener=myStreamListener())
TypeError: 'MyStreamListener' object is not callable
PROBLEM SOLVED! SEE SOLUTION BELOW
After another round of debug, here is the solution for one who may have interest in the same topic:
# Import Tweepy, sys, sleep, credentials.py
try:
import json
except ImportError:
import simplejson as json
import tweepy, sys
from time import sleep
from credentials import *
# Access and authorize our Twitter credentials from credentials.py
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
# Assign coordinates to the variable
box = [-74.0,40.73,-73.0,41.73]
import tweepy
#override tweepy.StreamListener to add logic to on_status
class MyStreamListener(tweepy.StreamListener):
def on_status(self, status):
print(status.text.encode('utf-8'))
def on_error(self, status_code):
if status_code == 420:
#returning False in on_data disconnects the stream
return False
myStreamListener = MyStreamListener()
myStream = tweepy.Stream(api.auth, listener=myStreamListener)
myStream.filter(track=['NYC'], locations=(box), async=True)
Core Problem:
I think you're misunderstanding what the Stream is here.
Tl;dr: Your code is working, you're just not doing anything with the data that gets back.
The rest API call is a single call for information. You make a request, Twitter sends back some information, which gets assigned to your variable.
The StreamObject (which you've created as stream) from Tweepy opens a connection to twitter with your search parameters, and Twitter, well, streams Tweets to it. Forever.
From the Tweepy docs:
The streaming api is quite different from the REST api because the
REST api is used to pull data from twitter but the streaming api
pushes messages to a persistent session. This allows the streaming api
to download more data in real time than could be done using the REST
API.
So, you need to build a handler (streamListener, in tweepy's terminology), like this one that prints out the tweets..
Additional
Word of warning, from bitter experience - if you're going to try and save the tweets to a database: Twitter can, and will, stream objects to you much faster than you can save them to the database. This will result in your Stream being disconnected, because the tweets back up at Twitter, and over a certain level of backed-up-ness (not an actual phrase), they'll just disconnect you.
I handled this by using django-rq to put save jobs into a jobqueue - this way, I could handle hundreds of tweets a second (at peak), and it would smooth out. You can see how I did this below. Python-rq would also work if you're not using django as a framework round it. The read both method is just a function that reads from the tweet and saves it to a postgres database. In my specific case, I did that via the Django ORM, using the django_rq.enqueue function.
__author__ = 'iamwithnail'
from django.core.management.base import BaseCommand, CommandError
from django.db.utils import DataError
from harvester.tools import read_both
import django_rq
class Command(BaseCommand):
args = '<search_string search_string>'
help = "Opens a listener to the Twitter stream, and tracks the given string or list" \
"of strings, saving them down to the DB as they are received."
def handle(self, *args, **options):
try:
import urllib3.contrib.pyopenssl
urllib3.contrib.pyopenssl.inject_into_urllib3()
except ImportError:
pass
consumer_key = '***'
consumer_secret = '****'
access_token='****'
access_token_secret_var='****'
import tweepy
import json
# This is the listener, responsible for receiving data
class StdOutListener(tweepy.StreamListener):
def on_data(self, data):
decoded = json.loads(data)
try:
if decoded['lang'] == 'en':
django_rq.enqueue(read_both, decoded)
else:
pass
except KeyError,e:
print "Error on Key", e
except DataError, e:
print "DataError", e
return True
def on_error(self, status):
print status
l = StdOutListener()
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret_var)
stream = tweepy.Stream(auth, l)
stream.filter(track=args)
Edit: Your subsequent problem is caused by calling the listener wrongly.
myStreamListener = MyStreamListener() #creates an instance of your class
Where you have this:
myStream = tweepy.Stream(auth = api.auth, listener=myStreamListener())
You're trying to call the listener as a function when you use the (). So it should be:
myStream = tweepy.Stream(auth = api.auth, listener=myStreamListener)
And in fact, can probably just be more succinctly written as:
myStream = tweepy.Stream(api.auth,myStreamListener)
I'm trying to use the tweepy library in one of my python projects. When I try the following code that creates a tweepy cursor to fetch a user's timeline status messages, the count parameter is always ignored.
def search(self, username, keyword, consumer_key, consumer_secret, access_token, access_token_secret):
#start twitter auth
try:
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
user = api.get_user(username)
except Exception as e:
print(str(e))
self.error = str(e)
return
self.followercount = user.followers_count
self.screenname = user.screen_name
results = []
for status in tweepy.Cursor(api.user_timeline, id=username, count=2).items():
try:
tweet = status._json
In this instance, the count is set to 2 in the Cursor object, yet it receives all of them. What am I doing wrong?
tweepy.Cursor() does not appear to recognize a count argument. In fact, count is not mentioned anywhere in tweepy/cursor.py, the module where tweepy.Cursor is defined. Instead, it looks like you might want to use:
for status in tweepy.Cursor(api.user_timeline, id=username).items(2):
passing the limit to items() instead of as the count keyword argument. See this section in the tweepy Cursor tutorial.
I want to do multiproccesing on twitter search API.
I have got the code below and it do 1 call every time instead of multiproccesing.
from multiprocessing import Process
from twitter import *
config = {}
exec(compile(open("config.py", "rb").read(), "config.py", 'exec'), config)
twitter = Twitter(
auth=OAuth(config["access_key"], config["access_secret"], config["consumer_key"], config["consumer_secret"]))
def twitterSearch(word):
tweetsWithWord = twitter.search.tweets(q=word, count=100)
print(tweetsWithWord)
if __name__ == '__main__':
for i in range(8):
p = Process(target=twitterSearch, args=('racist',))
p.start()
p.join()
Please kindly help me to fix this.
If I understand correctly, what you want is to have a continuous stream of results for your search term. I don't know the package you're working with, but i do know that both twython and tweepy can use twitter's streaming API.
In any case, you'll need to process each and every tweet that comes up your stream when it arrives, and you can use processes/threads at that stage if needed.
Code example for streaming:
from threading import Thread
from queue import Queue
from twython import TwythonStreamer
from requests.exceptions import ChunkedEncodingError
CONSUMER_KEY = 'AAA'
CONSUMER_SECRET = 'BBB'
ACCESS_KEY = 'CCC'
ACCESS_SECRET = 'DDD'
class TwitterStream(TwythonStreamer):
def __init__(self, consumer_key, consumer_secret, token, token_secret, tqueue):
self.tweet_queue = tqueue
super(TwitterStream, self).__init__(consumer_key, consumer_secret, token, token_secret)
def on_success(self, data):
if 'text' in data:
self.tweet_queue.put(data)
def on_error(self, status_code, data):
#print(status_code)
#with open(logfile,'a') as f:
# f.write(time.asctime(time.gmtime()) + ' ' + status_code + '\n')
# Want to stop trying to get data because of the error?
# Uncomment the next line!
# self.disconnect()
pass
def stream_tweets(tweets_queue,track):
# Input your credentials below
consumer_key = CONSUMER_KEY
consumer_secret = CONSUMER_SECRET
token = ACCESS_KEY
token_secret = ACCESS_SECRET
try:
stream = TwitterStream(consumer_key, consumer_secret, token, token_secret, tweets_queue)
stream.statuses.filter(track=track)
except ChunkedEncodingError:
# Sometimes the API sends back one byte less than expected which results in an exception in the
# current version of the requests library
stream_tweets(tweet_queue)
def process_tweets(tweets_queue, reply_dict, api, logfile):
while True:
twt = tweets_queue.get()
# Do something with the tweet
# You can start a new thread for actually proccessing each tweet
tweets_queue.task_done()
tweet_queue = Queue()
track = 'whatever you want to filter by' # Search terms go here
Thread(target=stream_tweets,
args=(tweet_queue, track,),
daemon=True).start()
process_tweets(tweet_queue, reply_dict, api, logfile)
I have the following simple snippet which worked fine before, but now isnt:
import sys
import tweepy
# Consumer keys and access tokens, used for OAuth
consumer_key=""
consumer_secret=""
access_key = ""
access_secret = ""
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_key, access_secret)
api = tweepy.API(auth)
class CustomStreamListener(tweepy.StreamListener):
def on_status(self, status):
print >>status.text
def on_error(self, status_code):
print >>sys.stderr, 'Encountered error with status code:', status_code
return True # Don't kill the stream
def on_timeout(self):
print >>sys.stderr, 'Timeout...'
return True # Don't kill the stream
sapi = tweepy.streaming.Stream(auth, CustomStreamListener())
sapi.filter(locations=[-180,-90,180,90])
This code should print tweets from all over the world. However, I get the error:
sapi = tweepy.streaming.Stream(auth, CustomStreamListener())
TypeError: __init__() missing 1 required positional argument: 'listener'
Please help and thank you!
EDIT:
So I changed
sapi = tweepy.streaming.Stream(auth, CustomStreamListener())
to:
sapi = tweepy.streaming.Stream(auth, listener=CustomStreamListener())
and now I get the following (similar) error:
sapi = tweepy.streaming.Stream(auth, listener=CustomStreamListener())
TypeError: __init__() missing 1 required positional argument: 'password'
maybe this will give someone insight into what is going on here, because I stll have no idea.
Thanks
I think the problem is that you are using an outdated version of Tweepy. Your code is correct, but older versions of Tweepy had a different constructor for the Stream class (it required passing a username and password instead of an OAuthHandler instance).
Version 2.2 is the latest (it's on PyPI). Your code should work correctly with it.