I have already Twitter Standard API (I got the approved recently and did not use Twitter API yet ) because I need to collect historical tweets.
So I have to upgrade to Premium API but should I choose API sandbox to test my code before paid and upgrade the premium API full archive? I am afraid to lose some tweets and reduce the requests.
I am a little confusing for understanding some operators
results_per_call=100 .. max_results=100 .. what are they meaning?
Can I choose any numbers to get more tweets?
How many requests can I use per day?
I find code in python that I will use it to collect ? is it correct? I am a beginner in python
where can I find the JSON file on my computer.? and how convert this file to .cvs?
!pip install searchtweets
!pip install yaml
import yaml
config = dict(
search_tweets_api = dict(
account_type = 'premium',
endpoint = 'https://api.twitter.com/1.1/tweets/search/fullarchive/YOUR_LABEL.json',
consumer_key = 'YOUR_CONSUMER_KEY',
consumer_secret = 'YOUR_CONSUMER_SECRET'
))
with open('twitter_keys_fullarchive.yaml', 'w') as config_file:
yaml.dump(config, config_file, default_flow_style=False)
from searchtweets import load_credentials
premium_search_args = load_credentials("twitter_keys_fullarchive.yaml",
yaml_key="search_tweets_api",
env_overwrite=False)
print(premium_search_args)
from searchtweets import gen_rule_payload
query = "(#COVID19 OR # Corona_virus) (pandemic OR corona OR infected OR vaccine)" rule = gen_rule_payload(query, results_per_call=100, from_date="2020-01-01", to_date="2020-01-30")` from searchtweets import ResultStream
rs = ResultStream(rule_payload=rule,
max_results=100,
**premium_search_args) print(rs)
mport json
with open('twitter_premium_api_demo.jsonl', 'a', encoding='utf-8') as f:
n = 0
for tweet in rs.stream():
n += 1
if n % 10 == 0:
print('{0}: {1}'.format(str(n), tweet['created_at']))
json.dump(tweet, f)
f.write('\n') print('done')
Very thank you in advance.
Once I had the same task that collect twitter data using different conditions,After lot of searching and tests,I had to create completely separate python twitter client API for my task.This is what I know regarding the API (documentation is little bit confusing)
Twitter API has 3 versions for search and download data.
Standard(free version with limitations)
Premium (paid version with some extended features)
Enterprise ( paid version with customize options for large scale operations)
Standard API
Free to use with correct authentication
Only return past 7 days data
Can use Standard search operators
You can send limited number of requests within given time period(ex 180 requests in 15min window for user auth and 450 requests in 15 min window for app auth)
one request return 100 data objects (100 tweets)
Premium API
Preimum APi includes 2 versions.
30-day Endpoint - Provide tweets posted within last 30 days
Full Archive endpoints - Provides tweets from starting from 2006
these 2 versions share the same endpoints and only difference is timeframe you can search.
Premium package returns maximum 500 data objects per request,Still you can limit the return count according to your use case.
Select requests per month by subscription (example 50 requests,250 requests (per month))
Answering your questions:
results_per_call=100 means how many tweet objects return by the API by default and max_results=100 is how many objects you need.
should I choose API sandbox to test my code before paid and upgrade the premium API full archive?
yes you can test basic logic and some search queries and check return object using free service.But if you need to search date difference more than 7days, or premium operators you have to use premium API.
these are some useful links
https://developer.twitter.com/en/docs/tweets/search/overview
operators
https://developer.twitter.com/en/docs/tweets/search/guides/standard-operators
https://developer.twitter.com/en/docs/tweets/search/guides/premium-operators
API
https://developer.twitter.com/en/docs/tweets/search/api-reference/premium-search
https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets
There are more hidden information in documentation please add more if you find anything useful.
Related
Im looking for ways to retrieve tweets from Twitter which contains certain hashtags.
I tried to use the official API and tweepy package in Python but even with academic access I was only able to retrieve tweets which are 7 days old. I want to retrieve tweets from 2019 till 2020 but Im not able to do so with tweepy.
I tried the following packages GetOldTweet3, twint but none of them seem to work due to some changes Twitter made last year.
Can someone suggest a way to get old tweets with certain hashtags. Thanks in advance for any help or suggestion provided.
If you have academic access, you are able to use the full archive search API available in the Twitter API v2. Tweepy has support for this via the tweepy.Client class. There's a full tutorial on DEV, but the code will be something like this:
import tweepy
client = tweepy.Client(bearer_token='REPLACE_ME')
# Replace with your own search query
query = 'from:andypiper -is:retweet'
tweets = client.search_all_tweets(query=query, tweet_fields=['context_annotations', 'created_at'], max_results=100)
for tweet in tweets.data:
print(tweet.text)
if len(tweet.context_annotations) > 0:
print(tweet.context_annotations)
You can use search query parameters to specify the date range.
I am trying to access historical weather data from an API. I obtained the API key from here: https://www.visualcrossing.com/weather/weather-data-services#/timeline
I am trying this but I keep getting an Error 404. I am not sure if this is because of a problem with the API or my code.
import requests
r = requests.get("https://weather.visualcrossing.com/VisualCrossingWebServices/rest/services/timeline/London,UK/2021-01-01/2021-03-28?key=AXSSS")
print(r)
Documentation: https://www.visualcrossing.com/resources/documentation/weather-api/timeline-weather-api/
How can I obtain the data?
i tested the site you gave and created a account and api key to get London whether data, you can use it too
Code :
import requests
r = requests.get("https://weather.visualcrossing.com/VisualCrossingWebServices/rest/services/timeline/London?unitGroup=metric&key=PPKBBJ7637X5SNDUG6HZA23X7")
print(r)
Output :
<Response [200]>
now you can access data by json() method too:
print(r.json())
the output is so Huge, but your problem is 2 things:
1-API key is not correct (i tested)
2-You should buy premium plan
for get a range of dates url will be like this:
https://weather.visualcrossing.com/VisualCrossingWebServices/rest/services/timeline/London/2021-1-1/2021-1-5?unitGroup=us&key=PPKBBJ7637X5SNDUG6HZA23X7
and the date range you give , has too much row per request, you should buy premium plan
Otherwise you will get this error in their own website:
Your plan allows up to 100 rows per request. This query will return (yyy) rows. Please smaller date range or fewer locations.
Our paid plans offer increased query limits
If you don`t want to pay for paid plan you can use this link to use github public apis, a lot of free and without api key you can use for it
Github public apis
I am looking for a method to get information of a "trend" regarding some hashtag/key word on Twitter. Let`s say I want to measure how often the hashtag/key word "Python" is tweeted in time. For instance, today, "Python" is tweeted on average every 1 minute but yesterday it was tweeted on average every 2 minutes.
I have tried various options but I am always bouncing off the twitter API limitations, i.e. if I try to download all tweets for a hashtag during the last (for example) day, only a certain franction of the tweets is downloaded (via tweepy.cursor).
Do you have any ideas / script examples of achieving similar results? Libraries or guides to recommend? I did not find any help searching on the internet. Thank you.
You should check twint repository.
Can fetch almost all Tweets (Twitter API limits to last 3200 Tweets only);
Fast initial setup;
Can be used anonymously and without Twitter sign up;
here is a sample code:
import twint
def scrapeData(search):
c = twint.Config()
c.Search = search
c.Since = '2021-03-05 00:00:00'
c.Until = '2021-03-06 00:00:00'
c.Pandas = True
c.Store_csv = True
c.Hide_output = True
c.Output = f'{search}.csv'
c.Limit = 10 # number of tweets want to fetch
print(f"\n#### Scraping from {c.Since} to {c.Until}")
twint.run.Search(c)
print("\n#### Preview: ")
print(twint.storage.panda.Tweets_df.head())
if __name__ == "__main__":
scrapeData(search="python")
Try a library called:
GetOldTweets or GetOldTweets3
Twitter Search, and by extension its API, are not meant to be an exhaustive source of tweets. The Twitter Streaming API places a limit of just one week on how far back tweets can be extracted from that match the input parameters. So in order to extract all historical tweets relevant to a set of search parameters for analysis, the Twitter Official API needs to be bypassed and custom libraries that mimic the Twitter Search Engine need to be used.
I am using the Twitter's premium API to download tweets restricted to a particular country and language. In order to do this, I am using search tweets, which access the mentioned premium API. However, when I collect the tweets I received an empty list.
I am starting by restricting the search to a given language ('es'), as shown below, and a string. However, this query produces zero matches. When I tried the same but removing the language information from the rule, gen_rule_payload('cr7', from_date='2017-06-03', to_date='2017-06-04', results_per_call=100), it provides the expected results.
So, my question is how can I query for both language and place_country simultaneously in either searchtweets or any other python wrapper of Twitter's premium API?
Any help is greatly appreciated!
Daniel
from searchtweets import gen_rule_payload, load_credentials,
collect_results
premium_creds = load_credentials(filename="filename.yaml",
yaml_key="yaml_key",
env_overwrite=False)
rule = gen_rule_payload('cr7 lang=es',
from_date="2017-06-03",
to_date="2017-06-04",
results_per_call=100)
tweets = collect_results(rule, max_results=100,
result_stream_args=premium_creds)
[print(tweet.all_text) for tweet in tweets]
[]
ok so im really new to python and I am trying to create to assist me in marketing my music via social media. I am trying to code it so that when I compare a users followers with my followers if I am not following one of their followers, it automatically follows them. here is what I have
import twitter
import time
now = time.time
username = raw_input("whos followers")
api = twitter.Api(...)
friendslist = api.GetFollowersPaged(screen_name=username, count=1,)
myfollowers = api.GetFollowersPaged(user_id=821151801785405441, count=1)
for u in friendslist:
if u not in myfollowers:
api.CreateFriendship(u.friendslist)
print 'you followed new people'
time.sleep(15)
I am using python 2.7 and the python-twitter api wrapper my error seems to start at the api.CreateFriendship line. also I set the count to 1 to try to avoid rate limiting but hae had them as high as 150, 200 being the max
The Twitter API has fairly subjective controls in place for Write operations. There are daily follow limits and they designed to limit exactly the sort of thing you are doing.
see https://support.twitter.com/articles/15364 and https://support.twitter.com/articles/15364
If you do reach a limit, we'll let you know with an error message
telling you which limit you've hit. For limits that are time-based
(like the direct messages, Tweets, changes to account email, and API
request limits), you'll be able to try again after the time limit has
elapsed.