I am working on a project using social media data. An important part of it is to understand the spatial distribution of the social data, like tweets. I used the getoldtweet3 to scrape tweets but all of them are devioded of geo data. Just wonder if there a way to get the spatial features of tweets; or there are other tools to scrape other social media data with geospatial feature.
First of all, the library you are using is not accessing the Twitter API, you probably know this already. It’s not official and it could stop working at any moment.
Secondly and more importantly for your question, there is only about a 1% use of geo pinning in the API. There’s no special way to go around this, it is up to a user to enable geo on their Tweets.
Does this answer your question?
Related
I am working on a project for which I want to extract the timelines of around 500 different twitter users (I am using this for historical analysis, so I'll only need to retrieve them all once- no need to update with incoming tweets).
While I know the Twitter API only allows the last 3,200 tweets to be retrieved, when I use the basic UserTimeline method of the R twitteR package, I only seem to fetch about 20 every time I try (for users with significantly more, recent, tweets). Is this because of rate limiting, or because I am doing something wrong?
Does anyone have tips for doing this most efficiently? I realize it might take a lot of time because of rate limiting, is there a way of automating/iterating this process in R?
I am quite stuck, so thank you very much for any help/tips you may have!
(I have some experience using the Twitter API/twitteR package to extract tweets using a certain hashtag over a couple of days. I have basic Python skills, if it turns out to be easier/quicker to do in Python).
It looks like the twitteR documentation suggests using the maxID argument for pagination. So when you get the first batch of results, you could use the minimum ID in that set minus one as the maxID for the next request, until you get no more results back (meaning you've gotten to the beginning of a user's timeline).
I've been looking into scraping, and I cant manage to scrape twitter searches that date long way back by using python code, i can do it however with an addon for chrome but it falls short since it will only let me obtain a very limited amount of tweets. can anybody point me in the right direction¿
I found a related answer here:
https://stackoverflow.com/a/6082633/1449045
you can get up to 1500 tweets for a search; 3200 for a particular user's timeline
source: https://dev.twitter.com/docs/things-every-developer-should-know
see "There are pagination limits"
Here you can find a list of libraries to simplify the use of the APIs,
in different languages, including python
https://dev.twitter.com/docs/twitter-libraries
You can use snscrape. It doesn't require a Twitter Developer Account it can go back many years.
I am working on a website for which it would be useful to know the number of links shared by a particular facebook page (e.g., http://www.facebook.com/cocacola) so that the user can know whether they are 'liking' a firehose of information or a dribble of goodness. What is the best way to get the number of links/status updates that are shared by a particular page?
+1 for implementations that use python (this is a django website) but any solutions are welcome! I tried using fbconsole to accomplish this but I have come up a little short.
For what it is worth, this unanswered question seems relevant. As does the fact that, as of 2012.04.18, you can export your data to csv from the insights management page on the facebook site. The information is in there I just don't know how to get it out...
Thanks for your help!
In the event that anyone else finds this useful, I thought I'd post my gist example here. fbconsole makes it fairly simple to extract data through the Facebook Graph API.
The caveat is that it was not terribly easy to programmatically extract data through fbconsole so I wrote the fbconsole.automatically_authenticate to make it much easier to access this information in a systematic way. This addition has not yet been incorporated into the master branch of fbconsole (it was just posted this morning), but it is available here in the meantime for those that are interested.
I am sorry for asking but I am new in writing crawler.
I would like to crawl Twitter space for Twitter users and follow relationship among them using python.
Any recommendation for starting points such as tutorials?
Thank you very much in advance.
I'm a big fan of Tweepy myself - https://github.com/tweepy/tweepy
You'll have to refer to the Twitter docs for the API methods that you're going to need. As far as I know, Tweepy wraps all of them, but I recommend looking at Twitter's own docs to find out which ones you need.
To construct a following/follower graph, you're going to need some of these:
GET followers/ids - grab followers (in IDs) for a user
GET friends/ids - grab followings (in IDs) for a user
GET users/lookup - grab up to 100 users, specified by IDs
besides reading the twitter api?
a good starting point would be the great python twitter library by mike verdona which personally I think is the the best one. (also an intorduction here)
also see this question in stackoverflow
I am building a project in python that needs to scrape huge and huge amounts of Twitter data. Something like 1 million users and all their tweets need to be scraped.
Previously I have used Tweepy and Twython, but hit the limit of Twitter very fast.
How do sentiment analysis companies etc. get their data? How do they get all those tweets? Do you buy this somewhere or build something that iterates through different proxies or something?
How do companies like Infochimps with for example Trst rank get all their data?
* http://www.infochimps.com/datasets/twitter-census-trst-rank
If you want the latest tweets from specific users, Twitter offers the Streaming API.
The Streaming API is the real-time sample of the Twitter Firehose. This API is for those developers with data intensive needs. If you're looking to build a data mining product or are interested in analytics research, the Streaming API is most suited for such things.
If you're trying to access old information, the REST API with its severe request limits is the only way to go.
I don't know if this will work for what you're trying to do, but the Tweets2011 dataset was recently released.
From the description:
As part of the TREC 2011 microblog track, Twitter provided identifiers
for approximately 16 million tweets sampled between January 23rd and
February 8th, 2011. The corpus is designed to be a reusable,
representative sample of the twittersphere - i.e. both important and
spam tweets are included.