I'm just being a bit of an idiot here, I think, but I've figured out how to fetch my timeline, but not how to modify that into performing a search. I've currently got:
consumer = oauth.Consumer(key=CONSUMER_KEY, secret=CONSUMER_SECRET)
access_token = oauth.Token(key=ACCESS_KEY, secret=ACCESS_SECRET)
client = oauth.Client(consumer, access_token)
response, data = client.request(searchURL)
I'm guessing it's the last line that'll change to work with the search, but I'm not sure how to format it, if I change the searchURL to the one used for actually searching (it's currently on timeline) it says it's in the wrong format.
Can anyone help?
Thanks.
Turns out it's off the form:
searchURL = https://api.twitter.com/1.1/search/tweets.json?q=obama&count=2&tresult_type=popular
That's an example search using the keyword "obama", setting the count to 2, and filtering for popular results.
response, data = client.request(searchURL)
tweets = json.loads(data)
The format of the returned tweets is a bit...awkward, but understandable with a bit of playing around.
Related
These days I encounter a really weird problem and cannot solve it.Please help!
I want to use google map api to get the longitude and latitude of an address.Here is my definition of the function to request api:
def get_coordinates(df):
if pd.notnull(df['geocode']):
address=df['geocode']
response = requests.get("https://maps.googleapis.com/maps/api/geocode/json?address="
+ address+"&key=my-key")
json_response = response.json()
if len(json_response['results'])==0:
return 'None'
else:
coordinates=json_response['results'][0]['geometry']['location']
latitude=coordinates['lat']
longitude=coordinates['lng']
l_l=[latitude, longitude]
return l_l
else:
return 'None'
I store the address in a dataframe.
Then I can use df.apply to request api for each address:
test2['coor1'] = test2.apply(get_coordinates,axis = 1)
But then the really weird thing is: I know that most of these address should be in Brazil, but when I use scatter to plot them, I notice many of the point are out of Brazil. So I guess there is something wrong with api request.Then I run apply again and this time, some address get a totally different location from the last time. I have no idea why. I wonder if someone had the same issue before.Thanks a lot!
I am so confused by all these levels of dicts I have to wade through it would be easier imo just to do it by scraping, however I guess it's a good excercise to learn dicts and will be quicker perhaps once I figure it out.
My code is as follows, where the assignment statement for cposts returns a 404:
import pytumblr
# Authenticate via OAuth
client = pytumblr.TumblrRestClient(
'xxxxxxxxxxxxxxxxxxxxxxxxxxxx'
)
f = client.followers('blog.tumblr.com')
users = f['users']
names = [b['name'] for b in f['users']]
print names
cposts = client.posts(names[0], 'notes_info=True')
print (cposts)
But the python api info says: client.posts('codingjester', **params) # get posts for a blog
and this SO post (Getting more than 50 notes with Tumblr API) says you should use notes_info to get the notes. But I don't know how to construct it in python rather than making a url.
I could use a request constructing a url but I figure there is a simpler way using the python/tumblr api I just haven't figured it out, if someone could illuminate please.
Remove the quotes around notes_info=True. You should be passing the value True to the notes_info argument of the client's posts() method. Instead, you're actually passing the string 'notes_info=True' as a positional argument, which is invalid and causes pytumblr to create an invalid URL, which is why you're getting a 404.
I had a python file that I used to pull geotagged tweets from twitter, but now it requires OAuth. I'm trying to work with 1.1 but I'm not sure what to do next. My code below is from the Twitter API documentation, with some minor changes since the documentation is wrong (force_auth_header=True was removed and set body="").
I searched around, but either the posts are outdated or confusing. I think I'm close, but not sure where to go next. Ultimately I'd like to use something like search_url3, but I know that format is incorrect because I can tell it is getting a 401 error.
In the mean time, if I can get the results in a json format, that'd be great.
I used to use something like this:
search = urllib.urlopen("http://search.twitter.com/search.json?q=&rpp=100&geocode=39.95256,-75.164,2mi)
for result in j["results"]:
...
My current code:
import oauth2 as oauth
consumer_key="123"
consumer_secret="345"
Access_token="567"
Access_token_secret="890"
def oauth_req(url, key, secret, http_method="GET",http_headers=None):
consumer = oauth.Consumer(key, secret)
token = oauth.Token(Access_token,Access_token_secret)
client = oauth.Client(consumer, token)
resp, content = client.request(url,method=http_method,body="",
headers=http_headers)
print resp
return content
search_url2="https://api.twitter.com/1.1/search/tweets.json?q=&geocode=39.95256,-75.164,2mi"
search_url3="https://api.twitter.com/1.1/search/tweets.json?q=%23freebandnames&since_id=24012619984051000&max_id=250126199840518145&result_type=mixed&count=4"
home_timeline = oauth_req(search_url3, consumer_key, consumer_secret)
Basically, unless this is somekind of homework, you are doing it wrong. there are terrific (and not so terrific) python-twitter libs and wrappers, unless building one is your core mission than use one of them:
python-twitter
python-twitter-tools
twython
tweepy
Etc..
I'm trying to make an FQL query with the following code:
def get_postData(to_post, access_token):
postData = {}
postData["method"] = "fql.query"
postData["query"] = to_post
postData["access_token"] = access_token
return postData
def make_request(url, to_post, access_token):
postData = get_postData(to_post, access_token)
return requests.post(url, data = postData).json()[u'data']
Using POST requests is not the best documented in the docs, and I'm unable to get this to work. With either "fql.query" or "fql" specified under method (taken from the Javascript specific example here: How can I execute a FQL query with Facebook Graph API), I get the response:
{u'error': {u'message': u'Unsupported method, fql.query', u'code': 100, u'type': u'GraphMethodException'}}
Which is, of course, not covered in the docs. Without that method specification, I get back:
{u'error': {u'message': u'Unsupported post request.', u'code': 100, u'type': u'GraphMethodException'}}
Which is also not covered in the docs. I'm not able to use a get request here (which is trivial), as I'm making a rather large query that at the moment doesn't overflow the get request limits but very well could in the near future.
Thanks for any help you may be able to give with regards to solving this problem.
EDIT: Should note I'm making the request to:
https://graph.facebook.com
First of all, what URL are you trying to access? I mean, why do you need POST request for FQL? FQL is for fetching data, not for posting.
According to docs (https://developers.facebook.com/docs/technical-guides/fql/) your request should look like that:
https://graph.facebook.com/fql?q=QUERY&access_token=TOKEN - where QUERY - is your urlencoded query to FQL, TOKEN - your valid access token.
All you need to do is understand how your requests are built, if you understand that, then the errors will make more sense to you.
postData = {}
postData["method"] = "fql.query"
postData["query"] = to_post
postData["access_token"] = access_token
requests.post(url, data = postData).json()[u'data']
Without even running this, I know the request looks like
POST https://graph.facebook.com/?method=fql.query&query=THE_QUERY&access_token=THE_TOKEN
Which is not the fql.method of method/fql.query as shown as the relative url in the doc https://developers.facebook.com/docs/reference/api/batch/ you presented.
Removing the specification (I don't know why you want to do this) will obviously result in an unknown error since this is now the request you are making
POST https://graph.facebook.com/?query=THE_QUERY&access_token=THE_TOKEN
The correct request will be
GET https://api-read.facebook.com/restserver.php?method=fql.query&query=THE_QUERY&access_token=THE_TOKEN
or
GET https://api.facebook.com/method/fql.query&query=THE_QUERY&access_token=THE_TOKEN
I'm not entire sure what endpoint the batch uses that allows an HTTP POST to method/fql.query so I wouldn't rely on it unless you are actually doing batch requests.
In the end using fql.query may not be the best way to go since it's on its way to deprecation.
I'm still unsure how your query could be so long that it exceeds the GET request limit. Consider re-evaluating how you structure your query as a multi-query or in batch.
Does anyone know if there is some parameter available for programmatic search on yahoo allowing to restrict results so only links to files of specific type will be returned (like PDF for example)?
It's possible to do that in GUI, but how to make it happen through API?
I'd very much appreciate a sample code in Python, but any other solutions might be helpful as well.
Yes, there is:
http://developer.yahoo.com/search/boss/boss_guide/Web_Search.html#id356163
Thank you.
I found myself that something like this works OK (file type is the first argument, and query is the second):
format = sys.argv[1]
query = " ".join(sys.argv[2:])
srch = create_search("Web", app_id, query=query, format=format)
Here's what I do for this sort of thing. It exposes more of the parameters so you can tune it to your needs. This should print out the first ten PDFs URLs from the query "resume" [mine's not one of them ;) ]. You can download those URLs however you like.
The json dictionary that gets returned from the query is a little gross, but this should get you started. Be aware that in real code you will need to check whether some of the keys in the dictionary exist. When there are no results, this code will probably throw an exception.
The link that Tiago provided is good for knowing what values are supported for the "type" parameter.
from yos.crawl import rest
APPID="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
base_url = "http://boss.yahooapis.com/ysearch/%s/v%d/%s?start=%d&count=%d&type=%s" + "&appid=" + APPID
querystr="resume"
start=0
count=10
type="pdf"
search_url = base_url % ("web", 1, querystr, start, count, type)
json_result = rest.load_json(search_url)
for url in [recs['url'] for recs in json_result['ysearchresponse']['resultset_web']]:
print url