I have a simple piece of Python code to run a Radar Search with Places API and return the results coordinates in a list. I run into three problems, first, the results pulled this way do not match with doing a search on Google Maps itself using the same coordinates and parameters. Specifically, I get MANY more results on Radar Search. In a radius of 1km, I get more than 200 results for a restaurant chain name.
Second, the results go beyond 1km, my specified radius. The furthest is 1.3km away using Haversine.
Third, the results are wrong. The keyword field has no effect on the results. For example, searching for "McDonalds" or "Car" with the same parameters yield the exact same results. One of the results points to an Adidas store when I use the Place ID to find the Google description.
This is code independent, these problems are there if I just C&P this into the url bar:
https://maps.googleapis.com/maps/api/place/radarsearch/json?location=39.876186,116.439424&radius=1000&keyword=McDonalds&key=KEY
I have seen another similar post on Places API malfunctioning recently. Any help is appreciated. Thanks
I have a support ticket open with Google about this, as we're Enterprise customers, and they have confirmed there is an issue and they're working on it. From my conversations with them over the last few days:
There have been a few other reports of this issue and we've reported
the problem to the Places API team. I'll get back to you as soon as we
have more information from them.
We've received some other reports of this and the API engineers are
looking at the issue with the highest priority. There's no obvious
cause yet, but we'll let you know when they're done investigating and
have determined a fix.
I'm sorry to hear about the complaints that you're receiving, but
unfortunately the engineers haven't been able to give me an ETA yet. I
expect to hear back from them soon but can't give an estimate yet.
I'll post updates here as I get them.
UPDATE 9/8: Google's support is saying this issue will be fixed by end of the week.
UPDATE 9/12: Google fixed it. It was being tracked here: https://code.google.com/p/gmaps-api-issues/issues/detail?id=7082
Related
I am just starting out to learning Python and have taken some online courses in my free time.
I am trying to find the data source for this website, making a daily count of departures from the airport, and eventually building a flights vs date plot.
Have spent two weeks investigating the page source, but am unable to find the json source. Would a kind soul please show me where the json source is? Thanks!
https://www.changiairport.com/en/flights/departures.html
https://www.changiairport.com/cag-web/flights/departures?lang=en&callback=JSON_CALLBACK&date=today
There you go. That will give you the flight schedule for today. The date parameter can probably be other things too, but I don't know what the options are. Any normal get request should work, it seems publicly accessible.
You just need to right click and go to "inspect" then hit the "network" tab and then just browse through the different requests.
Just a note:
Just for the record, this is called scraping and it's often in a legal
gray area where, so long as you aren't using it too extensively or
making a profit off of it you probably won't get in any trouble, but
just make sure you have permission from the company if you plan to
make a lot of calls to an open API like this. It's usually against their terms of service, but as an unenforced clause that they will only use if you become a nuisance.
I am aware that this can't be done with bash script only, or it isn't as far as I know (and I'm still learning). This is why I'm asking for help. What do I need more ? Are there specific tools ?
This is what I'd like to do:
Upload an image to https://www.google.com/searchbyimage/upload
Then find all the identical images
Download the one which has the greatest resolution
So far I've been able to upload an image to Searchbyimage through curl. This uploaded image then creates a very long token that is used to search similar images, with some supplementary keywords.
The uploaded image creates a link composed like so:
https://www.google.com/search?tbs=sbi:
After this is the awfully long token: AMhZZith3JfR2OzwmuyQjufBifvdFWNjMShRMypWIE2-g005QfYLeTATLhGHAWz8MLI-tbgHzZp-bREPlJbsNWhY7U4Z2_19bu0oHII6VJPIVVJSPANODqnrJXp6X5VKKoXHMLcBCmI9eIpxS_1EX9g9YJPFL2XFEfJqIApLX83erP5mlRM7rSiIF5Te_1RPNyVkp4IPZPBRtoOKGhpDw2xad-JZsqd2ai4F5sMvyO2A_18PMFKg21nTRH_1jVeOeUhz8U5zkL4lycIg3kafAYlNy8YwmjSFcmc2nZB_10t9MFyi2BnBmemDRp4DCACI0FVM6pLTIB8VCBpU9A
And it adds this at the end: &hl=fr.
Finally the image is searched, and I have the choice between clicking "similar images" or "all sizes" (it's "all sizes" I want, as similar images doesn't ensure it will be identical). This will add some keywords from google's analysis of the picture (here, a photography of Émile Zola) and create a second token:
The picture I searched here
https://www.google.com/search?safe=strict&hl=fr&
q=emile+zola&tbm=isch
&tbs=simg:
CAQSmQEJthA57uIOXdcajQELEKjU2AQaBggXCD0IQgwLELCMpwgaYgpgCAMSKLQZ9QH3BLMZ2A6xGdcO3w70Ad0OwjrEOqEuwzqiLsE67iSTLoM4oC4aMIk1iw7XQn7Wu55hLB2k-bnfW3_1yf24eA0N-w-baKvWkDj48J67yZZS-uQ-BgjCRQyAEDAsQjq7-CBoKCggIARIEnfZWUgw&sa=X&ved=0ahUKEwi965ashtrhAhWI3eAKHSmRCBwQ2A4IKygB
&biw=1920&bih=944
With at the end the resolution of the picture. The idea is to recreate this second link, to then download the highest resolution image amongst what google has found. I have to get the token, but everything else can be found on the picture file itself: the file is properly named after the picture, and thus could make for keywords, and its resolution is also easily known. I'd like to make it a script, to download higher resolution images of many paintings - over a thousand - that are in low quality. Ideally I'd use it quite often. So far I had found how to upload a picture with curl, and it had gave me back a token, but uncomplete. Beyond this, I was completely lost.
In theory this doesn't seem impossible. The problem is I'm too much of a newbie: I enjoy a lot so far Linux and bash, but I only know so few. I have of course done some hours of googling before, nothing showed up that I knew I could use. There is nothing alike neither on github: a lot of scripts that search for similar images, but none for identical. None of them that also compares the sizes of these images. There's also a python API for reverse image searching, but it didn't seem like it could search for identical images, and it seems related to the google API, which is problematic. All of this is probably dumbly hard for me because I'm only a beginner, and I don't know enough to build this script: but in another way - maybe due to my lack of knowledge - it doesn't seem impossible at all, and I'm very willing to try, fail, try again: learn. So here I am, to ask: how do I do that ? Can it be done in bash only ? If not, what must I include ? Or perhaps it cannot be done ?
Lastly, I know there is a google API for reverse image searching. That'd be very useful, if it wasn't limited to a hundred image searches a day: if you want more, you've got to pay. And by a 100 images a day, it'd take me around eleven days to reverse search all the images I wanted in a better quality: in the end, I'd be done as fast by searching all that myself, by hand. But neither these options seems to be a solution: and this script doesn't seem impossible. It is only beyond my current capacities.
Thank you in advance, if anyone has got an idea !
PS: I can use linux wether through WSL, or a virtual machine. Both work very fine so far, including whatever command or package. WSL is much faster. And sorry for my english, I'm french !
Second PS: I've been asked to show what I had as code, but this doesn't get beyond this:
curl -i -F sch=sch -F encoded_image=#path/to/my/imagefile.jpg https://www.google.com/searchbyimage/upload
Which was a partial answer to my question I had found here:
How to use google search by image in curl
There's two fundamental ways to use the web programmatically:
via API: this is purpose built for computers to access web resources and always preferred. You follow strict rules and get well defined results back.
by crawling: this is when the computer pretends to be a user, emulating the clicking on links done in a browser. Basically curl, but over and over again with state stored in between, parameters generated correctly, encoding applied, etc.
As you say, there's an API available so if it does what you want then it's the right way to go. The fact that it does what you want, but enforces limits, is a very useful sign that was you're trying to do has limits. Those limits will have been carefully set to incentivise you to work within them. Trying to crawl for the same results will likely either breach Google's service term limits, or your sanity limits.
So if you really want to work around the API, then use a crawler library such as Python Scrapy. But note that the API limits might be a useful indication of how far you can expect to get without paying.
There are several posts already about getting directions to Google Maps API if your waypoints exceed 23, but the answers about it do not account for if optimize_waypoints is True. The difference is, adding one stop to the list of addresses could throw off the whole route optimization equation. Does anyone have any suggested work-arounds for it?
I was thinking maybe running it once, dropping off the smallest mileage different stops (say if I send 26, drop off the 3 smallest), then running it again, then somehow reconcile the two routes? Is there a more efficient or cleaner way? Any thoughts? Thanks!
I am using Python, but this is not a question to specific Python. More of a general programming and Google Maps directions question. But answers referencing Python would be most appreciated!
We have an entry for this feature request for including more than 23 waypoints in Directions API requests in the Google Issue Tracker so that we can provide information on it to all Google Maps APIs users, as the technical aspects of this issue are of interest to a number of our customers,
Issues in the Issue Tracker are directly curated by Directions API specialists, who will provide updates in the Issue Tracker whenever there's news from the engineering team.
We would like to warmly invite you to view the issue in the Issue Tracker, and to star it to register your interest. This will subscribe you to receive technical updates on the issue. Starring the issue also provides us with valuable feedback on the importance of the issue to our customers, and increases the issue's priority with the product engineering team.
You can view and star the issue here:
- https://issuetracker.google.com/35824756
This Issue Tracker entry is the authoritative source for public information regarding this issue, and all publicly-relevant updates will be posted there.
I did try this API: (insert)
https://developers.google.com/youtube/v3/docs/playlistItems/insert#try-it
and got a 200OK and a json revising what I've insert correctly.
what I specified is the video resource (kind, id) and a playlistId which is my 'watchhistory'.
The strange thing is I cannot see the new result when I do a list (GET) with the corresponding api calls. However, if I actually go to youtube.com, I see the new item which I never watch, in my watch history appearing the right order (newest). But my objective is to still be able to get this information with the api calls. Anybody experience similar things?
I think this is related to an internal engineering issue concerning the API not reporting the full watch history. It's something that they've been working on for a few months now, with limited success (they've pushed a couple of changes, but it hasn't remedied the problem). Basically, until it is fixed, the v3 watch history only returns an older, incomplete subset of the actual history.
Here's the issue if you want to track it or contribute to the data to help resolve it:
https://code.google.com/p/gdata-issues/issues/detail?id=4642
What ready available algorithms could I use to data mine twitter to find out the degrees of separation between 2 people on twitter.
How does it change when the social graph keeps changing and updating constantly.
And then, is there any dump of twitter social graph data which I could use rather than making so many API calls to start over.
From the Twitter API
What's the Data Mining Feed and can I have access to it?
The Data Mining Feed is an expanded version of our /statuses/public_timeline REST API method. It returns 600 recent public statuses, cached for a minute at a time. You can request it up to once per minute to get a representative sample of the public statuses on Twitter. We offer this for free (and with no quality of service guarantees) to researchers and hobbyists. All we ask is that you provide a brief description of your research or project and the IP address(es) you'll be requesting the feed from; just fill out this form. Note that the Data Mining Feed is not intended to provide a contiguous stream of all public updates on Twitter; please see above for more information on the forthcoming "firehose" solution.
and also see: Streaming API Documentation
There was a company offering a dump of the social graph, but it was taken down and no longer available. As you already realized - it is kind of hard, as it is changing all the time.
I would recommend checking out their social_graph api methods as they give the most info with the least API calls.
There might be other ways of doing it but I've just spent the past 10 minutes looking at doing something similar and stumbled upon this Q.
I'd use an undirected (& weighted - as I want to look at location too) graph - use JgraphT or similar in py; JGraphT is java based but includes different prewritten algos.
You can then use an algorithm called BellmanFord; takes an integer input and searches the graph for the shortest path with the integer input, and only integer input, unlike Dijkstras.
http://en.wikipedia.org/wiki/Bellman%E2%80%93Ford_algorithm
I used it recently in a project for flight routing, iterating up to find shortest path with shortest 'hops' (edges).