I want to get info from API via Python, which has infinite info updating (it is live - for example live video or live monitoring). So I want to stop this GET request after interval (for example 1 second), then process these information and then repeat this cycle.
Any ideas? (now I am using requests module, but I do not know, how to stop receiving data and then get them)
I might be off here, but if you hit an endpoint at a specific time, it should return the JSON at that particular moment. You could then store it and use it in whatever process you have created.
If you want to hit it again, you would just use requests to hit the endpoint.
Related
I would like to know what is the most elegant way to scrape live webcam (traffic) data, ideally using Python. The webcam feed is represented by an API, with each get request yielding a image of the currently available feed from the webcam. The feed in question has a 2/3 second delay and therefore there are ~ 30 images per minute that can be requested.
My current (trivial) solution simply queries the API in a loop (perhaps with a sleep timer) and then cleans up any duplicated images. However, this seems quite dirty and I was wondering if there is a cleaner/more elegant solution available.
In principal I would like the solution to (if at all possible) avoid:
downloading duplicated images
sleep timers
looping
Is something like this possible?
To avoid sleep timers in your code, you can write a process that is triggered by cron. Cron will handle running your script at defined intervals, such as every 2 seconds (60s / 30 images per minute).
An example process might call the API using requests. Assuming an image is passed back, the following example code might work. If a JSON string is passed back then you will need to parse it and extract the image URL.
r = requests.get('https://traffic-cam-site.com/cam', auth=('user', 'pass'))
if r.status == 200:
image = r.content
To avoid downloading duplicate images, you would need to know when a new image is present on the API. You will need to periodically check the cam site API for this purpose. Store a hash of the collected images in a database (or text file), and send that with your request to the API. Then hash the image that is currently present on the cam site - if the hashes match, don't download the image to your server.
Alternatively, if the cam site API does push notifications then you may be notified when a new image is present.
I have written a python script that helps me get the prices from alphavantage using an api key. I have 2 different alphavantage api keys (mine and my brother's) and the python script requests data from alphavantage separately from both keys but from the same laptop. But even though I request it separately from separate keys, I get the maximum api call frequency exceeded error (5 api calls per minute per key).
My best assumption is that alphavantage knows whether the request is coming from the same location or the same system. Is there any workaround this problem? Maybe bounce my signal (sort of found an answer but don't know if that's the problem though!) or pretend like the request is going from different system?
Your IP is limited to 5 requests per minute regardless of how many keys you cycle through, so you would need to cycle through that as well.
I went through the same issue and just ended up buying their premium key. It's well worth it. You can also alternatively add a time delay and just run the script in the background so it doesn't time out.
You can also try polygon API its 60 requests per minute for free
https://polygon.io/pricing
So I'm trying to get data from an API which has a max call limit of 60/min. If I go over this it will return a response [429] too many requests.
I thought maybe a better way was to keep requesting the API until I get a response [200] but I am unsure how to do this.
import requests
r = requests.get("https://api.steampowered.com/IDOTA2Match_570/GetTopLiveGame/v1/?key=" + key + "&partner=0")
livematches = json.loads(r.text)['game_list']
So usually it runs but if it returns anything other than a response [200], my program will fail and anyone who uses the website won't see anything.
Generally with cases like this, I would suggest adding a layer of cache indirection here. You definitely don't want (and can't) try solving this using frontend like you suggested since that's not what the frontend of your app is meant for. Sure, you can add the "wait" ability, but all someone has to do is pull up Chrome Developer tools to grab your API key and then call that as many times as they want. Think of it like this: say you have a chef that can only cook 10 things per hour. Someone can easily come into the restaurant, order 10 things, and then nobody else can order anything for the next hour, which isn't fair. Instead, adding a "cache" layer, which means that you only call the steam API every couple seconds. If 5 people request your site within, say, 5 seconds, you only go to the steam API on the first call, then you save that response. To the other 4 (and anyone who comes within those next few seconds), you return a "cached" version.
The two main reasons for adding a cache API layer are the following:
You would stop exposing key from the frontend. You never want to expose your API key to the frontend directly like this since anyone could just take your key and run many requests and then bam, your site is now down (denial of service attack is trivial). You would instead have the users hit your custom mysite.io/getLatestData endpoint which doesn't need the key since that would be securely stored in your backend.
You won't run into the rate limiting issue. Essentially if your cache only hits the API once every minute, you'll not run into any time where users can't access your site due to an API limit, since it'll return cached results to the user.
This may be a bit tricky to visualize, so here's how this works:
You write a little API in your favorite server-side language. Let's say NodeJS. There are lots of resources for learning the basics of ExpressJS. You'd write an endpoint like /getTopGame that is attached to a cache like redis. If there's a cached entry in the redis cache, return that to the user. If not, go to the steam API and get the latest data. Store that with an expiration of, say, 5 seconds. Boom, you're done!
It seems a bit daunting at first but as you said being a college student, this is a great way to learn a lot about how backend development works.
How to get the properties as shown on the image (Blocked, DNS resolution, Connecting ...) after sending the request?
From firefox, the waiting time = ~650ms
From python, requests.Response.elapsed.total_seconds() = ~750ms
Since the result is difference, i want to have a more details result as shown on firefox developer mode.
You can only get the total time of the request because the response doesn´t know more itself.
More informations are only logged by the programms which does handle the request and start-stopping a timer for some steps.
You need to track times in/on you connection-framework or you can have a look on the FireFox API for "timings" - there are some more APIs so maybe you find something you are able to use for your case - but main fact is you can´t do it directly and only with your script because request and response are fired/catched and logging/getting data does happen between other components then.
I'm using the Instagram API to retrieve all photos from a list of hashtags. Starting today, I've been hitting the API rate limits (429 error, specifically). To debug this, I've been trying to see how to get the number of calls I have left per hour and incorporate that into my script.
The Instagram API says that you can access the number of calls left in the hour through the HTTP header, but I'm not sure how to access that in Python.
The following fields are provided in the header of each response and their values are related to the type of call that was made (authenticated or unauthenticated):
X-Ratelimit-Remaining: the remaining number of calls available to your app within the 1-hour window
X-Ratelimit-Limit: the total number of calls allowed within the 1-hour window
http://instagram.com/developer/limits
How would I access this data in Python?
I assumed it was much a much fancier solution based on a few other answers on SO, but after some researching, I found a really simple solution!
import requests
r = requests.get('URL of the API response here')
r.headers