I have generally wondering if it is possible to access data displayed on graphs such as these: https://coinmarketcap.com/currencies/bitcoin/
I have looked at the .htm file and other links within this, yet none seem to be containing this. Would anyone know how to retrieve and print information from the graph using python?
You can retrieve the information from coinmarketcap using their API.
https://coinmarketcap.com/api/
This will allow you to request data from them, sometimes with given parameters (e.g. the currency) and you'll retrieve the data in a JSON format, which you can then again parse and display with python.
There are plenty of API tutorials out there for python on the web. Good luck!
Related
I'm trying to gather metadata from a youtube channel using python via youtube data api-v3.
I've managed to get data in json format but the texts like titles, descriptions, comments etc are in Korean language and they are shown as
ex : "title": "\ud55c \uc5ec\ub984\ubc24\uc758 \uafc8",
Is there a way to set encoding for specific feature/keys?
https://developers.google.com/youtube/v3/docs/i18nLanguages/list
The above seems to be the way to solve the problem, but I am new to api and I do not know how to apply it... can anyone give me an example or a link? so I can try myself?
or is there a way to decode back to original language? from the broken language output?
Thank you very much for your time.
Im currently trying to retrieve data from the google trends site directly into python. By inspecting the "download csv"-button I was able to extract the underlying url of the file, which looks like this:
https://trends.google.com/trends/api/widgetdata/multiline/csv?req=%7B%22time%22%3A%222018-08-30%202019-08-30%22%2C%22resolution%22%3A%22WEEK%22%2C%22locale%22%3A%22de%22%2C%22comparisonItem%22%3A%5B%7B%22geo%22%3A%7B%7D%2C%22complexKeywordsRestriction%22%3A%7B%22keyword%22%3A%5B%7B%22type%22%3A%22BROAD%22%2C%22value%22%3A%22trump%22%7D%5D%7D%7D%5D%2C%22requestOptions%22%3A%7B%22property%22%3A%22%22%2C%22backend%22%3A%22IZG%22%2C%22category%22%3A0%7D%7D&token=APP6_UEAAAAAXWrEbGVLW-ssfeQvOJgr9938DRgYO1sm&tz=-120
Unquoted :
https://trends.google.com/trends/api/widgetdata/multiline/csv?req={"time":"2018-08-30 2019-08-30","resolution":"WEEK","locale":"de","comparisonItem":[{"geo":{},"complexKeywordsRestriction":{"keyword":[{"type":"BROAD","value":"trump"}]}}],"requestOptions":{"property":"","backend":"IZG","category":0}}&token=APP6_UEAAAAAXWrEbGVLW-ssfeQvOJgr9938DRgYO1sm&tz=-120
I can now easily get this csv into a pandas dataframe. Ideally, I would now just manipulate the url in order to make custom requests and load new data. The problem I have is that I cannot use the same token parameter, since it is somehow generated newly for each individual csv-request. I think that the answer from shaochuancs in Origin of tokens in Google trends API call describes the problem as I am facing it. Can anyone explain how I can request this token which I could then use for the second request ( the actual csv download )? :/
I would like to download the data in this table:
http://portal.ujn.gov.rs/Izvestaji/IzvestajiVelike.aspx
I know how to use selenium to go through the pages and the CSS selectors are helpful enough that it shouldn't be too difficult to get all the data...
However, I am curious if anyone knows some way of getting to a json or whatever intermediary object is used to make the html? As in, whatever the raw data format file that gets exported by the server is? Is this possible with aspnet frameworks?
I have found such solutions in the past, but with much simpler web pages and web pages with get requests...
Thank you!
Taking a look at the website (I have no experience with Russian at all but not like it maters much.) It looks to me like it is pulling the information from a database via PHP (In my book the "old" way of doing it) not a JSON file. Which means that your basically stuck doing it the normal web scraping route like you said OR to find a SQL injection (which I am in NO WAY SUGGESTING as it is illegal?) to be able to bypass the limitations of there crappy search page.
I just wanted to ask what can I do to solve this issue I have.
Essentially I am making a stock checker for sneakers from Adidas, I know the endpoint to obtain the stock but the JSON data given back to me whilst readable and contains what I need also contains a bunch of other information that is unnecessary to what I am trying to do.
Example of a link to an endpoint:
http://production.store.adidasgroup.demandware.net/s/adidas-GB/dw/shop/v16_9/products/(BZ0221)?client_id=c1f3632f-6d3a-43f4-9987-9de920731dcb&expand=availability,variations,prices
This is a link to the JSON containing the stock of the shoe, price and availability. However, if you try to open it you'll see that it responds a bunch of useless info such as the description of the shoe and the price which I do not need.
A github repository that I was using to try and get to grips with the requests I am trying to make is:
https://github.com/yzyio/adidas-stock-checker/blob/master/assets/index.js
I can get it to give me the JSON response I am just trying to strip what I don't need and keep what I do need which I am finding very difficult especially in python.
Many Thanks!
Since you've said you can get a JSON response from the server than the first think you need to do is tell python to load it as JSON.
import json
data = json.loads(response_from_server)
After doing this you can now access the values in your JSON object the way you would access them via a Python dict.
data["artist"]["id"]
I am looking to replace Yahoo Query Language with something more manageable and dependable. Right now we use it to scrape public CSV files and use the information in our web app.
Currently I am having trouble trying to find an alternative and it seems that scraping websites with Python is the best bet. However I don't even know where to start.
My question is what is needed to scrape a CSV, save the data and use it elsewhere in a web application using Python? Do I need a dedicated database or can I save the data a different way?
A simple explanation is appreciated
This is a bit broad, but let's divide it in separate tasks
My question is what is needed to scrape a CSV
If you mean downloading CSVs files from already known URLs, you can simply use urllib. If you don't have the CSVs URLs you'll have to obtain them somehow. If you want to get the URLs from webpages, beautifulsoup is commonly used to parse HTML. scrapy is used for larger-scale scraping.
save the data.
Do I need a dedicated database or can I save the data a different way?
Not at all. You can save the CSV files directly to your disk., store them with pickle, serialize them to JSON or use a relational or NoSQL database. What you should use depends heavily on what you want to do and what of access you need to the data (local/remote, centralized/distributed).
and use it elsewhere in a web application using Python
You'll probably want to learn how to use a web framework for that (django, flask and cherrypy are common choices). If you don't need concurrent write access, any of the storage approaches I mentioned would work with these