I would like to download float data of stocks using the following code, which then has to write to a json file.
import requests
import json
filename='float.json'
url = "https://api.iextrading.com/1.0/stock/aapl/stats"
response = requests.get(url).json()
data = (response['symbol'], response['float'])
with open(filename, 'a+') as outfile:
json.dump(data, outfile, indent=4)
Now i would like to download the data for multiple stocks, so where it says "aapl" in the url, i would like to have mutliple stocks, like "tsla", "goog", etc..
Could someone explain to me how to achieve this?
Kind Regards
Can you try the following:
import json
import requests
stk_list = ['aapl', 'tsla', 'goog']
for stk_name in stk_list:
try:
url = "https://api.iextrading.com/1.0/stock/{}/stats".format(stk_name)
response = requests.get(url).json()
data = (response['symbol'], response['float'])
filename = 'float_{}.json'.format(stk_name)
with open(filename, 'a+') as outfile:
json.dump(data, outfile, indent=4)
except:
pass
you could try:
import json
import requests
stocks = ['appl', 'goog']
base_url = 'https://api.iextrading.com/1.0/stock/{}/stats'
filename='float.json'
for stock in stocks:
try:
response = requests.get(base_url.format(stock))
except:
continue
if response.status_code == 200:
response_json = response.json()
data = (response_json['symbol'], response_json['float'])
with open(filename, 'a+') as outfile:
json.dump(data, outfile, indent=4)
Related
How do I make below working code to iterate over data["#odata.nextLink"] and append the data["value"] to sample.json file?
import requests
import json
import datetime
def get_data():
bearerAccessToken = '*************'
now = datetime.datetime.now()-datetime.timedelta(days=10)
dt_string = now.strftime("%Y-%m-%dT%H:%M:%S-04:00")
print(dt_string)
resourceUrl = "https://retsapi.raprets.com/CIN/RESO/OData/Property?Class=Residential&$count=true"
query_params = {"$filter":"ModificationTimestamp ge "+dt_string}
print(query_params)
r = requests.get(resourceUrl, params=query_params, headers={'Authorization' : 'Bearer '+ bearerAccessToken})
data = r.json()
with open("sample.json", "w") as outfile:
json.dump(data["value"], outfile)
print(data["#odata.nextLink"])
get_data()
This scraper class is supposed to loop through the CATEGORIES array and then loop through the JSON_FILES concurrently and create individual JSON files. I am trying to use a stack and just append the data to the stack and then pop it off the top so that my stack array only has the current data from the most recent scrape. The issue is that each JSON file gets created but it overwrites all of them each time and in the end, they all produce the last element.
How do I make it so the JSON files only save the data that share the same index as the CATEGORIES array?
import requests
import json
from bs4 import BeautifulSoup
class Scraper:
def scrape_category():
try:
stack = []
for url in CATEGORIES:
soup = BeautifulSoup(requests.get(BASE_URL + url).text, "lxml")
stack.append(Scraper.extract_product_info(soup))
Scraper.create_json_file(stack.pop())
except ConnectionError:
print("Couldn't connect to ebay.com! Please try again.")
exit(1)
def extract_product_info(soup):
title = soup.find_all("h3", class_="s-item__title")
bid = soup.find_all("span", class_="s-item__price")
link = soup.find_all("a", class_="s-item__link", href=True)
bid_time = soup.find_all("span", class_="s-item__time-left")
bid_count = soup.find_all("span", class_="s-item__bids s-item__bidCount")
data = []
for title, bid, link, bid_time, bid_count in zip(
title, bid, link, bid_time, bid_count
):
data.append(
{
"Title": title.getText(),
"Bid": bid.getText(),
"Link": link["href"],
"Bid_Time": bid_time.getText(),
"Bid_Count": bid_count.getText(),
}
)
return data
def create_json_file(data):
for file in JSON_FILES:
json_file = open(file, "w")
json.dump(data, json_file, indent=2)
json_file.close()
if __name__ == "__main__":
BASE_URL = "https://www.ebay.com/b/"
CATEGORIES = [
"Apple-Cell-Phones-Smartphones/9355/bn_319682?LH_Auction=1&rt=nc",
"Samsung-Cell-Phones-and-Smartphones/9355/bn_352130?LH_Auction=1&rt=nc",
"Google-Cell-Phones-Smartphones/9355/bn_3904160?LH_Auction=1&rt=nc",
"LG-Cell-Phones-and-Smartphones/9355/bn_353985?LH_Auction=1&rt=nc",
]
JSON_FILES = ["apple.json", "samsung.json", "google.json", "lg.json"]
Scraper.scrape_category()
I worked through it, and decided to get rid of the create_json_file function entirely, and instead made the scrape_category function look like this:
def scrape_category():
try:
for url, filename in zip(CATEGORIES, JSON_FILES):
soup = BeautifulSoup(requests.get(BASE_URL + url).text, "lxml")
data = Scraper.extract_product_info(soup)
with open(filename, "w") as json_file:
json.dump(data, json_file, indent=2)
print("Files created successfully!")
except ConnectionError:
print("Couldn't connect to ebay.com! Please try again.")
exit(1)
I'm collecting tweets from Twitter's API. My code is returning a string which I have transformed into a dictionary. I am looking to create a CSV where I store this data by creating columns. I have attached an image of my CSV currently looks like.
current CSV image:
.
What suggestions do you suggest for creating something like the following;
desired outcome:
with open('dict.csv', 'w') as csv_file:
writer = csv.writer(csv_file)
for key, value in y.items():
writer.writerow([key, value])
#with open('my_file.csv', 'w') as f:
# [f.write('{0},{1}\n'.format(key, value)) for key, value in y.items()]
Full code:
import requests
import os
import json
import pandas as pd
import csv
import sys
import time
bearer_token = "insert here"
search_url = "https://api.twitter.com/2/tweets/search/all"
query_params = {'query': '(Johnson&Johnson) -is:retweet -is:verified -baby -lotion -shampoo','tweet.fields': 'text', 'tweet.fields':'created_at', 'start_time':'2021-01-20T00:00:01.000Z', 'end_time':'2021-02-17T23:30:00.000Z'}
#query_params={'query':'(vaccine OR vaccinated) -is:retweet -is:verified -RT -baby -lotion -shampoo&start_time=2021-01-20T00:00:01.000Z&end_time=2021-02-20T23:30:00.000Z&max_results=10&tweet.fields=author_id,conversation_id,created_at,geo,id,lang,source,text&expansions=author_id&place.fields=full_name&user.fields=created_at,description,entities,id,location,name,url,username'}
def create_headers(bearer_token):
headers = {"Authorization": "Bearer {}".format(bearer_token)}
return headers
def connect_to_endpoint(url, headers, params):
response = requests.request("GET", search_url, headers=headers, params=params)
print('first:', response.status_code)
if response.status_code != 200:
raise Exception(response.status_code, response.text)
return response.json()
def main():
headers = create_headers(bearer_token)
json_response = connect_to_endpoint(search_url, headers, query_params)
x = json.dumps(json_response,sort_keys=True)
y = json.loads(x)
if __name__ == "__main__":
main()
Try Using DictWriter,
import csv
with open(csv_file, 'w') as csvfile:
writer = csv.DictWriter(csvfile, fieldnames=csv_columns)
writer.writeheader()
for data in dict_data:
writer.writerow(data)
For more info refer below link,
How to save a Python Dictionary to a CSV File?
I would like to fetch all videos from YouTube API for given ChannelId using Python. Below is the code I have tried, but I cannot figure out how to append the data in the while loop.
import requests
import json
mykey = 'myGoogleKey'
channelid = 'someRandomChannelId'
# First request
r = requests.get("https://www.googleapis.com/youtube/v3/search?part=snippet&maxResults=50&channelId="+channelid+"&order=date&key="+mykey)
json_data = r.json()
nextPageToken = json_data.get("nextPageToken")
# Retrieve all the rest of the pages
while nextPageToken:
r = requests.get("https://www.googleapis.com/youtube/v3/search?part=snippet&maxResults=50&channelId="+channelid+"&order=date&key="+mykey+"&pageToken="+nextPageToken)
json_data.append(r.json()) # this part needs to be modified/adjusted
nextPageToken = json_data.get("nextPageToken")
with open('myJsonFile.json', 'w') as outfile:
json.dump(json_data, outfile, sort_keys=True, indent=4)
print("All Done!")
json_data.update(r.json())
Should do the trick.
I've written some code using python to scrape some titles and price from a webpage and write the results in a csv file. The script is running awesome. As I'm appending data to a csv file the script is writing headers in such a way that if it runs 4 loops then the headers will be written 4 times. How to fix it so that the headers will be written once. Thanks.
This is the script:
import csv
import requests
from bs4 import BeautifulSoup
diction_page = ['http://www.bloomberg.com/quote/SPX:IND','http://www.bloomberg.com/quote/CCMP:IND']
for link in diction_page:
res = requests.get(link).text
soup = BeautifulSoup(res,'lxml')
title = soup.select_one('.name').text.strip()
price = soup.select_one('.price').text
print(title,price)
with open('item.csv','a',newline='') as outfile:
writer = csv.writer(outfile)
writer.writerow(["Title","Price"])
writer.writerow([title, price])
As an option you can try this:
import csv
import requests
from bs4 import BeautifulSoup
diction_page = ['http://www.bloomberg.com/quote/SPX:IND','http://www.bloomberg.com/quote/CCMP:IND']
for i,link in enumerate(diction_page):
res = requests.get(link).text
soup = BeautifulSoup(res,'lxml')
title = soup.select_one('.name').text.strip()
price = soup.select_one('.price').text
print(title,price)
with open('item.csv','a',newline='') as outfile:
writer = csv.writer(outfile)
if (i == 0):
writer.writerow(["Title","Price"])
writer.writerow([title, price])
Don't write the headers in the for loop:
import csv
import requests
from bs4 import BeautifulSoup
diction_page = ['http://www.bloomberg.com/quote/SPX:IND','http://www.bloomberg.com/quote/CCMP:IND']
outfile = open('item.csv','w',newline='')
writer = csv.writer(outfile)
writer.writerow(["Title","Price"])
for link in diction_page:
res = requests.get(link).text
soup = BeautifulSoup(res,'lxml')
title = soup.select_one('.name').text.strip()
price = soup.select_one('.price').text
print(title,price)
writer.writerow([title, price])
outfile.close()