I'm collecting tweets from Twitter's API. My code is returning a string which I have transformed into a dictionary. I am looking to create a CSV where I store this data by creating columns. I have attached an image of my CSV currently looks like.
current CSV image:
.
What suggestions do you suggest for creating something like the following;
desired outcome:
with open('dict.csv', 'w') as csv_file:
writer = csv.writer(csv_file)
for key, value in y.items():
writer.writerow([key, value])
#with open('my_file.csv', 'w') as f:
# [f.write('{0},{1}\n'.format(key, value)) for key, value in y.items()]
Full code:
import requests
import os
import json
import pandas as pd
import csv
import sys
import time
bearer_token = "insert here"
search_url = "https://api.twitter.com/2/tweets/search/all"
query_params = {'query': '(Johnson&Johnson) -is:retweet -is:verified -baby -lotion -shampoo','tweet.fields': 'text', 'tweet.fields':'created_at', 'start_time':'2021-01-20T00:00:01.000Z', 'end_time':'2021-02-17T23:30:00.000Z'}
#query_params={'query':'(vaccine OR vaccinated) -is:retweet -is:verified -RT -baby -lotion -shampoo&start_time=2021-01-20T00:00:01.000Z&end_time=2021-02-20T23:30:00.000Z&max_results=10&tweet.fields=author_id,conversation_id,created_at,geo,id,lang,source,text&expansions=author_id&place.fields=full_name&user.fields=created_at,description,entities,id,location,name,url,username'}
def create_headers(bearer_token):
headers = {"Authorization": "Bearer {}".format(bearer_token)}
return headers
def connect_to_endpoint(url, headers, params):
response = requests.request("GET", search_url, headers=headers, params=params)
print('first:', response.status_code)
if response.status_code != 200:
raise Exception(response.status_code, response.text)
return response.json()
def main():
headers = create_headers(bearer_token)
json_response = connect_to_endpoint(search_url, headers, query_params)
x = json.dumps(json_response,sort_keys=True)
y = json.loads(x)
if __name__ == "__main__":
main()
Try Using DictWriter,
import csv
with open(csv_file, 'w') as csvfile:
writer = csv.DictWriter(csvfile, fieldnames=csv_columns)
writer.writeheader()
for data in dict_data:
writer.writerow(data)
For more info refer below link,
How to save a Python Dictionary to a CSV File?
Related
How do I make below working code to iterate over data["#odata.nextLink"] and append the data["value"] to sample.json file?
import requests
import json
import datetime
def get_data():
bearerAccessToken = '*************'
now = datetime.datetime.now()-datetime.timedelta(days=10)
dt_string = now.strftime("%Y-%m-%dT%H:%M:%S-04:00")
print(dt_string)
resourceUrl = "https://retsapi.raprets.com/CIN/RESO/OData/Property?Class=Residential&$count=true"
query_params = {"$filter":"ModificationTimestamp ge "+dt_string}
print(query_params)
r = requests.get(resourceUrl, params=query_params, headers={'Authorization' : 'Bearer '+ bearerAccessToken})
data = r.json()
with open("sample.json", "w") as outfile:
json.dump(data["value"], outfile)
print(data["#odata.nextLink"])
get_data()
I am working on an example but stuck with some points. I am a beginner for python and try to improve myself. I am trying to use openweather api and get some data from it and then write these data to a csv file. The aim of my code is input a txt file which contains city names, I want to get City name, Country code, lat long, Temperature, Wind speed, Wind direction. and then write them to a csv file. I can input the txt file or get the data with input from the command line but can not do both. And also I want to write the data to a csv file. Could you please help me? I can write it to the console, but I need to write them to the csv file. But, I can not convert my json object to csv
My input.txt
Los Angeles
San Francisco
...
My code:
import requests
from pprint import pprint
import csv
import pandas as pd
file = input("Input the filepath: ")
with open(file) as f:
for line in f:
line = line.strip()
API_key = "MYAPIKEY"
base_url = "http://api.openweathermap.org/data/2.5/weather?"
headers = {'content-type': 'application/json'}
city_name = line
Final_url = base_url + "appid=" + API_key + "&q=" + city_name
weather_data = requests.get(Final_url).json()
print("\nCurrent Weather" + city_name + ":\n")
weather_data = requests.get(Final_url, headers=headers)
f = open('weather_data_file.csv', "w")
f.write(weather_data.text)
f.close()
print(f)
The problem after edit:
The CSV file just contains the last city data and data is not in a
proper form if I open with excel
The data it outputs:
{"coord":{"lon":-122.42,"lat":37.77},"weather":[{"id":802,"main":"Clouds","description":"scattered clouds","icon":"03d"}],"base":"stations","main":{"temp":284.74,"feels_like":280.59,"temp_min":283.15,"temp_max":286.48,"pressure":1024,"humidity":76},"visibility":10000,"wind":{"speed":5.1,"deg":260},"clouds":{"all":40},"dt":1609003065,"sys":{"type":1,"id":5817,"country":"US","sunrise":1608996226,"sunset":1609030632},"timezone":-28800,"id":5391959,"name":"San Francisco","cod":200}
To write your JSON file to a CSV file:
import pandas as pd
if __name__ == "__main__":
df = pd.read_json (r'ajsonfile.json')
df.to_csv (r'filename.csv', index = None)
To write your JSON data from an API:
# Get JSON Data
weather_data = requests.get(yourURL, headers=yourHeaders)
# Write to .CSV
f = open('weather_data_file.csv', "w")
f.write(weather_data.text)
f.close()
JavaScript Object Notation (JSON) is a serialized representation of data. pandas.read_json tries to decode the JSON and build a dataframe from it. After you've read the the data with requests and deserialized it into python with the .json() call, its not JSON anymore and pandas.read_json won't work.
Sometimes you can build a dataframe directly from the python object, but in this case you've got an additional problem. You are only asking for one row of data at a time (one city) and its information is nested in multiple dictionaries. You can use python to flatten the received city data into the subset of data you want. And since you are only working row by row anyway, use the csv module to write the rows as you go.
A solution is
import requests
from pprint import pprint
import csv
openweathermap_city_csv_header = ["City Name", "Country Code", "Lat", "Long", "Temperature",
"Wind Speed", "Wind Direction"]
def openweathermap_city_flatten_record(record):
coord = record["coord"]
wind = record["wind"]
return { "City Name":record["name"],
"Country Code":record["Code"],
"Lat":coord["lat"],
"Long":coord["lon"],
"Temperature":record["main"]["temp"],
"Wind Speed":wind["speed"],
"Wind Direction":wind["deg"] }
file = input("Input the filepath: ")
with open(file) as cities, open('weather_data_file.csv', "w") as outfile:
writer = csv.DictWriter(outfile, openweathermap_city_csv_header)
for line in f:
line = line.strip()
API_key = "MYAPIKEY"
base_url = "http://api.openweathermap.org/data/2.5/weather?"
headers = {'content-type': 'application/json'}
city_name = line
Final_url = base_url + "appid=" + API_key + "&q=" + city_name
weather_data = requests.get(Final_url, headers=headers).json()
print("\nCurrent Weather" + city_name + ":\n")
writer.writerow(openweathermap_city_flatten_record(weather_data))
I have the following code which utilises boto3 for AWS.
import boto3
from trp import Document
# Document
s3BucketName = "bucket"
documentName = "doc.png"
# Amazon Textract client
textract = boto3.client('textract')
# Call Amazon Textract
response = textract.analyze_document(
Document={
'S3Object': {
'Bucket': s3BucketName,
'Name': documentName
}
},
FeatureTypes=["FORMS"])
#print(response)
doc = Document(response)
for page in doc.pages:
# Print fields
print("Fields:")
for field in page.form.fields:
print("Key: {}, Value: {}".format(field.key, field.value))
I am trying to save the output of that function as dict, JSON, or CSV, but I am not an experienced python programmer yet.
I tried this:
key_map = {}
filepath = 'output.txt'
with open(filepath) as fp:
line = fp.readline()
cnt = 1
while line:
for page in doc.pages:
# Print fields
print("Fields:")
for field in page.form.fields:
#print("Key: {}, Value: {}".format(field.key, field.value))
key_map[str(field.key, field.value)] = cnt
line = fp.readline()
cnt +=1
But I don't think that this solution is working. Any tips on how to save the output of that for loop as a JSON?
If you want as a csv output, you can use csv module as:
import csv
doc = Document(response)
with open('aws_doc.csv', mode='w') as aws_field_file:
field_write = csv.writer(aws_field_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)
for page in doc.pages:
for field in page.form.fields:
# This will write it as your <key>, <value>
field_write.writerow([field.key, field.value])
In case you want headers in the file you can also use the DictWriter which would make it easy for you to just pass a dictionary:
https://docs.python.org/3.4/library/csv.html#csv.DictWriter
I would like to download float data of stocks using the following code, which then has to write to a json file.
import requests
import json
filename='float.json'
url = "https://api.iextrading.com/1.0/stock/aapl/stats"
response = requests.get(url).json()
data = (response['symbol'], response['float'])
with open(filename, 'a+') as outfile:
json.dump(data, outfile, indent=4)
Now i would like to download the data for multiple stocks, so where it says "aapl" in the url, i would like to have mutliple stocks, like "tsla", "goog", etc..
Could someone explain to me how to achieve this?
Kind Regards
Can you try the following:
import json
import requests
stk_list = ['aapl', 'tsla', 'goog']
for stk_name in stk_list:
try:
url = "https://api.iextrading.com/1.0/stock/{}/stats".format(stk_name)
response = requests.get(url).json()
data = (response['symbol'], response['float'])
filename = 'float_{}.json'.format(stk_name)
with open(filename, 'a+') as outfile:
json.dump(data, outfile, indent=4)
except:
pass
you could try:
import json
import requests
stocks = ['appl', 'goog']
base_url = 'https://api.iextrading.com/1.0/stock/{}/stats'
filename='float.json'
for stock in stocks:
try:
response = requests.get(base_url.format(stock))
except:
continue
if response.status_code == 200:
response_json = response.json()
data = (response_json['symbol'], response_json['float'])
with open(filename, 'a+') as outfile:
json.dump(data, outfile, indent=4)
I am trying to loop through a CSV and run a web request. I can't get it past the first row in the CSV.
The CSV is being exported from Mac Excel as a list of 10 items in 10 rows / 1 column.
def AddTokens(request):
import csv
tokenList = []
output = 0
apikey = "12345"
restkey = "12345"
URL = "https://api.web.com/1/install/"
headers = {'content-type': 'application/json', 'X-web-Application-Id': apikey, 'X-web-REST-API-Key': restkey}
with open('/Users/name/Desktop/test.csv', 'rU') as csvfile:
deviceTokens = csv.reader(csvfile, delimiter=',')
for token in deviceTokens:
deviceToken = token[0].replace("/", "")
deviceType = "ios"
pushToken = "pushtoken_" + deviceToken
payload = {"deviceType": deviceType, "deviceToken": deviceToken, "channels": ["", pushToken]}
r = requests.post(URL, data=json.dumps(payload), headers=headers)
t = get_template('addpush.html')
html = t.render(Context({'output': output, 'tokenList': tokenList, 'deviceTokens': deviceTokens, 'token': token}))
return HttpResponse(html)