I have a problem with my Python script.
Basically what my script does is first it does a GET request to our API and extracts all ID's from the endpoint and then saves it in a CSV file.
Atm im having problems inserting data in the csv file. What i want is for my csv file to look like this after inserting data in:
id
1
2
3
...
Basically I want every id in their own row.
But what ends up being inserted is this:
id
1,2,3,...
I have tried for looping and few other things and nothing seemed to work out. I would love if anyone can help me with this problem. It's probably something really simple I just missed out.
My script code:
import requests
import json
import csv
from jsonpath_ng import jsonpath, parse
url = 'url'
headers = {
"Authorization": "Bearer token"
}
response = requests.get(url_v1, headers=headers)
JsonResponse = response.json()
converted = json.dumps(JsonResponse)
Data = json.loads(converted)
ParseData = parse('$..id')
Id = ParseData.find(Data)
open_file = open('C:/File/test.csv','w', newline='')
writer = csv.writer(open_file)
list_id = []
fields = ['id']
for i in range(0, len(Id)):
result = Id[i].value
list_id.append(result)
writer.writerow(fields)
writer.writerow(list_id)
open_file.close()
Related
I'm trying to pull the average of temperatures from this API from a bunch of different ZIP codes. I can currently do so by manually changing the ZIP code in the URL for the API, but I was hoping it to be able to loop through a list of ZIP codes or ask for input and use those zip codes.
However, I'm rather new and have no idea on how to add variables and stuff to a link, either that or I'm overcomplicating it. So basically I was searching for some methods to add a variable to the link or something to the same effect so I can change it whenever I want.
import urllib.request
import json
out = open("output.txt", "w")
link = "http://api.openweathermap.org/data/2.5/weather?zip={zip-code},us&appid={api-key}"
print(link)
x = urllib.request.urlopen(link)
url = x.read()
out.write(str(url, 'utf-8'))
returnJson = json.loads(url)
print('\n')
print(returnJson["main"]["temp"])
import urllib.request
import json
zipCodes = ['123','231','121']
out = open("output.txt", "w")
for i in zipCodes:
link = "http://api.openweathermap.org/data/2.5/weather?zip=" + i + ",us&appid={api-key}"
x = urllib.request.urlopen(link)
url = x.read()
out.write(str(url, 'utf-8'))
returnJson = json.loads(url)
print(returnJson["main"]["temp"])
out.close()
You can achieve what you want by looping through a list of zipcodes and creating a new URL from them.
I am using python3.7 and running my code in Unix environment.
In my code, I have to hit some api and retrieve the data in json format, however when my source data has the japenese or non ascii characters, then it is not able to get the data form the request. Same api call when i make through the postman, it is returning me data.
Do i need to make any encoding changes if i have non -ascii characters in the api request?
bash-4.2$ more sourcefile.csv
"ひとみ","Abràmoff","70141558"
import requests
import csv
with open('sourcefile.csv', "r" ) as csv_file:
csv_reader = csv.reader(csv_file, delimiter=',')
for lines in csv_reader:
FRST_NM = (lines[0])
LST_NM = (lines[1])
ID = (lines[2])
URL2 ='<base url>?filter=(equals(FirstName,"' + FRST_NM + '"))and(equals(LastName,"' + LST_NM + '"))and(equals(ID,"' + ID + '"))'
full_data = requests.get(URL2)
print(full_data.json())
This is returning [], however it should return the data.
You can use BeautifulSoup https://www.dataquest.io/blog/web-scraping-beautifulsoup/
from that you can extract the information .
So I'm trying to read an excel file and pass it into a curl command. So in my curl command, I need 4 things. I need the name, path,Tname, and format.
So for me to run this script all I need to do is parse the excel file. It should get the data by parsing it by column. So in the column, I have the headers such as name, path, and format. I'm trying to make it so that it reads the column and takes the data from there and sends it into the curl command. Once it passes then it should go to the next row and do the same until it gets to the end.
So my curl command I know the problem that needs to be fixed. I'm not concerned about the curl command at the moment. I'm more concern on how to get the data and sending it into the curl command.
I've gotten the excel sheet by using pandas. once I did that I appended each column into an array. Then in my curl command, I'm trying to pass it.
I'm using * in my url and user and pass for privacy reasons. Like I said i'm not concern about the curl command. I'm trying to fix my code on getting the data and sending it into the curl command.
import requests
import json
import pprint
import urllib
import sys
import pandas as pd
path='data.xlsx'
excel=pd.read_excel(path)
name = []
path = []
Tname = []
formatG = []
for cell in excel:
name.append(cell[:0]),
path.append(cell[:1]),
Tname.append(cell[:2]),
formatG.append(cell[:3])
url='*'
data={
'name':'{}'.format(name),
'path':'{}'.format(path),
'TName':'{}'.format(Tname),
'formatG':'{}'.format(formatG)
}
djson=json.dumps(data)
headers={'content-type':'application/json','Authorization':'token' {}.format(fk)}
response=requests.post(url,data=data_json,headers=headers)
pprint.pprint(response.json())
if you have a csv file has comma separated data like below for example:
name,path,Tname,formatG
tom,vv,teto,mp4
mane,bb,memo,png
then you can use a code like below:
data = []
with open('myfile.txt') as f:
for line in f:
data.append(line.strip().split(','))
data = data[1:] # get the data without the first row which is data headers
print(data)
for entry in data:
name, path, Tname, formatG = entry
# do your curl stuff
data looks like:
[['tom', 'vv', 'teto', 'mp4'], ['mane', 'bb', 'memo', 'png']]
Try this.
data_list = []
for cell in excel: data_list.append({'name':cell[:0],'path':cell[:1]),'TName':cell[:2]),'formatG':cell[:3]})
url='*'
headers={'content-type':'application/json','Authorization':'token' {}.format(fk)}
for i in data_list: response=requests.post(url,data=i,headers=headers)
Let me know if it worked.
I have a question regarding appending to text file. I have written a script and what this script does is that it will read the URL in JSON format and extract the list of titles and write into the file "WordsInCategory.text".
As this code will be used in a loop thus I used f1 = open('WordsInCategory.text', 'a').
But I encountered a problem, that is it will add in already existing title into the file.
I am having trouble coming out with a solution to solve this problem and using 'w' will overwrite what it is written.
My code is as follows:
import urllib2
import json
url1 ='https://en.wikipedia.org/w/api.php?action=query&format=json&list=categorymembers&cmtype=page&cmtitle=Category:Geography&cmlimit=100'
json_obj = urllib2.urlopen(url1)
data1 = json.load(json_obj)
f1 = open('WordsInCategory.text', 'a')
for item in data1['query']:
for i in data1['query']['categorymembers']:
f1.write((i['title']).encode('utf8')+"\n")
Please advice on how I should modify my code.
Thank you.
I would suggest saving every title in an array, before writing to a file (and hence writing only once to the given file). You can modify your code this way :
import urllib2
import json
data = []
f1 = open('WordsInCategory.text', 'w')
url1 ='https://en.wikipedia.org/w/api.php?\
action=query&format=json&list=categorymembers\
&cmtype=page&cmtitle=Category:Geography&cmlimit=100'
json_obj = urllib2.urlopen(url1)
data1 = json.load(json_obj)
for item in data1['query']:
for i in data1['query']['categorymembers']:
data.append(i['title'].encode('utf8')+"\n")
# Do additional requests, and append the new titles to the data array
f1.write(''.join(set(data)))
f1.close()
set allows me to delete any duplicate entry.
If keeping the titles in memory is a problem, you can check if the title already exists before writing it to the file, but it may be awfully time consuming :
import urllib2
import json
data = []
url1 ='https://en.wikipedia.org/w/api.php?\
action=query&format=json&list=categorymembers\
&cmtype=page&cmtitle=Category:Geography&cmlimit=100'
json_obj = urllib2.urlopen(url1)
data1 = json.load(json_obj)
for item in data1['query']:
for i in data1['query']['categorymembers']:
title = (i['title'].encode('utf8')+"\n")
with open('WordsInCategory.text', 'r') as title_check:
if title not in title_check:
data.append(title)
with open('WordsInCategory.text', 'a') as f1:
f1.write(''.join(set(data)))
# Handle additional requests
Hope it'll be helpful.
You can track the titles you added.
titles = []
and then add each title to the list when writing
if title not in titles:
# write to file
titles += title
I’m trying to write a python script that sends a query to TweetSentiments.com API.
The idea is that it will perform like this –
Reads CSV tweet file > construct query from tweet file > Interrogates API > format JSON response > write API results to CSV file.
So far I’ve come up with this –
import csv
import urllib
import simplejson as json
import os
Tweets=[] ## Creates empty list to store tweets.
TweetWriter = csv.writer(open('test.csv', 'w'), dialect='excel', delimiter=' ',quotechar='|')
TweetReader = csv.reader(open("C:\StoredTweets.csv", "r"))
for row in TweetReader:
#TweetList.append(rows)
Tweets.append({ 'tweet': row[0], 'date': row[1] }) ## Stores from CSV in list.
for rows in Tweets:
#print TweetList
data = urllib.urlencode({'Tweet': row[0], 'Date': row[1]}) ##Takes Tweet and date to construct query.
#print data
API_request = urllib.urlopen("http://data.tweetsentiments.com:8080/api/analyze.json?q=", data) ## Adds query to end of URL and queries API
result = json.load(API_request('{"sentiment":}') ## "Sentiment": is a heading in the JSON response
TweetWriter.writerow(result) ## Writes API Response to CSV file.
I’m having trouble decoding the json. I’m confused with the json.load part … should I be referring to a heading in the json response and then trying to store it to a variable? Meaning that I can write this to a CSV file?
The response I get from the API is –
{"sentiment":{"value":1,"name":"Positive"}}
So I figure referring to “Sentiment” would store the whole reply.
Sorry, I’m just getting really confused. Read so many different forum posts and tutorials.
Your advice and help would be greatly appreciated!