Hi I have created a piece of code that downloads data from a api end point and also loads in the apikeys.
I am trying to achieve downloading the api data into csv files into their own folder based on the input.csv I have tried to achieve this by adding the following section at the end. The problem is that it does not download the file its to be receiving from the api end point.
Please assist?
with open('filepath/newfile.csv', 'w+') as f:
f.write(r.text)
import csv
import sys
import requests
def query_api(business_id, api_key):
headers = {
"Authorization": api_key
}
r = requests.get('https://api.link.com', headers=headers)
print(r.text)
# get filename from command line arguments
if len(sys.argv) < 2:
print ("input.csv")
sys.exit(1)
csv_filename = sys.argv[1]
with open(csv_filename) as csv_file:
csv_reader = csv.DictReader(csv_file, delimiter=',')
for row in csv_reader:
business_id = row['BusinessId']
api_key = row['ApiKey']
query_api(business_id, api_key)
with open('filepath/newfile.csv', 'w+') as f:
f.write(r.text)
Related
I am currently trying to use a csv file (input.csv) to load multiple api keys so i can download multiple difference documents using the same the api link but the system changes the api key automatically based of input.csv. The input.csv is in the same location as the python script. Also trying to get python to save to a specific location. Any help would be massively appreciated.
I am currently getting the following error when running the script:
import csv
import sys
import requests
def query_api(business_id, api_key):
headers = {
"Authorization": api_key
}
r = requests.get('https://api.link.com', headers=headers)
print(r.text)
# get filename from command line arguments
if len(sys.argv) < 2:
print "input.csv"
sys.exit(1)
csv_filename = sys.argv[1]
with open(csv_filename) as csv_file:
csv_reader = csv.DictReader(csv_file, delimiter=',')
for row in csv_reader:
business_id = row['BusinessId']
api_key = row['ApiKey']
query_api(business_id, api_key)
I am currently getting the following error when running the script:
line 12
print "input.csv"
^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("input.csv")?
I'm having trouble organizing my CSV file full of urls and downloading each image per url.
https://i.imgur.com/w1slgf6.png
It's quite hell, but the goal is to:
Write the src of these images into a csv file, splitting each url per line.
And download each image
from selenium import webdriver
from bs4 import BeautifulSoup
from time import sleep
import urllib.request
import pandas as pd
import requests
import urllib
import csv
# BeautifulSoup4 findAll src from img
print ('Downloading URLs to file')
sleep(1)
with open('output.csv', 'w', newline='\n', encoding='utf-8') as csvfile:
writer = csv.writer(csvfile)
writer.writerow(srcs)
print ('Downloading images to folder')
sleep(1)
filename = "output"
with open("{0}.csv".format(filename), 'r') as csvfile:
# iterate on all lines
i = 0
for line in csvfile:
splitted_line = line.split(',')
# check if we have an image URL
if splitted_line[1] != '' and splitted_line[1] != "\n":
urllib.request.urlretrieve(splitted_line[1], "img_" + str(i) + ".png")
print ("Image saved for {0}".format(splitted_line[0]))
i += 1
else:
print ("No result for {0}".format(splitted_line[0]))
Base on the limited resources that you provided, I think this is the code that you need:
import requests
with open('output.csv', 'r') as file:
oldfile = file.read()
linkslist = oldfile.replace("\n", "") # Because your file is wrongly splitted by new lines so I removed it
links = linkslist.split(",")
with open('new.csv', 'w') as file: # Writing all your links to a new file, this can combine with the below code but I think open file and requests at the same time will make it slower
for link in links:
file.write(link + "\n")
for link in links:
response = requests.get(link) # This is to save image
file = open("(yourfilenamehere).png", "wb") # Replace the name that you want for the picture in here
file.write(response.content)
file.close()
Please find comments of explanation inside the code, If you have any problem, just ask, I haven't tested it because I don't have your exact CSV but it should work
I'm new to python and really struggling with writing a program to download a CSV from this page
So far I have:
import csv
import requests
import os
output_dir = 'C:/Users/Moshe/Downloads'
output_file = 'Covid_19_uk_timeseries.csv'
CSV_URL = 'https://coronavirus.data.gov.uk/downloads/csv/coronavirus-deaths_latest.csv'
assert os.path.exists(output_dir)# test that we can write to output_dir
with requests.Session() as s:
download = s.get(CSV_URL)
decoded_content = download.content.decode('utf-8')
cr = csv.reader(decoded_content.splitlines(), delimiter=',')
my_list = list(cr)
for row in my_list:
print(row)
Its output show that it gets the CSV fine, and has access to the output directory. But for now I just want to save it as it is, and can't seem to work out how to do that.
def main():
import requests
url = "https://coronavirus.data.gov.uk/downloads/csv/coronavirus-deaths_latest.csv"
with requests.get(url, stream=True) as response:
response.raise_for_status()
with open("deaths_latest.csv", "wb") as file:
for chunk in response.iter_content(chunk_size=8192):
file.write(chunk)
file.flush()
print("Done")
return 0
if __name__ == "__main__":
import sys
sys.exit(main())
You can use
csv.writerows()
to write the csv file. Check this site for more. https://www.programiz.com/python-programming/writing-csv-files
The code below downloads the pictures as .png filetype to my computer. However, when I try to open the images, the image viewer says: "It looks like we don't support this file format". I used another app to open the image and the problem persisted. It seems that the files are downloaded as 'bytes object' instead of as images..
import csv
import requests
with open('rlth.csv', 'r', newline='') as file:
has_header = csv.Sniffer().has_header(file.read(1024))
file.seek(0) # Rewind.
reader = csv.reader(file)
if has_header:
next(reader) # Skip header row.
csvrows = csv.reader(file, delimiter=',', quotechar='"')
for row in csvrows:
filename = row[0]
url = row[2]
print(url)
result = requests.get(url, stream=True)
if result.status_code == 200:
image = result.raw.read()
open("{}.png".format(filename),"wb").write(image)
I'm trying to automatically download data from the following website; however I just get the html and no data:
http://tcplus.com/GTN/OperationalCapacity#filter.GasDay=02/02/19&filter.CycleType=1&page=1&sort=LocationName&sort_direction=ascending
import csv
import urllib2
downloaded_data = urllib2.urlopen('http://tcplus.com/GTN/OperationalCapacity#filter.GasDay=02/02/19&filter.CycleType=1&page=1&sort=LocationName&sort_direction=ascending')
csv_data = csv.reader(downloaded_data)
for row in csv_data:
print row
The code below will only fetch data from provided url, but if you tweak parameters you can get other reports as well.
import requests
parameters = {'serviceTypeName': 'Ganesha.InfoPost.Service.OperationalCapacity.OperationalCapacityService, Ganesha.InfoPost.Service',
'filterTypeName': 'Ganesha.InfoPost.ViewModels.GasDayAndCycleTypeFilterViewModel, Ganesha.InfoPost',
'templateType': 6,
'exportType': 1,
'filter.GasDay': '02/02/19',
'filter.CycleType': 1}
response = requests.post('http://tcplus.com/GTN/Export/Generate', data=parameters)
with open('result.csv', 'w') as f:
f.write(response.text)