Develop Reference

How can I get my API upload multiple files with a certain extension to work?

I'm new to programming and am trying to resolve this script to upload data, but I can't get it to work and I don't know why. I am modifying a script I already had to work with a single file, but I need to upload over 3,000 files with a specific extension (.json) in the specific file directory. I am getting an error on the 'f.read(open(files, 'r')) line. The error I get is there is no read extension for f. Not sure what I am doing incorrectly. I've researched it and still can't fix it. Any help would be appreciated.
import requests
import time
import csv
import json
import glob
from glob import glob
# function to post data
def postData(xactData):
url = 'api address'
headers = {
'Content-Type': 'application/json',
'Content-Length': str(len(xactData)),
'Request-Timeout': '2000000000'
}
return requests.post(url, headers=headers, data=xactData)
# read data
f = r'path to file'
# iterate over all files with extension path .json
for files in glob(f + '/*.json'):
my_data = f.read(open(files, 'r'))
print(files) # print the files that have been uploaded to the api
print(my_data) # print the data uploaded to the api
# post the data to the database
result = postData(my_data)
print(result)

How to close a file which is opened to read in an API Call

import glob
import os
import requests
import shutil
class file_service:
def file():
dir_name = '/Users/TEST/Downloads/TU'
os.chdir(dir_name)
pattern='TU_*.csv'
for x in glob.glob(pattern):
file_name=os.path.join(dir_name,x)
print (file_name)
from datetime import date
dir_name_backup = '/Users/Zill/Downloads/backup'
today = date.today()
backup_file_name = f'Backup_TU_{today.year}{today.month:02}{today.day:02}.csv'
backup_file_name_directory= os.path.join(dir_name_backup,backup_file_name)
print(backup_file_name_directory)
newPath = shutil.copy(file_name, backup_file_name_directory)
url = "google.com"
payload = {'name': 'file'}
files = [
('file', open(file_name,'rb'))
]
headers = {
'X-API-TOKEN': '12312'
}
response = requests.request("POST", url, headers=headers, data = payload, files = files)
print(response.text.encode('utf8'))
files.close()
os.remove(file_name)
file()
To provide an overall context, I am trying to retrieve a file from my OS and using POST method I am trying to post the content of the file into an application. Its working as expected so far, the details are getting pushed into application as expected. As part of my next step I am trying to remove the file from my existing directory using os.remove(). But I am getting a Win32 error as my file is not closed when it was opened in read-only mode in the POST call. I am trying to close it but I am unable to do so.
Can anyone please help me out with it.
Thanks!

I'm not sure I understand your code correctly. Could you try replacing
files.close()
with
for _, file in files:
file.close()
and check if it works?
Explanation:
In
files = [('file', open(file_name,'rb'))]
you create a list containing exactly one tuple that has the string 'file' as first element and a file object as second element:
[('file', file_object)]
The loop takes the tuple from the list, ignores its first element (_), takes its second element, the file object, and uses its close method to close it.
I've just now realised the list contains only one tuple. So there's no need for a loop:
files[0][1].close()
should do it.
The best way would be to use with (the file gets automatically closed once you leave the with block):
payload = {'name': 'file'}
with open(file_name, 'rb') as file:
files = [('file', file)]
headers = {'X-API-TOKEN': '12312'}
response = requests.request("POST", url, headers=headers, data = payload, files = files)
print(response.text.encode('utf8'))
os.remove(file_name)

Using python to download table data without .csv file address provided

My purpose is to download the data from this website:
http://transoutage.spp.org/
When opening this website, in the bottom of web, there is a description used to illustrate how to auto-download the data. For example:
http://transoutage.spp.org/report.aspx?download=true&actualendgreaterthan=3/1/2018&includenulls=true
The code I wrote is this:
import requests
ul_begin = 'http://transoutage.spp.org/report.aspx?download=true'
timeset = '3/1/2018' #define the time, m/d/yyyy
fn = ['&actualendgreaterthan='] + [timeset] + ['&includenulls=true']
fn = ''.join(fn)
ul = ul_begin+fn
r = requests.get(ul, verify=False)
Since, if you input the web address,
http://transoutage.spp.org/report.aspx?download=true&actualendgreaterthan=3/1/2018&includenulls=true,
into the Chrome, it will auto-download the data in .csv file. I do not know how to continue my code.
Please help me!!!!

You need to write the response you receive to a file:
r = requests.get(ul, verify=False)
if 200 >= r.status_code <= 300:
# If the request has succeeded
file_path = '<path_where_file_has_to_be_downloaded>`
f = open(file_path, 'w+')
f.write(r.content)
f.close()
This will work properly if the csv file is small in size. but for large files, you need to use stream param to download: http://masnun.com/2016/09/18/python-using-the-requests-module-to-download-large-files-efficiently.html

How do I upload multiple files using the Flask test_client?

How can I use the Flask test_client to upload multiple files to one API endpoint?
I'm trying to use the Flask test_client to upload multiple files to a web service that accepts multiple files to combine them into one large file.
My controller looks like this:
#app.route("/combine/file", methods=["POST"])
#flask_login.login_required
def combine_files():
user = flask_login.current_user
combined_file_name = request.form.get("file_name")
# Store file locally
file_infos = []
for file_data in request.files.getlist('file[]'):
# Get the content of the file
file_temp_path="/tmp/{}-request.csv".format(file_id)
file_data.save(file_temp_path)
# Create a namedtuple with information about the file
FileInfo = namedtuple("FileInfo", ["id", "name", "path"])
file_infos.append(
FileInfo(
id=file_id,
name=file_data.filename,
path=file_temp_path
)
)
...
My test code looks like this:
def test_combine_file(get_project_files):
project = get_project_files["project"]
r = web_client.post(
"/combine/file",
content_type='multipart/form-data',
buffered=True,
follow_redirects=True,
data={
"project_id": project.project_id,
"file_name": "API Test Combined File",
"file": [
(open("data/CC-Th0-MolsPerCell.csv", "rb"), "CC-Th0-MolsPerCell.csv"),
(open("data/CC-Th1-MolsPerCell.csv", "rb"), "CC-Th1-MolsPerCell.csv")
]})
response_data = json.loads(r.data)
assert "status" in response_data
assert response_data["status"] == "OK"
However, I can't get the test_client to actually upload both files. With more than one file specified, the file_data is empty when the API code loops. I have tried my own ImmutableDict with two "file" entries, a list of file tuples, a tuple of file tuples, anything I could think of.
What is the API to specify multiple files for upload in the Flask test_client? I can't find this anywhere on the web! :(

The test client takes a list of file objects (as returned by open()), so this is the testing utility I use:
def multi_file_upload(test_client, src_file_paths, dest_folder):
files = []
try:
files = [open(fpath, 'rb') for fpath in src_file_paths]
return test_client.post('/api/upload/', data={
'files': files,
'dest': dest_folder
})
finally:
for fp in files:
fp.close()
I think if you lose your tuples (but keeping the open()s) then your code might work.

You should just send data object with your files named as you want:
test_client.post('/api/upload',
data={'title': 'upload sample',
'file1': (io.BytesIO(b'get something'), 'file1'),
'file2': (io.BytesIO(b'forthright'), 'file2')},
content_type='multipart/form-data')

Another way of doing this- if you want to explicitly name your file uploads here (my use case was for two CSVs, but could be anything) with test_client is like this:
resp = test_client.post(
'/data_upload_api', # flask route
file_upload_one=[open(FILE_PATH, 'rb')],
file_upload_two=[open(FILE_PATH_2, 'rb')]
)
Using this syntax, these files would be accessible as:
request.files['file_upload_one'] # etc.

Python: POST multiple multipart-encoded files

As described here, it is possible to send multiple files with one request:
Uploading multiple files in a single request using python requests module
However, I have a problem generating these multiple filehandlers from a list.
So let's say I want to make a request like this:
sendfiles = {'file1': open('file1.txt', 'rb'), 'file2': open('file2.txt', 'rb')}
r = requests.post('http://httpbin.org/post', files=sendfiles)
How can I generate sendfiles from the list myfiles?
myfiles = ["file1.txt", "file20.txt", "file50.txt", "file100.txt", ...]

Use a dictionary comprehension, using os.path.splitext() to remove those extensions from the filenames:
import os.path
sendfiles = {os.path.splitext(fname)[0]: open(fname, 'rb') for fname in myfiles}
Note that a list of 2-item tuples will do too:
sendfiles = [(os.path.splitext(fname)[0], open(fname, 'rb')) for fname in myfiles]
Beware; using the files parameter to send a multipart-encoded POST will read all those files into memory first. Use the requests-toolbelt project to build a streaming POST body instead:
from requests_toolbelt import MultipartEncoder
import requests
import os.path
m = MultipartEncoder(fields={
os.path.splitext(fname)[0]: open(fname, 'rb') for fname in myfiles})
r = requests.post('http://httpbin.org/post', data=m,
headers={'Content-Type': m.content_type})