Develop Reference

Why does server returns 500 response to post request

I am trying to download file from steamworkshopdownloader.io with requests but it always returns 500 error. What am I doing wrong? I am not very familiar with requests.
Code:
import requests
def downloadMap(map_id):
session = requests.session()
file = session.post("https://backend-02-prd.steamworkshopdownloader.io/api/details/file",
data={"publishedfileid": map_id})
print(file)
downloadMap("814218628")

If you want to download a file from this API try this code, it's adapted from the link in the comment I posted earlier (https://greasyfork.org/en/scripts/396698-steam-workshop-downloader/code) and converted into Python:
import requests
import json
import time
def download_map(map_id):
s = requests.session()
data = {
"publishedFileId": map_id,
"collectionId": None,
"extract": True,
"hidden": False,
"direct": False,
"autodownload": False
}
r = s.post('https://backend-01-prd.steamworkshopdownloader.io/api/download/request', data=json.dumps(data))
print(r.json())
uuid = r.json()['uuid']
data = f'{{"uuids":["{uuid}"]}}'
while True:
r = s.post('https://backend-01-prd.steamworkshopdownloader.io/api/download/status', data=data)
print(r.json())
if r.json()[uuid]['status'] == 'prepared':
break
time.sleep(1)
params = (('uuid', uuid),)
r = s.get('https://backend-01-prd.steamworkshopdownloader.io/api/download/transmit', params=params, stream=True)
print(r.status_code)
with open(f'./{map_id}.zip', 'wb') as f:
for chunk in r.iter_content(chunk_size=1024):
if chunk:
f.write(chunk)
download_map(814218628)
The code demonstrates how to use the API and downloads a file named 814218628.zip (or whatever map_id was provided) into the directory the script is run from, the zip archive contains the .udk file (Game map design created by the Unreal Engine Development Kit).

Upload a file and handle redirection [PYTHON]

I need to submit an image to the site https://zxing.org/w/decode.jspx, and read the result page : https://zxing.org/w/decode.
I tried this, but it does not work :
def decode_qrcode(path):
s = requests.Session()
url = "https://zxing.org/w/decode.jspx"
files = {'file': open(path, 'rb')}
s.post(url, files=files)
return s.get("https://zxing.org/w/decode").text
I know that there are librairies to read QR code, but I did not find any that works for the kind of QR codes that I work with (they might have an error rate not supported).

You have to use the allow_redirects argument when making the POST request
import requests
def decode_qrcode(path):
s = requests.Session()
url = "https://zxing.org/w/decode.jspx"
files = {'file': open(path, 'rb')}
s.post(url, files=files, allow_redirects = True)
return s.get("https://zxing.org/w/decode").text

Cannot Post CSV file in Python

I'm having problems with a CSV file upload with requests.post method in python 3.
from requests.auth import HTTPBasicAuth
import csv
import requests
user='myuser'
pw='mypass'
advertiserid='10550'
campaignid='12394'
url='http://example.example.com/api/edc/upload/'+advertiserid+'/'+campaignid+'/'+'?encoding=utf-8&fieldsep=%3B&decimalsep=.&date=DD%2FMM%2FYYYY&info=1&process=1'
csv="myfile.csv"
with open(csv, 'r') as f:
r = requests.post(url, files={csv: f})
print(r)
The output is 'Response [502]'
Any idea of what could be the problem?
Many thanks!

You can refer the documentation of Requests library here: post-a-multipart-encoded-file
Change your request line to:
r = requests.post(url, files={'report.csv': f})

Try opening it in binary mode? And with specific 'text/csv' mime type?
with open(csv, 'rb') as f:
r = requests.post(url, files={'file': ('myfile.csv', f, 'text/csv', {'Expires': '0'})})
print(r.text)
If it still does not work, try without the binary, but still with the rest.
If it stiiill does not work, print the exact error message. And 502 (Bad Gateway) might just mean that you're not targetting the right url. (you're not targetting example.com, right?

csv="myfile.csv"
url='http://example.example.com/api/edc/upload/'+advertiserid+'/'+campaignid+'/'+'?encoding=utf-8&fieldsep=%3B&decimalsep=.&date=DD%2FMM%2FYYYY&info=1&process=1'
files = {'upload_file': open(csv,'rb')}
r = requests.post(url, files=files)

Imagine I have a rest API to import the CSV file (Multipart encoded file)
corresponding python request should be like below.
import requests
​
hierarchy_file_name = '/Users/herle/ws/LookoutLab/data/monitor/Import_Hierarchy_Testcase_v2.csv'
headers = {
'x-api-key': **REST_API_KEY**,
'x-api-token': **REST_API_TOKEN**,
'accept': 'application/json'
}
files = {'file': (hierarchy_file_name, open(hierarchy_file_name, 'rb'), 'text/csv')}
url = "https://abcd.com"
response = requests.post(url +'/api/v2/core/workspaces/import/validate',
files=files, verify=False, headers=headers)
print("Created")
print(response)
print(response.text)
Note:
Make sure that you don't add 'Content-Type': 'multipart/form-data' in the header

subprocess.call() formatting issue

I am very new to subprocess, and I have a hard debugging it without any error code.
I'm trying to automatically call an API which respond to :
http -f POST https://api-adresse.data.gouv.fr/search/csv/ columns=voie columns=ville data#path/to/file.csv > response_file.csv
I've tried various combination with subprocess.call, but I only manage to get "1" as an error code.
What is the correct way to format this call, knowing that the answer from the API has to go in a csv file, and that I send a csv (path after the #data)?
EDIT: Here are my attempts :
ret = subprocess.call(cmd,shell=True)
ret = subprocess.call(cmd.split(),shell=True)
ret = subprocess.call([cmd],shell=True)
The same with shell = False, and with stdout = myFileHandler (open inside a with open(file,"w") as myFileHandler:)
EDIT2 : still curious about the answer, but I managed to go around with Request, as #spectras suggested
file_path = "PATH/TO/OUTPUT/FILE.csv"
url = "https://api-adresse.data.gouv.fr/search/csv/"
files = {'data': open('PATH/TO/CSV/FILE.csv','rb')}
values = {'columns': 'Adresse', 'columns': 'Ville', 'postcode': 'CP'}
r = requests.post(url, files=files, data=values)
with open(file_path, "w") as myFh:
myFh.write(r.content)

Since you are attempting to send a form, may I suggest you do it straight from python?
import requests
with open('path/to/file', 'rb') as fd:
payload = fd.read()
r = requests.post(
'https://api-adresse.data.gouv.fr/search/csv/',
data=(
('columns', 'voie'),
('columns', 'ville'),
),
files={
'data': ('filename.csv', payload, 'text/csv'),
}
)
if r.status_code not in requests.codes.ok:
r.raise_for_status()
with open('response_file.csv', 'wb') as result:
result.write(r.content)
This uses the ubiquitous python-requests module, and especially the form file upload part of the documentation.
It's untested. Basically, I opened httpie documentation and converted your command line arguments into python-requests api arguments.

Python append json to json file in a while loop

I'm trying to get all users information from GitHub API using Python Requests library. Here is my code:
import requests
import json
url = 'https://api.github.com/users'
token = "my_token"
headers = {'Authorization': 'token %s' % token}
r = requests.get(url, headers=headers)
users = r.json()
with open('users.json', 'w') as outfile:
json.dump(users, outfile)
I can dump first page of users into a json file by now. I can also find the 'next' page's url:
next_url = r.links['next'].get('url')
r2 = requests.get(next_url, headers=headers)
users2 = r2.json()
Since I don't know how many pages yet, how can I append 2nd, 3rd... page to 'users.json' sequentially in a while loop as fast as possible?
Thanks!

First, you need to open file in 'a' mode, otherwise subsequence write will overwrite everything
import requests
import json
url = 'https://api.github.com/users'
token = "my_token"
headers = {'Authorization': 'token %s' % token}
outfile = open('users.json', 'a')
while True:
r = requests.get(url, headers=headers)
users = r.json()
json.dump(users, outfile)
url = r.links['next'].get('url')
# I don't know what Github return in case there is no more users, so you need to double check by yourself
if url == '':
break
outfile.close()

Append the data you get from the requests query to a list and move on to the next query.
Once you have all of the data you want, then proceed to try to concatenate the data into a file or into an object. You can also use threading to do multiple queries in parallel, but most likely there is going to be rate limiting on the api.