I am posting to Hudson server using curl from the command line using the following--
curl -X POST -d '<run><log encoding="hexBinary">4142430A</log><result>0</result><duration>2000</duration></run>' \
http://user:pass#myhost/hudson/job/_jobName_/postBuildResult
as shown in the hudson documentation..can I emulate the same thing using python..i don't want to use pyCurl or send this line through os.system()..is there ny way out using raw python??
import urllib2
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
result = response.read()
where data is the encoded data you want to POST.
You can encode a dict using urllib like this:
import urllib
values = { 'foo': 'bar' }
data = urllib.urlencode(values)
The modern day solution to this is much simpler with the requests module (tagline: HTTP for humans! :)
import requests
r = requests.post('http://httpbin.org/post', data = {'key':'value'}, auth=('user', 'passwd'))
r.text # response as a string
r.content # response as a byte string
# gzip and deflate transfer-encodings automatically decoded
r.json() # return python object from json! this is what you probably want!
Related
Can you please help me convert the following CURL command into a command for Python's requests library?
curl -F pdb_file[pathvar]=#/path/myfile.pdb -X POST https://proteins.plus/api/pdb_files_rest -H "Accept: application/json"
I've tried many things, like this:
import requests
file = open("3w32.pdb", "rb")
url = "https://proteins.plus/api/pdb_files_rest"
response = requests.post(url, data=file)
response.json()
or this:
import requests
file = open("3w32.pdb", "rb")
url = "https://proteins.plus/api/pdb_files_rest"
response = requests.post(url, data={"pdb_file[pathvar]": file}, json={"Accept": "application/json"})
response.json()
but I can't make it work. I either get an error from the server, or a JSONDecodeError in Python.
Try this answer https://stackoverflow.com/a/22567429/4252013
Instead of using "data" argument try with "file"
Ok, I managed to solve it at last.
The problem was that the 'pdb_file[pathvar]' part in the curl command should've been added to the request, like this:
import requests
file = open("3w32.pdb", "rb")
url = "https://proteins.plus/api/pdb_files_rest"
response = requests.post(url, files={'pdb_file[pathvar]':file})
response.json()
Now it works!
The problem in your code is that you have only opened the file and passed the pointer. you can pass the file contents by having file.read() in place.
Please refer this block of code:
import requests
file = open("3w32.pdb", "rb")
url = "https://proteins.plus/api/pdb_files_rest"
response = requests.post(url, data=file.read())
response.json()
I am trying to translate a specific curl method into Python's requests module to upload a file to to an api. My standard method that works for non-file requests looks like this:
import requests
requestObject = requests.Session()
standard_headers = {header1:headerValue1,header2:headerValue2}
payload = {key1:value1,key2:value2}
url = 'https://myUrl.com/apiCall'
requestObject.post(url,headers=standard_headers, json=payload)
This works for non-file requests that I need to make to the API. However for file uploads, the API documentation shows a method using curl:
curl -XPOST -H 'header1' -H 'header2 'https://myUrl.com/apiCall' \
-F 'attachment=#/path/to/my/file' \
-F '_json=<-;type=application/json' << _EOF_
{
"key1":"keyValue1",
"key2":"keyValue2"
}
_EOF_
I tested the curl command and it works successfully.
My question is how do I translate that curl method using the << _EOF_ method in Python requests. One idea I had was simply to use the 'files' option in the requests module:
requestObject = requests.Session()
standard_headers = {header1:headerValue1,header2:headerValue2}
payload = {key1:keyValue1,key2:keyValue2}
url = 'https://myUrl.com/apiCall'
file_to_upload = {'filename': open('/path/to/my/file', 'rb')}
requestObject.post(url,headers=standard_headers, files=file_to_upload, json=payload)
But that does not seem to work as the necessary json parameters (the values in payload) do not appear to get passed to the file upload
I also tried specifying the json parameters directly into the file_to_upload variable:
requestObject = requests.Session()
standard_headers = {header1:headerValue1,header2:headerValue2}
url = 'https://myUrl.com/apiCall'
file_to_upload = {'attachment': open('/path/to/my/file', 'rb'),'{"key1":"keyValue1","key2":"keyValue2"}'}
requestObject.post(url,headers=standard_headers, files=file_to_upload)
Similar result, it seems as though I am not passing the necessary json values correctly. I tried a few other ways but I am overlooking something. Any insight into how I should structure my request is appreciated.
Ok I managed to get it to work and posting for anyone who might need help in the future.
The trick was to include the _json key in the data field. My code ended up looking like so:
import requests
requestObject = requests.Session()
standard_headers = {header1:headerValue1,header2:headerValue2}
json_fields = json.dumps({
"key1": "key1Value",
"key2": "key2Value"
})
payload = {"_json":json_fields)
file = {"attachment": /path/to/my/file}
url = 'https://myUrl.com/apiCall'
requestObject.post(url,headers=standard_headers, files=file, data=payload)
Hope that helps some future person.
I am using requests to make a POST request to create a user. The request succeeds with 201 created when I use curl, however fails with a 500 response when I use requests. My curl command is
curl --user administrator:password -H "Content-Type: application/json" https://localhost:8080/midpoint/ws/rest/users -d #user.json -v
And my python script is:
import requests
import json
headers = {
'Content-Type': 'application/json',
}
with open('user.json') as j:
data = json.load(j)
response = requests.post('https://localhost:8080/midpoint/ws/rest/users', headers=headers, data=str(data), auth=('Administrator', 'password'))
print(response)
Can anyone see a reason why my python script would be failing? I am at a loss.
str(data) returns the Python representation of data, not its JSON representation. These two forms can differ in things like ' vs. ", True vs. true, and None vs. null. To properly JSONify data, call json.dumps() on it:
response = requests.post(..., data=json.dumps(data))
or let requests do the JSONification:
response = requests.post(..., json=data)
or use the JSON as it appears in user.json directly:
with open('user.json') as j:
data = j.read()
response = requests.post(..., data=data)
Using curl I get the hash
$ curl "http://127.0.0.1:8002/hash/" -X POST -d 'data={"key":"value"}'
{
"hash": "9724c1e20e6e3e4d7f57ed25f9d4efb006e508590d528c90da597f6a775c13e5"
}
Using Python requests return 400
import json
import requests
r = requests.post('http://127.0.0.1:8002/hash/', data = 'data={"key":"value"}', allow_redirects=False)
Try passing it in with just {"key":"value"} as your data.
So that would be:
r = requests.post('http://127.0.0.1:8002/hash/', data = '{"key":"value"}', allow_redirects=False)
requests can also accept a dictionary object or a list of tuples as it's data argument, which is probably easier and more intuitive than directly entering a string. Also note that you can work with a dictionary and turn it into a json string with json.dumps.
I am experimenting with the census bulk geocode API documentation
The following curl command works:
curl --form addressFile=#Addresses.csv --form benchmark=9 http://geocoding.geo.census.gov/geocoder/locations/addressbatch --output geocoderesult.csv
But when I attempt to port this to python requests:
url = 'http://geocoding.geo.census.gov/geocoder/geographies/addressbatch'
payload = {'benchmark':9}
files = {'addressFile': ('Addresses.csv', open('Addresses.csv', 'rb'), 'text/csv')}
r = requests.post(url, files=files, data = payload)
print r.text
I am apparently not sending a well formed request and only receiving "There was an internal error" in response. Any idea what I am doing wrong in forming this request?
Got it! Turns out that the geographies request type required some parameters that the locations type did not. Working solution:
url = 'http://geocoding.geo.census.gov/geocoder/geographies/addressbatch'
payload = {'benchmark':'Public_AR_Current','vintage':'ACS2013_Current'}
files = {'addressFile': ('Addresses.csv', open('Addresses.csv', 'rb'), 'text/csv')}
r = requests.post(url, files=files, data = payload)
print r.text
May be this is a simpler way to do the same thing.
You will get a clean output in pandas dataframe :)
# pip install censusgeocode
import censusgeocode
import pandas as pd
cg = censusgeocode.CensusGeocode(benchmark='Public_AR_Census2010',vintage='Census2010_Census2010')
k = cg.addressbatch('D:\WORK\Addresses.csv')
# Bonus
# Get clean output in Dataframe
df = pd.DataFrame(k, columns=k[0].keys())
# PS: I tried with 9990 records in single batch
Reference:
https://pypi.org/project/censusgeocode/
https://geocoding.geo.census.gov/geocoder/benchmarks
https://geocoding.geo.census.gov/geocoder/vintages?form
https://geocoding.geo.census.gov/geocoder/geographies/addressbatch?form
Works great. Today I just used the code shown below.
url = 'https://geocoding.geo.census.gov/geocoder/locations/addressbatch'
payload = {'benchmark':'Public_AR_Current','vintage':'ACS2013_Current'}
files = {'addressFile': ('19067.csv', open('19067.csv', 'rb'), 'text/csv')}
r = requests.post(url, files=files, data = payload)