I have this simple working curl command:
curl -k -d X-User=user -d X-Password=password https://12.12.12.21
This is my example:
import urllib2
opener = urllib2.build_opener()
opener.addheaders = [('X-User', 'user'),('X-Password', 'password')]
rr = opener.open("https://12.12.12.21")
print rr.read()
It's not working as i expected ( result: wrong password/user name ), can you help me understand why?
Your curl command is using the -d flag which sends the data using POST, not using headers.
If you meant to use headers then you need to use the -H argument:
curl -k -H X-User=user -H X-Password=password https://12.12.12.21
Here is how to do a POST request in case that is what you need:
values = {'X-User' : 'user', 'X-Password' : 'password'}
data = urllib.urlencode(values)
req = urllib2.Request("https://12.12.12.21", data)
rr = urllib2.urlopen(req)
Related
I am using FastAPI to serve some ML models and I have a Streamlit basic UI using Python Requests module.
One of my service is getting an image through a POST request and it is working like a charm.
Server side
#app.post("/segformer")
async def segformer(file: Optional[UploadFile] = None):
curl given by {BASE_URI}/docs
curl -X 'POST' \
'http://localhost:8080/segformer' \
-H 'accept: application/json' \
-H 'Content-Type: multipart/form-data' \
-F 'file=#image1.jpg;type=image/jpeg'
Client side using Python Requests
response = requests.post(f"{BASE_URI}/segformer", files=files)
As soon as I want to add additional parameters, it gets creepy. For example:
Server side
#dataclass
class ModelInterface:
model_name: str = "detr"
tags: List[str] = Query(...)
#app.post("/detr")
async def detr(file: UploadFile = File(...), model: ModelInterface = Depends()):
curl -- given by {BASE_URI}/docs
curl -X 'POST' \
'http://localhost:8080/detr?model_name=detr&tags=balloon&tags=dog' \
-H 'accept: application/json' \
-H 'Content-Type: multipart/form-data' \
-F 'file=#image1.jpg;type=image/jpeg'
Until that point everything works fine. Once I want to convert that call using Python Requests, I have some issues, which always end up in /detr HTTP/1.1" 400 Bad Request error.
Here is what I've tried:
Client side using Python Requests
headers = {
"Content-type": "multipart/form-data",
"accept": "application/json"
}
payload = {
"tags": ["balloon", "dog"]
}
response = requests.post(
f"{BASE_URI}/detr",
headers=headers,
json=payload, <- also *data*
files=files,
)
At the end it seems that the problem narrows on how to convert this:
curl -X 'POST' \
'http://localhost:8080/detr?tags=balloon&tags=dog' \
-H 'accept: application/json' \
-H 'Content-Type: multipart/form-data' \
-F 'file=#image1.jpg;type=image/jpeg'
to a valid Python Requests call!
I also faced the issue that the following FastAPI code:
#dataclass
class ModelInterface:
model_name: str = "detr"
tags: List[str] = None
#app.post("/detr2")
async def detr2(file: UploadFile = File(...), model: ModelInterface = Form(...)):
...
translates to this curl command:
curl -X 'POST' \
'http://localhost:8080/detr2' \
-H 'accept: application/json' \
-H 'Content-Type: multipart/form-data' \
-F 'file=#image1.jpg;type=image/jpeg' \
-F 'model={
"model_name": "detr",
"tags": [
"balloon", "dog"
]
}'
which fails with a "POST /detr2 HTTP/1.1" 422 Unprocessable Entity error
To pass query parameters in Python requests, you should use params key instead. Hence:
response = requests.post(url='<your_url_here>', params=payload)
Additionally, there is no need to set the Content-type in the headers, as it will automatically be added, based on the parameters you pass to requests.post(). Doing so, the request will fail, as, in addition to multipart/form-data,"Content-type must include the boundary value used to delineate the parts in the post body. Not setting the Content-Type header ensures that requests sets it to the correct value" (ref). Have a look at this and this. Also, make sure that in files you use the same key name you gave in your endpoint, i.e., file. Thus, in Python requests it should look something like the below:
files = {('file', open('my_file.txt', 'rb'))}
response = requests.post(url='<your_url_here>', files=files, params=payload)
You could also find more options as to how to send additional data along with files at this answer.
I would like to "translate" my curl request into python using request or urllib.
I try to post a file.
My curl request is :
curl -X POST -H "Content-Type: multipart/form-data" -H "Authorization: Bearer $1" -F "data=#$2;filename=$3" --cert certificateprivate.pem --cacert MyDigiCertSHA2.crt.pem <my_url>
I tried the code below ( doesn't work , response 400)
def upload_file(token,file_path):
ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH,cafile='MyDigiCertSHA2.crt.pem')
ctx.load_cert_chain(certfile='certificateprivate.pem')
url = <my_url>
hdr = {"Content-Type": "multipart/form-data","Authorization":"Bearer "+token}
data = '{"filterList":[{}]}'
with open(file_path, 'rb') as f:
from urllib import request, parse
data = parse.urlencode( {'filename':file_path,"data": f.read()}).encode()
req = request.Request( url, data = data,headers=hdr)
resp = request.urlopen(req, context=ctx)
content = resp.read()
data = json.loads(content.decode('utf-8'))
print(data)
The curl command has "Content-Type: multipart/form-data", but your python code has "Content-Type": "application/json".
I'm trying to convert the following working request in curl to a python request (using Requests).
curl --data 'query={"tags":["test1","test2"]}' http://www.test.com/match
(I've used a fake url but the command does work with the real url)
The receiving end (ran in Flask) does this:
#app.route("/match", methods=['POST'])
def tagmatch():
query = json.loads(request.form['query'])
tags = query.get('tags')
# ... does stuff ...
return json.dump(stuff)
In curl (7.30), ran on Mac OS X (10.9) the command above properly returns a JSON list that's filtered using the tag query.
My Python script is as follows, it returns a 400 Bad Request error.
import requests
payload = {"tags":["test1", "test2"]}
# also tried payload = 'query={"tags":["test1","test2"]}'
url = 'http://www.test.com/match'
r = requests.post(url, data=payload)
if __name__=='__main__':
print(r.text)
There is an open source cURL to Python Requests conversion helper at https://curlconverter.com/. It isn't perfect, but helps out a lot of the time. Especially for converting Chrome "Copy as cURL" commands. There is also a node library if you need to do the conversions programmatically
Your server is expecting JSON, but you aren't sending it. Try this:
import requests
import json
payload = {'query': json.dumps({"tags":["test1", "test2"]})}
url = 'http://www.test.com/match'
r = requests.post(url, data=payload)
if __name__=='__main__':
print r.text
Save your life
A simpler approach would be:
Open POSTMAN
Click on the "import" tab on the upper left side.
Select the Raw Text option and paste your cURL command.
Hit import and you will have the command in your Postman builder!
Hope this helps!
credit: Onkaar Singh
Try to use uncurl library. It is pretty nice to do its job. I've tried it.
u = uncurl.parse(
"curl -X GET 'https://mytesturl.com/' -H 'accept: application/json' -H 'Authorization: 1234567890'")
print(u)
It prints,
requests.get("https://mytesturl.com/",
headers={
"Authorization": "1234567890",
"accept": "application/json"
},
cookies={},
)
try this:
https://github.com/spulec/uncurl
import uncurl
print uncurl.parse("curl 'https://pypi.python.org/pypi/uncurl' -H
'Accept-Encoding: gzip,deflate,sdch'")
I wrote an HTTP client plugin for Sublime Text called Requester, and one of its features is to convert calls to cURL to Requests, and vice versa.
If you're using Sublime Text this is probably your fastest, easiest option. If not, here's the code that actually handles the conversion from cURL to Requests. It's based uncurl, but with various improvements and bug fixes.
import argparse
import json
try:
from urllib.parse import urlencode, parse_qsl
except ImportError: # works for Python 2 and 3
from urllib import urlencode
from urlparse import parse_qsl
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('command')
parser.add_argument('url')
parser.add_argument('-X', '--request', default=None)
parser.add_argument('-d', '--data', default=None)
parser.add_argument('-G', '--get', action='store_true', default=False)
parser.add_argument('-b', '--cookie', default=None)
parser.add_argument('-H', '--header', action='append', default=[])
parser.add_argument('-A', '--user-agent', default=None)
parser.add_argument('--data-binary', default=None)
parser.add_argument('--compressed', action='store_true')
parsed_args = parser.parse_args()
method = 'get'
if parsed_args.request:
method = parsed_args.request
base_indent = ' ' * 4
post_data = parsed_args.data or parsed_args.data_binary or ''
if post_data:
if not parsed_args.request:
method = 'post'
try:
post_data = json.loads(post_data)
except ValueError:
try:
post_data = dict(parse_qsl(post_data))
except:
pass
cookies_dict = {}
if parsed_args.cookie:
cookies = parsed_args.cookie.split(';')
for cookie in cookies:
key, value = cookie.strip().split('=')
cookies_dict[key] = value
data_arg = 'data'
headers_dict = {}
for header in parsed_args.header:
key, value = header.split(':', 1)
if key.lower().strip() == 'content-type' and value.lower().strip() == 'application/json':
data_arg = 'json'
if key.lower() == 'cookie':
cookies = value.split(';')
for cookie in cookies:
key, value = cookie.strip().split('=')
cookies_dict[key] = value
else:
headers_dict[key] = value.strip()
if parsed_args.user_agent:
headers_dict['User-Agent'] = parsed_args.user_agent
qs = ''
if parsed_args.get:
method = 'get'
try:
qs = '?{}'.format(urlencode(post_data))
except:
qs = '?{}'.format(str(post_data))
print(post_data)
post_data = {}
result = """requests.{method}('{url}{qs}',{data}\n{headers},\n{cookies},\n)""".format(
method=method.lower(),
url=parsed_args.url,
qs=qs,
data='\n{}{}={},'.format(base_indent, data_arg, post_data) if post_data else '',
headers='{}headers={}'.format(base_indent, headers_dict),
cookies='{}cookies={}'.format(base_indent, cookies_dict),
)
print(result)
You could make a script with this code, e.g. curl_to_request.py, and call this script from the command line like so. It will work for both Python 2 and Python 3.
python curl_to_request.py curl -X POST -d 'key2=value2&key1=value1' 'http://httpbin.org/post'
python curl_to_request.py curl -X POST -H 'Content-Type: application/json' -d '{"key2": "value2", "key1": "value1"}' 'http://httpbin.org/post'
python curl_to_request.py curl -X POST -H 'Content-Type: application/json' -d '[1, 2, 3]' 'http://httpbin.org/post'
python curl_to_request.py curl -X POST -H 'Content-Type: application/json' -d '{"name": "Jimbo", "age": 35, "married": false, "hobbies": ["wiki", "pedia"]}' 'http://httpbin.org/post'
python curl_to_request.py curl -X GET 'http://httpbin.org/get?key2=value2&key1=value1'
python curl_to_request.py curl -X GET -H 'key1: value1' -H 'key2: value2' 'http://httpbin.org/headers'
python curl_to_request.py curl -X GET -b 'key1=value1;key2=value2' 'http://httpbin.org/cookies'
From your code using requests and in Flask, it seems like you don't post the right data format. The payload should be like this:
payload = {'query': {'tags': ['test1', 'test2']},}
This seems not normal as post data when using requests.post(). So if you have posted the html form here, it may have been more clear to solve the problem.
Here is another similar question: Using Python Requests to pass through a login/password
I have this curl command and I would like to know if I have converted it correctly to pycurl.
curl command
curl -D- -u fred:fred -X PUT --data{see below} -H "Content-Type:application/json" http://kelpie9:8081/rest/api/2/issue/QA-31
{
"fields":
{
"assignee":{"name":"harry"}
}
}
python code
def assign(self, key, name):
data = json.dumps({"fields":{"assignee":{"name":name}}})
c= pycurl.Curl()
c.setopt(pycurl.VERBOSE, 1)
c.setopt(pycurl.URL, "http://xxx/rest/api/2/issue/"+ key )
c.setopt(pycurl.HTTPHEADER, ['Content-Type: application/json', 'Accept: application/json'])
c.setopt(pycurl.USERPWD, "****")
c.setopt(pycurl.PUT, 1)
c.setopt(pycurl.POSTFIELDS,data)
c.perform(
Nopes. First off: use curl's --libcurl option to get a first template.
Then, "-X PUT" translates to CUSTOMREQUEST set to "PUT", only changing the actual method keyword.
I would personally accomplish this by using the remarkable requests library
import requests
import requests.auth
import json
def assign(key, name):
url = "http://xxx/rest/api/2/issue/" + key
headers = {'Content-type': 'application/json', 'Accept': 'application/json'}
data = json.dumps({"fields": {"assignee": {"name": name}}})
r = requests.put(url, data=data, headers=headers, auth=requests.auth.HTTPBasicAuth('fred', 'fred'))
print(r.status_code)
print(r.json())
curl -j --libcurl git.txt -D- -u fred:fred -X PUT --data"{\"fields\":{\"assignee\":{\"name\":\"FRED\"}}}" -H "Content-Type:application/json" http://kelpie9:8081/rest/api/2/issue/QA-31
this gives the template in git.txt .
Also in the data field the inner quotes must be escaped as shown above.
The working code is attached below.
def assign(self, key, name):
self._startCurl()
self.c.setopt(pycurl.URL, "http://xxx/rest/api/2/issue/"+ key )
self.c.setopt(pycurl.HTTPHEADER, ['Content-Type: application/json', 'Accept: application/json'])
self.c.setopt(pycurl.USERPWD, "fred:fred")
self.c.setopt(pycurl.CUSTOMREQUEST, "PUT")
data = json.dumps({"fields":{"assignee":{"name":name}}})
self.c.setopt(pycurl.POSTFIELDS,data)
self.c.perform()
self.c.close()
Thanks to Daniel Stenberg for pointing it out
I have a request that gets PayPal authentication. It is written in Curl and it works perfectly. Trying to rewrite it in Python leads to an error response(500000 internal error). Can anyone please direct me on how I would rewrite it or correct my existing code.
CURL
curl -s --insecure -H "X-PAYPAL-SECURITY-USERID: <user_id>" -H "X-PAYPAL-SECURITY-PASSWORD: <user_password>" -H "X-PAYPAL-SECURITY-SIGNATURE: <user_signature>" -H "X-PAYPAL-REQUEST-DATA-FORMAT: JSON" -H "X-PAYPAL-RESPONSE-DATA-FORMAT: JSON" -H "X-PAYPAL-APPLICATION-ID: APP-80W284485P519543T" https://svcs.sandbox.paypal.com/Permissions/RequestPermissions -d "{\"scope\":\"EXPRESS_CHECKOUT\", \"callback\":\"<callback_url>", \"requestEnvelope\": {\"errorLanguage\":\"en_US\"}}"
PYTHON
import settings
import urllib
import urllib2
from django.utils import simplejson
def home(request):
headers = {
"X-PAYPAL-SECURITY-USERID": settings.USERNAME,
"X-PAYPAL-SECURITY-PASSWORD": settings.PASSWORD,
"X-PAYPAL-SECURITY-SIGNATURE": settings.SIGNATURE,
"X-PAYPAL-REQUEST-DATA-FORMAT": "JSON",
"X-PAYPAL-RESPONSE-DATA-FORMAT": "JSON",
"X-PAYPAL-APPLICATION-ID": "APP-80W284485P519543T"
}
data = {"scope":"EXPRESS_CKECKOUT", callback":"http://www.example.com/success.html", "requestEnvelope": {"errorLanguage":"en_US"}}
req = urllib2.Request("https://svcs.sandbox.paypal.com/Permissions/RequestPermissions/", simplejson.dumps(data), urllib.urlencode(data), headers)
res = urllib2.urlopen(req).read()
typo in "EXPRESS_CKECKOUT" instead of "EXPRESS_CHECKOUT" and third argument urllib.urlencode(data) for urllib2.Request is not required.
data = {"scope":"EXPRESS_CHECKOUT", "callback":"http://www.example.com/success.html", "requestEnvelope": {"errorLanguage":"en_US"}}
req = urllib2.Request("https://svcs.sandbox.paypal.com/Permissions/RequestPermissions/", simplejson.dumps(data), headers)
res = urllib2.urlopen(req).read()