I am trying to convert curl commands to python requests using python scripts. I could find the uncurl modules and could use it to convert few curl commands successfully. But I am facing issues to make my script generic for all curl commands. Following issues are holding back my work. I am trying to write the utility in python which will take curl commands from the text file and will convert commands to python requests one by one.
Commands types like GET / POST etc are not accepted by uncurl.
Curl command options like -u , -X etc are rejected.
For DELETE requests, how should I use uncurl.
curl -v -k -H "Content-Type: application/json" -H "sync_id: 00000000-
0000-0000-0000-000000000001" -H "sync_token:
NhbSzPhbtFlZ9Gm1nLr5f8e0WLGQitG4o00jb006m5Vcs00XVqzRdHcFtyv4YOzd5S02Z3x1iR5OWQINgLP2Og" -H "instance_id: instance_id" -X GET 'example.com'
curl -v -k -H "Content-Type: application/json" -H "sync_id: 00000000-0000-0000-0000-000000000001" -H "sync_token: NhbSzPhbtFlZ9Gm1nLr5f8e0WLGQitG4o00jb006m5Vcs00XVqzRdHcFtyv4YOzd5S02Z3x1iR5OWQINgLP2Og" -H "instance_id: instance_id" -X POST -d '[{"id":"1", "name":"1", "env_mapping_name":"a", "env_mapping_id":"a1"}]' example.com'
The python code which I used for convertsion.
import uncurl
command = 'curl -v -k -H "Content-Type: application/json" -H "sync_id: 00000000-0000-0000-0000-000000000001" -H "sync_token: NhbSzPhbtFlZ9Gm1nLr5f8e0WLGQitG4o00jb006m5Vcs00XVqzRdHcFtyv4YOzd5S02Z3x1iR5OWQINgLP2Og" -H "instance_id: instance_id" -X GET 'example.com'
print uncurl.parse(command)
I have stripped off the tags and curl options which were giving me exceptions as follows.
def stripTags(command):
'''Strip off the unwanted tags from the curl command'''
print '\nCommand is *********************************', command
command = command.replace('-X', '')
command = command.replace('-k', '')
command = command.replace('-v', '')
command = command.replace(' \'', ' \"')
command = command.replace('\' ', '\" ')
command = command.replace('\'', '\"')
command = command.replace(' GET ', '')
command = command.replace(' POST ', '')
command = command.replace(' PUT ', '')
command = command.replace(' POST ', '')
command = command.replace(' DELETE ', '')
print '\nStripped string is =============', command
return command
When I used the following curl command for conversion, I got an exception.
'curl -v --proxy-user "00dcf6e7-4513-4e46-bbaf-ef4cac8f8d47":"XUmh8pIG68Zo" -X GET "example.com" --proxy 127.0.0.1:8080 -k'
The -U option here is bothering. Likewise there are other options which I observed exceptions on. So I am not sure if I am using uncurl correctly or not.
All the curl commands are converted when I used online convertor from curl.trillworks.com.
Can you please provide me the pointer on how to convert curl commands in to python requests?
**** :- I could able to send the curl command using 'runcurl' package. As per my understanding, the uncurl package has limitation of sending the curl request which the runcurl bridges by providing the 'execute' method. I used the following code to do so.
import runcurl
cmd = "curl -v -u "00dcf6e7-4513-4e46-bbaf-ef4cac8f8d47":"XUmh8pIG68Zo" -X GET "https://example.com" -k"
#strip off the curl options like -k, -v, -X, GET which causes runcurl to #throw exception.
try:
code = runcurl.execute(command)
except:
code = 400
//Write the failed cases in some file for further references
As earlier, I have to strip off the options supported by curl like '-k', '-v'. In the above curl command, the option for authentication is provided with the '-u' option. But the runcurl is throwing an exception.
I am seeking help on this point. My expectation is that 'runcurl' should not error out for the required options supported by curl. For ex:- '-u' option. Did I miss anything here? Is there any better way of handling curl command above?
Note :- I am trying to avoid the use of 'subprocess.call()' to call the curl command in my code.
Related
I have an API that is showing me an example that uses
curl -X POST -d "#-" -H "Content-Type: application/json" https://localhost.com/api/ <<EOF
{
"origin_id": "test_user_id"
}
EOF
And I'm not sure how to do this in python.
According to man curl:
If you start the data with the letter #, the rest should be a
file name to read the data from, or - if you want curl to read
the data from stdin.
The <<EOF is a bash way of saying "send everything after EOF to standard in". So another way to write this would be:
curl -X POST -d '{"origin_id": "test_user_id"}' -H "Content-Type: application/json" https://localhost.com/api/
So in python this would be:
import requests
requests.post('https://localhost.com/api/', data={"origin_id": "test_user_id"})
This is a followup to this Stackoverflow post
How to use Python variables in Google Colab terminal command?
Which asks how to send python variables to the command line.
The answer is you need to place a $ in front of it. In other cases, you need to quote everything like this '$variable'
This works for me except for curl commands. For example
emailBody1 = ' this is some test text'
!curl -X POST 'http://localhost.yxy' -H "Content-Type: application/json" -d '{"emails": [{"emailBody": '$emailBody1'}}'
I tried no quotes, single quotes, and double quotes. No quotes results in an error, and single/double quotes just sends whatever is inside as a string; no reference to the variable.
This works for me
-d '{{"emails": [{{"emailBody": "$emailBody1" }}]}}'
or
-d '{{"emails": [{{"emailBody": "{emailBody1}" }}]}}'
All string is inside ' ' (or " ") without spliting to 'string' $var 'string'
And normal { } has to be converted to {{ }} because it uses { } to put values from variables.
I tested it with http://httpbin.org/post which sends back all data so I could see what was send.
emailBody1 = ' this is some test text'
!curl -X POST 'http://httpbin.org/post' -H "Content-Type: application/json" -d '{{"emails": [{{"emailBody": "{emailBody1}" }}]}}'
If you really insist on using curl for this, use subprocess to run a copy explicitly so you don't depend on Jupyter trying to do magic.
import subprocess, json
emailBody1 = ' this is some test text'
subprocess.run([
'curl',
'-X', 'POST',
'-H', 'Content-Type: application/json',
'-d', json.dumps({"emails": [{"emailBody": emailBody1}]}),
'http://localhost.yxy',
])
This also has the advantage of using Python's json.dumps() to generate your JSON text, which won't break down in a number of cases where the other approach would (f/e, if your email text contains newlines, literal double quotes, or other special characters).
i have made a simple python file which want to submit in Livy.Livy is currently running in local mode.Also I have mentioned following property in livy.conf file.
Property name: livy.file.local-dir-whitelist,
value "/usr/local/livy/scripts"
My file is kept in following path "/usr/local/livy/scripts"
import json, pprint, requests, textwrap
host = 'http://localhost:8998'
data = {'kind': 'spark'}
headers = {'Content-Type': 'application/json'}
r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
r.json()
I am submitting it using curl as follows:
curl -X POST --data '{"file": "/usr/local/livy/scripts/pi.py"}' -H "Content-Type: application/json" 10.140.178.24:8999/batches
It is giving me following error:
requirement failed: Local path /usr/local/livy/scripts/pi.py cannot be added to user sessions.
My Ubuntu system only have following things:
a)Spark
b)Livy
c)Java
What am I doing wrong here?
For people using incubating mode of livy for first time,kindly check that the template file is renamed with stripping off .template in livy.conf.template.Then make sure that the following configurations are present in it.
livy.spark.master = local
livy.file.local-dir-whitelist = /path/to/script/folder/
Kindly make sure that forward slash is present in end of path
Then write url in following manner for
Python:
curl -v -X POST --data '{"file": "/path/to/script/folder/name-of-python-file.py"}' -H "Content-Type: application/json" localhost:8998/batches
Note:It will not accept relative path,whole absolute path needs to be defined in it.
curl -X POST --data '{"file": "/usr/local/livy/scripts/pi.py"}' -H "Content-Type: application/json" 10.140.178.24:8999/batches
{"id":2,"state":"starting","log":[]}
When i use curl under proxy for rest framework api testing in django i'm getting this error Exception Value: Unable to parse connection string: ""
i have used
curl -v --noproxy localhost -X POST -H "Authorization: Token <token id >" -d "email=d#c.co" "http://localhost:4000/api/....."
and also tried
export http_proxy=""
while was working properly when i was not under proxy
curl -X POST -H "Authorization: Token <token id >" -d "email=d#c.co" "http://localhost:4000/api/....."
From the manual of --noproxy, it expects -
Comma-separated list of hosts which do not use a proxy, if one is specified. The only wildcard is a single * character, which matches all hosts, and effectively disables the proxy.
So you must specify a comma separated list of hosts. That may be either specific hosts such as "localhost:4000" or "*".
So the following should work-
curl -v --noproxy "*" localhost -X POST -H "Authorization: Token <token id >" -d "email=d#c.co" "http://localhost:4000/api/....."
I'm sending a POST request via terminal to my flask backend server by doing the following:
curl -v -H "Accept: application/json" -H "Content-type: application/json" -X POST -d '{“Email”:”email#email.com”,”Password”:”testpass”}' http://127.0.0.1:5000/auth/login
I can print out the request data in flask by doing the following:
print(request.data)
However, when I try to input the data into a form to validate the information, the process comes to a halt and fails.
I'm pretty sure the issue is caused by this code, since nothing gets executed after this line:
data = MultiDict(mapping=request.json)
Any idea why?
I'm not sure that your quotes are all the same in your curl command. I tried adapting that to one of my apis and it returned an error. Are you using the right quotes in your json? Are those smart quotes? I was able to replace them and it worked.
curl -v -H "Accept: application/json" -H "Content-type: application/json" -X POST -d '{"email":"email#email.com","password":"testpass"}' http://127.0.0.1:5000/auth/login
If you are then taking the dictionary and then loading them into a form (form = LoginForm(**data)) you also need to make sure your keys match the naming of your form. I see in your json you have Email with E uppercase. Make sure that is the name of the form field in your form. If it is email = StringField(...) it won't map.