I have already gone through few StackOverflow existing links for this query, did not help me.
I would like to run few curl command(4) and each curl commands give output. From that output, I would like to parse the few group ids for next command.
curl --basic -u admin:admin -d \'{ "name" : "test-dev" }\' --header \'Content-Type: application/json\' http://localhost:8080/mmc/api/serverGroups
I have tried with as ,
#!/usr/bin/python
import subprocess
bash_com = 'curl --basic -u admin:admin -d '{ "name" : "test-dev" }' --header 'Content-Type: application/json' http://localhost:8080/mmc/api/serverGroups'
subprocess.Popen(bash_com)
output = subprocess.check_output(['bash','-c', bash_com]) # subprocess has check_output method
It gives me the syntax error, though I have changed from a single quote to double quote for that curl command.
I have been trying with Pycurl but i have to look more into that. Is there any way we can run curl commands in python and can parse the output values and pass it to next curl command.
You can use os.popen with
fh = os.popen(bash_com, 'r')
data = fh.read()
fh.close()
Or you can use subprocess like this
cmds = ['ls', '-l', ]
try:
output = subprocess.check_output(cmds, stderr=subprocess.STDOUT)
retcode = 0
except subprocess.CalledProcessError, e:
retcode = e.returncode
output = e.output
print output
There you have to organize your command and params in a list.
Or you just go the easy way and use requests.get(...).
And do not forget: Using popen you can get shell injections via parameters of your command!
Better output using os.open(bash_com,'r') and then fh.read()
python api.py
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
199 172 0 172 0 27 3948 619 --:--:-- --:--:-- --:--:-- 4027
{"href":"http://localhost:8080/mmc/api/serverGroups/39a28908-3fae-4903-adb5-06a3b7bb06d8","serverCount":0,"name":"test-dev","id":"39a28908-3fae-4903-adb5-06a3b7bb06d8"}
trying understand that fh.read() has executed the curl command? please correct me
I am trying to redirect the curl commands output to text file and then parse the file via JSON. All I am trying to get "id" from about output.
fh = os.popen(bash_com,'r')
data = fh.read()
newf = open("/var/tmp/t1.txt",'w')
sys.stdout = newf
print data
with open("/var/tmp/t1.txt") as json_data:
j = json.load(json_data)
print j['id']
I have checked the files content in JSONlint.com and got VALID JSON on it. It is throwing "ValueError: No JSON object could be decoded" at json.load line. Is there anything need to perform before parsing the redirected file.
Related
I run a curl command on windows command line prompt. It produces a json output. The command looks like this:
curl --data "action=details&user=user&project=project1&problemid=2021" https://website:9020/
I issue the same command in python as following:
import subprocess
output = subprocess.run(
[
"curl",
"--data",
"\"action=details&user=user&project=project1&problemid=2021\""
"https://website:9020/",
],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
shell=True,
)
print(output.stdout.decode("utf-8"))
The output is the following:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 199 100 64 100 123 64 123 0:00:01 --:--:-- 0:00:01 1254
{"status":400,"message":"Action parameter is missing"}
But the command line produces a json output. Yet the same command issued through subprocess.run produces this error. I also tried it with subprocess.Popen and subprocess.check_output. Same issue persists. What am I doing wrong here that is causing this error?
In some cases (e.g. if they require running command line as administrator), subprocess might not be the right choice to execute the command line commands.
You can use os.system() to see the output or os.popen() to read and store the output.
import os
import json
# to see the output
print(os.system("curl --data \"action=details&user=user&project=project1&problemid=2021\" https://website:9020/")
output = os.popen("curl --data \"action=details&user=user&project=project1&problemid=2021\" https://website:9020/").read()
outputjson = json.loads(output)
then you can access the json information.
Did you try running your code without the quotation marks around the action parameter? Because the API is complaining about missing action parameter. I think it's just not recognising it due to the escaped quotation marks.
import subprocess
output = subprocess.run(
[
"curl",
"--data",
"action=details&user=user&project=project1&problemid=2021"
"https://website:9020/",
],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
shell=True,
)
print(output.stdout.decode("utf-8"))
EDIT: I'm not sure if this is the case, but it could be that subprocess implicitly surrounds all its parameters with quotation marks in order to avoid code injection and globbing.
I was just googling again and found this answer, which states it quite well: why would you use curl, if you could use requests?
No guarantee, but I think something like this should work:
import requests
url = "https://website:9020/"
payload = {
"action": "details",
"user": "user",
"project": "project1",
"problemid": "2021"
}
res = requests.get(url, params=payload)
I am trying to get the response code for three sites using the below python code snippet. But wondering how I can parse each object in the array to pass through the for loop within the curl call.
import os
servers = ["google", "yahoo", "nonexistingsite"]
for i in range(len(servers)):
print(os.system('curl --write-out "%{http_code}\n" --silent --output'
' /dev/null "https://servers[i].com"'))
With the above code, it's not getting passed through servers[i].
You need to perform string formatting, like:
import os
servers = ["google", "yahoo", "nonexistingsite"]
for server in servers:
print(os.system('curl --write-out "%{{http_code}}\\n" --silent --output /dev/null "https://{}.wellsfargo.com"'.format(server)))
The above can however still go wrong, if for example the servers contain quotes, etc.
It might be better to here use subprocess.run and pass it a list of parameters, like:
servers = ["google", "yahoo", "nonexistingsite"]
for server in servers:
p = subprocess.run(
[
'curl'
'--write-out',
'%{http_code}\\n',
'--silent',
'--output'
'/dev/null',
'https://{}.wellsfargo.com'.format(server)
],
shell=True,
capture_output=True
)
Try using Python's string formatting, something like:
"This string uses an %s" %(argument) would become "This string uses an argument"
Something like this:
print(os.system('curl --write-out "%%{http_code}\n" --silent --output /dev/null "https://%s.wellsfargo.com"') % (servers[i])
More here: https://powerfulpython.com/blog/python-string-formatting/
Just use the requests library instead of shelling out to run curl:
for s in servers:
resp = requests.get(s)
print(resp.status_code)
Since you don't care about the body of the response, only whether it responds, you could save bandwidth by using the head function instead of get to retrieve only the headers from the server.
I'm trying to write a script that imitates a cURL command I make to change data on the target web page:
curl -u username:password "https://website.com/update/" --data "simChangesList=%5B%7B%22simId%22%3A760590802%2C%22changeType%22%3A2%2C%22targetValue%22%3A%220003077%22%2C%22effectiveDate%22%3Anull%7D%5D" --compressed
As you can see above, I am POSTing a url-encoded string to the target web page.
The following code does not work:
import requests
import urllib
enc = urllib.quote('[{"simId":760590802,"changeType":2,"targetValue":000307,"effectiveDate":null}]')
simChangesList = 'simChangesList=' + enc
print simChangesList
auth = s.post(url, data=simChangesList)
print auth.text
Even though I'm fairly certain the above code imitates my cURL command previously, but it obviously isn't.
I am getting a Required List parameter 'simChangesList' is not present error.
What is the equivalent of the cURL command to POST a url-encoded string with the requests module in Python?
EDIT:
I've tried to make multiple dictionaries with simChangesList as the key, but I cannot seem to do it.
Here are my attempts:
simChangesList: [{"simId":760590802,"changeType":2,"targetValue":000307,"effectiveDate":null}]
data = {'simChangesList': ['simId': 760590802, 'changeType': 2, 'targetValue': '0003077', 'effectiveDate': null]}
data['simChangesList'] = ['simId': 760590802, 'changeType': 2, 'targetValue': '0003077', 'effectiveDate': null]
simChangesList:[{"simId":760590802,"changeType":2,"targetValue":"000307","effectiveDate":null}]
payload = {
'simChangesList':
[{'simId': '760590802',
'changeType': '2',
'targetValue': '0003077',
'effectiveDate': 'null'}]
}
so when I run the python code the server (google) give me a different response than when I run curl command. Can someone tell me where I'm wrong please?
code:
import urllib2, simplejson
def MapsWIFI(card):
req = urllib2.Request("https://www.googleapis.com/geolocation/v1/geolocate?key=AI...")
jWifi = """
{
"wifiAccessPoints": [
{
"macAddress": "64:D1:A3:0A:11:65",
"channel": 6,
},
... #some AP here
]
}
"""
print jWifi
req.add_header("Content-Type", "application/json")
jWifiReport = urllib2.urlopen(req,simplejson.dumps(jWifi)).read()
print jWifiReport
APdetected = str(len(wifiCell))
mapsDict = simplejson.loads(jWifiReport)
location = str(mapsDict.get("location",{}))[1:-1]
accuracy = "Accuracy: "+str(mapsDict.get("accuracy",{}))[1:-1]
mapMe = "|---"+location.split(",")[0]+"\n|---"+location.split(",")[1][1:]+"\n|---$
return mapMe
MapsWIFI("wlp8s0")
And the command is:
curl -d #file2.json -H "Content-Type: application/json" -i "https://www.googleapis.com/geolocation/v1/geolocate?key=AI..."
where file2.json contains exactly jWifi in that format.
The problem is that, as said, the location returned by the code is different from the location returned by curl. I don't get error code so I thing that the syntax is correct.
The data is already a JSON encoded string, you don't want to encode it twice.
Pass it in without encoding it again:
jWifiReport = urllib2.urlopen(req, jWifi).read()
You only need to encode if you have a Python data structure (a dictionary in this case).
I have generated a list of Soundcloud track id's with the following python code:
import soundcloud
import urllib
client = soundcloud.Client(client_id='XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
client_secret='XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
username='XXXXXXXXXXXXXXXXXXXXXXXXXX',
password='XXXXXXXXXXXXXXXXXX')
f=open('soundcloud-track-ids', 'w+')
count = 0
while count < 6000:
tracks = client.get('/me/tracks', limit=200, offset=count)
for track in tracks:
print >>f, track.id, "\t", track.title .encode('utf-8')
count += 200
f.close()
I have then run a bash script to backup the entire archive to contents to a hard drive:
#!/bin/bash
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
while read line; do
if [ ! -f /mnt/drobo_1/Soundcloud/$(echo $line | cut -f 2- | sed 's,/,\ ,g').mp3 ]; then
wget https://api.soundcloud.com/tracks/"$(echo $line | awk '{print $1}')"/download?oauth_token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX \
-O /mnt/drobo_1/Soundcloud/"$(echo $line | cut -f 2- | sed 's,/,\ ,g').mp3"
fi
done < ./soundcloud-track-ids
IFS=$SAVEIFS
Nearly all of the 5317 tracks are private, and most have downloaded without a problem, however about 600 tracks have failed to download with the following error:
--2015-01-05 12:46:09-- https://api.soundcloud.com/tracks/152288957/download?oauth_token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Resolving api.soundcloud.com (api.soundcloud.com)... 93.184.220.127
Connecting to api.soundcloud.com (api.soundcloud.com)|93.184.220.127|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2015-01-05 12:46:10 ERROR 404: Not Found.
Does anyone know what the error could be?
That 404 error is saying the file couldn't be found at SoundCloud's end. It could be SoundCloud's rate limiter doing this, preventing you from hammering it so much.
See https://developers.soundcloud.com/docs/api/terms-of-use#quotas
If you try those failed downloads later, do they work?