Develop Reference

RestApi json results into a csv using python

I'm trying to write the response returned from RestApi URL into a csv file for provided port number(Interactive mode) and selected users users_list.txt (Script mode). I have the below code to do the job.
import json
import csv
import urllib.request
import subprocess
portvalue = input("Please enter an Port Number:\n")
portvalue = int(portvalue)
print(f'You entered {portvalue}')
tooluser='admin'
toolpassword='password'
user = open('users-list.txt')
for line in user:
bash_com = 'curl --user {tooluser}:{toolpassword} http://198.98.99.12:46567/{portvalue}/protects/{user} \
| jq --arg a_port {portvalue} --arg a_userid {user} 'map(.+{"userid":{user}}+{"port":{portvalue}})'' as url:
subprocess.Popen(bash_com)
output = subprocess.check_output(['bash','-c', bash_com])
print(line)
myfile.close()
# with urllib.request.urlopen("curl --user admin:password http://198.98.99.12:46567/{port}/protects/{user} | jq") as url:
data = json.loads(url.read().decode())
fname = "output.csv"
with open(fname, "w") as file:
csv_file = csv.writer(file,lineterminator='\n')
csv_file.writerow(["depotFile","host","isgroup","line","perm","user","port","userid"])
for item in data["raw_data"]:
csv_file.writerow([item['depotFile'],item['host'],item['isgroup'],item['line'],item['perm'],item['user'],item['port'],item['userid']])
Curl with URL to get data for single user - curl --user admin:password http://198.98.99.12:46567/2324/protects/sanchez.ricardo | jq
users_list.txt consists of users in below format.
sanchez.ricardo
varun.sharma
daniel.vel
One of the json output format as follows,
[
{
"depotFile": "//LIB/Include/...",
"host": "*",
"isgroup": "",
"line": "19",
"perm": "open",
"user": "5G",
"port": "2324",
"userid": "sanchez.ricardo"
},
....
......
.........
]
Expected output csv file:-
Sno depotFile host isgroup line perm user port userid
1 //LIB/Include/... * 19 open 5G 2324 sanchez.ricardo
2 //LIB/... * 19 write 6G 2324 varun.sharma
3 //AND/RIO/... * 20 write AND 2324 daniel.vel
I'm unable to process the RestApi URl in above code. Please help me to achieve this in python. Thanks in advance for your help.

I like to use requests to grab stuff as it is so easy now, it has improved drastically over its beginnings. It is so much easier than pyCurl now that I am in the process of refactoring code to use it as I make improvements to the code over time.
Here is a really cool website to convert a curl command to requests code:
https://curl.trillworks.com/
so:
curl --user admin:password http://198.98.99.12:46567/2324/protects/sanchez.ricardo
is:
import requests
response = requests.get('http://198.98.99.12:46567/2324/protects/sanchez.ricardo', auth=('admin', 'password'))
Now it looks to me like your loop isn't going to work. It loops and assigns the stuff to bash but then doesnt do anything until the loop is done. Probably you accidentally unindented it though. At any rate, here is a loop that should work, using the suggestion of f-strings as suggested in another answer.
I am not completely sure of what you get back from the api, seems like you need to make multiple requests to get json for each, so I wrote it that way. I am sure you need to tweak some stuff but this should get you closer.
import requests
import json
import csv
user_csv = '''sanchez.ricardo
varun.sharma
daniel.vel'''
# I cannot tell where you get the ports from, so I will put a list here to show the code working
# and you can fill it differently, maybe from the csv?? not sure
ports = [2324, 2324, 2324]
users = user_csv.split('\n')
fname = "output.csv"
with open(fname, "w") as file:
csv_file = csv.writer(file,lineterminator='\n')
csv_file.writerow(["depotFile", "host", "isgroup", "line", "perm", "user", "port", "userid"])
for user, port in zip(users, ports):
print(f'from csv and other method here is the user and port: {user}, {port}')
url = f'http://198.98.99.12:46567/{port}/protects/{user}'
print(f'grab data from rest api: {url}')
# cannot tell if there is one user:password for the REST API, this code assumes so
response = requests.get(url, auth=('admin', 'password'))
# assuming response returns json like this:
# response = '''[
# {
# "depotFile": "//LIB/Include/...",
# "host": "*",
# "isgroup": "",
# "line": "19",
# "perm": "open",
# "user": "5G",
# "port": "2324",
# "userid": "sanchez.ricardo"
# }]'''
data = json.loads(response)
for item in list(data):
csv_file.writerow(item.values())

In this case, f-string is probably what you're looking for. Consider this:-
user='sanchez.ricardo'
port=2324
url = f'http://198.98.99.12:46567/{port}/protects/{user}'

The second snippet you have only covers the part
curl --user $USERID:$PASSWORD http://198.98.99.12:46567/$PORT/protects/$u \
jq --arg a_port $PORT --arg a_userid .....
From what I undrtstand you want both funcionalities
from user input - interactive mode.
from file - script mode.
Many things can be done but what I would do is use Argeparse and maybe have a switch param -ulf|--users-list-file to pass a file or simply no param for interactive mode.
And have the port be passed via environment variable, and read that using os.environ, aswell as a -p|--port as an optional param
A quick'n dirty way is to use sys.argv to read passed arguments and write your conditions accordingly.

I have been going through your code, you can simply use request and use pandas to store data in CSV. This will make easy output.
import json
import pandas as pd
json_string = '{ "name":"John", "age":30, "car":"None" }'
a_json = json.loads(json_string)
print(a_json)
dataframe = pd.DataFrame.from_dict(a_json)

Python requests module: Equivalent of cURL --data command

I'm trying to write a script that imitates a cURL command I make to change data on the target web page:
curl -u username:password "https://website.com/update/" --data "simChangesList=%5B%7B%22simId%22%3A760590802%2C%22changeType%22%3A2%2C%22targetValue%22%3A%220003077%22%2C%22effectiveDate%22%3Anull%7D%5D" --compressed
As you can see above, I am POSTing a url-encoded string to the target web page.
The following code does not work:
import requests
import urllib
enc = urllib.quote('[{"simId":760590802,"changeType":2,"targetValue":000307,"effectiveDate":null}]')
simChangesList = 'simChangesList=' + enc
print simChangesList
auth = s.post(url, data=simChangesList)
print auth.text
Even though I'm fairly certain the above code imitates my cURL command previously, but it obviously isn't.
I am getting a Required List parameter 'simChangesList' is not present error.
What is the equivalent of the cURL command to POST a url-encoded string with the requests module in Python?
EDIT:
I've tried to make multiple dictionaries with simChangesList as the key, but I cannot seem to do it.
Here are my attempts:
simChangesList: [{"simId":760590802,"changeType":2,"targetValue":000307,"effectiveDate":null}]
data = {'simChangesList': ['simId': 760590802, 'changeType': 2, 'targetValue': '0003077', 'effectiveDate': null]}
data['simChangesList'] = ['simId': 760590802, 'changeType': 2, 'targetValue': '0003077', 'effectiveDate': null]
simChangesList:[{"simId":760590802,"changeType":2,"targetValue":"000307","effectiveDate":null}]
payload = {
'simChangesList':
[{'simId': '760590802',
'changeType': '2',
'targetValue': '0003077',
'effectiveDate': 'null'}]
}

Python Flask - Passing hash input in API call

I am a beginner in making API. I have followed the blog https://blog.miguelgrinberg.com/post/designing-a-restful-api-with-python-and-flask and able to create the Get - Post API method.
I am using Flask to making Rest API. Please see the code below in which I want to take question as input in API and return an answer in JSON format by making my code as an API.
app = Flask(__name__)
#app.route('/match/api/v1', methods = ['POST'])
def my_form_post():
if not request.json or not 'question' in request.json:
abort(400)
input_text_p = request.json['question'] # access input from curl request
reference_data = request.json['data'] # to access data field from the API request
path = 'airtel_faq.xlsx'
question_list, answer_list = read_excel_file(path) # reading some reference data from an excel file
input_text = input_text_p.translate(None, string.punctuation) # remove punctuation
final_answer = find_similarity(input_text, answer_list, question_list)
print "Final Answer is : ", final_answer
values = [
{'id' : 1,
'answer' : final_answer # answer I want in JSON
'done' : False
}
]
return jsonify({'values': values}), 201
if __name__ == '__main__':
app.run(debug = True)
I am trying to pass hash input for data tag in the request to API. I am not aware how to do this. The curl request I am making is giving error:
Failed to decode JSON object: Expecting ',' delimiter or '}': line 1 column 80 (char 79)
The curl request I am making :
curl -i -H "Content-Type: application/json" -X POST -d '{"question":"Hi","cutoff":"0.3", "data":"{"q":"hello whats going on","a":"I am fine","type":"1"}"}' http://localhost:5000/match/api
Please let me know how to do this. Do I have to include anything in my script. Or is there a way to pass the JSON file in API call by giving path to the file.

In order to parse the JSON request, it needs to be valid JSON. Try this request:
curl -i -H "Content-Type: application/json" -X POST -d '{"question": "Hi", "cutoff": "0.3", "data": {"q": "hello whats going on", "a": "I am fine", "type": "1"}}' http://localhost:5000/match/api

Flask Restful PUT update list with object

I want to use Flask Restful to update a list with an object using a PUT request.
The resulting JSON should look like:
{"EmployeeID":"12345", "firstname":"Joe","lastname":"Bloggs","SupervisorName":"Name","SupervisorID":"2468","role":"Role","active":"True","hours":["{'date':'01/01/2017','qty':'3','project':'Project 1'}"]"}
The Hours field in the JSON is a list. The aim is to append an object to the list on each PUT request.
The Parser for hours is:
parser.add_argument('hours', action='append')
The Put method code is:
def put(self, EmployeeID=None):
data = parser.parse_args()
if not data:
data = {'ERROR': 'No Data'}
return jsonify(data)
else:
if EmployeeID:
if mongo.db.employee.find_one({'EmployeeID': EmployeeID}):
mongo.db.employee.update_one({'EmployeeID': EmployeeID}, {set: {"hours": data.get('hours')}})
return {'response': 'Employee:'+str(EmployeeID)+' updated'}
else:
return {'Error': 'employee ' + str(EmployeeID) + ' not found'}
else:
return {'response': 'Employee ID missing'}
Is the update_one method the right one to use here?
using curl PUT request :
curl -H "Content-type: application/json" -X PUT -d '{"EmployeeID":"1234",...,"hours":{'time':'','qty':'3','project':'Project 1'}}' http://127.0.0.1:5000/api/people/1234
Gave the error:
{
"message": "Failed to decode JSON object: Expecting property name enclosed in double quotes: line 1 column 168 (char 167)"
}
But When I add the quotations into the request it return an error:
{
"message": "Failed to decode JSON object: Unterminated string starting at: line 1 column 167 (char 166)"
}
I can't figure out whether there is an issue with the requests or with the actual Flask-Restful code.
What is the correct way to go about updating a Mongo document list with an object?

I don't have much experience with Flask, but from the error messages you've posted, it looks like the JSON in the request cannot be correctly decoded by Flask. I believe this is because you are using single quotes in your JSON string and are wrapping the entire JSON with yet another set of single quotes.
Try to just use double quotes in your JSON string and wrap the entire JSON string in single quotes like so:
curl -H "Content-type: application/json" -X PUT -d '{"EmployeeID":"1234","hours":{"time":"","qty":"3","project":"Project 1"}}' http://127.0.0.1:5000/api/people/1234
I think that should solve the issue of not being able to decode JSON form the request.

If anyone runs into this issue I finally got there.
I had to change the RequestParser:
parser.add_argument('hours')
Removing the action='append'
Then using the mongo update:
mongo.db.employee.update_one({'EmployeeID': EmployeeID}, {'$addToSet': {"hours": data.get('hours')}})

Launch Jenkins parametrized build from script

I am trying to launch a Jenkins parametrized job from a python script. Due to environment requirements, I can't install python-jenkins. I am using raw requests module.
This job I am trying to launch has three parameters:
string (let's call it payload)
string (let's call it target)
file (a file, optional)
I've searched and search, without any success.
I managed to launch the job with two string parameters by launching:
import requests
url = "http://myjenkins/job/MyJobName/buildWithParameters"
target = "http://10.44.542.62:20000"
payload = "{payload: content}"
headers = {"Content-Type": "application/x-www-form-urlencoded"}
msg = {
'token': 'token',
'payload': [ payload ],
'target': [ target ],
}
r = requests.post(url, headers=headers, data=msg)
However I am unable to send a file and those arguments in single request.
I've tried requests.post file argument and failed.

It turns out it is impossible to send both data and file in a single request via HTTP.

import jenkinsapi
from jenkinsHandler import JenkinsHandler
in your python script
Pass parameters to buildJob() , (like < your JenkinsHandler object name>.buildJob())
JenkinsHandler module has functions like init() , buildJob(), isRunning() which helps in triggering the build

Here is an example:
curl -vvv -X POST http://127.0.0.1:8080/jenkins/job/jobname/build
--form file0='#/tmp/yourfile'
--form json='{"parameter": [{"name":"file", "file":"file0"}]}'
--form json='{"parameter": [{"name":"payload", "value":"123"}]
--form json='{"parameter": [{"name":"target", "value":"456"}]}'