I am kind of newbie to REST and testing dept. I needed to write automation scripts to test our REST services.We are planning to run these scripts from a Jenkins CI job regularly. I prefer writing these in python as we already have UI functionality testing scripts in python generated by selenium IDE, but I am open to any good solution.I checked httplib,simplejson and Xunit, but looking for better solutions available out there.
And also, I would prefer to write a template and generate actual script for each REST API by reading api info from xml or something. Advance thanks to all advices.
I usually use Cucumber to test my restful APIs. The following example is in Ruby, but could easily be translated to python using either the rubypy gem or lettuce.
Start with a set of RESTful base steps:
When /^I send a GET request for "([^\"]*)"$/ do |path|
get path
end
When /^I send a POST request to "([^\"]*)" with the following:$/ do |path, body|
post path, body
end
When /^I send a PUT request to "([^\"]*)" with the following:$/ do |path, body|
put path, body
end
When /^I send a DELETE request to "([^\"]*)"$/ do |path|
delete path
end
Then /^the response should be "([^\"]*)"$/ do |status|
last_response.status.should == status.to_i
end
Then /^the response JSON should be:$/ do |body|
JSON.parse(last_response.body).should == JSON.parse(body)
end
And now we can write features that test the API by actually issuing the requests.
Feature: The users endpoints
Scenario: Creating a user
When I send a POST request to "/users" with the following:
"""
{ "name": "Swift", "status": "awesome" }
"""
Then the response should be "200"
Scenario: Listing users
Given I send a POST request to "/users" with the following:
"""
{ "name": "Swift", "status": "awesome" }
"""
When I send a GET request for "/users"
Then the response should be "200"
And the response JSON should be:
"""
[{ "name": "Swift", "status": "awesome" }]
"""
... etc ...
These are easy to run on a CI system of your choice. See these links for references:
http://www.anthonyeden.com/2010/11/testing-rest-apis-with-cucumber-and-rack-test/
http://jeffkreeftmeijer.com/2011/the-pain-of-json-api-testing/
http://www.cheezyworld.com/2011/08/09/running-your-cukes-in-jenkins/
import openpyxl
import requests
import json
from requests.auth import HTTPBasicAuth
urlHead='https://IP_ADDRESS_HOST:PORT_NUMBER/'
rowStartAt=2
apiColumn=2
#payloadColumn=3
responseBodyColumn=12
statusCodeColumn=13
headerTypes = {'Content-Type':'application/json',
'Accept':'application/json',
'Authorization': '23324'
}
wb = openpyxl.load_workbook('Excel_WORKBOOK.xlsx')
# PROCESS EACH SHEET
for sheetName in (wb.get_sheet_names()):
print ('Sheet Name = ' + sheetName)
flagVar = input('Enter N To avoid APIs Sheets')
if (flagVar=='N'):
print ('Sheet got skipped')
continue
#get a sheet
sheetObj = wb.get_sheet_by_name(sheetName)
#for each sheet iterate the API's
for i in range(2, sheetObj.max_row+1):
#below is API with method type
apiFromSheet = (sheetObj.cell(row=i, column=apiColumn).value)
if apiFromSheet is None:
continue
#print (i, apiFromSheet)
#Let's split the api
apiType = apiFromSheet.split()[0]
method = apiFromSheet.split()[1]
if (apiType!='GET'):
continue
#lets process GET API's
absPath = urlHead + method
print ("REQUESTED TYPE AND PATH = ", apiType, absPath)
print('\n')
res = requests.get(absPath, auth=HTTPBasicAuth(user, pwd), verify=False, headers=headerTypes)
#LET's write res body into relevant cell
sheetObj.cell(row=i, column=responseBodyColumn).value = (res.text)
sheetObj.cell(row=i, column=statusCodeColumn).value = (res.status_code)
wb.save('Excel_WORKBOOK.xlsx')
`#exit(0)`
Related
I am working with an app that sends data to a server with a POST request,
POST https://www.somedomain.com//sendImage HTTP/2.0
looking like this:
{
"user": {
"consent": true,
"currentNumberIs": 1,
"images": {
"data": "BASE64ENCODED IMAGE",
"docType": "avatar"
},
"totalNumberOfImages": 1
}
}
I want to replace the data part of this Json, but only if the docType is avatar. Trying to use a python script for that, that I found here and edited:
def response(flow: http.HTTPFlow) -> None:
if "somedomain.com" in flow.request.pretty_url:
request_data = json.loads(flow.request.get_text())
if request_data["user"]["images"]["docType"] == "avatar":
data = json.loads(flow.response.get_text())
data["user"]["images"]["data"] = "NEWDATA"
flow.response.text = json.dumps(data)
Launched mitmproxy with -s script.py, but according to the web console, the specific request does not trigger the script at all. Which kinda limits the scope to debug.
Would glady appreciate any help.
As #nneonneo mentioned in the comments, I would first recommend to make extensive use of mitmproxy.ctx.log() to make sure that your event hook is triggered properly. Second, if I understand things correctly, you intend to modify the request and not the response? If you want to modify request contents before they are sent to the server, you need to use the request hook and not the response hook:
def request(flow: http.HTTPFlow) -> None:
# this is executed after we have received the request
# from the client, but before it is sent to the server.
def response(flow: http.HTTPFlow) -> None:
# this is executed after we have sent the request
# to the server and received the response at the proxy.
Finally, you currently read from flow.request.text and then later assign to flow.response.text. I don't know your specific use case, but usually that should be flow.request.text as well.
You're altering the flow variable in a function, but not using the edited flow. If you return the new flow you can then use it and post it.
def response(flow: http.HTTPFlow) -> http.HTTPFlow:
if "somedomain.com" in flow.request.pretty_url:
request_data = json.loads(flow.request.get_text())
if request_data["user"]["images"]["docType"] == "avatar":
data = json.loads(flow.response.get_text())
data["user"]["images"]["data"] = "NEWDATA"
flow.response.text = json.dumps(data)
return flow
The problem: I want to interact with Jupyter from another application via Jupyter API, in particular I want to run my notebooks from the app at least (Perfect variant for me is to edit some paragraphs before running it). I've read the API documentation but haven't found what I need.
I've used for that purpose Apache Zeppelin which have the same structure (Notebooks and paragraphs).
Does anybody used Jupyter for the purpose I've just described?
Ignoring if the use of Jupyter API is the best solution for the problem (not clearly described in the question), the code below does what you have asked for: it will execute remotely a Jupyter notebook over http and get some results. It is not production ready, it more an example of how it can be done. Did not test it with cells that generate lots of output - think it will need adjustments.
You can also change/edit the code programmatically by altering the code array.
You will need to change the notebook_path, base and headers according to your configuration, see code for details.
import json
import requests
import datetime
import uuid
from pprint import pprint
from websocket import create_connection
# The token is written on stdout when you start the notebook
notebook_path = '/Untitled.ipynb'
base = 'http://localhost:9999'
headers = {'Authorization': 'Token 4a72cb6f71e0f05a6aa931a5e0ec70109099ed0c35f1d840'}
url = base + '/api/kernels'
response = requests.post(url,headers=headers)
kernel = json.loads(response.text)
# Load the notebook and get the code of each cell
url = base + '/api/contents' + notebook_path
response = requests.get(url,headers=headers)
file = json.loads(response.text)
code = [ c['source'] for c in file['content']['cells'] if len(c['source'])>0 ]
# Execution request/reply is done on websockets channels
ws = create_connection("ws://localhost:9999/api/kernels/"+kernel["id"]+"/channels",
header=headers)
def send_execute_request(code):
msg_type = 'execute_request';
content = { 'code' : code, 'silent':False }
hdr = { 'msg_id' : uuid.uuid1().hex,
'username': 'test',
'session': uuid.uuid1().hex,
'data': datetime.datetime.now().isoformat(),
'msg_type': msg_type,
'version' : '5.0' }
msg = { 'header': hdr, 'parent_header': hdr,
'metadata': {},
'content': content }
return msg
for c in code:
ws.send(json.dumps(send_execute_request(c)))
# We ignore all the other messages, we just get the code execution output
# (this needs to be improved for production to take into account errors, large cell output, images, etc.)
for i in range(0, len(code)):
msg_type = '';
while msg_type != "stream":
rsp = json.loads(ws.recv())
msg_type = rsp["msg_type"]
print(rsp["content"]["text"])
ws.close()
Useful links based on which this code is made (that I recommend reading if you want more info):
https://jupyter-client.readthedocs.io/en/latest/messaging.html#python-api
https://github.com/jupyter/jupyter/wiki/Jupyter-Notebook-Server-API
Note that there is also https://jupyter-client.readthedocs.io/en/stable/index.html, but as far as I could tell it does not support HTTP as a transport.
For reference this works with notebook-5.7.4, not sure about other versions.
Extending the code by #vladmihaisima
from websocket import create_connection, WebSocketTimeoutException
while msg_type != "stream":
try:
rsp = json.loads(ws.recv())
print(rsp["msg_type"])
print(rsp)
msg_type = rsp["msg_type"]
if msg_type == "error":
raise Exception(rsp['content']['traceback'][0])
except WebSocketTimeoutException as _e:
print("No output")
return
I believe that using of remote Jupyter Notebook is over-engineering in your case.
I see good way is pass necessary parameters to python program with well logging.
The code that I included to call a API in AWS lambda is given below. urlilb3 python library is uploaded as a zip folder successfully. But when I try to access the particular intent it shows
When I included the API call in AWS lambda (python 3.6), I got
"The remote endpoint could not be called, or the response it returned was invalid" .
Why is it so? What are the prerequisites to be done before including the API calls in python 3.6. I used urllib3 python library and upload as zip folder.?? Is any other things required to do??
def get_weather(session):
should_end_session = False
speech_output = " "
reprompt_text = ""
api = "some url ...."
http = urllib3.PoolManager()
response = http.request('GET',api)
weather_status = json.loads(response.data.decode('utf-8'))
for weather in weather_status:
final_weather = weather["WeatherText"]
return build_response(session_attributes, build_speechlet_response(speech_output, reprompt_text, should_end_session))
Scenario : To obtain weather using an third party API
import urllib3
def get_weather():
api = "some url ...."
http = urllib3.PoolManager()
response = http.request('GET',api)
weather_status = json.loads(response.data.decode('utf-8'))
for weather in weather_status:
final_weather = weather["WeatherText"] ## The attribute "WeatherText" will varies depending upon the weather API you are using.
return final_weather
get_weather() # simple function call
Try printing response.data so you can see it in the logs. That might give you a clue. I would also try to switch to Python Requests instead of URLLib3. You may also need to set the Content Type depending on the implementation of the API you're calling.
from __future__ import print_function
import json
from botocore.vendored import requests
def lambda_handler(event, context):
print('received request: ' + str(event))
doctor_intent = event['currentIntent']['slots']['doctor']
email_intent = event['currentIntent']['slots']['email']
print(doctor_intent, email_intent)
print(type(doctor_intent), type(email_intent))
utf8string = doctor_intent.encode("utf-8")
utf8string1 = email_intent.encode("utf-8")
print(type(utf8string))
print(type(utf8string1))
car1 = {"business_name": utf8string , "customer_email": utf8string1 }
r = requests.post('https://postgresheroku.herokuapp.com/update',
json=car1)
#print ("JSON : ", r.json())
print(r.json())
data = str(r.json())
print(type(data))
return {
"dialogAction": {
"type": "Close",
"fulfillmentState": "Fulfilled",
"message": {
"contentType": "PlainText",
"content": "Thank you for booking appointment with {doctor}
{response}".format(doctor=doctor_intent,response=data)
}
}
}
I am trying to use python for my jenkins job, this job downloads and refreshes a line in the project then commits and creates a pull request, I am trying read the documentation for GitPython as hard as I can but my inferior brain is not able to make any sense out of it.
import git
import os
import os.path as osp
path = "banana-post/infrastructure/"
repo = git.Repo.clone_from('https://github.myproject.git',
osp.join('/Users/monkeyman/PycharmProjects/projectfolder/', 'monkey-post'), branch='banana-refresh')
os.chdir(path)
latest_banana = '123456'
input_file_name = "banana.yml"
output_file_name = "banana.yml"
with open(input_file_name, 'r') as f_in, open(output_file_name, 'w') as f_out:
for line in f_in:
if line.startswith("banana_version:"):
f_out.write("banana_version: {}".format(latest_banana))
f_out.write("\n")
else:
f_out.write(line)
os.remove("deploy.yml")
os.rename("deploy1.yml", "banana.yml")
files = repo.git.diff(None, name_only=True)
for f in files.split('\n'):
repo.git.add(f)
repo.git.commit('-m', 'This an Auto banana Refresh, contact bannana#monkey.com',
author='moneky#banana.com')
After committing this change I am trying to push this change and create a pull request from branch='banana-refresh' to branch='banana-integration'.
GitPython is only a wrapper around Git. I assume you are wanting to create a pull request in a Git hosting service (Github/Gitlab/etc.).
You can't create a pull request using the standard git command line. git request-pull, for example, only Generates a summary of pending changes. It doesn't create a pull request in GitHub.
If you want to create a pull request in GitHub, you can use the PyGithub library.
Or make a simple HTTP request to the Github API with the requests library:
import json
import requests
def create_pull_request(project_name, repo_name, title, description, head_branch, base_branch, git_token):
"""Creates the pull request for the head_branch against the base_branch"""
git_pulls_api = "https://github.com/api/v3/repos/{0}/{1}/pulls".format(
project_name,
repo_name)
headers = {
"Authorization": "token {0}".format(git_token),
"Content-Type": "application/json"}
payload = {
"title": title,
"body": description,
"head": head_branch,
"base": base_branch,
}
r = requests.post(
git_pulls_api,
headers=headers,
data=json.dumps(payload))
if not r.ok:
print("Request Failed: {0}".format(r.text))
create_pull_request(
"<your_project>", # project_name
"<your_repo>", # repo_name
"My pull request title", # title
"My pull request description", # description
"banana-refresh", # head_branch
"banana-integration", # base_branch
"<your_git_token>", # git_token
)
This uses the GitHub OAuth2 Token Auth and the GitHub pull request API endpoint to make a pull request of the branch banana-refresh against banana-integration.
It appears as though pull requests have not been wrapped by this library.
You can call the git command line directly as per the documentation.
repo.git.pull_request(...)
I have followed frederix's answer also used https://api.github.com/repos/ instead of https://github.com/api/v3/repos/
Also use owner_name instead of project_name
I'm in the process of writing a very simple Python application for a friend that will query a service and get some data in return. I can manage the GET requests easily enough, but I'm having trouble with the POST requests. Just to get my feet wet, I've only slightly modified their example JSON data, but when I send it, I get an error. Here's the code (with identifying information changed):
import urllib.request
import json
def WebServiceClass(Object):
def __init__(self, user, password, host):
self.user = user
self.password = password
self.host = host
self.url = "https://{}/urldata/".format(self.host)
mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
mgr.add_password(None, "https://{}".format(self.host), self.user, self.password)
self.opener = urllib.request.build_opener(urllib.request.HTTPDigestAuthHandler(mgr))
username = "myusername"
password = "mypassword"
service_host = "thisisthehostinfo"
web_service_object = WebServiceClass(username, password, service_host)
user_query = {"searchFields":
{
"First Name": "firstname",
"Last Name": "lastname"
},
"returnFields":["Entity ID","First Name","Last Name"]
}
user_query = json.dumps(user_query)
user_query = user_query.encode("ascii")
the_url = web_service_object.url + "modules/Users/search"
try:
user_data = web_service_object.opener.open(the_url, user_query)
user_data.read()
except urllib.error.HTTPError as e:
print(e.code)
print(e.read())
I got the class data from their API documentation.
As I said, I can do GET requests fine, but this POST request gives me a 500 error with the following text:
Content type 'application/x-www-form-urlencoded;charset=UTF-8' not supported
In researching this error, my assumption has become that the above error means I need to submit the data as multipart/form-data. Whether or not that assumption is correct is something I would like to test, but stock Python doesn't appear to have any easy way to create multipart/form-data - there are modules out there, but all of them appear to take a file and convert it to multipart/form-data, whereas I just want to convert this simple JSON data to test.
This leads me to two questions: does it seem as though I'm correct in my assumption that I need multipart/form-data to get this to work correctly? And if so, do I need to put my JSON data into a text file and use one of those modules out there to turn it into multipart, or is there some way to do it without creating a file?
Maybe you want to try the requests lib, You can pass a files param, then requests will send a multipart/form-data POST instead of an application/x-www-form-urlencoded POST. You are not limited to using actual files in that dictionary, however:
import requests
response = requests.post('http://httpbin.org/post', files=dict(foo='bar'))
print response.status_code
If you want to know more about the requests lib, and specially in sending multipart forms take a look at this:
http://docs.python-requests.org/en/master/
and
http://docs.python-requests.org/en/master/user/advanced/?highlight=Multipart-Encoded%20Files