We are using grpc as the RPC protocol for all our internal system. Most of the system is written in Java.
In Java, we can use InprocessServerBuilder for unittest. However, I haven't find a similar class in Python.
Can any one provide a sample code for how to do GRPC unittest in python?
How serendipitous that you have asked this question today; our unit test framework just entered code review. So for the time being the way to test is to use the full production stack to connect your client-side and server-side code (or to violate the API and mock a lot of internal stuff) but hopefully in days to weeks the much better solution will be available to you.
Some example code to get started:
proto
syntax = "proto3";
service MyLibrary {
rpc Search (Request) returns (Response);
}
message Request {
string id = 1;
}
message Response {
string status = 1;
}
python unit test
#!/usr/bin/env python
# coding=utf-8
import unittest
from grpc import StatusCode
from grpc_testing import server_from_dictionary, strict_real_time
import mylibrary_pb2
class TestCase(unittest.TestCase):
def __init__(self, methodName) -> None:
super().__init__(methodName)
myServicer = MyLibraryServicer()
servicers = {
mylibrary_pb2.DESCRIPTOR.services_by_name['MyLibrary']: myServicer
}
self.test_server = server_from_dictionary(
servicers, strict_real_time())
def test_search(self):
request = mylibrary_pb2.Request(
id=2,
)
method = self.test_server.invoke_unary_unary(
method_descriptor=(mylibrary_pb2.DESCRIPTOR
.services_by_name['Library']
.methods_by_name['Search']),
invocation_metadata={},
request=request, timeout=1)
response, metadata, code, details = method.termination()
self.assertTrue(bool(response.status))
self.assertEqual(code, StatusCode.OK)
if __name__ == '__main__':
unittest.main()
I find pytest-grpc is easy to follow and get it works in few minutes.
src: https://pypi.org/project/pytest-grpc/
Related
I am working on a cloud based API implemented using Lambda functions on AWS. We are scanning it for securities issues and Veracode SAST tool has identified a potential XSS on one of the endpoints.
The code looks as follow:
def lambda_handler(event: dict, _):
event = Event(event)
body = event.body
id_xss = body[“id”]
…
return {
status_code: 200,
body:{
"message": f"Successfully endpoint execution ${id_xss}.”
}
}
I am trying to fix it by using the Python module Bleach. So my new code would look as follows:
def lambda_handler(event: dict, _):
event = Event(event)
body = event.body
# SANITIZE THE INPUT
id_xss = bleach.clean(body[“id”], BLEACH_VALID_TAGS,
BLEACH_VALID_ATTRS, BLEACH_VALID_STYLES)
…
return {
status_code: 200,
body:{
"message": f"Successfully endpoint execution ${id_xss}.”
}
}
I've also tried checking that body[“id”] is a natural number.
However, Veracode SAST tool still consider it an issue. Am I using it properly? What would be the right approach?
Any comment or feedback will be welcomed.
UPDATE
For me the Problem got fixed as soon as I was putting "encoding: URLEncoding(destination: .queryString)" in my request. Maybe this helps somebody else. link
I struggled the whole day to find the problem in my Alamofire PUT Request or the Flask Restful API. Request like GET, DELETE and POST are working fine with Alamofire, except the PUT Request.
When I'm using PUT Requests in combination with Postman and Flask-Restful everything is also working fine. But as soon as I'm trying to achieve the same Result with Alamofire, I'm not getting any parameters in Flask. I tried to illustrate this in the code examples.
So in short my example illustrates the following:
DELETE Request(Same with GET and POST)
Postman: success
Alamofire: success
PUT Request
Postman: success
Alamofire: failure (parameter dictionary empty in Flask-Restful)
Here is my Python Code [API Server]:
from flask import Flask, request, jsonify
from flask_restful import Resource, Api, reqparse
app = Flask(__name__)
api = Api(app)
class Stackoverflow(Resource):
def delete(self):
print(request.args)
if request.args.get('test-key') is None:
return jsonify({"message": "failure"})
else:
return jsonify({"message": "success"})
def put(self):
print(request.args)
if request.args.get('test-key') is None:
return jsonify({"message": "failure"})
else:
return jsonify({"message": "success"})
api.add_resource(Stackoverflow, '/stackoverflow')
if __name__ == '__main__':
app.run(debug=True, host='0.0.0.0')
If I'm using Postman, I get this result (like expected):
Result in Postman
But now I'm trying to do the same with Alamofire in Swift. Same Server, nothing changed.
SWIFT demo Code [IOS APP]:
import UIKit
import Alamofire
import SwiftyJSON
class ViewController: UIViewController {
override func viewDidLoad() {
super.viewDidLoad()
// Do any additional setup after loading the view
simplePUTRequest()
simpleDELETERequest()
}
func simplePUTRequest(){
AF.request("http://localhost:5000/stackoverflow", method: .put, parameters: ["test-key":"testvalue"])
.validate(statusCode: 200..<300)
.responseJSON { response in
if let data = response.data {
print("Result PUT Request:")
print(String(data: data, encoding: .utf8)!)
//print(utf8Text)
}else{
}
}
}
func simpleDELETERequest(){
AF.request("http://localhost:5000/stackoverflow", method: .delete, parameters: ["test-key":"testvalue"])
.validate(statusCode: 200..<300)
.responseJSON { response in
if let data = response.data {
print("Result DELETE Request:")
print(String(data: data, encoding: .utf8)!)
//print(utf8Text)
}else{
}
}
}
Xcode Console:
Result PUT Request:
{
"message": "failure"
}
Result DELETE Request:
{
"message": "success"
}
python Console (both Alamofire Requests):
ImmutableMultiDict([])
127.0.0.1 - - [15/Jun/2019 21:17:31] "PUT /stackoverflow HTTP/1.1" 200 -
ImmutableMultiDict([('test-key', 'testvalue')])
127.0.0.1 - - [15/Jun/2019 21:17:31] "DELETE /stackoverflow?test-key=testvalue HTTP/1.1" 200 -
As you can see, I'm getting the success message only while using the DELETE method.
Till now I tried using different encodings like URLEncoding.httpbody and URLEncoding.default, but nothing really helped.
For me it seems like it's a Alamofire/Swift Problem, because in Postman the same request method is working fine.
I would really appreciate your help, because I'm stuck and don't know anything further to do. I hope I didn't misunderstood something essential.
Thank you in advance!
I am currently using the same version AlamoFire, and when I use the PUT method, I use it as follows:
let request = AF.request(url, method: .put, parameters: ["uid": uid],
encoding: JSONEncoding.default, headers: headers)
request.responseJSON(completionHandler: { response in
guard response.error == nil else {
//Handle error
}
if let json = response.value as? [String: Any]
// Handle result.
}
The only difference to your post is that I used the encoding option. You can try to put the option and see what happens.
It looks like your server is expecting your PUT parameters to be URL form encoded into the URL. You may be hitting the version of the request method that uses JSON encoding by default, so adding encoder: URLEncodedFormParameterEncoder.default at the end of your request call should fix that. A future release will make that the default, as it's safe across all request types.
If that's not the issue, I suggest you investigate more closely to see what the differences between the requests may be. Since you control the server you should have easy access to the traffic.
The problem: I want to interact with Jupyter from another application via Jupyter API, in particular I want to run my notebooks from the app at least (Perfect variant for me is to edit some paragraphs before running it). I've read the API documentation but haven't found what I need.
I've used for that purpose Apache Zeppelin which have the same structure (Notebooks and paragraphs).
Does anybody used Jupyter for the purpose I've just described?
Ignoring if the use of Jupyter API is the best solution for the problem (not clearly described in the question), the code below does what you have asked for: it will execute remotely a Jupyter notebook over http and get some results. It is not production ready, it more an example of how it can be done. Did not test it with cells that generate lots of output - think it will need adjustments.
You can also change/edit the code programmatically by altering the code array.
You will need to change the notebook_path, base and headers according to your configuration, see code for details.
import json
import requests
import datetime
import uuid
from pprint import pprint
from websocket import create_connection
# The token is written on stdout when you start the notebook
notebook_path = '/Untitled.ipynb'
base = 'http://localhost:9999'
headers = {'Authorization': 'Token 4a72cb6f71e0f05a6aa931a5e0ec70109099ed0c35f1d840'}
url = base + '/api/kernels'
response = requests.post(url,headers=headers)
kernel = json.loads(response.text)
# Load the notebook and get the code of each cell
url = base + '/api/contents' + notebook_path
response = requests.get(url,headers=headers)
file = json.loads(response.text)
code = [ c['source'] for c in file['content']['cells'] if len(c['source'])>0 ]
# Execution request/reply is done on websockets channels
ws = create_connection("ws://localhost:9999/api/kernels/"+kernel["id"]+"/channels",
header=headers)
def send_execute_request(code):
msg_type = 'execute_request';
content = { 'code' : code, 'silent':False }
hdr = { 'msg_id' : uuid.uuid1().hex,
'username': 'test',
'session': uuid.uuid1().hex,
'data': datetime.datetime.now().isoformat(),
'msg_type': msg_type,
'version' : '5.0' }
msg = { 'header': hdr, 'parent_header': hdr,
'metadata': {},
'content': content }
return msg
for c in code:
ws.send(json.dumps(send_execute_request(c)))
# We ignore all the other messages, we just get the code execution output
# (this needs to be improved for production to take into account errors, large cell output, images, etc.)
for i in range(0, len(code)):
msg_type = '';
while msg_type != "stream":
rsp = json.loads(ws.recv())
msg_type = rsp["msg_type"]
print(rsp["content"]["text"])
ws.close()
Useful links based on which this code is made (that I recommend reading if you want more info):
https://jupyter-client.readthedocs.io/en/latest/messaging.html#python-api
https://github.com/jupyter/jupyter/wiki/Jupyter-Notebook-Server-API
Note that there is also https://jupyter-client.readthedocs.io/en/stable/index.html, but as far as I could tell it does not support HTTP as a transport.
For reference this works with notebook-5.7.4, not sure about other versions.
Extending the code by #vladmihaisima
from websocket import create_connection, WebSocketTimeoutException
while msg_type != "stream":
try:
rsp = json.loads(ws.recv())
print(rsp["msg_type"])
print(rsp)
msg_type = rsp["msg_type"]
if msg_type == "error":
raise Exception(rsp['content']['traceback'][0])
except WebSocketTimeoutException as _e:
print("No output")
return
I believe that using of remote Jupyter Notebook is over-engineering in your case.
I see good way is pass necessary parameters to python program with well logging.
Lets say I have a very simple web app which is presented as blue if the current president is a democrat and red if they are a republican. A REST API is used to get the current president, via the endpoint:
/presidents/current
which currently returns the json object:
{name: "Donald Trump", party: "Republican"}
So when my page loads I call the endpoint and I show red or blue depending on who is returned.
I wish to test this HTML/javascript page and I wish to mock the back-end so that I can control from within the test environment the API responses. For example:
def test_republican():
# configure the response for this test that the web app will receive when it connects to this endpoint
configure_endpoint(
"/presidents/current",
jsonify(
name="Donald Trump",
party="Republican"
)
)
# start the web app in the browser using selenium
load_web_app(driver, "http://localhost:8080")
e = driver.find_element_by_name("background")
assert(e.getCssValue("background-color") == "red")
def test_democrat():
# configure the response for this test that the web app will receive when it connects to this endpoint
configure_endpoint(
"/presidents/current",
jsonify(
name="Barack Obama",
party="Democrat"
)
)
# start the web app in the browser using selenium
load_web_app(driver, "http://localhost:8080")
e = driver.find_element_by_name("background")
assert(e.getCssValue("background-color") == "blue")
So the question is how should I implement the function configure_endpoint() and what libraries can you recommend me?
As #Kie mentioned, configure_endpoint implementation won't be enough, if you're going to stub the whole server-side within Selenium Python code. You would need a web server or whatever that will response via HTTP to requests from within testing environment.
It looks like the question is partially about testing of client-side code. What I see is that you're trying to make unit-test for client-side logic, but use integration testing suite in order to check this logic (it's strange).
The main idea is as follows.
You're trying to test client-side code. So, let's make mocks client-side too! Because this part of code is completely client-side related stuff.
If you actually want to have mocks, not stubs (watch the difference here: https://stackoverflow.com/a/3459491/882187) it is a better way to mock out HTTP requests inside your Javascript code. Just because you're testing a client-side piece of code, not some parts of server-side logic.
Having it isolated from whatever server-side is - is a great idea that you would love when your project become grow, while more and more endpoints will be appearing.
For example, you can use the following approach:
var restResponder = function() { // the original responder your client-side app will use
this.getCurrentPresident = function(successCallback) {
$.get('/presidents/current', callback);
}
};
var createMockResponder = function(president, party){ // factory that creates mocks
var myPresident = president;
var myParty = party;
return function() {
this.getCurrentPresident = function (successCallback) {
successCallback({"name": myPresident, "party": myParty});
}
};
}
// somewhere swap the original restResponder with new mockResponder created by 'createMockResponder'
// then use it in your app:
function drawColor(restResponder, backgroundEl) {
restResponder.getCurrentPresident(function(data){
if (data.party == "Democrat") $(backgroundEl).style('background-color', 'blue')
else if (data.party == "Republican") $(backgroundEl).style('background-color', 'red')
else console.info('Some strange response from server... Nevermind...');
});
}
Practically, this implementation depends on what do you have at the client-side as a framework. If jQuery, then my example is enough, but it looks very wordy. In case you have something more advanced, like AngularJS, you can do the same in 2-3 lines of code:
// Set up the mock http service responses
$httpBackend = $injector.get('$httpBackend');
// backend definition common for all tests
authRequestHandler = $httpBackend.when('GET', '/auth.py')
.respond({userId: 'userX'}, {'A-Token': 'xxx'});
Check out the docs: https://docs.angularjs.org/api/ngMock/service/$httpBackend
If you're still stick to the idea, that you need mocks inside Selenium tests, please
try this project: https://turq.readthedocs.io/en/latest/
It serves with Python DSL for describing REST responders.
Using turq your mocks will look as follows:
path('/presidents/current').json({'name':'Barack Obama', 'party': 'Democrat'}, jsonp=False)
Also, I would recommend to try stubs instead of mocks and use this Python module: mock-server https://pypi.python.org/pypi/mock-server/0.3.7
You are required to create the directory layout containing corresponding pre-populated JSON responses and to add some boilerplate code in order to make the mock-server respond on 'localhost:8080'. The directory layout for your example will look like this:
stub_obama/
presidents/
current/
GET_200.json # will contain {"name": "Barack Obama", "party": "Democrat"}
stub_trump/
presidents/
current/
GET_200.json # will contain {"name": "Donald Trump", "party": "Republican"}
But the mock_server is based on Tornado, it is very heavy solution for using in tests I think.
I hope, my answer is helpful and informative. Welcome to discuss it! I made tons of projects with Selenium, big and small tests, tested client-side and server-side.
I would use tornado web framework.
import json
import functools
import operator
from tornado import ioloop, web, gen
from tornado.options import define, options
define("data_file", default='default/mock.json', type=str)
class Handler(web.RequestHandler):
def data_received(self, chunk):
pass
def initialize(self, data):
self.data = data
#gen.coroutine
def get(self, *args, **kwargs):
path = self.request.path.split("/")[1:]
path = functools.reduce(
operator.add,
[[k, v[0].decode("utf-8")] for k, v in self.request.query_arguments.items()],
path
)
try:
self.write(functools.reduce(operator.getitem, path, self.data))
except KeyError:
self.set_status(404)
class Application(web.Application):
def __init__(self):
data = {}
with open(options.data_file) as data_file:
data = json.load(data_file)
handlers = [
('(.*)', Handler, {"data": data})
]
settings = dict(
gzip=True,
static_hash_cache=True,
)
web.Application.__init__(self, handlers, **settings)
def main():
io_loop = ioloop.IOLoop.instance()
backend_application = Application()
backend_application.listen(8001)
io_loop.start()
if __name__ == "__main__":
main()
This is a code I used for mocking a REST-API which is a standalone script, but it can be embedded into your test environment as well.
I defined a JSON file which defines the different path components and what should be returned. Like this:
{
"presidents": {
"current": {
"name": "Donald Trump",
"party": "Republican"
}
}
}
I saved this to a mock.json and called the script with a parameter mock_rest.py --data-file="./mock.json".
I hope that gives you a starting point and a good example.
If your load_web_app function uses the requests library to access the REST API, using requests-mock is a convenient way to fake that library's functionality for test purposes.
For those who stumble upon this question, and do not want to end up writing the code to create their own mock server implementations of the API, you can use Mocktastic, which is a downloadable desktop application for Windows, MacOS and Linux, which provides an easy to use GUI to setup your mock API servers.
The title is fairly self-explanatory, so I'll just show some code I've tried so far:
From https://stackoverflow.com/a/713950,
import cherrypy
from cherrypy import expose
cherrypy.config.update({'server.socket_port': 80})
class Test:
#expose
def test_call(self):
return "Testing"
cherrypy.quickstart(Test())
Also, from another SO post, two variants on the following:
cherrypy.config.update({
'server.socket_port': 80,
'/': {
'request.dispatch': cherrypy.dispatch.MethodDispatcher(),
'tools.trailing_slash.on': False
}
})
class Test:
def test_call(self, *args):
return json.dumps(args)
test_call.exposed = True
class API:
def __init__(self):
self.Test = Test()
class Root:
def __init__(self):
self.API = API()
cherrypy.tree.mount(Root())
cherrypy.quickstart(Root())
with a variation suggested here: Path Not Found in CherryPy
cherrypy.quickstart(cherrypy.Application(Root(), '/', {}))
I run these and access http://mysite.com/test_call or, in the other case, mysite.com/api/test/test_call, and neither of these seem to be doing much of anything except returning a 404. Ideas?
I am completely open to trying a different framework if it'll just let me expose a few function calls to dump JSON. I don't need anything fancy or cool, just functioning.
EDIT: Apparently my problem was that the server was by default expecting to be localhost, which basically makes me an idiot. Adding cherrypy.server.socket_host = "mydomain.com" fixes this.
The title doesn't match with examples and is telling that you were likely misguided by REST adepts, in the answer's comments you linked, who tend call everything a "RPC", which deviates from their CRUD-limited resource perspective. JSON RPC is the certain specification, which defines JSON structures for request and response. It looks like this.
--> {"jsonrpc": "2.0", "method": "subtract", "params": [42, 23], "id": 1}
<-- {"jsonrpc": "2.0", "result": 19, "id": 1}
Your examples have nothing to do with it, as well as with REST. They are a copy-paste without understanding the subject. Let's sort out a little.
CherryPy has serveral dispatching options. Default dispatcher maps request URL segments to Python object tree, like /API/Test to root.API.Test in your second example. Its common use is regular GET/POST web flow.
If you want to implement a RESTful API, here's dedicated manual page for it: Creating RESTful applications in CherryPy. It briefly explains the style and usage of MethodDispatcher.
If actual JSON RPC is what you want, then you can look at python-jsonrpc package, which has CherryPy adapter.
Though, if all you want to achieve is to return a JSON string with appropriate content-type header, CherryPy has a specific tool for it, so there's no point to do it manually.
Here follows example for #4.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import cherrypy
config = {
'global' : {
'server.socket_host' : '127.0.0.1',
'server.socket_port' : 8080,
'server.thread_pool' : 8
}
}
class Api:
#cherrypy.expose
#cherrypy.tools.json_out()
def oneway(self):
'''Just open http://127.0.0.1:8080/api/oneway'''
return {
'foo' : 'bar',
'baz' : 'another one'
}
#cherrypy.expose
#cherrypy.tools.json_in()
#cherrypy.tools.json_out()
def twoway(self):
'''You can call it like:
curl -X POST -H "Content-Type: application/json" \
-d '{"foo":123,"bar":"baz"}' http://127.0.0.1:8080/api/twoway
'''
data = cherrypy.request.json
return data.items()
if __name__ == '__main__':
cherrypy.quickstart(Api(), '/api', config)
I tried the first script as below (with 2 notes, using root user if using port 80). Access it by "http:// 127.0.0.1 /test_call". It works.
You should be more specific to raise your question (giving your codes), so the audience knows how to help solve it.
#! /usr/bin/python3
import cherrypy
from cherrypy import expose
## NOTE 1 - using 0.0.0.0
cherrypy.config.update({'server.socket_host' : '0.0.0.0', 'server.socket_port': 80})
class Test:
#expose
def test_call(self):
return "Testing"
## NOTE 2 - a work around for python3 and cherrypy v3.x
## work around OSERR issue - OSError: Port 7766 not bound on '10.220.203.233'
## refer to http://stackoverflow.com/questions/767575/crp-hello-world-error
def fake_wait_for_occupied_port(host, port): return
cherrypy.process.servers.wait_for_occupied_port = fake_wait_for_occupied_port
cherrypy.quickstart(Test())