I have written my own custom dispatcher that uses regular expressions to map routes, however, I can no longer host static files in /static. Here is the dispatcher and the config:
class Dispatcher(object):
def __init__(self):
self.urls = {}
def __call__(self, path_info):
print('Dispatcher called: ' + path_info)
func = self.find_handler(path_info)
cherrypy.serving.request.handler = func
def find_handler(self, path_info):
request = cherrypy.serving.request
request.config = cherrypy.config.copy()
for url in self.urls:
args = re.findall(url, path_info)
if len(args) > 0:
# in the case that the route is just a URL, we don't want
# an extra argument in the method function
try:
args.remove(path_info)
except ValueError:
pass
controller = self.urls[url]
method = request.method.lower()
return cherrypy._cpdispatch.LateParamPageHandler(getattr(controller, method), *args)
return cherrypy.NotFound()
def connect(self, url, controller):
if not url.endswith("$"):
url += "$"
self.urls[url] = controller
And the config:
config = {
'global': {
'server.socket_host': '0.0.0.0',
'server.socket_port': port,
},
'/static': {
'tools.staticdir.on': True,
'tools.staticdir.dir': os.path.join(os.getcwd(), 'static'),
},
'/': {
'request.dispatch': self.dispatcher,
}
}
If I use the standard dispatcher, static files work as they should, however if I use my own, they no longer work. Having done debugging in the dispatcher, static files go through the dispatcher, even though I have specific that only in '/' does the dispatcher get used.
I'm not familiar with cherrypy, but it seems obvious: everything in /static is also in /, so it is anyone's guess which config entry it will use. I would hope that "more specific has priority", but according to your description, this is not the case. Looking at the documentation also doesn't help, there's no mention of ambiguous path handling.
You would think that changing the order might help, but since this is a dictionary, the order isn't preserved.
It seems that cherrypy is unable to do this. If it has a default dispatcher which is overloaded with others, that could solve the problem. Another option is that your custom dispatcher can call the static one if it detects the path.
Finally, the documentation talks about "mounting an application to a path". If you do this, you may want to change the order. If you don't do this, it might be done automatically, and doing it manually may solve your problem.
Not all of this may make sense, since as I wrote I'm not familiar with cherrypy, but I hope it helps you a bit anyway.
Related
Overview
When creating a post request from my website to my Python server running CherryPy, I receive the error Access to XMLHttpRequest has been blocked by CORS policy: Request header field content-type is not allowed by Access-Control-Allow-Headers in preflight response. . I was able to get away with the problem temporarily with one of the "CORS Everywhere" browser extensions, but
Due to recent updates, the extensions have not yet been updated to be working again.
The website involved needs to eventually be used by many in my local complex without the browser extension, so once the extensions get updated, it does not really matter one way or another, as I cannot rely on these extensions, and force everyone to use them (when there is obviously a fix that would make an extension not necessary).
I figure that perhaps the solutions are outdated, but am not sure.
Here is the relevant code:
On the server side (CherryPy/Python):
The CherryPy Python function being called, from the website post request
#cherrypy.expose
#cherrypy.tools.json_in()
def add_meeting(self):
data = None
id = None
start_time = None
end_time = None
title = None
userlist = None
result = {"operation": "request", "result": "success"}
if cherrypy.request.method == "POST":
data = cherrypy.request.json
id = data["id"]
start_time = data["start_time"]
end_time = data["end_time"]
title = data["title"]
userlist = data["userlist"]
# Rest of relevant code in function is left out, to take up less
# space and not post irrelevant code. That being said, I am
# positive the logic is correct, as it originally ran smoothly
# with a "Cors Everywhere" Browser Extension.
return result
Here is the area where I set up and run CherryPy
def main():
# Create the configuration file parser object and start the CherryPy server
config = ConfigParser.ConfigParser()
config.read(CONFIG_FILE)
port = config.getint('Meta', 'port')
host = config.get('Meta', 'host')
cherrypy.config.update({'server.socket_port': port,
'server.socket_host': host,
'tools.CORS.on': True})
cherrypy.quickstart(Coordinator(config))
main()
Here is the config file mentioned in the code above (CONFIG_FILE)
[Meta]
host = 0.0.0.0
port = 3000
# Rest is left out, as it is irrelevant with problem
The solutions I have tried implementing
The inclusion of the following function above the main function:
def CORS():
cherrypy.response.headers["Access-Control-Allow-Origin"] = "*"
with cherrypy.tools.CORS = cherrypy.Tool('before_handler', CORS)
2. Adding " 'cors.expose.on': True " to cherrypy.config.update above
3. Using this cherrypy-cors Python library I found online: https://pypi.org/project/cherrypy-cors/
4. The inclusion of headers in the config.update portion of the Python file
5. Adding "#cherrypy.tools.accept(media='application/json')" before "def add_meeting"
Conclusion
I've tried the solutions above together, separately, some with and without the others, and I am still stuck. Maybe some of these solutions are partially correct, and there is something extra needed with my code. I am not sure; I just cannot get it working. I do not have much experience with web development before this, so maybe (and hopefully) the solution is extremely simple. I know the code works, I just cannot get it running without a working "Cors Everywhere" browser extension for every user.
As for the versions I am running: I am using CherryPy 14.2.0 and Python 2.7.6
Any help would mean the absolute world to me, thank you.
So first, you need to set pre-flight headers when processing OPTIONS request, you can list allowed methods there.
Then, you also need to enable the cors.expose tool.
There's some usage hints in the docstring of cherrypy-cors. For example, when using a MethodDispatcher, you could just decorate an OPTIONS handler method with #cherrypy_cors.tools.preflight() instead of doing this in every HTTP handler.
Here's a simple traversal example (without a method dispatcher). To test it, visit http://127.0.0.1/ and it will make requests against http://localhost:3333/add_meeting which is a different Origin in terms of CORS ('localhost' != '127.0.0.1').
"""Example of CORS setup using cherrypy-cors library."""
import cherrypy
import cherrypy_cors
# Python 2 compat: make all classes new-style by default
__metaclass__ = type # pylint: disable=invalid-name
class WebRoot:
"""Root node for HTTP handlers."""
#cherrypy.expose
def index(self): # pylint: disable=no-self-use
"""Render a web page handling request against ``/``.
Contains client JS snippet which will query the API endpoint.
It will be executed by the browser while loading the page.
"""
return """<html>
<script type="text/javascript">
async function addMeeting() {
/*
* Example coroutine for querying /add_meeing
* HTTP endpoint. It uses localhost as in the URL.
* For testing CORS, make sure to visit
* http://127.0.0.1/ which is a different origin
* from browser's perspective.
* /
const request_payload = {
some: 'data',
listed: ['h', 'er', 'e'],
}
try {
const resp = await fetch(
'http://localhost:3333/add_meeting',
{
method: 'POST',
mode: 'cors', // Required for customizing HTTP request headers
credentials: 'same-origin',
headers: {
'Content-Type': 'application/json; charset=UTF-8', // Required for ``cherrypy.tools.json_in`` to identify JSON payload and parse it automatically
},
body: JSON.stringify(request_payload),
},
)
const json_resp = await resp.json()
console.log(json_resp) // Will print: {"method": "POST", "payload": {"listed": ["h", "er", "e"], "some": "data"}}
} catch (e) {
console.warn('Exception: ' + e)
}
}
async function main() {
await addMeeting()
}
main() // Entry point
</script>
</html>""" # noqa: E501
#cherrypy.expose
#cherrypy.tools.json_in() # turn HTTP payload into an object; also checking the Content-Type header
#cherrypy.tools.json_out() # turn ``return``ed Python object into a JSON string; also setting corresponding Content-Type
def add_meeting(self):
"""Handle HTTP requests against ``/add_meeting`` URI."""
if cherrypy.request.method == 'OPTIONS':
# This is a request that browser sends in CORS prior to
# sending a real request.
# Set up extra headers for a pre-flight OPTIONS request.
cherrypy_cors.preflight(allowed_methods=['GET', 'POST'])
if cherrypy.request.method == 'POST':
return {'method': 'POST', 'payload': cherrypy.request.json}
return {'method': 'non-POST'}
def main():
"""Set up and run the web app.
Initializes CORS tools.
Sets up web server socket.
Enables the CORS tool.
"""
cherrypy_cors.install()
cherrypy.config.update({
'server.socket_host': '127.0.0.1',
'server.socket_port': 3333,
'cors.expose.on': True,
})
cherrypy.quickstart(WebRoot())
__name__ == '__main__' and main() # pylint: disable=expression-not-assigned
Lets say I have a very simple web app which is presented as blue if the current president is a democrat and red if they are a republican. A REST API is used to get the current president, via the endpoint:
/presidents/current
which currently returns the json object:
{name: "Donald Trump", party: "Republican"}
So when my page loads I call the endpoint and I show red or blue depending on who is returned.
I wish to test this HTML/javascript page and I wish to mock the back-end so that I can control from within the test environment the API responses. For example:
def test_republican():
# configure the response for this test that the web app will receive when it connects to this endpoint
configure_endpoint(
"/presidents/current",
jsonify(
name="Donald Trump",
party="Republican"
)
)
# start the web app in the browser using selenium
load_web_app(driver, "http://localhost:8080")
e = driver.find_element_by_name("background")
assert(e.getCssValue("background-color") == "red")
def test_democrat():
# configure the response for this test that the web app will receive when it connects to this endpoint
configure_endpoint(
"/presidents/current",
jsonify(
name="Barack Obama",
party="Democrat"
)
)
# start the web app in the browser using selenium
load_web_app(driver, "http://localhost:8080")
e = driver.find_element_by_name("background")
assert(e.getCssValue("background-color") == "blue")
So the question is how should I implement the function configure_endpoint() and what libraries can you recommend me?
As #Kie mentioned, configure_endpoint implementation won't be enough, if you're going to stub the whole server-side within Selenium Python code. You would need a web server or whatever that will response via HTTP to requests from within testing environment.
It looks like the question is partially about testing of client-side code. What I see is that you're trying to make unit-test for client-side logic, but use integration testing suite in order to check this logic (it's strange).
The main idea is as follows.
You're trying to test client-side code. So, let's make mocks client-side too! Because this part of code is completely client-side related stuff.
If you actually want to have mocks, not stubs (watch the difference here: https://stackoverflow.com/a/3459491/882187) it is a better way to mock out HTTP requests inside your Javascript code. Just because you're testing a client-side piece of code, not some parts of server-side logic.
Having it isolated from whatever server-side is - is a great idea that you would love when your project become grow, while more and more endpoints will be appearing.
For example, you can use the following approach:
var restResponder = function() { // the original responder your client-side app will use
this.getCurrentPresident = function(successCallback) {
$.get('/presidents/current', callback);
}
};
var createMockResponder = function(president, party){ // factory that creates mocks
var myPresident = president;
var myParty = party;
return function() {
this.getCurrentPresident = function (successCallback) {
successCallback({"name": myPresident, "party": myParty});
}
};
}
// somewhere swap the original restResponder with new mockResponder created by 'createMockResponder'
// then use it in your app:
function drawColor(restResponder, backgroundEl) {
restResponder.getCurrentPresident(function(data){
if (data.party == "Democrat") $(backgroundEl).style('background-color', 'blue')
else if (data.party == "Republican") $(backgroundEl).style('background-color', 'red')
else console.info('Some strange response from server... Nevermind...');
});
}
Practically, this implementation depends on what do you have at the client-side as a framework. If jQuery, then my example is enough, but it looks very wordy. In case you have something more advanced, like AngularJS, you can do the same in 2-3 lines of code:
// Set up the mock http service responses
$httpBackend = $injector.get('$httpBackend');
// backend definition common for all tests
authRequestHandler = $httpBackend.when('GET', '/auth.py')
.respond({userId: 'userX'}, {'A-Token': 'xxx'});
Check out the docs: https://docs.angularjs.org/api/ngMock/service/$httpBackend
If you're still stick to the idea, that you need mocks inside Selenium tests, please
try this project: https://turq.readthedocs.io/en/latest/
It serves with Python DSL for describing REST responders.
Using turq your mocks will look as follows:
path('/presidents/current').json({'name':'Barack Obama', 'party': 'Democrat'}, jsonp=False)
Also, I would recommend to try stubs instead of mocks and use this Python module: mock-server https://pypi.python.org/pypi/mock-server/0.3.7
You are required to create the directory layout containing corresponding pre-populated JSON responses and to add some boilerplate code in order to make the mock-server respond on 'localhost:8080'. The directory layout for your example will look like this:
stub_obama/
presidents/
current/
GET_200.json # will contain {"name": "Barack Obama", "party": "Democrat"}
stub_trump/
presidents/
current/
GET_200.json # will contain {"name": "Donald Trump", "party": "Republican"}
But the mock_server is based on Tornado, it is very heavy solution for using in tests I think.
I hope, my answer is helpful and informative. Welcome to discuss it! I made tons of projects with Selenium, big and small tests, tested client-side and server-side.
I would use tornado web framework.
import json
import functools
import operator
from tornado import ioloop, web, gen
from tornado.options import define, options
define("data_file", default='default/mock.json', type=str)
class Handler(web.RequestHandler):
def data_received(self, chunk):
pass
def initialize(self, data):
self.data = data
#gen.coroutine
def get(self, *args, **kwargs):
path = self.request.path.split("/")[1:]
path = functools.reduce(
operator.add,
[[k, v[0].decode("utf-8")] for k, v in self.request.query_arguments.items()],
path
)
try:
self.write(functools.reduce(operator.getitem, path, self.data))
except KeyError:
self.set_status(404)
class Application(web.Application):
def __init__(self):
data = {}
with open(options.data_file) as data_file:
data = json.load(data_file)
handlers = [
('(.*)', Handler, {"data": data})
]
settings = dict(
gzip=True,
static_hash_cache=True,
)
web.Application.__init__(self, handlers, **settings)
def main():
io_loop = ioloop.IOLoop.instance()
backend_application = Application()
backend_application.listen(8001)
io_loop.start()
if __name__ == "__main__":
main()
This is a code I used for mocking a REST-API which is a standalone script, but it can be embedded into your test environment as well.
I defined a JSON file which defines the different path components and what should be returned. Like this:
{
"presidents": {
"current": {
"name": "Donald Trump",
"party": "Republican"
}
}
}
I saved this to a mock.json and called the script with a parameter mock_rest.py --data-file="./mock.json".
I hope that gives you a starting point and a good example.
If your load_web_app function uses the requests library to access the REST API, using requests-mock is a convenient way to fake that library's functionality for test purposes.
For those who stumble upon this question, and do not want to end up writing the code to create their own mock server implementations of the API, you can use Mocktastic, which is a downloadable desktop application for Windows, MacOS and Linux, which provides an easy to use GUI to setup your mock API servers.
I'm using python-raven to automatically capture all 500 in my django project, and it works great. I'm also forwarding some exceptions that I handle and append them with a special tag to be able to filter them out. The problem is that I can not filter for messages that is missing a particular tag so I'd like to set a default tag for all messages, but can't get it to work.
I've tried the following but it's just ignored.
RAVEN_CONFIG = {
'dsn': 'udp://x:y#z:q/w',
'tags': {'testtag': 'value'},
}
Does anyone know how to send a default tag to sentry?
The following works for me in the settings module.
It adds the environment variable DJANGO_ENVDIR to every message being sent.
from raven.contrib.django.client import DjangoClient as RavenDjangoClient
class SentryDjangoClient(RavenDjangoClient):
def build_msg(self, *args, **kwargs):
data = super(RavenDjangoClient, self).build_msg(*args, **kwargs)
data['tags']['ENVDIR'] = os.environ.get('DJANGO_ENVDIR', 'unset')
return data
if get_env_var('SENTRY_DSN', False):
RAVEN_CONFIG = {
'dsn': get_env_var('SENTRY_DSN'),
# NOTE: timeout set via DSN
}
SENTRY_CLIENT = 'project.settings.base.SentryDjangoClient'
You need to adjust the SENTRY_CLIENT setting, according to where you put the SentryDjangoClient class, which extends the build_msg method.
You could assign a different name to the logger, before logging the message. That would allow you to filter by the "logger" in the Sentry server.
After looking into it a bit more it seems like this is not at all possible ATM.
The title is fairly self-explanatory, so I'll just show some code I've tried so far:
From https://stackoverflow.com/a/713950,
import cherrypy
from cherrypy import expose
cherrypy.config.update({'server.socket_port': 80})
class Test:
#expose
def test_call(self):
return "Testing"
cherrypy.quickstart(Test())
Also, from another SO post, two variants on the following:
cherrypy.config.update({
'server.socket_port': 80,
'/': {
'request.dispatch': cherrypy.dispatch.MethodDispatcher(),
'tools.trailing_slash.on': False
}
})
class Test:
def test_call(self, *args):
return json.dumps(args)
test_call.exposed = True
class API:
def __init__(self):
self.Test = Test()
class Root:
def __init__(self):
self.API = API()
cherrypy.tree.mount(Root())
cherrypy.quickstart(Root())
with a variation suggested here: Path Not Found in CherryPy
cherrypy.quickstart(cherrypy.Application(Root(), '/', {}))
I run these and access http://mysite.com/test_call or, in the other case, mysite.com/api/test/test_call, and neither of these seem to be doing much of anything except returning a 404. Ideas?
I am completely open to trying a different framework if it'll just let me expose a few function calls to dump JSON. I don't need anything fancy or cool, just functioning.
EDIT: Apparently my problem was that the server was by default expecting to be localhost, which basically makes me an idiot. Adding cherrypy.server.socket_host = "mydomain.com" fixes this.
The title doesn't match with examples and is telling that you were likely misguided by REST adepts, in the answer's comments you linked, who tend call everything a "RPC", which deviates from their CRUD-limited resource perspective. JSON RPC is the certain specification, which defines JSON structures for request and response. It looks like this.
--> {"jsonrpc": "2.0", "method": "subtract", "params": [42, 23], "id": 1}
<-- {"jsonrpc": "2.0", "result": 19, "id": 1}
Your examples have nothing to do with it, as well as with REST. They are a copy-paste without understanding the subject. Let's sort out a little.
CherryPy has serveral dispatching options. Default dispatcher maps request URL segments to Python object tree, like /API/Test to root.API.Test in your second example. Its common use is regular GET/POST web flow.
If you want to implement a RESTful API, here's dedicated manual page for it: Creating RESTful applications in CherryPy. It briefly explains the style and usage of MethodDispatcher.
If actual JSON RPC is what you want, then you can look at python-jsonrpc package, which has CherryPy adapter.
Though, if all you want to achieve is to return a JSON string with appropriate content-type header, CherryPy has a specific tool for it, so there's no point to do it manually.
Here follows example for #4.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import cherrypy
config = {
'global' : {
'server.socket_host' : '127.0.0.1',
'server.socket_port' : 8080,
'server.thread_pool' : 8
}
}
class Api:
#cherrypy.expose
#cherrypy.tools.json_out()
def oneway(self):
'''Just open http://127.0.0.1:8080/api/oneway'''
return {
'foo' : 'bar',
'baz' : 'another one'
}
#cherrypy.expose
#cherrypy.tools.json_in()
#cherrypy.tools.json_out()
def twoway(self):
'''You can call it like:
curl -X POST -H "Content-Type: application/json" \
-d '{"foo":123,"bar":"baz"}' http://127.0.0.1:8080/api/twoway
'''
data = cherrypy.request.json
return data.items()
if __name__ == '__main__':
cherrypy.quickstart(Api(), '/api', config)
I tried the first script as below (with 2 notes, using root user if using port 80). Access it by "http:// 127.0.0.1 /test_call". It works.
You should be more specific to raise your question (giving your codes), so the audience knows how to help solve it.
#! /usr/bin/python3
import cherrypy
from cherrypy import expose
## NOTE 1 - using 0.0.0.0
cherrypy.config.update({'server.socket_host' : '0.0.0.0', 'server.socket_port': 80})
class Test:
#expose
def test_call(self):
return "Testing"
## NOTE 2 - a work around for python3 and cherrypy v3.x
## work around OSERR issue - OSError: Port 7766 not bound on '10.220.203.233'
## refer to http://stackoverflow.com/questions/767575/crp-hello-world-error
def fake_wait_for_occupied_port(host, port): return
cherrypy.process.servers.wait_for_occupied_port = fake_wait_for_occupied_port
cherrypy.quickstart(Test())
I have 2 files compiled by django-pipeline along with s3boto: master.css and master.js. They are set to "Public" in my buckets. However, when I access them, sometimes master.css is served, sometimes it errs with SignatureDoesNotMatch. The same with master.js. This doesn't happen on Chrome. What could I be missing?
EDIT: It now happens on Chrome too.
Happened to me too...
Took a few hours to find, but I figured it out eventually.
Turns out that if the right signature is :
ssCNsAOxLf5vA80ldAI3M0CU2%2Bw=
Then AWS will NOT accept:
ssCNsAOxLf5vA80ldAI3M0CU2+w=
Where the only difference is the translation of %2B to '+'.
S3BotoStorage actually yields it correctly but the encoding happens on CachedFilesMixin in the final line of the url method (return unquote(final_url)).
To fix it, I derived a new CachedFilesMixin to undo the "damage" (I should mention that I don't know why this unquote exists in the first place, so undoing it might cause other problems)
class MyCachedFilesMixin(CachedFilesMixin):
def url(self, *a, **kw):
s = super(MyCachedFilesMixin, self).url(*a, **kw)
if isinstance(s, unicode):
s = s.encode('utf-8', 'ignore')
scheme, netloc, path, qs, anchor = urlparse.urlsplit(s)
path = urllib.quote(path, '/%')
qs = urllib.quote_plus(qs, ':&=')
return urlparse.urlunsplit((scheme, netloc, path, qs, anchor))
Where I used the code I found here.
Hope this helps...
I had a similar issue causing SignatureDoesNotMatch errors when downloading files using an S3 signed URL and the python requests HTTP library.
My problem ended up being a bad content-type. The documentation at AWS on Authenticating REST Requests helped me figure it out, and has examples in Python.
I was struggling with this for a while, and I didn't like the idea of messing up with CachedFilesMixin (seemed like an overkill to me).
Until a proper fix is issued to the django platform, I've found quoting the signature two times is a good option. I know it's not pretty, but it works and it's simple.
So you'll just have to do something like this:
signature = urllib.quote_plus(signature.strip())
signature = urllib.quote_plus(signature.strip())
Hope it helps!
This article on Flask is a good resource on getting your signatures right: https://devcenter.heroku.com/articles/s3-upload-python
#app.route('/sign_s3/')
def sign_s3():
AWS_ACCESS_KEY = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
S3_BUCKET = os.environ.get('S3_BUCKET')
object_name = request.args.get('s3_object_name')
mime_type = request.args.get('s3_object_type')
expires = int(time.time()+10)
amz_headers = "x-amz-acl:public-read"
put_request = "PUT\n\n%s\n%d\n%s\n/%s/%s" % (mime_type, expires, amz_headers, S3_BUCKET, object_name)
signature = base64.encodestring(hmac.new(AWS_SECRET_KEY,put_request, sha).digest())
signature = urllib.quote_plus(signature.strip())
url = 'https://%s.s3.amazonaws.com/%s' % (S3_BUCKET, object_name)
return json.dumps({
'signed_request': '%s?AWSAccessKeyId=%s&Expires=%d&Signature=%s' % (url, AWS_ACCESS_KEY, expires, signature),
'url': url
})
Simple workaround for me was to generate a new access key with only alphanumeric characters (ie no special characters such as "/", "+", etc. which AWS sometimes adds to the keys).