I have a script that continually runs, processing data that it gets from an external device. The core logic follows something like:
from external_module import process_data, get_data, load_interesting_things
class MyService:
def __init__(self):
self.interesting_items = load_interesting_things()
self.run()
def run(self):
try:
while True:
data = get_data()
for item in self.interesting_items:
item.add_datapoint(process_data(data, item))
except KeyboardInterrupt:
pass
I would like to add the ability to request information for the various interesting things via a RESTful API.
Is there a way in which I can add something like a Flask web service to the program such that the web service can get a stat from the interesting_items list to return? For example something along the lines of:
#app.route("/item/<idx>/average")
def average(idx: int):
avg = interesting_items[idx].getAverage()
return jsonify({"average":avg})
Assuming there is the necessary idx bounds checking and any appropriate locking implemented.
It does not have to be Flask, but it should be light weight. I want to avoid using a database. I would prefer to use a webservice, but if it is not possible without completely restructuring the code base I can use a socket instead, but this is less preferable.
The server would be running on a local network only and usually only handling a single user, sometimes it may have a few.
I needed to move the run() method out of the __init__() method, so that I could have a global reference to the service, and start the run method in a separate thread. Something along the lines of:
service = MyService()
service_thread = threading.Thread(target=service.run, daemon=True)
service_thread.start()
app = flask.Flask("appname")
...
#app.route("/item/<idx>/average")
def average(idx: int):
avg = service.interesting_items[idx].getAverage()
return jsonify({"average":avg})
...
app.run()
Related
I'm using django-rq in my project.
What I want to achieve:
I have a first view that loads a template where an image is acquired from webcam and saved on my pc. Then, the view calls a second view, where an asynchronous task to process the image is enqueued using rq. Finally, after a 20-second delay, a third view is called. In this latter view I'd like to retrieve the result of the asynchronous task.
The problem: the job object is correctly created, but the queue is always empty, so I cannot use queue.fetch_job(job_id). Reading here I managed to find the job in the FinishedJobRegistry, but I cannot access it, since the registry is not iterable.
from django_rq import job
import django_rq
from rq import Queue
from redis import Redis
from rq.registry import FinishedJobRegistry
redis_conn = Redis()
q = Queue('default',connection=redis_conn)
last_job_id = ''
def wait(request): #second view, starts the job
template = loader.get_template('pepper/wait.html')
job = q.enqueue(processImage)
print(q.is_empty()) # this is always True!
last_job_id = job.id # this is the expected job id
return HttpResponse(template.render({},request))
def ceremony(request): #third view, retrieves the result
template = loader.get_template('pepper/ceremony.html')
print(q.is_empty()) # True
registry = FinishedJobRegistry('default', connection=redis_conn)
finished_job_ids = registry.get_job_ids() #here I have the correct id (last_job_id)
return HttpResponse(template.render({},request))
The question: how can I retrieve the result of the asynchronous job from the finished job registry? Or, better, how can I correctly enqueue the job?
I have found an other way to do it: I'm simply using a global list of jobs, that I'm modifying in the views. Anyway, I'd like to know the right way to do this...
My Bash script using kubectl create/apply -f ... to deploy lots of Kubernetes resources has grown too large for Bash. I'm converting it to Python using the PyPI kubernetes package.
Is there a generic way to create resources given the YAML manifest? Otherwise, the only way I can see to do it would be to create and maintain a mapping from Kind to API method create_namespaced_<kind>. That seems tedious and error prone to me.
Update: I'm deploying many (10-20) resources to many (10+) GKE clusters.
Update in the year 2020, for anyone still interested in this (since the docs for the python library is mostly empty).
At the end of 2018 this pull request has been merged,
so it's now possible to do:
from kubernetes import client, config
from kubernetes import utils
config.load_kube_config()
api = client.ApiClient()
file_path = ... # A path to a deployment file
namespace = 'default'
utils.create_from_yaml(api, file_path, namespace=namespace)
EDIT: from a request in a comment, a snippet for skipping the python error if the deployment already exists
from kubernetes import client, config
from kubernetes import utils
config.load_kube_config()
api = client.ApiClient()
def skip_if_already_exists(e):
import json
# found in https://github.com/kubernetes-client/python/blob/master/kubernetes/utils/create_from_yaml.py#L165
info = json.loads(e.api_exceptions[0].body)
if info.get('reason').lower() == 'alreadyexists':
pass
else
raise e
file_path = ... # A path to a deployment file
namespace = 'default'
try:
utils.create_from_yaml(api, file_path, namespace=namespace)
except utils.FailToCreateError as e:
skip_if_already_exists(e)
I have written a following piece of code to achieve the functionality of creating k8s resources from its json/yaml file:
def create_from_yaml(yaml_file):
"""
:param yaml_file:
:return:
"""
yaml_object = yaml.loads(common.load_file(yaml_file))
group, _, version = yaml_object["apiVersion"].partition("/")
if version == "":
version = group
group = "core"
group = "".join(group.split(".k8s.io,1"))
func_to_call = "{0}{1}Api".format(group.capitalize(), version.capitalize())
k8s_api = getattr(client, func_to_call)()
kind = yaml_object["kind"]
kind = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', kind)
kind = re.sub('([a-z0-9])([A-Z])', r'\1_\2', kind).lower()
if "namespace" in yaml_object["metadata"]:
namespace = yaml_object["metadata"]["namespace"]
else:
namespace = "default"
try:
if hasattr(k8s_api, "create_namespaced_{0}".format(kind)):
resp = getattr(k8s_api, "create_namespaced_{0}".format(kind))(
body=yaml_object, namespace=namespace)
else:
resp = getattr(k8s_api, "create_{0}".format(kind))(
body=yaml_object)
except Exception as e:
raise e
print("{0} created. status='{1}'".format(kind, str(resp.status)))
return k8s_api
In above function, If you provide any object yaml/json file, it will automatically pick up the API type and object type and create the object like statefulset, deployment, service etc.
PS: The above code doesn't handler multiple kubernetes resources in one file, so you should have only one object per yaml file.
I see what you are looking for. This is possible with other k8s clients available in other languages. Here is an example in java. Unfortunately the python client library does not support that functionality yet. I opened a new feature request requesting the same and you can either choose to track it or contribute yourself :). Here is the link for the issue on GitHub.
The other way to still do what you are trying to do is to use java/golang client and put your code in a docker container.
I'm writing test cases for code that is called via a route under Flask. I don't want to test the code by setting up a test app and calling a URL that hits the route, I want to call the function directly. To make this work I need to mock flask.request and I can't seem to manage it. Google / stackoverflow searches lead to a lot of answers that show how to set up a test application which again is not what I want to do.
The code would look something like this.
somefile.py
-----------
from flask import request
def method_called_from_route():
data = request.values
# do something with data here
test_somefile.py
----------------
import unittest
import somefile
class SomefileTestCase(unittest.TestCase):
#patch('somefile.request')
def test_method_called_from_route(self, mock_request):
# want to mock the request.values here
I'm having two issues.
(1) Patching the request as I've sketched out above does not work. I get an error similar to "AttributeError: 'Blueprint' object has no attribute 'somefile'"
(2) I don't know how to exactly mock the request object if I could patch it. It doesn't really have a return_value since it isn't a function.
Again I can't find any examples on how to do this so I felt a new question was acceptable.
Try this
test_somefile.py
import unittest
import somefile
import mock
class SomefileTestCase(unittest.TestCase):
def test_method_called_from_route(self):
m = mock.MagicMock()
m.values = "MyData"
with mock.patch("somefile.request", m):
somefile.method_called_from_route()
unittest.main()
somefile.py
from flask import request
def method_called_from_route():
data = request.values
assert(data == "MyData")
This is going to mock the entire request object.
If you want to mock only request.values while keeping all others intact, this would not work.
A few years after the question was asked, but this is how I solved this with python 3.9 (other proposed solutions stopped working with python 3.8 see here). I'm using pytest and pytest-mock, but the idea should be the same across testing frameworks, as long as you are using the native unittest.mock.patch in some capacity (pytest-mock essentially just wraps these methods in an easier to use api). Unfortunately, it does require that you set up a test app, however, you do not need to go through the process of using test_client, and can just invoke the function directly.
This can be easily handled by using the Application Factory Design Pattern, and injecting application config. Then, just use the created app's .test_request_context as a context manager to mock out the request object. using .test_request_context as a context manager, gives everything called within the context access to the request object. Here's an example below.
import pytest
from app import create_app
#pytest.fixture
def request_context():
"""create the app and return the request context as a fixture
so that this process does not need to be repeated in each test
"""
app = create_app('module.with.TestingConfig')
return app.test_request_context
def test_something_that_requires_global_request_object(mocker, request_context):
"""do the test thing"""
with request_context():
# mocker.patch is just pytest-mock's way of using unittest.mock.patch
mock_request = mocker.patch('path.to.where.request.is.used')
# make your mocks and stubs
mock_request.headers = {'content-type': 'application/json'}
mock_request.get_json.return_value = {'some': 'json'}
# now you can do whatever you need, using mock_request, and you do not
# need to remain within the request_context context manager
run_the_function()
mock_request.get_json.assert_called_once()
assert 1 == 1
# etc.
pytest is great because it allows you to easily setup fixtures for your tests as described above, but you could do essentially the same thing with UnitTest's setUp instance methods. Happy to provide an example for the Application Factory design pattern, or more context, if necessary!
with help of Gabrielbertouinataa on this article: https://medium.com/#vladbezden/how-to-mock-flask-request-object-in-python-fdbc249de504:
code:
def print_request_data():
print(flask.request.data)
test:
flask_app = flask.Flask('test_flask_app')
with flask_app.test_request_context() as mock_context:
mock_context.request.data = "request_data"
mock_context.request.path = "request_path"
print_request_data()
Here is an example of how I dealt with it:
test_common.py module
import pytest
import flask
def test_user_name(mocker):
# GIVEN: user is provided in the request.headers
given_user_name = "Some_User"
request_mock = mocker.patch.object(flask, "request")
request_mock.headers.get.return_value = given_user_name
# WHEN: request.header.get method is called
result = common.user_name()
# THEN: user name should be returned
request_mock.headers.get.assert_called_once_with("USERNAME", "Invalid User")
assert result == given_user_name
common.py module
import flask
def user_name():
return flask.request.headers.get("USERNAME", "Invalid User")
What you're trying to do is counterproductive. Following the RFC 2616 a request is:
A request message from a client to a server includes, within the first line of that message, the method to be applied to the resource, the identifier of the resource, and the protocol version in use.
Mocking the Flask request you need to rebuild its structure, what certainly, you will not to want to do!
The best approach should be use something like Flask-Testing or use some recipes like this, and then, test your method.
In flask, I can do this:
render_template("foo.html", messages={'main':'hello'})
And if foo.html contains {{ messages['main'] }}, the page will show hello. But what if there's a route that leads to foo:
#app.route("/foo")
def do_foo():
# do some logic here
return render_template("foo.html")
In this case, the only way to get to foo.html, if I want that logic to happen anyway, is through a redirect:
#app.route("/baz")
def do_baz():
if some_condition:
return render_template("baz.html")
else:
return redirect("/foo", messages={"main":"Condition failed on page baz"})
# above produces TypeError: redirect() got an unexpected keyword argument 'messages'
So, how can I get that messages variable to be passed to the foo route, so that I don't have to just rewrite the same logic code that that route computes before loading it up?
You could pass the messages as explicit URL parameter (appropriately encoded), or store the messages into session (cookie) variable before redirecting and then get the variable before rendering the template. For example:
from flask import session, url_for
def do_baz():
messages = json.dumps({"main":"Condition failed on page baz"})
session['messages'] = messages
return redirect(url_for('.do_foo', messages=messages))
#app.route('/foo')
def do_foo():
messages = request.args['messages'] # counterpart for url_for()
messages = session['messages'] # counterpart for session
return render_template("foo.html", messages=json.loads(messages))
(encoding the session variable might not be necessary, flask may be handling it for you, but can't recall the details)
Or you could probably just use Flask Message Flashing if you just need to show simple messages.
I found that none of the answers here applied to my specific use case, so I thought I would share my solution.
I was looking to redirect an unauthentciated user to public version of an app page with any possible URL params. Example:
/app/4903294/my-great-car?email=coolguy%40gmail.com to
/public/4903294/my-great-car?email=coolguy%40gmail.com
Here's the solution that worked for me.
return redirect(url_for('app.vehicle', vid=vid, year_make_model=year_make_model, **request.args))
Hope this helps someone!
I'm a little confused. "foo.html" is just the name of your template. There's no inherent relationship between the route name "foo" and the template name "foo.html".
To achieve the goal of not rewriting logic code for two different routes, I would just define a function and call that for both routes. I wouldn't use redirect because that actually redirects the client/browser which requires them to load two pages instead of one just to save you some coding time - which seems mean :-P
So maybe:
def super_cool_logic():
# execute common code here
#app.route("/foo")
def do_foo():
# do some logic here
super_cool_logic()
return render_template("foo.html")
#app.route("/baz")
def do_baz():
if some_condition:
return render_template("baz.html")
else:
super_cool_logic()
return render_template("foo.html", messages={"main":"Condition failed on page baz"})
I feel like I'm missing something though and there's a better way to achieve what you're trying to do (I'm not really sure what you're trying to do)
You can however maintain your code and simply pass the variables in it separated by a comma: if you're passing arguments, you should rather use render_template:
#app.route("/baz")
def do_baz():
if some_condition:
return render_template("baz.html")
else:
return render_template("/foo", messages={"main":"Condition failed on page baz"})
Just found the Queue module which is helping me adapt the pyftpdlib module. I'm running an very strict FTP server, and my goal is to restrict the filenames available to upload. This is to prevent people from uploading whatever they want (it's actually the backend of an upload client, not a complete FTP server).
I have this in the ftpserver Authorizer:
def fetch_worlds(queue, username):
while queue.empty():
worlds = models.World.objects.filter(is_synced=True, user__username=username)
print worlds
queue.put(worlds, timeout=1)
class FTPAuthorizer(ftpserver.DummyAuthorizer):
def __init__(self):
self.q = Queue.Queue()
self.t = None # Thread
self.world_item = None
def has_perm(self, username, perm, path=None):
print "Checking permission\n"
if perm not in ['r','w']:
return False
# Check world name
self.t = threading.Thread(target=fetch_worlds, args=(self.q, username))
self.t.daemon = True
self.t.start()
self.world_item = self.q.get()
print "WORLDITEM: %s" % self.world_item
if path is not None:
path = os.path.basename(path)
for world in self.world_item:
test = "{0}_{1}.zip".format(username, world.name)
if path == test:
print "Match on %s" % test
return True
return False
My issue is, after the server starts, the first time I STOR a file, it does an initial db call and gets all the worlds properly. But when I then add another world (for example, set is_synced=True on one, it still returns the old data from self.q.get(). has_perm() is called every time a file is uploaded, and it needs to return live data (to check if a file is allowed).
For example, brand new server:
STOR file.zip, self.q.get() returns <World1, World2>
Update the database via other methods etc
STOR file2.zip, inside fetch_worlds, print worlds returns <World1, World2, World3> but self.q.get() returns <World1, World2>
Just found the Queue module and it seemed like it would be helpful but I can't get the implementation right.
(also couldn't add tag pyftpdlib)
i think this is what could be happening here:
when has_perm is called, you create a Thread that will query a database (?) to add elements to the queue
after calling start the call to the database will take some time
meanwhile in your main thread you entered q.get which will block.
the db call finishes and the result is added to the queue
and is immediately removed from the queue again by the blocking q.get
the queue is now empty, your thread enters the while-loop again and executes the same query again and puts the result onto the queue.
the next call to q.get will return that instead of what it expects.
you see, you could have a race condition here, that already is aparent from the fact that you're adding something to a queue in a loop while you don't have a loop when pulling.
also you assume the element you get from the queue is the result to what you have put onto it before. that doesn't have to be true. if you call has_perm two times this will result in two calls to fetch_worlds with the possibility that the queue.empty() check for one of the calls fails. so only one result will be put onto the queue. now you have two threads waiting on q.get, but only one will get a result, while the oter one waits until one becomes ready...
has_perm looks like it should be a blocking call anyway.