Creating state in a Flask webapplication - python

I'm building a webapplication with the Flask framework of python. On the server I would like to preserve some state. I think the following code example makes my goal clear (and was also my initial idea):
name = ""
#app.route('/<input_name>')
def home(input_name):
global name
name = input_name
return "name set"
#app.route('/getname')
def getname():
global name
return name
Though when I deployed my website the response for an /getname request behaves inconsistent because there are multiple thread-instances of the code (I could be wrong). I have some plausible solutions to overcome this problem but I wonder if there would be a more 'clean' solution:
Solution 1: read and write the name from a database (a database seems like overkill if a only want to store 1 variable)
Solution 2: store the value of name in a file and setup a locking mechanism so that only one process/thread could write to the file at the same moment.
Goal: when client 'A' requests www.website.com/sven and after that client 'B' requests www.website.com/getname I want the response for client B to be 'sven'
Any suggestions?

Your example should not be done with global state, it will not work for the reason you mentioned - requests might land into different processes that will have different global values.
You can handle storing global variables by using a key-value cache, such as memcached or Redis, or file-system based cache - check Flask-Chaching package and particular docs https://flask-caching.readthedocs.io/en/latest/#built-in-cache-backends

Related

Simple way to delete existing pods from Python

I have end-to-end tests written in pytest, running on a Kubernetes cluster in Namespace foo. Now I want to add simple chaos engineering to the tests to check the resilience of my services. For this, I only need to delete specific pods within foo -- since K8s restarts the corresponding service, this simulates the service being temporarily unreachable.
What is a simple way to delete a specific pod in the current namespace with Python?
What I have tried so far:
Since I did not find a suitable example in https://github.com/kubernetes-client/python/tree/master/examples, but the ones using pods looked quite complex,
I looked into kubetest, which seems very simple and elegant.
I wanted to use the kube fixture and then do something like this:
pods = kube.get_pods()
for pod in pods:
if can_be_temporarily_unreachable(pod):
kube.delete(pod)
I thought calling pytest with parameter --in-cluster would tell kubetest to use the current cluster setup and not create new K8s resources.
However, kubetest wants to create a new namespace for each test case that uses the kube fixture, which I do not want.
Is there a way to tell kubetest not to create new namespaces but do everything in the current namespace?
Though kubetest looks very simple and elegant otherwise, I am happy to use another solution, too.
A simple solution that requires little time and maintenance and does not complicate (reading) the tests would be awesome.
you can use delete_namespaced_pod(from CoreV1Api) to delete specific pods within a namespace.
Here is an example:
from kubernetes import client, config
from kubernetes.client.rest import ApiException
config.load_incluster_config() # or config.load_kube_config()
configuration = client.Configuration()
with client.ApiClient(configuration) as api_client:
api_instance = client.CoreV1Api(api_client)
namespace = 'kube-system' # str | see #Max Lobur's answer on how to get this
name = 'kindnet-mpkvf' # str | Pod name, e.g. via api_instance.list_namespaced_pod(namespace)
try:
api_response = api_instance.delete_namespaced_pod(name, namespace)
print(api_response)
except ApiException as e:
print("Exception when calling CoreV1Api->delete_namespaced_pod: %s\n" % e)
A continuation to an existing answer by Tibebes. M
How do I find out the namespace that the code itself is running in?
There's two ways to do this:
Use Downward API to pass pod namespace to a pod environment variable:
https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#the-downward-api
https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/#capabilities-of-the-downward-api See metadata.namespace
Use your template engine / helm to additionally pass environment variable namespace to a pod. Example:
env:
- name: CURRENT_NAMESPACE
value: "{{ .Values.namespace }}"

How to integrate Lambda, Alexa, and my code (Python - Tweepy)?

I am trying to tweet something by talking to Alexa. I want to put my code on AWS Lambda, and trigger the function by Alexa.
I already have a Python code that can tweet certain string successfully. And I also managed to create a zip file and deploy it on Lambda (code depends on the "tweepy" package). However, I could not get to trigger the function by Alexa, I understand that I need to use handlers and ASK-SDK (Alexa Service Kit), but I am kind of lost at this stage. Could anyone please give me an idea about how the handlers work and help me see the big picture?
Alexa ASK_SDK Psuedo Code:
This is pseudo code of the new ASK_SDK, which is the predecessor to the ALEXA_SDK.
Also note I work in NodeJS but the structure is likely the same
Outer Function with Call Back - Lambda Function Handler
CanHandle Function
Contains logic to determine if this handler is the right handler. The HandlerInput variable contains the request data so you can check and see if the intent == "A specific intent" then return true. Else return false. Or you can go way more specific. (Firing handlers by intent is pretty basic. you can take it a step further and fire handlers based on Intent and State.
Handle Function
Which ever "canHandle" function returns true this is the code that will be run. The handler has a few functions it can perform. It can read the session attributes, change the session attributes based on the intent that was called, formulate a string response, read and write to a more persistant attribute store like dynamodb and create and fire an alexa response.
The handerInput contains everything you'll need. I'd highly recommend running your test code in Pycharm with the debugger and then examining the handlerInput variable.
The response builder is also very important and is what allows you to add speech, follow up prompts, cards, elicit slot values etc.
handler_input.response_builder
Example to Examine
https://github.com/alexa/skill-sample-python-helloworld-classes/blob/master/lambda/py/hello_world.py
class HelloWorldIntentHandler(AbstractRequestHandler):
"""Handler for Hello World Intent."""
def can_handle(self, handler_input):
# type: (HandlerInput) -> bool
return ask_utils.is_intent_name("HelloWorldIntent")(handler_input)
def handle(self, handler_input):
# type: (HandlerInput) -> Response
speak_output = "Hello Python World from Classes!"
return (
handler_input.response_builder
.speak(speak_output)
# .ask("add a reprompt if you want to keep the session open for the user to respond")
.response
)
For your question about capturing user's input for tweeting, use AMAZON.SearchQuery slot type. You may run into limitations about how much text can be collected and the quality of the capture but SearchQuery slot is the place to start.

Python and YAML- GAE app settings file

I was reading about using YAML to store settings in a GAE application and I think I want to go this way. I'm talking about my own constants like API keys and temp variables, NOT about standard GAE config files like app.yaml.
So far I used a simple settings.py file for it (which included separate configs for production/testing/etc environment), but it doesn't seem to do the job well enough.
I even had some serious problems when a git merge overwrote some settings (hard to control it).
Eventually I want to store as much data as I can in the data store but as for now I'm looking for ideas.
So does anybody have any ideas or examples for simple storing and access of, sometimes protected, config data?
Storing the API keys and configuration variables in static files is usually always a bad idea for a few reasons:
Hard to update: you need to deploy a new app in order to update the values
Not secure: you might commit it to the version control system and there is no way back
Don't scale: very often you have different keys for different domains
So why not storing all these values right from the very beginning securely in the datastore, especially when it's actually so much easier and here is how:
All you have to do is to create one new model for your configuration values and have only one record in it. By using NDB, you have caching out of the box and because of the nature of this configuration file you can even cache it per running instance to avoid regular reads to the datastore.
class Config(ndb.Model):
analytics_id = ndb.StringProperty(default='')
brand_name = ndb.StringProperty(default='my-awesome-app')
facebook_app_id = ndb.StringProperty(default='')
facebook_app_secret = ndb.StringProperty(default='')
#classmethod
def get_master_db(cls):
return cls.get_or_insert('master')
I'm also guessing that you already have one file that most likely is called config.py where you have some constants like these ones:
PRODUCTION = os.environ.get('SERVER_SOFTWARE', '').startswith('Google App Engine')
DEVELOPMENT = not PRODUCTION
If you don't create one or if you do just add this to that file as well:
import model # Your module that contains the models
CONFIG_DB = model.Config.get_master_db()
and finally to be able to read your config values you can simply do that, wherever in your app (after importing the config.py of course):
config.CONFIG_DB.brand_name
From now on you don't even have to create any special forms to update these values (it is recommended though), because you can do that from the admin console locally or the Dashboard on the production.
Just remember that if you're going to store that record in a local variable you will have to restart the instances after updating the values to see the changes.
All of the above you can see it in action, in one of my open sourced projects called gae-init.
You should not store config values in source code, for the reasons outlined in the other answers. In my latest project, I put config data in the datastore using this class:
from google.appengine.ext import ndb
class Settings(ndb.Model):
name = ndb.StringProperty()
value = ndb.StringProperty()
#staticmethod
def get(name):
NOT_SET_VALUE = "NOT SET"
retval = Settings.query(Settings.name == name).get()
if not retval:
retval = Settings()
retval.name = name
retval.value = NOT_SET_VALUE
retval.put()
if retval.value == NOT_SET_VALUE:
raise Exception(('Setting %s not found in the database. A placeholder ' +
'record has been created. Go to the Developers Console for your app ' +
'in App Engine, look up the Settings record with name=%s and enter ' +
'its value in that record\'s value field.') % (name, name))
return retval.value
Your application would do this to get a value:
AMAZON_KEY = Settings.get('AMAZON_KEY')
If there is a value for that key in the datastore, you will get it. If there isn't, a placeholder record will be created and an exception will be thrown. The exception will remind you to go to the Developers Console and update the placeholder record.
I find this takes the guessing out of setting config values. If you are unsure of what config values to set, just run the code and it will tell you!

Global variable in Python server

Background: I am a complete beginner when it comes to servers, but I know my way around programming in Python.
I am trying to setup a simple server using the basic Python 2.7 modules (SimpleHTTPServer, CGIHTTPServer, etc). This server needs to load a global, read-only variable with several GB of data from a file when it starts; then, when each user accesses the page, the server uses the big data to generate some output which is then given to the user.
For the sake of example, let's suppose I have a 4 GB file names.txt which contains all possible proper nouns of English:
Jack
John
Allison
Richard
...
Let's suppose that my goal is to read the whole list of names into memory, and then choose 1 name at random from this big list of proper nouns. I am currently able to use Python's native CGIHTTPServer module to accomplish this. To start, I just run the CGIHTTPServer module directly, by executing from a terminal:
python -m CGIHTTPServer
Then, someone accesses www.example-server.net:8000/foo.py and they are given one of these names at random. I have the following code in foo.py:
#!/usr/bin/env python
import random
name_list = list()
FILE = open('names.txt','r')
for line in FILE:
name = line[:-1]
name_list.append(name)
FILE.close()
name_to_return = random.choice(name_list)
print "Content-type: text/html"
print
print "<title>Here is your name</title>"
print "<p>" + name_to_return + "</p>"
This does what I want; however, it is extremely inefficient, because every access forces the server to re-read a 4 GB file.
How can I make this into an efficient process, where the variable name_list is created as global immediately when the server starts, and each access only reads from that variable?
Just for future reference, if anyone ever faces the same problem: I ended up sub-classing CGIHTTPServer's request handler and implementing a new do_POST() function. If you had a working CGI script without global variables, something like this should get you started:
import CGIHTTPServer
import random
import sys
import cgi
class MyRequestHandler(CGIHTTPServer.CGIHTTPRequestHandler):
global super_important_list
super_important_list = range(10)
random.shuffle(super_important_list)
def do_POST(s):
"""Respond to a POST request."""
form = cgi.FieldStorage(fp=s.rfile,headers=s.headers,environ={'REQUEST_METHOD':'POST','CONTENT_TYPE':s.headers['Content-Type'],})
s.wfile.write("<html><head><title>Title goes here.</title></head>")
s.wfile.write("<body><p>This is a test.</p>")
s.wfile.write("<p>You accessed path: %s</p>" % s.path)
s.wfile.write("<p>Also, super_important_list is:</p>")
s.wfile.write(str(super_important_list))
s.wfile.write("<p>Furthermore, you POSTed the following info: ")
for item in form.keys():
s.wfile.write("<p>Item: " + item)
s.wfile.write("<p>Value: " + form[item].value)
s.wfile.write("</body></html>")
if __name__ == '__main__':
server_address = ('', 8000)
httpd = CGIHTTPServer.BaseHTTPServer.HTTPServer(server_address, MyRequestHandler)
try:
httpd.serve_forever()
except KeyboardInterrupt:
sys.exit()
Whenever someone fills out your form and performs a POST, the variable form will be a dictionary-like object with key-value pairs which may differ for each user of your site, but the global variable super_important_list will be the same for every user.
Thanks to everyone who answered my question, especially Mike Steder, who pointed me in the right direction!
CGI works by spawning a process to handle each request. You need to run a server process that stays in memory handles HTTP requests.
You could use a modified BaseHTTPServer, just define your own Handler class. You'd load the dataset once in your code and then the do_GET method of your handler would just pick one randomly.
Personally, I'd look into something like CherryPy as a simple solution that is IMO a lot nicer than BaseHTTPServer. There are tons of options other than CherryPy like bottle, flask, twisted, django, etc. Of course if you need this server to be behind some other webserver you'll need to look into setting up a reverse proxy or running CherryPy as a WSGI app.
You may want to store the values of the names in a db and store the names according to the letter that they start with. Then you can do a random for a letter between a and z and from there randomize again to get a random name from your random beginning letter.
Build a prefix tree (a.k.a. trie) once and generate a random walk whenever you receive a query.
That should be pretty efficient.

Using response.out.write from within an included file in Google App Engine

I think this is quite an easy question to answer, I just haven't been able to find anywhere detailing how to do it.
I'm developing a GAE app.
In my main file I have a few request handlers, for example:
class Query(webapp.RequestHandler):
def post(self):
queryDOI = cgi.escape(self.request.get('doiortitle'))
import queryCosine
self.response.out.write(queryCosine.cosine(queryDOI))
In that handler there I'm importing from a queryCosine.py script which is doing all of the work. If something in the queryCosine script fails, I'd like to be able to print a message or do a redirect.
Inside queryCosine.py there is just a normal Python function, so obviously doing things like
self.response.out.write("Done")
doesn't work. What should I use instead of self or what do I need to include within my included file? I've tried using Query.self.response.out.write instead but that doesn't work.
A much better, more modular approach, is to have your queryCosine.cosine function throw an exception if something goes wrong. Then, your handler method can output the appropriate response depending on the return value or exception. This avoids unduly coupling the code that calculates whatever it is you're calculating to the webapp that hosts it.
Pass it to the function.
main file:
import second
...
second.somefunction(self.response.out.write)
second.py:
def somefunction(output):
output('Done')

Categories

Resources