Background: I am a complete beginner when it comes to servers, but I know my way around programming in Python.
I am trying to setup a simple server using the basic Python 2.7 modules (SimpleHTTPServer, CGIHTTPServer, etc). This server needs to load a global, read-only variable with several GB of data from a file when it starts; then, when each user accesses the page, the server uses the big data to generate some output which is then given to the user.
For the sake of example, let's suppose I have a 4 GB file names.txt which contains all possible proper nouns of English:
Jack
John
Allison
Richard
...
Let's suppose that my goal is to read the whole list of names into memory, and then choose 1 name at random from this big list of proper nouns. I am currently able to use Python's native CGIHTTPServer module to accomplish this. To start, I just run the CGIHTTPServer module directly, by executing from a terminal:
python -m CGIHTTPServer
Then, someone accesses www.example-server.net:8000/foo.py and they are given one of these names at random. I have the following code in foo.py:
#!/usr/bin/env python
import random
name_list = list()
FILE = open('names.txt','r')
for line in FILE:
name = line[:-1]
name_list.append(name)
FILE.close()
name_to_return = random.choice(name_list)
print "Content-type: text/html"
print
print "<title>Here is your name</title>"
print "<p>" + name_to_return + "</p>"
This does what I want; however, it is extremely inefficient, because every access forces the server to re-read a 4 GB file.
How can I make this into an efficient process, where the variable name_list is created as global immediately when the server starts, and each access only reads from that variable?
Just for future reference, if anyone ever faces the same problem: I ended up sub-classing CGIHTTPServer's request handler and implementing a new do_POST() function. If you had a working CGI script without global variables, something like this should get you started:
import CGIHTTPServer
import random
import sys
import cgi
class MyRequestHandler(CGIHTTPServer.CGIHTTPRequestHandler):
global super_important_list
super_important_list = range(10)
random.shuffle(super_important_list)
def do_POST(s):
"""Respond to a POST request."""
form = cgi.FieldStorage(fp=s.rfile,headers=s.headers,environ={'REQUEST_METHOD':'POST','CONTENT_TYPE':s.headers['Content-Type'],})
s.wfile.write("<html><head><title>Title goes here.</title></head>")
s.wfile.write("<body><p>This is a test.</p>")
s.wfile.write("<p>You accessed path: %s</p>" % s.path)
s.wfile.write("<p>Also, super_important_list is:</p>")
s.wfile.write(str(super_important_list))
s.wfile.write("<p>Furthermore, you POSTed the following info: ")
for item in form.keys():
s.wfile.write("<p>Item: " + item)
s.wfile.write("<p>Value: " + form[item].value)
s.wfile.write("</body></html>")
if __name__ == '__main__':
server_address = ('', 8000)
httpd = CGIHTTPServer.BaseHTTPServer.HTTPServer(server_address, MyRequestHandler)
try:
httpd.serve_forever()
except KeyboardInterrupt:
sys.exit()
Whenever someone fills out your form and performs a POST, the variable form will be a dictionary-like object with key-value pairs which may differ for each user of your site, but the global variable super_important_list will be the same for every user.
Thanks to everyone who answered my question, especially Mike Steder, who pointed me in the right direction!
CGI works by spawning a process to handle each request. You need to run a server process that stays in memory handles HTTP requests.
You could use a modified BaseHTTPServer, just define your own Handler class. You'd load the dataset once in your code and then the do_GET method of your handler would just pick one randomly.
Personally, I'd look into something like CherryPy as a simple solution that is IMO a lot nicer than BaseHTTPServer. There are tons of options other than CherryPy like bottle, flask, twisted, django, etc. Of course if you need this server to be behind some other webserver you'll need to look into setting up a reverse proxy or running CherryPy as a WSGI app.
You may want to store the values of the names in a db and store the names according to the letter that they start with. Then you can do a random for a letter between a and z and from there randomize again to get a random name from your random beginning letter.
Build a prefix tree (a.k.a. trie) once and generate a random walk whenever you receive a query.
That should be pretty efficient.
Related
New to coding here. I am trying to make an application to call a number and give a set of instructions, this part is easy. After the call hangs up I would like to call again and give a different set of instructions. While testing to see if it's possible I am only calling myself and playing DTMF tones so I can hear that it is functioning as I need. I am trying to pass the instructions to TwiML as a variable so I don't have to write multiple functions to perform similar instructions. However, XML doesn't take variables like that. I know the code I have included is completely wrong but is there a method to perform the action I am trying to get.
def dial_numbers(code):
client.calls.create(to=numberToCall, from_=TWILIO_PHONE_NUMBER, twiml='<Response> <Play digits=code></Play> </Response>')
if __name__ == "__main__":
dial_numbers("1234")
dial_numbers("2222")
As I understand from the question: do you need to define a function to send Twilio instructions to the call?
In order to play digit tones, you need to import from twilio.twiml.voice_response import Play, VoiceResponse from Twilio and create XML command for it.
EASY WAY: And then you create a POST request to the Twilio Echo XML service and put it as URL into call function
HARD WAY: There is an alternative - to use Flask or FastAPI framework as a web server and create a global link via DDNS service like ngrok, if you are interested there is official manual.
Try this one:
def dial_numbers(number_to_call, number_from, digit_code):
from twilio.twiml.voice_response import Play, VoiceResponse # Import response module
import urllib.parse # Import urllib to create url for new xml file
response = VoiceResponse() # Create VoiceResponse instance
response.play('', digits=digit_code) # Create xml string of the digit code
url_of_xml = "http://twimlets.com/echo?Twiml=" # Now use twimlet echo service to create simple xml
string_to_add = urllib.parse.quote(str(response)) # Encode xml code to the url
url_of_xml = url_of_xml + string_to_add # Add our xml code to the service
client.calls.create(to=number_to_call, from_=number_from, url=url_of_xml) # Make a call
dial_numbers(number_to_call = numberToCall, number_from = TWILIO_PHONE_NUMBER, digit_code = "1234")
I have HTML code that talks to a server(just a php script) and writes user input to a json file. However, now I want to write python that notices when the json file is modified and reads the new value. I looked around and found references to watchdog and a lot of examples, but all of the examples seemed to throw the same error every time I modified the file. Here is one of the examples(the others are not noticeably different):
import time
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
class EventHandler(FileSystemEventHandler):
def on_any_event(self, event):
print event
if __name__ == "__main__":
path = "/PATH/TO/YOUR/FOLDER"
event_handler = EventHandler()
observer = Observer()
observer.schedule(event_handler, path, recursive=True)
observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
observer.join()
The error gave the path to the json file, then Bad file descriptor(it did this once for each directory in the file path). I looked into that, and it is caused by my modifying the Json file from outside the code(which I was doing from a text editor for testing purposes), which was what was going to happen anyway with the PHP Script. The way I see it, there are two ways to go on this problem. Either somehow connect the python and php so that the python is a part of the server, or find a new method of event watching in the file system. What do I do?
I got it. It is possible to connect php to python. Essentially, both can use a standardized output stream, such that when you print in python, it gets put into the php, so the connection works both ways. Here's how you do it. In the php, get your variables set up, then use the shell_exec command:
$variableToPassToPython="hi";
$result=shell_exec('Path to python that you use Path to py file that you want to run ' . $variableToPassToPython);
//that will concatenate the variable to the terminal command(shell_exc runs terminal commands) to be sent to the python as an arg
now, to access variables passed to the python in the command line, do this in the python:
import sys
hi=sys.argv[1]#argv[0] is the path of the python file, and argv returns a list of all arguments passed through command line
print(hi+' world!')
#remember that in this case, printing means sending something back to the php file
This lets you do cool stuff in the php(the below code assumes you already ran the python in the above code):
echo $result
//result was the variable that was assigned to the shell_exec command, and that command returns whatever python prints to it
The above code will look like this:
hi world!
You can see how this opens up many more doors than just that
IMPORTANT:
I had a problem with doing this initially. If you are using input from a POST request in your php, that is an array, and shell_exec casts to a string. In php, when you cast an array to a string, it becomes just the word "Array". To fix this, use the implode() function, which properly turns an array into a string so it can be passed as a shell command
I'm building a webapplication with the Flask framework of python. On the server I would like to preserve some state. I think the following code example makes my goal clear (and was also my initial idea):
name = ""
#app.route('/<input_name>')
def home(input_name):
global name
name = input_name
return "name set"
#app.route('/getname')
def getname():
global name
return name
Though when I deployed my website the response for an /getname request behaves inconsistent because there are multiple thread-instances of the code (I could be wrong). I have some plausible solutions to overcome this problem but I wonder if there would be a more 'clean' solution:
Solution 1: read and write the name from a database (a database seems like overkill if a only want to store 1 variable)
Solution 2: store the value of name in a file and setup a locking mechanism so that only one process/thread could write to the file at the same moment.
Goal: when client 'A' requests www.website.com/sven and after that client 'B' requests www.website.com/getname I want the response for client B to be 'sven'
Any suggestions?
Your example should not be done with global state, it will not work for the reason you mentioned - requests might land into different processes that will have different global values.
You can handle storing global variables by using a key-value cache, such as memcached or Redis, or file-system based cache - check Flask-Chaching package and particular docs https://flask-caching.readthedocs.io/en/latest/#built-in-cache-backends
I think this is quite an easy question to answer, I just haven't been able to find anywhere detailing how to do it.
I'm developing a GAE app.
In my main file I have a few request handlers, for example:
class Query(webapp.RequestHandler):
def post(self):
queryDOI = cgi.escape(self.request.get('doiortitle'))
import queryCosine
self.response.out.write(queryCosine.cosine(queryDOI))
In that handler there I'm importing from a queryCosine.py script which is doing all of the work. If something in the queryCosine script fails, I'd like to be able to print a message or do a redirect.
Inside queryCosine.py there is just a normal Python function, so obviously doing things like
self.response.out.write("Done")
doesn't work. What should I use instead of self or what do I need to include within my included file? I've tried using Query.self.response.out.write instead but that doesn't work.
A much better, more modular approach, is to have your queryCosine.cosine function throw an exception if something goes wrong. Then, your handler method can output the appropriate response depending on the return value or exception. This avoids unduly coupling the code that calculates whatever it is you're calculating to the webapp that hosts it.
Pass it to the function.
main file:
import second
...
second.somefunction(self.response.out.write)
second.py:
def somefunction(output):
output('Done')
I am trying to make a simple command line client for accessing shares via the Python bindings of gio (yes, the main requirement is to use gio).
I can see that comparing with it's predecessor gnome-vfs, it provides some means to do authentication stuff (subclassing MountOperation), and even some methods which are quite specific to samba shares, like set_domain().
But I'm stuck with this code:
import gio
fh = gio.File("smb://server_name/")
If that server needs authentication, I suppose that a call to fh.mount_enclosing_volume() is needed, as this methods takes a MountOperation as a parameter. The problem is that calling this methods does nothing, and the logical fh.enumerate_children() (to list the available shares) that comes next fails.
Anybody could provide a working example of how this would be done with gio ?
The following appears to be the minimum code needed to mount a volume:
def mount(f):
op = gio.MountOperation()
op.connect('ask-password', ask_password_cb)
f.mount_enclosing_volume(op, mount_done_cb)
def ask_password_cb(op, message, default_user, default_domain, flags):
op.set_username(USERNAME)
op.set_domain(DOMAIN)
op.set_password(PASSWORD)
op.reply(gio.MOUNT_OPERATION_HANDLED)
def mount_done_cb(obj, res):
obj.mount_enclosing_volume_finish(res)
(Derived from gvfs-mount.)
In addition, you may need a glib.MainLoop running because GIO mount functions are asynchronous. See the gvfs-mount source code for details.