The example code makes use of this oauth2_client which it immediately locks. The script does not work without these lines. What's the correct way to integrate this into a Flask app? Do I have to manage these locks? Does it matter if my web server spawns multiple threads? Or if I'm using gunicorn+gevent? Is there documentation on this anywhere?
It's not actually locking, it's just instantiating a lock object inside the module. The lock is actually acquired/released internally by oauth2_client; you don't need to manage it yourself. You can see this by looking at the source code, here: https://github.com/GoogleCloudPlatform/gsutil/blob/master/gslib/third_party/oauth2_plugin/oauth2_client.py
In fact, based on the source code linked above, you should be able to simply call oauth2_client.InitializeMultiprocessingVariables() instead of the try/except block, since that is ultimately doing almost the exact same thing.
Related
I am still somewhat new to Python, and have begun learning to use the Twisted framework so that I may set up an asynchronous web server. The details about storing stateful information in the Session object is pretty straightforward, but there is something that is lacking in the documentation that is throwing me off. The first line in the script on this tutorial reads:
cache()
...rest of the script goes here
This is something that only works in what is called an rpy script - more about that here. The problem is, I don't really want to use an rpy script, and it allegedly is not a requirement. The page I referenced describes rpy scripts as being mainly for experimenting with new ideas AND NOT MUCH ELSE.
My issue is that when I try to run a non-rpy version of my script, I get this error:
NameError: name 'cache' is not defined
Some additional research has told me that cache() is a part of the globals for every rpy script, so there is no need to import. However, the documentation doesn't describe how to use cache() in a non-rpy file. So, there is my question - how can I use cache() in a non-rpy file? I am pretty sure it is just a matter of knowing which module to import, which I do not. Any help will be appreciated.
A distinctive feature of the handling of rpy scripts by Twisted Web is that the source code is re-evaluated on each request.
cache is an API specifically for rpy scripts to tell the runtime not to re-evaluate the source again. If cache is called, the results of evaluating the source are saved and used to satisfy the next request for that resource.
Since this feature is unique to handling of rpy scripts, there is no need or value in using cache when defining resources for Twisted Web in a different way.
Apparently, if you aren't using an rpy file, you simply don't need to use cache(). I simply removed that line from the code and it seems to be working fine. Any additional input on this is still appreciated, because the documentation is lacking.
It is easy to create highly integrated code in python, so the need to quit and restart an application when its code is changed is understandable.
However, it surely must exist some strategies and models to be able to isolate parts of the code so it can be updated on-the-fly without the need to quit and restart.
For the applications I am working on many features will be independant background tasks that the main application will talk to, present status information and also instruct to perform tasks based on the current state. In many ways these background tasks can be seen as independent programs, just that they share some of the codebase with the main application and other tools, tasks, etc built on-top it.
While it probably is hard to make the whole shebang live updateable, I'm sure there must be ways where it is possible to roll out updates and have the running code to notice and update itself as needed.
Since I'm also keen on taking advantage of multithreading and asyncio (in Python 3.5), as well as explore making things stateless, it seems logically possible to do some interesting stuff here that at least makes it possible to avoid some forced hard restarts when rolling out new code.
Would be very grateful for tips and pointers to information about how to make this working.
There's a built-in function reload() that will reload a module, but it's easy to mess up. You'd have to be very careful about what holds references to objects created by the old version of the module, and then be sure to replace those when you reload the module.
I use Django a lot, and when it's running its web server in debug mode, it reloads the entire web server process whenever a source file changes. It's a nice compromise between a manual restart and reloading a single module.
I haven't used it, but watchdog might be helpful to monitor the file system for changes to trigger a reload.
My understanding of Gevent is that it's merely concurrency and not parallelism. My understanding of concurrency mechanisms like Gevent and AsyncIO is that, nothing in the Python application is ever executing at the same time.
The closest you get is, calling a non-blocking IO method, and while waiting for that call to return other methods within the Python application are able to be executed. Again, none of the methods within the Python application ever actually execute Python code at the same time.
With that said, why is there a need for gevent.queue? It sounds to me like the Python application doesn't really need to worry about more than one Python method accessing a queue instance at a time.
I'm sure there's a scenario that I'm not seeing that gevent.queue fixes, I'm just curious what that is.
Although you are right that no two statements execute at the same time within a single Python process, you might want to ensure that a series of statements execute atomically, or you might want to impose an order on the execution of certain statements, and in that case things like gevent.queue become useful. A tutorial is here.
Hello i want to know the best way to re import or re execute a module, because i have a web server with just one Apache session for all my domains and applications, and i if i need to make some changes on one application restart the server will affect the others, so looking for the best way to recall a module. If i choose subprocess i will need to print the response but i don' t know is that most secure way of communication. Please tell me in your experience which is the best way?
Thanks in advance!
Reloading a module is rarely a good idea in a production environment; it's a mechanism intended for debugging. When you reload a module, the module's contents (classes, function, data) get replaced, but existing references to these items from other modules are not affected. This is particularly important for classes: existing objects in memory still refer to the old class, whereas objects generated after the reload refer to the new class.
There is another alternative you might want to consider: load Python code from a file and exec it. Less overhead than a complete subprocess, and less tightly coupled to the rest of a program than a module. In principle the same caveats apply to re-exec-ing as to reloading a module, but you are much less tempted to have references to exec'd code because it's more work.
I'm trying to create a construct in Python 3 that will allow me to easily execute a function on a remote machine.
Assuming I've already got a python tcp server that will run the functions it receives, running on the remote server, I'm currently looking at using a decorator like
#execute_on(address, port)
This would create the necessary context required to execute the function it is decorating and then send the function and context to the tcp server on the remote machine, which then executes it. Firstly, is this somewhat sane? And if not could you recommend a better approach? I've done some googling but haven't found anything that meets these needs.
I've got a quick and dirty implementation for the tcp server and client so fairly sure that'll work. I can get a string representation the function (e.g. func) being passed to the decorator by
import inspect
string = inspect.getsource(func)
which can then be sent to the server where it can be executed. The problem is, how do I get all of the context information that the function requires to execute? For example, if func is defined as follows,
import MyModule
def func():
result = MyModule.my_func()
MyModule will need to be available to func either in the global context or funcs local context on the remote server. In this case that's relatively trivial but it can get so much more complicated depending on when and how import statements are used. Is there an easy and elegant way to do this in Python? The best I've come up with at the moment is using the ast library to pull out all import statements, using the inspect module to get string representations of those modules and then reconstructing the entire context on the remote server. Not particularly elegant and I can see lots of room for error.
Thanks for your time
The approach you outline is extremely risky unless the remote server is somehow very strongly protected or "extremely sandboxed" (e.g a BSD "jail") -- anybody who can send functions to it would be able to run arbitrary code there.
Assuming you have an authentication system that you trust entirely, comes the "fragility" problem that you realized -- the function can depend on any globals defined in its module at the moment of execution (which can be different from those you can detect by inspection: determining the set of imported modules, and more generally of globals, at execution time, is a Turing-complete problem).
You can deal with the globals problem by serializing the function's globals, as well as the function itself, at the time you send it off for remote execution (whether you serialize all this stuff in readable string form, or otherwise, is a minor issue). But that still leaves you with the issue of imports performed inside the function.
Unless you're willing to put some limitations on the "remoted" function, such as "no imports inside the function (and functions called from it)", I'm thinking you could have the server override __import__ (the built-in function that is used by all import statements and is designed to be overridden for peculiar needs, such as yours;-) to ask for the extra module from the sending client (of course, that requires that said client also have "server-like" capabilities, in that it must be able to respond to such "module requests" from the server).
Can't you impose some restrictions on functions that are remoted, to bring this task back into the domain of sanity...?
You may interested in the execnet project.
execnet provides carefully tested means to easily interact with Python interpreters across version, platform and network barriers. It has a minimal and fast API targetting the following uses:
distribute tasks to local or remote CPUs
write and deploy hybrid multi-process applications
write scripts to administer a bunch of exec environments
http://codespeak.net/execnet/example/test_info.html#get-information-from-remote-ssh-account
I've seen a demo of it. But never used it myself.
Its' not clear from your question whether there is some system limitation/requirement for you to solve your problem in this way. If not there may be much easier and quicker ways of doing this using some sort of messaging infrastructure.
For example you might consider whether [Celery][1]
[1]: http://ask.github.com/celery/getting-started/introduction.html will meet your needs.
What's your end goal with this? From your description I can see no reason why you can't just create a simple messaging class and send instances of that to command the remote machine to do 'stuff'.. ?
Whatever you do, your remote machine is going to need the Python source to execute so why not distribute the code there and then run it? You could create a simple server which would accept some python source files, extract them import the relevent modules and then run a command?
This will probably be hard to do and you'll run into issues with security, arguments etc.
Maybe you can just run a Python interpreter remotely that can be given code using a socket or an HTTP service rather than do it on a function by function level?