Develop Reference

Interactive python daemon

Is there a way to run a python that can process inputs interactively? Without user input. That way methods can be called without needed to import and initialize the script.
What I have:
import very_heavy_package
very_heavy_package.initialize()
if very_heavy_package.check(input_file):
do_something()
else:
do_something_else()
I want something like:
import very_heavy_package
very_heavy_package.initialize()
#entry_point()
def check_something(input_file):
if very_heavy_package.check(input_file):
do_something()
else:
do_something_else()
import and initialize() lines take a very long time, but check_something() is pretty much instantaneous. I want to be able to check_something() multiple times on demand, without executing the full script all over.
I know this could be achieved with a server built in flask, but it seems a little overkill. Is there a more "local" way of doing this?
This example in particular is about running some Google Vision processing in an image from a surveillance camera on a Raspberry Pi Zero. Initializing the script takes a while (~10s), but making the API request is very fast(<100ms). I'm looking to achieve fast response time.

I don't think the webserver is overkill. By using an HTTP server with a REST api, you are using standards that most people will find easy to understand, use and extend. As an additional advantage, should you ever want to automate the usage of your tool, most automation tools already know how to speak REST and JSON.
Therefore, I would suggest you to follow your initial idea and use http.server or a library such as flask to create a small, no-frills web server with a REST api.

Try python -im your_module
The 'i' flag is for interactive and the 'm' flag is for module. And leave off the '.py'.

I have managed to solve my whim using signals. As I didn't need to pass any information, only trigger a function, that functionality solves my needs. So using signal python library and SIGUSR1
import signal
import time
import very_heavy_package
very_heavy_package.initialize()
def check_something():
input_file = get_file()
if very_heavy_package.check(input_file):
do_something()
else:
do_something_else()
signal.signal(signal.SIGUSR1, check_something)
while True:
# Waits for SIGUSR1
time.sleep(600)
now I can start the daemon from bash with
nohup python myscript.py &
and make the wake up call with kill
pkill -SIGUSR1 -f myscript.py
Disclaimer:
The pkill command is somewhat dangerous and it can have undesirable effects (i.e. killing a text editor with myscript.py opened). I should look into fancier ways of killing processes.

How would a Python script running on Linux call a routine in a Python script running under Wine?

I have a Python (3) script running on Linux, referred to as the main script, which has to call a routine from a proprietary DLL. So far, I have solved this with Wine using the following construct:
# Main script running on Linux
import subprocess
# [...]
subprocess.Popen('echo "python dll_call.py %s" | wine cmd &' % options, shell = True)
# [...]
The script dll_call.py is executed by a Windows Python (3) interpreter installed under Wine. It dumps the return values into a file which is then picked up by the waiting main script. It's not exactly reliable and agonizingly slow if I have to do this a few times in a row.
I'd like to start the script dll_call.py once, offering some type of a simple server, which should expose the required routine in some sort of way. At the end of the day, I'd like to have a main script looking somewhat like this:
# Main script running on Linux
import subprocess
# [...]
subprocess.Popen('echo "python dll_call_server.py" | wine cmd &', shell = True)
# [...]
return_values = call_into_dll(options)
How can this be implemented best (if speed is required and security not a concern)?
Thank you #jsbueno and #AustinHastings for your answers and suggestions.
For those having similar problems: Inspired by the mentioned answers, I wrote a small Python module for calling into Windows DLLs from Python on Linux. It is based on IPC between a regular Linux/Unix Python process and a Wine-based Python process. Because I have needed it in too many different use-cases / scenarios, I designed it as a "generic" ctypes module drop-in replacement, which does most of the required plumbing automatically in the background.
Example: Assume you're in Python on Linux, you have Wine installed, and you want to call into msvcrt.dll (the Microsoft C runtime library). You can do the following:
from zugbruecke import ctypes
dll_pow = ctypes.cdll.msvcrt.pow
dll_pow.argtypes = (ctypes.c_double, ctypes.c_double)
dll_pow.restype = ctypes.c_double
print('You should expect "1024.0" to show up here: "%.1f".' % dll_pow(2.0, 10.0))
Source code (LGPL), PyPI package & documentation. It's still a bit rough around the edges (i.e. alpha and insecure), but it does handle most types of parameters (including pointers).

You can use the XMLRPC client and servers built-in Python's stdlib to do what you want. Just make your Wine-Python expose the desired functions as XMLRPC methods, and make an inter-process call from any other Python program to that.
It also works for calling functions running in Jython or IronPython from CPython, and also across Python2 and Python3 - the examples included in the module documentation themselves should be enough.Just check the docs: https://docs.python.org/2/library/xmlrpclib.html
If you need the calls to be asynchronous on the client side, or the server site to respond to more than one process, you can find other frameworks over which to build the calls - Celery should also work across several different Pythons while preserving call compatibility, and it is certainly enough performance-wise.

You want to communicate between two processes, where one of them is obscured by being under the control of the WINE engine.
My first thought here is to use a very decoupled form of IPC. There are just too many things that can go wrong with tight coupling and something like WINE involved.
And finally, how can this be made easy for someone new to this kind of stuff?
The obvious answer is to set up a web server. There are plenty of tutorials using plenty of packages in Python to respond to HTTP requests, and to generate HTTP requests.
So, set up a little HTTP responder in your WINE process, listen to some non-standard port (not 8080 or 80), and translate requests into calls to your DLL. If you're clever, you'll interpret web requests (http://localhost:108000/functionname?arg1=foo&arg2=bar) into possibly different DLL calls.
On the other side, create a HTTP client in your non-WINE code and make requests to your server.

How to make a python program open itself as text in safari?

I was trying to make a program in Python that would use os.system to open a file in Safari. In this case, I was trying to have it open a text copy of itself. The files name is foo.py.
import os, socket
os.system("python -m SimpleHTTPServer 4000")
IP = socket.gethostbyname(socket.gethostname())
osCommand = "open -a safari http://"+IP+":4000/foo.py"
os.system(osCommand)

system runs a program, then waits for it to finish, before returning.
So it won't get to the next line of your code until the server has finished serving. Which will never happen. (Well, you can hit ^C, and then it will stop serving—but then when you get to the next line that opens safari, it'll have no server to connect to anymore.)
This is one of the many reasons the docs for system basically tell you not to use it:
The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function. See the Replacing Older Functions with the subprocess Module section in the subprocess documentation for some helpful recipes.
For example:
import subprocess, socket
server = subprocess.Popen(['python', '-m', 'SimpleHTTPServer', '4000'])
IP = socket.gethostbyname(socket.gethostname())
safari = subprocess.Popen(['open', '-a', 'safari', 'http://'+IP+':4000/foo.py'])
server.wait()
safari.wait()
That will start both programs in the background, and then wait for both to finish, instead of starting one, waiting for it to finish, starting the other, and waiting for it to finish.
All that being said, this is kind of a silly way to do what you want. What's wrong with just opening a file URL (like 'file:///{}'.format(os.path.abspath(sys.argv[0]))) in Safari? Or in the default web browser (which would presumably be Safari for you, but would also work on other platform, and for Mac users who used Chrome or Firefox, and so on) by using webbrowser.open on that URL?

Program does not exit. How to find out what python is doing?

I have a python script which is working fine so far. However, my program does not exit properly. I can debug until and I'm returning to the end, but the programm keeps running.
main.main() does a lot of stuff: it downloads (http, ftp, sftp, ...) some csv files from a data provider, converts the data into a standardized file format and loads everyting into the database.
This works fine. However, the program does not exit. How can I find out, where the programm is "waiting"?
There exist more than one provider - the script terminates correctly for all providers except for one (sftp download, I'm using paramiko)
if __name__ == "__main__":
main.log = main.log2both
filestoconvert = []
#filestoconvert = glob.glob(r'C:\Data\Feed\ProviderName\download\*.csv')
main.main(['ProviderName'], ['download', 'convert', 'load'], filestoconvert)
I'm happy for any thoughts and ideas!

If your program does not terminate it most likely means you have a thread still working.
To list all the running threads you can use :
threading.enumerate()
This function lists all Thread that are currently running (see documentation)
If this is not enough you might need a bit of script along with the function (see documentation):
sys._current_frames()
So to print stacktrace of all alive threads you would do something like :
import sys, traceback, threading
thread_names = {t.ident: t.name for t in threading.enumerate()}
for thread_id, frame in sys._current_frames().iteritems():
print("Thread %s:" % thread_names.get(thread_id, thread_id))
traceback.print_stack(frame)
print()
Good luck !

You can involve the python debugger for a script.py with
python -m pdb script.py
You find the pdb commands at http://docs.python.org/library/pdb.html#debugger-commands

You'd better use GDB, which allows to pinpoint hung processes, like jstack in Java

This question is 10 years old, but I post my solution for someone with a similar issue with a non-finishing Python script like mine.
In my case, the debugging process didn't help. All debugging outputs showed only one thread. But the suggestion by #JC Plessis that some work should be going on helped me find the cause.
I was using Selenium with the chrome driver, and I was finishing the selenium process after closing the only tab that was open with
driver.close()
But later, I changed the code to use a headless browser, and the Selenium driver wasn't closed after driver.close(), and the python script was stuck indefinitely. It results that the right way to shutdown the Selenium driver was actually.
driver.quit()
That solved the problem, and the script was finally finishing again.

You can use sys.settrace to pinpoint which function blocks. Then you can use pdb to step through it.

can a python script know that another instance of the same script is running... and then talk to it?

I'd like to prevent multiple instances of the same long-running python command-line script from running at the same time, and I'd like the new instance to be able to send data to the original instance before the new instance commits suicide. How can I do this in a cross-platform way?
Specifically, I'd like to enable the following behavior:
"foo.py" is launched from the command line, and it will stay running for a long time-- days or weeks until the machine is rebooted or the parent process kills it.
every few minutes the same script is launched again, but with different command-line parameters
when launched, the script should see if any other instances are running.
if other instances are running, then instance #2 should send its command-line parameters to instance #1, and then instance #2 should exit.
instance #1, if it receives command-line parameters from another script, should spin up a new thread and (using the command-line parameters sent in the step above) start performing the work that instance #2 was going to perform.
So I'm looking for two things: how can a python program know another instance of itself is running, and then how can one python command-line program communicate with another?
Making this more complicated, the same script needs to run on both Windows and Linux, so ideally the solution would use only the Python standard library and not any OS-specific calls. Although if I need to have a Windows codepath and an *nix codepath (and a big if statement in my code to choose one or the other), that's OK if a "same code" solution isn't possible.
I realize I could probably work out a file-based approach (e.g. instance #1 watches a directory for changes and each instance drops a file into that directory when it wants to do work) but I'm a little concerned about cleaning up those files after a non-graceful machine shutdown. I'd ideally be able to use an in-memory solution. But again I'm flexible, if a persistent-file-based approach is the only way to do it, I'm open to that option.
More details: I'm trying to do this because our servers are using a monitoring tool which supports running python scripts to collect monitoring data (e.g. results of a database query or web service call) which the monitoring tool then indexes for later use. Some of these scripts are very expensive to start up but cheap to run after startup (e.g. making a DB connection vs. running a query). So we've chosen to keep them running in an infinite loop until the parent process kills them.
This works great, but on larger servers 100 instances of the same script may be running, even if they're only gathering data every 20 minutes each. This wreaks havoc with RAM, DB connection limits, etc. We want to switch from 100 processes with 1 thread to one process with 100 threads, each executing the work that, previously, one script was doing.
But changing how the scripts are invoked by the monitoring tool is not possible. We need to keep invocation the same (launch a process with different command-line parameters) but but change the scripts to recognize that another one is active, and have the "new" script send its work instructions (from the command line params) over to the "old" script.
BTW, this is not something I want to do on a one-script basis. Instead, I want to package this behavior into a library which many script authors can leverage-- my goal is to enable script authors to write simple, single-threaded scripts which are unaware of multi-instance issues, and to handle the multi-threading and single-instancing under the covers.

The Alex Martelli approach of setting up a communications channel is the appropriate one. I would use a multiprocessing.connection.Listener to create a listener, in your choice. Documentation at:
http://docs.python.org/library/multiprocessing.html#multiprocessing-listeners-clients
Rather than using AF_INET (sockets) you may elect to use AF_UNIX for Linux and AF_PIPE for Windows. Hopefully a small "if" wouldn't hurt.
Edit: I guess an example wouldn't hurt. It is a basic one, though.
#!/usr/bin/env python
from multiprocessing.connection import Listener, Client
import socket
from array import array
from sys import argv
def myloop(address):
try:
listener = Listener(*address)
conn = listener.accept()
serve(conn)
except socket.error, e:
conn = Client(*address)
conn.send('this is a client')
conn.send('close')
def serve(conn):
while True:
msg = conn.recv()
if msg.upper() == 'CLOSE':
break
print msg
conn.close()
if __name__ == '__main__':
address = ('/tmp/testipc', 'AF_UNIX')
myloop(address)
This works on OS X, so it needs testing with both Linux and (after substituting the right address) Windows. A lot of caveats exists from a security point, the main one being that conn.recv unpickles its data, so you are almost always better of with recv_bytes.

The general approach is to have the script, on startup, set up a communication channel in a way that's guaranteed to be exclusive (other attempts to set up the same channel fail in a predictable way) so that further instances of the script can detect the first one's running and talk to it.
Your requirements for cross-platform functionality strongly point towards using a socket as the communication channel in question: you can designate a "well known port" that's reserved for your script, say 12345, and open a socket on that port listening to localhost only (127.0.0.1). If the attempt to open that socket fails, because the port in question is "taken", then you can connect to that port number instead, and that will let you communicate with the existing script.
If you're not familiar with socket programming, there's a good HOWTO doc here. You can also look at the relevant chapter in Python in a Nutshell (I'm biased about that one, of course;-).

Perhaps try using sockets for communication?

Sounds like your best bet is sticking with a pid file but have it not only contain the process Id - have it also include the port number that the prior instance is listening on. So when starting up check for the pid file and if present see if a process with that Id is running - if so send your data to it and quit otherwise overwrite the pid file with the current process's info.