Develop Reference

Python daemon shows no output for an error

I read How do you create a daemon in Python? and also this topic, and tried to write a very simple daemon :
import daemon
import time
with daemon.DaemonContext():
while True:
with open('a.txt', 'a') as f:
f.write('Hi')
time.sleep(2)
Doing python script.py works and returns immediately to terminal (that's the expected behaviour). But a.txt is never written and I don't get any error message. What's wrong with this simple daemon?

daemon.DaemonContext() has option working_directory that has default fault value / i.e. your program probably doesn't have permission to create a new file there.

The problem described here is solved by J.J. Hakala's answer.
Two additional (important) things :
Sander's code (mentioned here) is better than python-daemon. It is more reliable. Just one example: try to start two times the same daemon with python-daemon : big ugly error. With Sander's code : a nice notice "Daemon already running."
For those who want to use python-daemon anyway: DaemonContext() only makes a daemon. DaemonRunner() makes a daemon + control tool, allowing to do python script.py start or stop, etc.

One thing that's wrong with it, is it has no way to tell you what's wrong with it :-)
A daemon process is, by definition, detached from the parent process and from any controlling terminal. So if it's got something to say – such as error messages – it will need to arrange that before becoming a daemon.
From the python-daemon FAQ document:
Why does the output stop after opening the daemon context?
The specified behaviour in PEP 3143_ includes the requirement to
detach the process from the controlling terminal (to allow the process
to continue to run as a daemon), and to close all file descriptors not
known to be safe once detached (to ensure any files that continue to
be used are under the control of the daemon process).
If you want the process to generate output via the system streams
‘sys.stdout’ and ‘sys.stderr’, set the ‘DaemonContext’'s ‘stdout’
and/or ‘stderr’ options to a file-like object (e.g. the ‘stream’
attribute of a ‘logging.Handler’ instance). If these objects have file
descriptors, they will be preserved when the daemon context opens.
Set up a working channel of communication, such as a log file. Ensure the files you open aren't closed along with everything else, using the files_preserve option. Then log any errors to that channel.

shell command from python script

I need you guys :D
I have a web page, on this page I have check some items and pass their value as variable to python script.
problem is:
I Need to write a python script and in that script I need to put this variables into my predefined shell commands and run them.
It is one gnuplot and one other shell commands.
I never do anything in python can you guys send me some advices ?
THx

I can't fully address your questions due to lack of information on the web framework that you are using but here are some advice and guidance that you will find useful. I did had a similar problem that will require me to run a shell program that pass arguments derived from user requests( i was using the django framework ( python ) )
Now there are several factors that you have to consider
How long will each job takes
What is the load that you are expecting (are there going to be loads of jobs)
Will there be any side effects from your shell command
Here are some explanation that why this will be important
How long will each job takes.
Depending on your framework and browser, there is a limitation on the duration that a connection to the server is kept alive. In other words, you will have to take into consideration that the time for the server to response to a user request do not exceed the connection time out set by the server or the browser. If it takes too long, then you will get a server connection time out. Ie you will get an error response as there is no response from the server side.
What is the load that you are expecting.
You will have probably figure that if a work that you are requesting is huge,it will take out more resources than you will need. Also, if you have multiple requests at the same time, it will take a huge toll on your server. For instance, if you do proceed with using subprocess for your jobs, it will be important to note if you job is blocking or non blocking.
Side effects.
It is important to understand what are the side effects of your shell process. For instance, if your shell process involves writing and generating lots of temp files, you will then have to consider the permissions that your script have. It is a complex task.
So how can this be resolve!
subprocesswhich ship with base python will allow you to run shell commands using python. If you want more sophisticated tools check out the fabric library. For passing of arguments do check out optparse and sys.argv
If you expect a huge work load or a long processing time, do consider setting up a queue system for your jobs. Popular framework like celery is a good example. You may look at gevent and asyncio( python 3) as well. Generally, instead of returning a response on the fly, you can retur a job id or a url in which the user can come back later on and have a look
Point to note!
Permission and security is vital! The last thing you want is for people to execute shell command that will be detrimental to your system
You can also increase connection timeout depending on the framework that you are using.
I hope you will find this useful
Cheers,
Biobirdman

Force bash executed in a pipe to enable echoing

I'm making a webshell in python, so actually, the user will use his favourite shell trough a web server. My idea is to create a subprocess.Popen with bash -i and to make two functions read and write in the webapp that, respectively, read the stdout or write in the stdin of the subprocess.
I start the shell with:
p = subprocess.Popen(["script","-f","-c","bash -i -l"],stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
The writing is Ok, but when I read the standard output I don't read the user typing:
while select.select([p.stdout],[],[],0)[0]!=[] or select.select([p.stderr],[],[],0)[0]!=[]:
if(select.select([p.stdout],[],[],0)[0]!=[]): data+=p.stdout.read(1)
if(select.select([p.stderr],[],[],0)[0]!=[]): data+=p.stderr.read(1)
I could force the echoing adding the user input to the output but that's not very elegant because, if the user uses some program that prevent echoing (like a password input), the user input would always be shown in the web page.
So, there is a way, like an option in the bash parameters, to force it adding the input to the output?
PS: If you wonder why I'm using script to run bash is because running bash alone will cause python to stop itself with
[1]+ Stopped python RPCServer.py
Altough I've not figured out WHY it happens, i've found how to prevent it from this question: Main Python Process is stopped using subprocess calls in SocketServer

So, there is a way, like an option in the bash parameters, to force it adding the input to the output?
Yes: you can use the -v command-line option ("Print shell input lines as they are read.").

How do I drive Ansible programmatically and concurrently?

I would like to use Ansible to execute a simple job on several remote nodes concurrently. The actual job involves grepping some log files and then post-processing the results on my local host (which has software not available on the remote nodes).
The command line ansible tools don't seem well-suited to this use case because they mix together ansible-generated formatting with the output of the remotely executed command. The Python API seems like it should be capable of this though, since it exposes the output unmodified (apart from some potential unicode mangling that shouldn't be relevant here).
A simplified version of the Python program I've come up with looks like this:
from sys import argv
import ansible.runner
runner = ansible.runner.Runner(
pattern='*', forks=10,
module_name="command",
module_args=(
"""
sleep 10
"""),
inventory=ansible.inventory.Inventory(argv[1]),
)
results = runner.run()
Here, sleep 10 stands in for the actual log grepping command - the idea is just to simulate a command that's not going to complete immediately.
However, upon running this, I observe that the amount of time taken seems proportional to the number of hosts in my inventory. Here are the timing results against inventories with 2, 5, and 9 hosts respectively:
exarkun#top:/tmp$ time python howlong.py two-hosts.inventory
real 0m24.285s
user 0m0.216s
sys 0m0.120s
exarkun#top:/tmp$ time python howlong.py five-hosts.inventory
real 0m55.120s
user 0m0.224s
sys 0m0.160s
exarkun#top:/tmp$ time python howlong.py nine-hosts.inventory
real 1m57.272s
user 0m0.360s
sys 0m0.284s
exarkun#top:/tmp$
Some other random observations:
ansible all --forks=10 -i five-hosts.inventory -m command -a "sleep 10" exhibits the same behavior
ansible all -c local --forks=10 -i five-hosts.inventory -m command -a "sleep 10" appears to execute things concurrently (but only works for local-only connections, of course)
ansible all -c paramiko --forks=10 -i five-hosts.inventory -m command -a "sleep 10" appears to execute things concurrently
Perhaps this suggests the problem is with the ssh transport and has nothing to do with using ansible via the Python API as opposed to from the comand line.
What is wrong here that prevents the default transport from taking only around ten seconds regardless of the number of hosts in my inventory?

Some investigation reveals that ansible is looking for the hosts in my inventory in ~/.ssh/known_hosts. My configuration has HashKnownHosts enabled. ansible isn't ever able to find the host entries it is looking for because it doesn't understand the hash known hosts entry format.
Whenever ansible's ssh transport can't find the known hosts entry, it acquires a global lock for the duration of the module's execution. The result of this confluence is that all execution is effectively serialized.
A temporary work-around is to give up some security and disabled host key checking by putting host_key_checking = False into ~/.ansible.cfg. Another work-around is to use the paramiko transport (but this is incredibly slow, perhaps tens or hundreds of times slower than the ssh transport, for some reason). Another work-around is to let some unhashed entries get added to the known_hosts file for ansible's ssh transport to find.

Since you have HashKnownHosts enabled, you should upgrade to the latest version of Ansible. Version 1.3 added support for hashed known_hosts, see the bug tracker and changelog. This should solve your problem without compromising security (workaround using host_key_checking=False) or sacrificing speed (your workaround using paramiko).

With Ansible 2.0 Python API, I switched off StrictHostKeyChecking with
import ansible.constants
ansible.constants.HOST_KEY_CHECKING = False
I managed to speed up Ansible considerably by setting the following on managed computers. Newer sshd have the default the other way around, I think, so it might not be needed in your case.
/etc/ssh/sshd_config
----
UseDNS no

Set a python process name

My python script needs to be killed every hour and after I need to restarted it. I need this to do because it's possible sometimes (I create screenshots) a browser window is hanging because of a user login popup or something.. Anyway. I created 2 files 'reload.py' and 'screenshot.py'. I run reload.py by cronjob.
I thought something like this would work
# kill process if still running
try :
os.system("killall -9 screenshotTaker");
except :
print 'nothing to kill'
# reload or start process
os.execl("/path/to/script/screenshots.py", "screenshotTaker")
The problem is, and what I read aswel the second argument of execl (the given process name) doesn't work? How can I set a process name for it to make the kill do it's work?
Thanks in advance!

The first argument to os.execl is the path to the executable. The remaining arguments are passed to that executable as if their where typed on the command-line.
If you want "screenshotTaker" become the name of the process, that is "screenshots.py" responsibility to do so. Do you do something special in that sense in that script?
BTW, a more common approach is to keep track (in /var/run/ usually) of the PID of the running program. And kill it by PID. This could be done with Python (using os.kill) At system-level, some distribution have helpers for that exact purpose. For example, on Debian there is start-stop-daemon. Here is a excerpt of the man:
start-stop-daemon(8) dpkg utilities start-stop-daemon(8)
NAME
start-stop-daemon - start and stop system daemon programs
SYNOPSIS
start-stop-daemon [options] command
DESCRIPTION
start-stop-daemon is used to control the creation and termination of
system-level processes. Using one of the matching options,
start-stop-daemon can be configured to find existing instances of a
running process.