Develop Reference

how to properly run Python script with crontab on every system startup

I have a Python script that should open my Linux terminal, browser, file manager and text editor on system startup. I decided crontab is a suitable way to automatically run the script. Unfortunately, it doesn't went well, nothing happened when I reboot my laptop. So, I captured the output of the script to a file in order to get some clues. It seems my script is only partially executed. I use Debian 8 (Jessie), and here's my Python script:
#!/usr/bin/env python3
import subprocess
import webbrowser
def action():
subprocess.call('gnome-terminal')
subprocess.call('subl')
subprocess.call(('xdg-open', '/home/fin/Documents/Learning'))
webbrowser.open('https://reddit.com/r/python')
if __name__ == '__main__':
action()
here's the entry in my crontab file:
#reboot python3 /home/fin/Labs/my-cheatcodes/src/dsktp_startup_script/dsktp_startup_script.py > capture_report.txt
Here's the content of capture_report.txt file (I trim several lines, since its too long, it only prints my folder structures. seems like it came from 'xdg-open' line on Python script):
Directory list of /home/fin/Documents/Learning/
Type Format Sort
[Tree ] [Standard] [By Name] [Update]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
/
... the rest of my dir stuctures goes here
I have no other clue what's possible going wrong here. I really appreciate your advice guys. thanks.

No, cron is not suitable for this. The cron daemon has no connection to your user's desktop session, which will not be running at system startup, anyway.
My recommendation would be to hook into your desktop environment's login scripts, which are responsible for starting various desktop services for you when you log in, anyway, and easily extended with your own scripts.

I'd do as tripleee suggested, but your job might be failing because it requires an X session, since you're trying to open a browser. You should put export DISPLAY=:0; after the schedule in your cronjob, as in
#reboot export DISPLAY=:0; python3 /home/fin/Labs/my-cheatcodes/src/dsktp_startup_script/dsktp_startup_script.py > capture_report.txt
If this doesn't work, you could try replacing :0 with the output of echo $DISPLAY in a graphical terminal.

Redirecting external function stdout to log file as they happen

Using python 2.7
I am calling an external library function and I want to print all the logs/prints that this function generates to a file. I would like to update the file with the logs as they happen.
I am using ipython notebook so if there is an easier way which is notebook specific I will be ok with that too.

It's probably better to perform the redirection in your shell, not from within the python code.
From a typical Unix command line, you can use the tee command:
python program.py | tee logfile

.pyw and pythonw does not run under Windows 7

Running a simple .py or .pyw python file causes python.exe to show up under Task Manager.
python myApp.py
python myApp.pyw
However when we try to run it without using the console, the script does not appear to run, nor does python.exe or pythonw.exe appears under Task Manager
pythonw myApp.pyw
pythonw myApp.py
How do we troubleshoot the problem? The system is running Python 2.7.8 x64.

tl;dr
To troubleshoot, use output redirection on invocation:
pythonw myApp.py 1>stdout.txt 2>stderr.txt
This will capture stdout output, such as from print(), in file stdout.txt, and stderr output (such as from unhandled exceptions), in file stderr.txt; from PowerShell, use
cmd /c pythonw myApp.py 1>stdout.txt 2>stderr.txt).
Note that the very act of redirecting stdout may actually make your script work again, if the only reason for its failure with pythonw was the use of print (in Python 2.x - see below).
Caveat: This output redirection technique seemingly does not work when invoking *.pyw scripts directly (as opposed to by passing the script file path to pythonw.exe). Do let me know if you know why and/or if it does work for you.
To fix your script:
Place the following at the top of any Python 2.x or 3.x script that you want to run with pythonw.exe:
import sys, os
if sys.executable.endswith("pythonw.exe"):
sys.stdout = open(os.devnull, "w");
sys.stderr = open(os.path.join(os.getenv("TEMP"), "stderr-"+os.path.basename(sys.argv[0])), "w")
This ensures the following when a script is run with pythonw.exe:
print() calls and explicit calls to sys.stdout() are effectively ignored (are no-ops).
Stderr output, including from an unhandled fatal exception, is sent to file
%TEMP%\stderr-<scriptFileName>; %TEMP% is a standard Windows environment variable that points to the current user's folder for temporary files.
In other words: With the above code in place, check file %TEMP%\stderr-<scriptFileName> after your script has failed silently when invoked with pythonw.exe.
For an explanation, read on.
On Windows, pythonw.exe is for launching GUI/no-UI-at-all scripts, which means that the
standard in- and output streams - sys.stdin, sys.stdout, sys.stderr are NOT available.
This has two nasty side effects:
Using print() - which targets sys.stdout by default - causes an exception in Python 2.x.
This problem has been fixed in Python 3.x.
Any unhandled exception - including one triggered by print() in 2.x - causes the script to abort silently.
Exception error messages go to sys.stderr by default, which is the very thing not available in this scenario.
The above code fixes these problems by:
sending stdout output to the null device, effectively ignoring any attempt to output to sys.stdout - whether explicitly, or implicitly via print().
sending all stderr output to a temporary file.
Differences between Python 2.x and Python 3.x:
When a script is run with pythonw.exe, sys.stdin, sys.stdout, and sys.stderr:
in Python 2.x: have invalid file descriptors
The eventual result when trying to write to sys.stdout or sys.stderr is the following exception: IOError: [Errno 9] Bad file descriptor
Pitfall: Due to output buffering, this exception may not surface until you've output, say, 4K bytes; you can provoke it instantly by invoking pythonw.exe with -u (for unbuffered output).
print() blindly tries to sys.stdout (by default), so it provokes this exception sooner or later.
in Python 3.x: are set to None
This is complemented with the 3.x print() function performing a no-op (doing nothing) when it finds that sys.stdout is None, so that print() statements can by default safely be used - they'll simply be ignored when run with pythonw.exe
However, it follows that trying to use sys.stdout.write() and sys.stderr.write() still results in an exception.
See here for more background.

Try adding the line import sys; sys.stderr = open("errlog.txt", "w") to the start of myApp.py. Then look in errlog.txt for a traceback or any other error messages.

I faced the same problem on a script of my own and found that when adding the output from Ross' answer the script would actually run.
It appears that for some reason that redirecting output fixes the problem. Since I'm not interested in writing the output to disk I've instead written it to /dev/null (or the platform equivalent) with:
if ( sys.platform == 'win32' and sys.executable.split( '\\' )[-1] == 'pythonw.exe'):
sys.stdout = open(os.devnull, 'w')
sys.stderr = open(os.devnull, 'w')
The if statement ensures it only happens when the script is launched from pythonw.exe. I'm not sure if it is related but it was important to do this before other imports (including e.g. import logging).

I was having similar problem.
After debugging step by step by writing to a log file, I discovered that pythonw.exe crashed after a statement that tried to use the call: sys.stdout.write(). It turns out, when run with pythonw.exe, sys.stdout is None.
If you are using functions of sys.stdout/stderr/stdin, and intend to use your program with pythonw.exe, adding a check for "None" is a good idea.

Im not sure I understand your problem but I think this is what you need to know
you need to right click on a py or pyw file and select open with ... find python.exe (probably C:\Python27\python.exe) .. check the box that says always open ... now you can just double click it if you want to run it
(usually the installer sets this up for you ...)

This is an old answer, but I want to leave my solution here also:
Open CMD (with elevated privileges or not - depends on your needs)
Change to directory of .py / .pyw script - this is important
Run pythonw with script as argument
cd E:\my\script\folder\
pythonw script.py

I had a similar problem after an upgrade to my computer RAM. Turns out I had to reinstall Pillow (library used for image processing). So make sure it is installed and if it's not, install it using "pip install Pillow" in cmd.

Running multiple Python scripts

I would like to create a simple Python program that will concurrently execute 2 independent scripts. For now, the two scripts just print a sequence of numbers but my intention is to use this program to concurrently run a few Twitter streaming programs in the future.
I suspect I need to use subprocess.Popen but I cannot quite get my head around what arguments I should put in there. There was a similar question on StackOverflow but the code provided there (pasted below) doesn't print anything. I will appreciate your help.
My files are:
thread1.py
thread2.py
import subprocess
subprocess.Popen(['screen', './thread1.py']))
subprocess.Popen(['screen', './thread2.py'])

Use supervisord
supervisord is process control system just for the purpose of running multiple command line scripts.
It features:
multiple controlled processes
autorestarting failed runs
log stdout and stderr output
starting scripts in order (using priority)
command line utility to view latest log output, stop, start, restart the processes
This solution works only on *nix based systems, it is not available on Windows.

As wanderlust mentioned, why do you want to do it this way and not via linux command line?
Otherwise, the solution you post is doing what it is meant to, i.e, you are doing this at the command line:
screen ./thread1.py
screen ./thread2.py
This will open a screen session and run the program and output within this screen session, such that you will not see the output on your terminal directly. To trouble shoot your output, just execute the scripts without the screen call:
import subprocess
subprocess.Popen(['./thread1.py'])
subprocess.Popen(['./thread2.py'])
Content of thread1.py:
#!/usr/bin/env python
def countToTen():
for i in range(10):
print i
countToTen()
Content of thread2.py:
#!/usr/bin/env python
def countToHundreds():
for i in range(10):
print i*100
countToHundreds()
Then don't forget to do this on the command line:
chmod u+x thread*.py

You can also just open several Command Prompt windows to run several Python programs at once - just run one in each of them:
In each Command Prompt window, go to the correct directory (such as C:/Python27) and then type 'python YourCodeNo1.py' in one Command Prompt window, 'python YourCodeNo2.py' in the next one ect. .
I'm currently running 3 codes at one time in this way, without slowing any of them down.

Run pdb without stdin/stdout using FIFO

I am developing FUSE filesystem with python. The problem is that after mounting a filesystem I have no access to stdin/stdout/stderr from my fuse script. I don't see anything, even tracebacks. I am trying to launch pdb like this:
import pdb
pdb.Pdb(None, open('pdb.in', 'r'), open('pdb.out', 'w')).set_trace()
All works fine but very inconvenient. I want to make pdb.in and pdb.out as fifo files but don't know how to connect it correctly. Ideally I want to type commands and see output in one terminal, but will be happy even with two terminals (in one put commands and see output in another). Questions:
1) Is it better/other way to run pdb without stdin/stdout?
2) How can I redirect stdin to pdb.in fifo (All what I type must go to pdb.in)? How can I redirect pdb.out to stdout (I had strange errors with "cat pdb.out" but maybe I don't understand something)

Ok. Exactly what I want, has been done in http://pypi.python.org/pypi/rpdb/0.1.1 .

Before starting the python app
mkfifo pdb.in
mkfifo pdb.out
Then when pdb is called you can interact with it using these two cat commands, one running in the background
cat pdb.out & cat > pdb.in
Note the readline support does not work (i.e. up arrow)

I just ran into a similar issue in a much simpler use-case:
debug a simple Python program running from the command line that had a file piped into sys.stdin, meaning, no way to use the console for pdb.
I ended up solving it by using wdb.
Quick rundown for my use-case. In the shell, install both the wdb server and the wdb client:
pip install wdb.server wdb
Now launch the wdb server with:
wdb.server.py
Now you can navigate to localhost:1984 with your browser and see an interface listing all Python programs running. The wdb project page above has instructions on what you can do if you want to debug any of these running programs.
As for a program under your control, you can you can debug it from the start with:
wdb myscript.py --script=args < and/stdin/redirection
Or, in your code, you can do:
import wdb; wdb.set_trace()
This will pop up an interface in your browser (if local) showing the traced program.
Or you can navigate to the wdb.server.py port to see all ongoing debugging sessions on top of the list of running Python programs, which you can then use to access the specific debugging session you want.
Notice that the commands for navigating the code during the trace are different from the standard pdb ones, for example, to step into a function you use .s instead of s and to step over use .n instead of n. See the wdb README in the link above for details.