Develop Reference

Crontab Python Script not running

I know this question has been asked before but I still haven't been able to get it to work. My crontab file just has this:
0 5 * * * /home/harry/my_env/bin/python /home/harry/compile_stats/process_tonight.py
Here's what my process_tonight.py looks like:
import datetime
import sys
sys.path.append('/home/harry/compile_stats/')
import compile # Module in above path
print("Processing last night\n")
date = str(datetime.datetime.today().year) + "-" + str(datetime.datetime.today().month) + "-" + str(datetime.datetime.today().day-1)
compile.process(date, date)
This file works perfectly fine when I just run it regularly from the command line but doesn't work when I schedule it.
I also looked at my /var/log/syslog file and the task I'm looking to run isn't showing up there.
Any ideas?
EDIT:
The time it's set to run in my example (5 A.M) is just a random time to put in. It's not running for any time I put in there.
EDIT 2#:
As per user speedyturkey I simplified my python script to better diagnose the problem:
import datetime
#import sys
#sys.path.append('/home/harry/compile_stats/')
#import compile # Module in above path
print("Processing last night\n")
date = str(datetime.datetime.today().year) + "-" + str(datetime.datetime.today().month) + "-" + str(datetime.datetime.today().day-1)
#compile.process(date, date)
Nothing is happening so I guess the problem isn't with the import.

As per the comments, I believe the problem is in how you are calling the python script in the crontab. Run the exact command you've given crontab and fix any problems it returns.

Ok, I was able to get it to work by creating a specific cron file, putting the info and there and loading it in.
So process_tonight.cron contains this:
0 5 * * * /home/harry/my_env/bin/python /home/harry/compile_stats/process_tonight.py
And I just loaded it into crontab:
crontab process_tonight.cron
Not really sure why this works and the other way doesn't (maybe someone else has an idea).

Instead of trying to modify the path from within your python script, you could do something like:
cd /home/harry/compile_stats/ && ./process_tonight.py
which would make it easier to import compile correctly. Note that this would also require making process_tonight.py executable (chmod +x process_tonight.py) and adding a shebang pointing to your Python interpreter (I guess... #!/home/harry/my_env/bin/python).
EDIT in response to Edit #2 above:
It is actually not possible to tell if it is running from the code you have written - the print statements are not being redirected. I suggest changing the code to perform some kind of side effect that you can check. For example, import subprocess and then do (example):
subprocess.call("date > /home/harry/compile_stats/date.txt")
If the script is executed properly, it will redirect the output of date to the file specified.

I know it's a bit silly, but checking your system time/timezone might be helpfull )))
I set my job to run at 5AM and when I logged in at 8AM there was no result of my script. So I spent over 1 hour trying to figure out what's the problem before I noticed that system time is incorrect and 5AM haven't come yet.

Have you tried running it from a shell script? I was having the same issue with my python script. I ended up putting the command in a shell script and running that. It threw an error that the library wasn't imported, so I installed it with pip and the --user flag. Now cron runs the shell script no issues.

Python script hangs without throwing exception/error on Windows

There isn't a particular point in the script where it hangs (I've seen it getting stuck at random points in the script), so checking the logs didn't yield much insight. It doesn't even throw an exception or an error. It just keeps running while stuck.
I'm basically calling this python script from a powershell file (which later gets called by Task scheduler).
$python = "C:\Python34\python.exe"
$python_path = "C:\Source\main.py"
cd (split-path $python_path)
while($true)
{
& $python $python_path
}
Is there something I need to do to make sure it doesn't get stuck?

You Will have to do your path strings like this path ="c:\\test\\file"
Because it ignorer the first backslash

Python: What does the following code do?

Can someone please explain to me piece by piece, what the following does? My code won't run and this part was provided. I've tested all of my code in iPython notebook and everything works, so I don't know if the problem is because of this block below.
def main():
args = sys.argv[1:]
if not args:
print 'usage: [--summaryfile] file [file ...]'
sys.exit(1)
summary = False
if args[0] == '--summaryfile':
summary = True
del args[0]
# ... my code ....
Update: I tried to do as Simon suggested. I opened up iPython and typed the following in the command line:
ipython 'assignment.py' --summaryfile
I tried variations of this and I keep getting a syntax error.
How do I run this?
Am I restricted to iPython only?

It basically checks to see if --summaryfile has been passed as an argument when you run the script
If no arguments have been passed, then it will print a line telling you how to use the script and then exit. summary is now set to false
If --summaryfile has been passed, then it will set summary = True and continue running the rest of your code
I'm not sure that you can enter arguments like that in ipython, so presumably your code will always exit because it doesnt find any arguments
EDIT:
For some reason I automatically associated ipython with ipython notebook. You can pass arguments with ipython. See here:
How to pass command line arguments to ipython
In your case, try adding --summaryfile as an argument when you run your script through ipython
If you're running this from ipython notebook, there are a couple of things you can try:
Remove that section of code and just set summary = True. The rest of your code should run, but without seeing everything its hard to say what impact this may have on the rest of your code
You can save your script as a python .py file, and use ipython magic to run the code from within notebook. You can pass arguments when you're running that script from within notebook. Check this: https://ipython.org/ipython-doc/dev/interactive/magics.html#magic-run
If you have all the code in a regular .py file (outside of a notebook) you can run and pass an argument using the command line. Navigate to the directory where the script file is and run ipython filename.py --summaryfile

A very rudimentary command line argument parser.
It checks for the command line except the executable's name itself (sys.argv[1:]). If it's empty, dump help message and fail. If the first argument is "--summaryfile", set some flag.

OS X & Python 3: Behavior while executing bash command in new terminal?

I've seen similar questions (e.g. Running a command in a new Mac OS X Terminal window ) but I need to confirm this command and its expected behavior in a mac (which I don't have). If anyone can run the following in Python 3 Mac:
import subprocess, os
def runcom(bashCommand):
sp = subprocess.Popen(['osascript'], stdin=subprocess.PIPE, stderr=subprocess.PIPE)
sp.communicate('''tell application "Terminal"\nactivate\ndo script with command "{0} $EXIT"\nend tell'''.format(bashCommand))
runcom('''echo \\"This is a test\\n\\nThis should come two lines later; press any key\\";read throwaway''')
runcom('''echo \\"This is a test\\"\n\necho \\"This should come one line later; press any key\\";read throwaway''')
runcom('''echo \\"This is testing whether I can have you enter your sudo pw on separate terminal\\";sudo ls;\necho \\"You should see your current directory; press any key\\";read throwaway''')
Firstly, and most basically, is the "spawn new terminal and execute" command correct? (For reference, this version of the runcom function came from this answer below, and is much cleaner than my original.)
As for the actual tests: the first one tests that internal double escaped \\n characters really work. The second tests that we can put (unescaped) newlines into the "script" and still have it work just like semicolon. Finally, the last one tests whether you can call a sudo process in a separate terminal (my ultimate goal).
In all cases, the new terminal should disappear as soon as you "press any key". Please also confirm this.
If one of these doesn't work, a correction/diagnosis would be most appreciated. Also appreciated: is there a more pythonic way of spawning a terminal on Mac then executing a (sudo, extended) bash commands on it?
Thanks!

[...] its expected behavior [...]
This is hard to answer, since those commands do what I expect them to do, which might not be what you expect them to do.
As for the actual tests: the first one tests that internal double escaped \n characters really work.
The \\n with the doubled backslash does indeed work correctly in that it causes echo to emit a newline character. However, no double quotes are emitted by echo.
The second tests that we can put (unescaped) newlines into the "script" and still have it work just like semicolon.
That works also.
Finally, the last one tests whether you can call a sudo process in a separate terminal (my ultimate goal).
There is no reason why this should not work also, and indeed it does.
In all cases, the new terminal should disappear as soon as you "press any key". Please also confirm this.
That will not work because of several reasons:
read in bash will by default read a whole line, not just one character
after the script you supply is executed, there is no reason for the shell within the terminal to exit
even if the shell would exit, the user can configure Terminal.app not to close a window after the shell exits (this is even the default setting)
Other problems:
the script you supply to osascript will appear in the terminal window before it is executed. in the examples above, the user will see every "This is a test [...]" twice.
I cannot figure out what $EXIT is supposed to do
The ls command will show the user "the current directory" only in the sense that the current working directory in a new terminal window will always be the user's home directory
throwaway will not be available after the script bashCommand exits
Finally, this script will not work at all under Python 3, because it crashes with a TypeError: communicate() takes a byte string as argument, not a string.
Also appreciated: is there a more pythonic way of spawning a terminal on Mac [...]
You should look into PyObjC! It's not necessarily more pythonic, but at least you would eliminate some layers of indirection.

I don't have Python 3, but I edited your runcom function a little and it should work:
def runcom(bashCommand):
sp = subprocess.Popen(['osascript'], stdin=subprocess.PIPE, stderr=subprocess.PIPE)
sp.communicate('''tell application "Terminal"\nactivate\ndo script with command "{0} $EXIT"\nend tell'''.format(bashCommand))

Running Another program from python

I want to call a program multiple times from a python code, and save the output of that program in a text file. My first problem right now is just calling the other code. I have to redirect to a different directory and call ./rank on output.txt. This is how Im trying to do it:
TheCommand = "~/src/rank-8-9-2011/rank output.txt"
os.system(TheCommand)
but im getting a parsing error.
[Parsing error on line ]Unknown error: 0
Im running python2.7 on Mac OS 10.5.8. Im not sure what the problem is. I also tried using subprocess:
subprocess.call(["~/src/rank-8-9-2011/rank", "output.txt"])
This does not find the directory (I have a feeling Im using the subprocess incorrectly), but I dont know what is wrong with the os.system.

the name of the program in the first argument to subprocess.Popen must not contain ~ as it doesn't pass the string to the shell for processing (which like always using parameterized queries in sql, protects one from string injection attacks, e.g. if instead of output.text one had ;rm -rf /, the system version would run rank and then run rm -rf . but the subprocess.Popen would only have rank open a file named ;rm -rf .), so one should expand it by calling os.path.expanduser:
subprocess.Popen([os.path.expanduser('~/src/rank-8-9-2011/rank'), "output.txt"])
although it is possible to turn shell processing on by passing shell=True, it is not recommended for the aforementioned reason.

you should try http://docs.python.org/library/os.path.html#os.path.expanduser
import os.path
subprocess.call([os.path.expanduser("~/src/rank-8-9-2011/rank"), "output.txt"])

I'm fairly certain your parsing error is coming from rank, not from your os.system command, as nothing there looks weird. What happens if you run rank manually?
subprocess seems to have a problem with '~', although I'm not immediately sure why. Put the full path and it should work (although you'll likely get that parsing error if it is indeed a problem with rank).