I am not expert in Bash and Python, so this question might appear silly.
I have a Python script called learn.py and I noticed two different behaviours of the standard output, when redirected to a log file.
If I call this from terminal, I can see the log file size growing while the script is running.
$ ./learn.py > file.log
However, if I create a bash file for the same purpose:
#!/bin/bash
./learn.py > file.log
the script starts (I checked with pgrep) but it does not seem to run, as the log file stays empty. Am I missing something?
I solved using the Logging facility for Python, by inserting
import logging
logging.basicConfig(filename='file.log',level=logging.INFO)
and replacing every occurrence of print "..." with
logging.info("...")
The final Bash script:
#!/bin/bash
./learn.py
you can use nohup:
Also you can use & to make it run in the background
#!/bin/bash
nohup python learn.py >> file.log &
Related
I am doing something need to use curl xxx | bash to run something.
I created a python script with sys.stdin.readline() to test like this:
[python3] test.py
import sys
def read_input():
input = sys.stdin.readline().rstrip()
print(input)
read_input()
It works directly run by python3 test.py
test
test
But if I use echo 'python3 test.py' | bash, it will not stop to let me input something.
Any tips?
When you pipe with |, you are redirecting the output from the first command into the input of the second. This means standard input of the second command doesn't connect to the terminal, and therefore cannot read keyboard input. The form curl xxx | bash therefore only functions for non-interactive scripts. This is in no way specific to Python.
You could in principle work around this by saving the input descriptor under another number, but it does get quite complex:
$ ( echo 'exec <&3 3<&- ; echo script starts ; read hello ; echo you entered $hello ; exit' | bash ) 3<&0
script starts
something
you entered something
Here I used () to create a subshell, in which stdin is duplicated on file descriptor 3 using 3<&0, and the script generated in the pipeline both renames that back as stdin with exec <&3 3<&- and exits to prevent further commands from being read from the restored stdin. This has side effects, such as descriptor 3 being open for the echo command.
Since the main reason to use curl address | bash in the first place is to keep the command simple, this is not what you're after. Besides, the pipe prevents you from handling if anything goes wrong during your download; your script could be interrupted anywhere. A traditional download then run isn't that much worse:
curl -O http://somewhere/somefile.py && python somefile.py
In comparison, this saves somefile.py to your filesystem. There are downsides to that, like requiring a writable filesystem and replacing that particular filename. On the upside, if anything goes wrong it stops there and doesn't run a corrupted script, due to &&.
One final possibility if the script you're downloading fits within the command line might be to put it there rather than in a pipe:
python -c "$(curl $url)"
This carries the same weaknesses to interrupted downloads, and additionally places the script contents in a command line which is generally public information (consider ps ax output). But if you're just downloading the script with curl, the information on how to get it likely was too. As this doesn't redirect stdin, it might be the answer to your immediate question.
In general, I recommend not to run any scripts straight off the internet without verification as this curl something | bash command line does. It's way too vulnerable to hijacking, as there's no verification involved for any step. It's better to use a package repository which checks signatures, such as apt.
Another method to get access to a terminal on Linux is via the device /dev/tty. This method is used for instance when ssh asks for a password. It might also be possible to reopen stdout or stderr for input, as in ( exec < /dev/null ; read foo <&2 ; echo $foo ).
Using python 2.7
I am calling an external library function and I want to print all the logs/prints that this function generates to a file. I would like to update the file with the logs as they happen.
I am using ipython notebook so if there is an easier way which is notebook specific I will be ok with that too.
It's probably better to perform the redirection in your shell, not from within the python code.
From a typical Unix command line, you can use the tee command:
python program.py | tee logfile
I have this python script:
#!/usr/bin/python
print 'hi'
I'm trying to send this script as a job to be executed on a computing cluster. I'm sending it with qsub like this: qsub myscript.py
Before running it I executed the following:
chmod +x myscript.py
However when I open the output file I find this:
Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
And when I open the error file I find this:
print: Command not found.
So what's wrong?!
Edit: I followed the instructions in this question
It looks like qsub isn't reading your shebang line, so is simply executing your script using the shell.
This answer provides a few options on how to deal with this, depending on your system: How can I use qsub with Python from the command line?
An option is to set the interpreter to python like so:
qsub -S /usr/bin/python myscript.py
I am quite sure there is an alternate way to do this without the -S option and have SGE execute the code based on interpreter in the shebang; however, this solution might be enough for you needs.
Also, concerning this output:
Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
It seems safe to ignore this:
http://www.linuxquestions.org/questions/linux-software-2/warning-no-access-to-tty-bad-file-descriptor-702671/
EDIT:
Also works:
qsub <<< "./myscript.py"
qsub <<< "python ./myscript.py"
I've a shell script with two shebangs, the first one tells #!/bin/sh and after a few lines the other one is #!/usr/bin/env python.
When this script is given executable permission and ran as ./script.sh, the script works fine, uses /bin/sh in first part and uses python interpreter in latter part.
But when the script is run as sh script.sh, the second shebang is not recognized and the script fails. Is there anyway I can force to change the interpreter if the script is run explicitly as sh script.sh.
The reason I need this is because I need to run the scripts through a tool which runs as sh script.sh
As far as I know you cannot have two shebang lines in one script. The shebang works only when -
it is on the first line
it starts in column one
If you need to run python code then have it in another script and then call the script by doing
python path/to/the/script.py
A better way to do this would be to use a shell here document. Something like this:
#!/bin/sh
curdir=`pwd`
/usr/bin/env python <<EOF
import os
print os.listdir("$curdir")
EOF
This way, you won't need to distribute the code on two separate files.
As you see, you can even access shell variables from the python code.
have your script.sh script invoke the python interpreter directly:
#!/bin/sh
# include full path if needed
python (your python interpreter arguments)
I am running a python script of web2py and want to log its output. I am using following command
nohup python /var/www/web2py/web2py.py -S cloud -M -N -R applications/cloud/private/process.py >>/var/log/web2pyserver.log 2>&1 &
The process is running but it is not logging into the file. I have tried without nohup also but it is still same.
The default logging of nohup in nohup.out is also not working.
Any suggestion what might be going wrong?
Nothing to worry. Actually the python process along with nohup was logging the file in batch mode and i could see the output only after quite some time and not instantaneously.
nohup will try to create the file in the local directory. Can you create a file in the folder you are running it from ?
If you've got commas in your print statements there's a good chance it's due to buffering. You can put a sys command (forget which) in your code or when you run the nohup, just add the option -u and you'll disable std(in|out|err) buffering
Don't worry about this, it is because of the buffering mechanism, run your Python script with the -u flag will solve the problem:
nohup python -u code.py > code.log &
or just
nohup python -u code.py &