So I know there are plenty of answers on stack overflow for ignoring errors in a bash script. None of them seem to work for the source command though.
I have tried the tried and true source ../bin/activate || true
I have tried setting set -e before running the command
I have tried source ../bin/activate 2>&1 /dev/null
I have tried setting set +e before running the command. This did not work either.
but on every run-through of this code, I receive
run.01: line 12: ../bin/activate: No such file or directory
The context for this problem is that I'm creating a simple bash script that runs some python code. The user is instructed for how to create a specific virtual environment and this line will automatically activate it if they set it up correctly, otherwise, this should ignore failing to activate running and continue running the commands in whatever environment is currently activated.
# Try virtual environment
source ../bin/activate || true
## Run.
code="../src-01/driver.v01.py"
## --------------------
graphInputFile="undirected_graph_01.inp"
graphType="undirected"
srcColId="0"
desColId="1"
degreeFind="2"
outFile="count.undirected.num.nodes.out"
python $code -inpGraphFile $graphInputFile -graphFormat $graphType -colSrcId $srcColId -colDesId $desColId -degreeFind $degreeFind -output_file $outFile
The python command should execute regardless of whether or not the source ../bin/activate command succeeds or not. I'm a little a loss for why none of these solutions are working and am currently under the assumption that source might do something different than a normal command given the circumstances.
EDIT:
I added the shebang #!/bin/bash -x to my file as requested but this did not do anything.
I this is my exact terminal output when I run this script.
Lucas-Macbook:test-01 lucasmachi$ sh run.01
run.01: line 14: ../bin/activate: No such file or directory
Lucas-Macbook:test-01 lucasmachi$
Where run.01 is the name of the bash script.
Also to clarify, the code I showed is not censored. that is the entire script (except now with the mentioned shebang at the top.)
This is a bug in Bash versions before 4.0 (macOS is stuck on 3.2).
Given this script:
#!/bin/bash
set -e
echo "Running $BASH_VERSION"
source "does not exist" || true
echo "Continuing"
Running on macOS will say:
macos$ ./myscript
Running 3.2.57(1)-release
./myscript: line 3: does not exist: No such file or directory
While on a modern system with an updated bash version, you get the expected behavior:
debian$ ./myscript
Running 5.0.11(1)-release
./myscript: line 3: does not exist: No such file or directory
Continuing
If you need to support macOS and Bash 3.2, run set +e first to disable errexit, and optionally re-enable it afterwards.
You could add a validation in your code to check whether the file exists before sourcing it :
[[ -f "../bin/activate" ]] && source ../bin/activate
Using the -f flag before a path will return true if the path file exists, and false if it doesn't.
The one liner syntax for bash if statements is as follow :
[[ Condition that returns true or false ]] && Exec if true
I ran into this issue with gitlab pipelines. None of the above worked for me, but combining the answers did:
[[ -f "${ENVIRONMENT}_test" ]] && source "${ENVIRONMENT}_test" || true
This gets me to where I need to go, hopefully it will help someone in the future.
Related
In python the first line of the script should be
#!/usr/bin/env/python{3}
Is there a way to do:
if python:
#!/usr/bin/env/python
else:
#!/usr/bin/env/python3
EDIT:
Running python in RHEL7 bring up python-2.7.5 (default).
Only running python3 will execute python3-3.6.8 on my RHEL7.
I can't say I recommend this at all, but it does work:
#!/bin/bash
_=""""
if [[ -x '/usr/bin/python3' ]]
then
exec /usr/bin/python3 "$0" "$#"
elif [[ -x '/usr/bin/python' ]]
then
exec /usr/bin/python "$0" "$#"
else
echo "No python available"
exit 1
fi
"""
import sys
sys.stdout.write("you made it\n")
Can't do from __future__ import print_function because it has to be the first line of python.
Principles here are to use bash to do the executable detection but also make it valid python so you can just re-execute with the python interpreter on the same file.
None of those are correct. The correct command is /usr/bin/env and the argument to that, after a space, is the name of the actual interpreter you want it to look up.
Your question is a bit of a "turtles all the way down" problem. You have to know what command you want to run. You could of course create yet another tool and call it something like py3orpy but then you need to know that that exists in the PATH before you try to ask env to find it.
I would expect us to eventually converge on python; if you can't rely on that, perhaps the least error-prone solution is to figure out what's correct for the target system at installation time, and have the installer write a correct shebang. Or just install a symlink /usr/local/bin/python3 if it doesn't exist already, and hardcode #!/usr/bin/env python3 everywhere.
I'm attempting to create a .sh file to batch a number of runs of a neural network on Python whilst on holidays.
At the moment I have been calling this from the command line:
python neural_network_trainer.py [args]
I now have a .sh script written:
#!/bin/bash
python neural_network_trainer.py [args]
# Repeated with varied args
That I am attempting to call in the same terminal as the original command line was running:
./august_hols.sh
I get the following error:
File "/data/Python-3.6.9/lib/python3.6/site.py", line 177
file=sys.stderr)
^
SyntaxError: invalid syntax
Where the Python install is in /data (for reasons).
Running which on the command line reports the correct Python directory set via an alias in ~/.bashrc:
alias python=/data/Python-3.6.9/bin/python3
But running which between the Bash shebang and the first python call reports /bin/python.
I've attempted to set the alias again at the start of the .sh script to no avail. I'm scratching my head as this is exact process I have used elsewhere, albeit not on this precise PC. I can copy the exact command from the top of the bash file into the terminal and it runs fine, try and call ./august_hols.sh and get the above Python error.
Where is Bash getting that path from, and why is it not using my expected route through ~/.bashrc?
Bash sub-shell does not inherit alias in the main shell
You can source the script (run in the main shell), instead of execute it (run in the sub-shell)
source script.sh
EDIT:
Solution 2:
Run bash as the login shell so ~/.bashrc is executed, so your alias is loaded before your script.
The subshell needs to be interactive to enable alias, because alias is enabled by default only for interactive shell, but script is non-interactive by default.
bash --login -i script.sh
Solution 3:
Similar to above, except alias is enabled explicitly
bash --login -O expand_aliases script.sh
Have you tried:
python=/data/Python-3.6.9/bin/python3 ./[your_bash].sh
In your .sh
Do this
#!/usr/bin/env bash
export PATH=/data/Python-3.6.9/bin:$PATH
exec python neural_network_trainer.py "$#"
Aliases are tricky.
A maybe more nasty solution
mapfile < <(declare -p | grep -m 1 BASH_ALIASES) && bash script.sh "${MAPFILE[#]}"
within your script you will need
shopt -s expand_aliases
eval $1
echo ${BASH_ALIASES[python]}
python --version
How about this:
#!/bin/bash
/data/Python-3.6.9/bin/python3 neural_network_trainer.py [args]
# Repeated with varied args
When a Python script is supposed to be run from a pyenv virtualenv, what is the correct shebang for the file?
As an example test case, the default Python on my system (OS X) does not have pandas installed. The pyenv virtualenv venv_name does. I tried getting the path of the Python executable from the virtualenv.
pyenv activate venv_name
which python
Output:
/Users/username/.pyenv/shims/python
So I made my example script.py:
#!/Users/username/.pyenv/shims/python
import pandas as pd
print 'success'
But when I tried running the script (from within 'venv_name'), I got an error:
./script.py
Output:
./script.py: line 2: import: command not found
./script.py: line 3: print: command not found
Although running that path directly on the command line (from within 'venv_name') works fine:
/Users/username/.pyenv/shims/python script.py
Output:
success
And:
python script.py # Also works
Output:
success
What is the proper shebang for this? Ideally, I want something generic so that it will point at the Python of whatever my current venv is.
I don't really know why calling the interpreter with the full path wouldn't work for you. I use it all the time. But if you want to use the Python interpreter that is in your environment, you should do:
#!/usr/bin/env python
That way you search your environment for the Python interpreter to use.
As you expected, you should be able to use the full path to the virtual environment's Python executable in the shebang to choose/control the environment the script runs in regardless of the environment of the controlling script.
In the comments on your question, VPfB & you find that the /Users/username/.pyenv/shims/python is a shell script that does an exec $pyenv_python. You should be able to echo $pyenv_python to determine the real python and use that as your shebang.
See also: https://unix.stackexchange.com/questions/209646/how-to-activate-virtualenv-when-a-python-script-starts
Try pyenv virtualenvs to find a list of virtual environment directories.
And then you might find a using shebang something like this:
#!/Users/username/.pyenv/python/versions/venv_name/bin/python
import pandas as pd
print 'success'
... will enable the script to work using the chosen virtual environment in other (virtual or not) environments:
(venv_name) $ ./script.py
success
(venv_name) $ pyenv activate non_pandas_venv
(non_pandas_venv) $ ./script.py
success
(non_pandas_venv) $ . deactivate
$ ./script.py
success
The trick is that if you call out the virtual environment's Python binary specifically, the Python interpreter looks around that binary's path location for the supporting files and ends up using the surrounding virtual environment. (See per *How does virtualenv work?)
If you need to use more shell than you can put in the #! shebang line, you can start the file with a simple shell script which launches Python on the same file.
#!/bin/bash
"exec" "pyenv" "exec" "python" "$0" "$#"
# the rest of your Python script can be written below
Because of the quoting, Python doesn't execute the first line, and instead joins the strings together for the module docstring... which effectively ignores it.
You can see more here.
To expand this to an answer, yes, in 99% of the cases if you have a Python executable in your environment, you can just use:
#!/usr/bin/env python
However, for a custom venv on Linux following the same syntax did not work for me since the venv created a link to the Python interpreter which the venv was created from, so I had to do the following:
#!/path/to/the/venv/bin/python
Essentially, however, you are able to call the Python interpreter in your terminal. This is what you would put after #!.
It's not exactly answering the question, but this suggestion by ephiement I think is a much better way to do what you want. I've elaborated a bit and added some more of an explanation as to how this works and how you can dynamically select the Python executable to use:
#!/bin/sh
#
# Choose the Python executable we need. Explanation:
# a) '''\' translates to \ in shell, and starts a python multi-line string
# b) "" strings are treated as string concatenation by Python; the shell ignores them
# c) "true" command ignores its arguments
# c) exit before the ending ''' so the shell reads no further
# d) reset set docstrings to ignore the multiline comment code
#
"true" '''\'
PREFERRED_PYTHON=/Library/Frameworks/Python.framework/Versions/2.7/bin/python
ALTERNATIVE_PYTHON=/Library/Frameworks/Python.framework/Versions/3.6/bin/python3
FALLBACK_PYTHON=python3
if [ -x $PREFERRED_PYTHON ]; then
echo Using preferred python $ALTERNATIVE_PYTHON
exec $PREFERRED_PYTHON "$0" "$#"
elif [ -x $ALTERNATIVE_PYTHON ]; then
echo Using alternative python $ALTERNATIVE_PYTHON
exec $ALTERNATIVE_PYTHON "$0" "$#"
else
echo Using fallback python $FALLBACK_PYTHON
exec python3 "$0" "$#"
fi
exit 127
'''
__doc__ = """What this file does"""
print(__doc__)
import platform
print(platform.python_version())
If you want just a single script with a simple selection of your pyenv virtualenv, you may use a Bash script with your source as a heredoc as follows:
#!/bin/bash
PYENV_VERSION=<your_pyenv_virtualenv_name> python - $# <<EOF
import sys
print(sys.argv)
exit
EOF
I did some additional testing. The following works too:
#!/usr/bin/env -S PYENV_VERSION=<virtual_env_name> python
/usr/bin/env python won't work, since it doesn't know about the virtual environment.
Assuming that you have main.py living next to a ./venv directory, you need to use Python from the venv directory. Or in other words, use this shebang:
#!venv/bin/python
Now you can do:
./main.py
Maybe you need to check the file privileges:
sudo chmod +x script.py
I'm having an issue with my .zshrc file while using oh-my-zsh. Recently, I've started trying to be more careful about mucking with my base OS environment, so I installed Python (2 and 3) and pyenv using homebrew. While trying to configure the autocomplete for pyenv, I switched on the pyenv plugin in oh-my-zsh.
This resulted in my shell shutting down during the launch. I found that I could prevent this from happening by commenting out most of the active portion of the pyenv oh-my-zsh plugin, and I'm not sure why that's causing the shell to exit.
To make this question as concise as possible, I'd like to know what the following function does:
if [ -d $pyenvdir/bin -a $FOUND_PYENV -eq 0 ]
The full code from the plugin follows:
_homebrew-installed() {
type brew &> /dev/null
}
_pyenv-from-homebrew-installed() {
brew --prefix pyenv &> /dev/null
}
FOUND_PYENV=0
pyenvdirs=("$HOME/.pyenv" "/usr/local/pyenv" "/opt/pyenv")
if _homebrew-installed && _pyenv-from-homebrew-installed ; then
pyenvdirs=($(brew --prefix pyenv) "${pyenvdirs[#]}")
fi
for pyenvdir in "${pyenvdirs[#]}" ; do
if [ -d $pyenvdir/bin -a $FOUND_PYENV -eq 0 ] ; then
FOUND_PYENV=1
export PYENV_ROOT=$pyenvdir
export PATH=${pyenvdir}/bin:$PATH
eval "$(pyenv init --no-rehash - zsh)"
function pyenv_prompt_info() {
echo "$(pyenv version-name)"
}
fi
done
unset pyenvdir
if [ $FOUND_PYENV -eq 0 ] ; then
function pyenv_prompt_info() { echo "system: $(python -V 2>&1 | cut -f 2 -d ' ')" }
fi
From what I can tell, it goes something like this:
Check to see if there is a bin directory in one of the pyenvdirs (-d $pyenvdir/bin), ???? (-a), check to see if we already found pyenv previously ($FOUND_PYENV -eq 0).
I tried searching through the zsh documentation, but I can't figure out what the -a is doing. Is it as simple as behaving as an AND statement? If so, why is my shell crashing? Is there an easy way to push the shell output to a log file (on OS X), or is this already done and I just don't know where to look?
What does -a do?
Here -a does indeed mean AND.
Why is it undocumented? Or is it?
The reason you did not find this in the zsh documentation is that the use of the [ builtin (aka test; it is not part of the zsh syntax) is discouraged in favour of conditional expressions (which are surrounded by [[ and ]]).
Here is the relevant part of zshbuiltins(1):
[ [ arg ... ] ]
Like the system version of test. Added for compatibility; use contitional expressions instead [...]
To find documentation on the parameter -a of [ have a look at test(1):
EXPRESSION1 -a EXPRESSION2
both EXPRESSION1 and EXPRESSION2 are true
What happens in the line with -a?
That means that this line
if [ -d $pyenvdir/bin -a $FOUND_PYENV -eq 0 ]
First checks if $pyenvdir/bin exists and is a directory, than it checks if $FOUND_PYENV is equal to 0. If both are true the following block is executed.
Should it kill the shell?
There is no reason why this line should immediatelly lead to the shell exiting.
Looking for the error
All shell output goes to the terminal, so you could just redirect it when starting it. As you are looking for error messages during initialisation, I'd suggest the following procedure:
Disable the problematic configurations
Open a terminal
Check the value of SHLVL: echo $SHLVL
Re-enable the configurations
Start a new z-shell from within the running shell with zsh 2> zsh-error.log, this redirects stderr to the file 'zsh-error.log'.
Check the value of SHLVL again. If it is bigger then previous value then exit the current shell (exit). (Explanation below)
Have a look at 'zsh-error.log' in the current directory.
If 'zsh-error.log' does not show anything, you may want to run zsh -x 2> zsh-error.log in step 5 instead. This provides a complete debug output of anything zsh does. This can get quite huge.
Explanation for SHLVL:
When a shell starts it looks if SHLVL is set on the environment. If so, it increases the value, else it initalizes SHLVL (usually with 1). If your shell started successfully in step 5, SHLVL should be increased. In that case you should stop the shell in order to keep the amount of output in the error log low. On the other hand, if SHLVL is unchanged, the shell terminated on its own and you are back in the original shell provided by the terminal in step 2.
I try to use supervisor with perlbrew, but I can not make it work. For perlbrew I just tried to set the environment variable that go well, but perhaps it is better to make a script that launches perlbrew and plackup, this my configuration file:
[program:MahewinSimpleBlog]
command = perlbrew use perl-5.14.2 && plackup -E deployment -s Starman --workers=10 -p 4000 -a bin/app.pl -D
directory = /home/hobbestigrou/MahewinSimpleBlog
environment = PERL5LIB ='/home/hobbestigrou/MahewinBlogEngine/lib',PERLBREW_ROOT='/home/hobbestigrou/perl5/perlbrew',PATH='/home/hobbestigrou/perl5/perlbrew/bin:/home/hobbestigrou/perl5/perlbrew/perls/perl-5.14.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games',MANPATH='/home/hobbestigrou/perl5/perlbrew/perls/perl-5.14.2/man:',PERLBREW_VERSION='0.43',PERLBREW_PERL='perl-5.14.2',PERLBREW_MANPATH='/home/hobbestigrou/perl5/perlbrew/perls/perl-5.14.2/man',PERLBREW_SKIP_INIT='1',PERLBREW_PATH='/home/hobbestigrou/perl5/perlbrew/bin:/home/hobbestigrou/perl5/perlbrew/perls/perl-5.14.2/bin',SHLVL='2'
user = hobbestigrou
stdout_file = /home/hobbestigrou/mahewinsimpleblog.log
autostart = true
In the log I see it's not looking at the right place:
Error while loading bin/app.pl: Can't locate Type/Params.pm in #INC (#INC contains: /home/hobbestigrou/MahewinSimpleBlog/lib /home/hobbestigrou/MahewinBlogEngine/lib /etc/perl /usr/local/lib/perl/5.14.2 /usr/local/share/perl/5.14.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.14 /usr/share/perl/5.14 /usr/local/lib/site_perl .) at /home/hobbestigrou/MahewinBlogEngine/lib/MahewinBlogEngine/Article.pm line 5.
I do not see the problem, maybe perlbrew use done other things
When you installed perlbrew, you added a command to your .bashrc. You're getting that message because that command wasn't run for the shell in question because it's not an interactive shell.
Why don't you explicitly use /home/hobbestigrou/perl5/perlbrew/perls/perl-5.14.2/bin/perl instead of using perlbrew use?