Develop Reference

Python script runs on command line but not from .sh file

I'm attempting to create a .sh file to batch a number of runs of a neural network on Python whilst on holidays.
At the moment I have been calling this from the command line:
python neural_network_trainer.py [args]
I now have a .sh script written:
#!/bin/bash
python neural_network_trainer.py [args]
# Repeated with varied args
That I am attempting to call in the same terminal as the original command line was running:
./august_hols.sh
I get the following error:
File "/data/Python-3.6.9/lib/python3.6/site.py", line 177
file=sys.stderr)
^
SyntaxError: invalid syntax
Where the Python install is in /data (for reasons).
Running which on the command line reports the correct Python directory set via an alias in ~/.bashrc:
alias python=/data/Python-3.6.9/bin/python3
But running which between the Bash shebang and the first python call reports /bin/python.
I've attempted to set the alias again at the start of the .sh script to no avail. I'm scratching my head as this is exact process I have used elsewhere, albeit not on this precise PC. I can copy the exact command from the top of the bash file into the terminal and it runs fine, try and call ./august_hols.sh and get the above Python error.
Where is Bash getting that path from, and why is it not using my expected route through ~/.bashrc?

Bash sub-shell does not inherit alias in the main shell
You can source the script (run in the main shell), instead of execute it (run in the sub-shell)
source script.sh
EDIT:
Solution 2:
Run bash as the login shell so ~/.bashrc is executed, so your alias is loaded before your script.
The subshell needs to be interactive to enable alias, because alias is enabled by default only for interactive shell, but script is non-interactive by default.
bash --login -i script.sh
Solution 3:
Similar to above, except alias is enabled explicitly
bash --login -O expand_aliases script.sh

Have you tried:
python=/data/Python-3.6.9/bin/python3 ./[your_bash].sh

In your .sh
Do this
#!/usr/bin/env bash
export PATH=/data/Python-3.6.9/bin:$PATH
exec python neural_network_trainer.py "$#"
Aliases are tricky.

A maybe more nasty solution
mapfile < <(declare -p | grep -m 1 BASH_ALIASES) && bash script.sh "${MAPFILE[#]}"
within your script you will need
shopt -s expand_aliases
eval $1
echo ${BASH_ALIASES[python]}
python --version

How about this:
#!/bin/bash
/data/Python-3.6.9/bin/python3 neural_network_trainer.py [args]
# Repeated with varied args

Change directory from python script for calling shell

I would like to build a python script, which can manipulate the state of it's calling bash shell, especially it's working directory for the beginning.
With os.chdir or os.system("ls ..") you can only change the interpreters path, but how can I apply the comments changes to the scripts caller then?
Thank you for any hint!

You can't do that directly from python, as a child process can never change the environment of its parent process.
But you can create a shell script that you source from your shell, i.e. it runs in the same process, and in that script, you'll call python and use its output as the name of the directory to cd to:
/home/choroba $ cat 1.sh
cd "$(python -c 'print ".."')"
/home/choroba $ . 1.sh
/home $

Executing shell command using docker-py

I'm trying to run a shell command with docker-py on an already-running container, but get an error:
exec: "export": executable file not found in $PATH
here's how I wrote the script:
exe = client.exec_create(container=my_container, cmd='export MYENV=1')
res = client.exec_start(exec_id=exe)
so my question is how can I run a shell command (inside the container) using docker-py?

You did it quite right. But you confused shell commands with linux executables. exec_create and and exec_start are all about running executables. Like for example bash. export in your example is a shell command. You can only use it in a shell like bash running inside the container.
Additionally what you are trying to achieve (setting a environment variable) is not going to work. As soon as your exec finishes (where you set the env var) the exec process will finish and its environment is been torn down.
You can only create global container environment variables upon creation of a container. If you want to change the env vars, you have to tear down the container and recreate it with your new vars. As you probably know, all data in the container is lost upon a remove unless you use volumes to store your data. Reconnect the volumes on container creation.
That said your example was nearly correct. This should work and create an empty /somefile.
exe = client.exec_create(container=my_container, cmd=['touch', '/somefile'])
res = client.exec_start(exec_id=exe)
To execute shell commands, use this example. It calls sh and tells it to run the interpreter on the given command string (-c)
exe = client.exec_create(container=my_container,
cmd=['/bin/sh', '-c', 'touch /somefile && mv /somefile /bla'])
res = client.exec_start(exec_id=exe)

For actually , when execute cmd docker exec in docker container export MYENV=1. It will fail and report this error
exec: "export": executable file not found in $PATH
Because export is a shell builtin, could run the cmd in shell.
whereis export
type export
can not find export in /usr/bin/ or somewhere else.
There is some ways to pass through this problem.
case1: use -c parameter
/bin/bash -c 'export MYENV=1 ; /bin/bash'
case2: append export cmds to a rcfile, then use this file.
echo "exprot MYENV=1" >> <some_file_path> ; /bin/bash --rcfile <some_file_path>
case3: open a terminal, then enter the cmds to export env parameters , then open a new terminal, the env parameters will work.
/bin/bash
exprot MYENV=1
/bin/bash # open a new terminal

Execute a python script as a command

I'm writing a python script which will be placed in a location. I want to execute it just like a command. for ex.
$ find_branch test
where find_branch is a script placed in anywhere in the system.
I would like to know how to achieve this. I can run it on the place where the script is present by chmod u+x on the script and removing the .py from the script

sudo nano /usr/bin/testpyscript
Then inside the script:
#!/usr/bin/python
print("I'm a python script")
Give it x permission:
sudo chmod +x /usr/bin/testpyscript
Now you can use it as a regular command:
bash-4.2$ testpyscript
I'm a python script
It doesn't have to be exactly at /usr/bin, any location that is inside your $PATH will do. Let's say you want it to be located at some folder inside your home directory, you could do something like this:
pwd
/home/brunorb
mkdir somedir
sudo mv /usr/bin/testpyscript somedir/
export PATH=$PATH:/home/brunorb/somedir/
testpyscript # from any folder in the system
I'm a python script

Make sure python has been added to your path and #!/usr/bin/python is located at the top of your script.
Note You could just try adding your script to your /usr/local/bin/ directory and give it the proper permissions.
sudo cp <your script> /usr/local/bin/

You have a number of options on how to achieve this.
Add the location where you put the script to your PATH environment variable, for example in your ~/.bashrc script:
export PATH="${PATH}:/folder/where/you/put/the/script"
Install the script to a location that is already on your path. It does not have to be a system folder like /usr/bin. Many default Bash setups will include ~/bin in your PATH.
Give the full path to your script on the command line:
/folder/where/you/put/the/script/find_branch test
Run the script through Python. This is very similar to option #2:
python /folder/where/you/put/the/script/find_branch test
Create an alias for the script in your environment. In bash you would do something like the following in your ~/.bashrc:
alias find_branch='/folder/where/you/put/the/script/find_branch'
OR
alias find_branch='python /folder/where/you/put/the/script/find_branch'
For options #1, #2, #3 and #5a to work properly, you should have a shebang with the version of python as the first line of the script. Any of the following will do, depending on how you have/want your environment set up:
#!/usr/bin/python
#!/usr/bin/python2
#!/usr/bin/python3
#!/usr/bin/env python
#!/usr/bin/env python2
#!/usr/bin/env python3
Finally, you do not have to remove the .py extension from the script if you do not want to. Many bash scripts have a .sh extension, for example, which does not prevent them from running as-is. You just have to include the extension in the name of the script when you run it.

From a python script, change user, set environment and run a couple of commands

I need to run a python script that changes user, sets a enviroment variable and executes a command and return the output.
1.) The way I am currently doing this is I am creating a shell script that does this for me:
tmpshell.sh
su - grid -c "echo +ASM1 | . oraenv; asmcmd volinfo -a"
The command fails because the environment is not being set.
2.) The second way I tried was by changing user is python script itself and then creating the shell script.
tmp.py
os.system('su - grid')
TMPFILE="/tmp/tmpfile.sh"
filehandle=open(TMPFILE,'w')
filehandle.write('+ASM1|. oraenv')
filehandle.write('asmcmd volinfo -a')
filehandle.close()
os.chmmod(TMPFILE,0755)
Here the problem is that the python script changes the user but the rest of the script doesn't run until I enter exit.
OUTPUT
[root#odadev1 oakvmclientlib]# python test.py
[grid#odadev1 ~]$ exit
[root#odadev1 oakvmclientlib]#
Any suggestions/better ways to do this ??
p.s.(edit) ". oraenv" is for setting the environment and +ASM1 is the environment variable it expects.

Try something like this:
$ sudo -u grid sh -c ". oraenv; echo +ASM1|asmcmd volinfo -a"
This will launch a shell as user grid, set up the environment in it and execute the command. I'm not sure what the second part of your command does, though - I suspect you want to pipe +ASM1 into the standard input of asmcmd, but you haven't given enough context to be sure.