I am trying to create a bash script that runs a python3 script with pdb trace set. As such, in my bash script I have the following lines:
python3 path/to/my_script.py
n
What I want to happen is for bash to run the python script, which will then open the python debugger. Then, the bash script will send the key 'n' to the pdb shell so that pdb executes the first line in the python script.
The script does not work as expected and bash waits until the python script has completed (or exited) to execute 'n' in the command line which just opens node.
I thought this might be a problem unique to pdb shells so I tried executing the following in bash:
python3
print("Hello")
However, again, we observe that the script creates a python3 shell and then waits for the shell to exit before executing print("Hello") in the terminal. I understand that I could use python3 -c for this case, but that does not address the case of passing commands to the pdb shell in the context of the running script.
Is there any way to send the 'n' command to the pdb shell that the python script generates?
Your code will try to run two commands. First, it will run your python script, then it will try to run a command called n. Assuming your script needs you read from stdin you can do one of the following:
Use a herestring:
python3 path/to/my_script.py <<< "n"
Use a pipeline:
One example echo "n" | python3 path/to/my_script.py
Echo is not the only command you can use. You can also use printf or even yes for this use case.
You can use a coprocess to send and receive from pdb.
#! /bin/bash
send() {
echo "$1" >&${PDB[1]}
}
recv() {
IFS= read -r -u ${PDB[0]} line
echo $line
}
coproc PDB { /path/to/my_script.py; }
send 'n'
recv
#...
send 'n'
recv
Related
I would like to run this multiline shell commands:
echo 'a=?'
read a
echo "a=$a"
from a python script, using the subprocess.call() method.
I wrote this, in test.py file:
import shlex, subprocess
args = ["echo", 'a=?',"read", "a", "echo", "a=$a"]
subprocess.call(args)
and when I execute it, I have in terminal this report:
Armonicus#MyMacs-iMac MyNewFolder % python test.py
a=? read a echo a=$a
which is not at least close to what I expect.
Can I have some support from anyone, please?
There are a couple of issues with your approach here.
First, if what you're trying to do is prompt the user for input from the command line, then you can use Python builtins instead of a subprocess:
a = input('a=?')
print(a)
If you do want to call a subprocess with multiple commands, you need to either make separate calls for each command, or invoke a shell and execute the commands within it. For example:
subprocess.call("echo 'a=?'; read a; echo $a", shell=True)
I have a script where I need to start a command, then pass some additional commands as commands to that command. I tried
su
echo I should be root now:
who am I
exit
echo done.
... but it doesn't work: The su succeeds, but then the command prompt is just staring at me. If I type exit at the prompt, the echo and who am i etc start executing! And the echo done. doesn't get executed at all.
Similarly, I need for this to work over ssh:
ssh remotehost
# this should run under my account on remotehost
su
## this should run as root on remotehost
whoami
exit
## back
exit
# back
How do I solve this?
I am looking for answers which solve this in a general fashion, and which are not specific to su or ssh in particular. The intent is for this question to become a canonical for this particular pattern.
Adding to tripleee's answer:
It is important to remember that the section of the script formatted as a here-document for another shell is executed in a different shell with its own environment (and maybe even on a different machine).
If that block of your script contains parameter expansion, command substitution, and/or arithmetic expansion, then you must use the here-document facility of the shell slightly differently, depending on where you want those expansions to be performed.
1. All expansions must be performed within the scope of the parent shell.
Then the delimiter of the here document must be unquoted.
command <<DELIMITER
...
DELIMITER
Example:
#!/bin/bash
a=0
mylogin=$(whoami)
sudo sh <<END
a=1
mylogin=$(whoami)
echo a=$a
echo mylogin=$mylogin
END
echo a=$a
echo mylogin=$mylogin
Output:
a=0
mylogin=leon
a=0
mylogin=leon
2. All expansions must be performed within the scope of the child shell.
Then the delimiter of the here document must be quoted.
command <<'DELIMITER'
...
DELIMITER
Example:
#!/bin/bash
a=0
mylogin=$(whoami)
sudo sh <<'END'
a=1
mylogin=$(whoami)
echo a=$a
echo mylogin=$mylogin
END
echo a=$a
echo mylogin=$mylogin
Output:
a=1
mylogin=root
a=0
mylogin=leon
3. Some expansions must be performed in the child shell, some - in the parent.
Then the delimiter of the here document must be unquoted and you must escape those expansion expressions that must be performed in the child shell.
Example:
#!/bin/bash
a=0
mylogin=$(whoami)
sudo sh <<END
a=1
mylogin=\$(whoami)
echo a=$a
echo mylogin=\$mylogin
END
echo a=$a
echo mylogin=$mylogin
Output:
a=0
mylogin=root
a=0
mylogin=leon
A shell script is a sequence of commands. The shell will read the script file, and execute those commands one after the other.
In the usual case, there are no surprises here; but a frequent beginner error is assuming that some commands will take over from the shell, and start executing the following commands in the script file instead of the shell which is currently running this script. But that's not how it works.
Basically, scripts work exactly like interactive commands, but how exactly they work needs to be properly understood. Interactively, the shell reads a command (from standard input), runs that command (with input from standard input), and when it's done, it reads another command (from standard input).
Now, when executing a script, standard input is still the terminal (unless you used a redirection) but the commands are read from the script file, not from standard input. (The opposite would be very cumbersome indeed - any read would consume the next line of the script, cat would slurp all the rest of the script, and there would be no way to interact with it!) The script file only contains commands for the shell instance which executes it (though you can of course still use a here document etc to embed inputs as command arguments).
In other words, these "misunderstood" commands (su, ssh, sh, sudo, bash etc) when run alone (without arguments) will start an interactive shell, and in an interactive session, that's obviously fine; but when run from a script, that's very often not what you want.
All of these commands have ways to accept commands by ways other than in an interactive terminal session. Typically, each command supports a way to pass it commands as options or arguments:
su root -c 'who am i'
ssh user#remote uname -a
sh -c 'who am i; echo success'
Many of these commands will also accept commands on standard input:
printf 'uname -a; who am i; uptime' | su
printf 'uname -a; who am i; uptime' | ssh user#remote
printf 'uname -a; who am i; uptime' | sh
which also conveniently allows you to use here documents:
ssh user#remote <<'____HERE'
uname -a
who am i
uptime
____HERE
sh <<'____HERE'
uname -a
who am i
uptime
____HERE
For commands which accept a single command argument, that command can be sh or bash with multiple commands:
sudo sh -c 'uname -a; who am i; uptime'
As an aside, you generally don't need an explicit exit because the command will terminate anyway when it has executed the script (sequence of commands) you passed in for execution.
If you want a generic solution which will work for any kind of program, you can use the expect command.
Extract from the manual page:
Expect is a program that "talks" to other interactive programs according to a script. Following the script, Expect knows what can be expected from a program and what the correct response should be. An interpreted language provides branching and high-level control structures to direct the dialogue. In addition, the user can take control and interact directly when desired, afterward returning control to the script.
Here is a working example using expect:
set timeout 60
spawn sudo su -
expect "*?assword" { send "*secretpassword*\r" }
send_user "I should be root now:"
expect "#" { send "whoami\r" }
expect "#" { send "exit\r" }
send_user "Done.\n"
exit
The script can then be launched with a simple command:
$ expect -f custom.script
You can view a full example in the following page: http://www.journaldev.com/1405/expect-script-example-for-ssh-and-su-login-and-running-commands
Note: The answer proposed by #tripleee would only work if standard input could be read once at the start of the command, or if a tty had been allocated, and won't work for any interactive program.
Example of errors if you use a pipe
echo "su whoami" |ssh remotehost
--> su: must be run from a terminal
echo "sudo whoami" |ssh remotehost
--> sudo: no tty present and no askpass program specified
In SSH, you might force a TTY allocation with multiple -t parameters, but when sudo will ask for the password, it will fail.
Without the use of a program like expect any call to a function/program which might get information from stdin will make the next command fail:
ssh use#host <<'____HERE'
echo "Enter your name:"
read name
echo "ok."
____HERE
--> The `echo "ok."` string will be passed to the "read" command
I'm attempting to create a .sh file to batch a number of runs of a neural network on Python whilst on holidays.
At the moment I have been calling this from the command line:
python neural_network_trainer.py [args]
I now have a .sh script written:
#!/bin/bash
python neural_network_trainer.py [args]
# Repeated with varied args
That I am attempting to call in the same terminal as the original command line was running:
./august_hols.sh
I get the following error:
File "/data/Python-3.6.9/lib/python3.6/site.py", line 177
file=sys.stderr)
^
SyntaxError: invalid syntax
Where the Python install is in /data (for reasons).
Running which on the command line reports the correct Python directory set via an alias in ~/.bashrc:
alias python=/data/Python-3.6.9/bin/python3
But running which between the Bash shebang and the first python call reports /bin/python.
I've attempted to set the alias again at the start of the .sh script to no avail. I'm scratching my head as this is exact process I have used elsewhere, albeit not on this precise PC. I can copy the exact command from the top of the bash file into the terminal and it runs fine, try and call ./august_hols.sh and get the above Python error.
Where is Bash getting that path from, and why is it not using my expected route through ~/.bashrc?
Bash sub-shell does not inherit alias in the main shell
You can source the script (run in the main shell), instead of execute it (run in the sub-shell)
source script.sh
EDIT:
Solution 2:
Run bash as the login shell so ~/.bashrc is executed, so your alias is loaded before your script.
The subshell needs to be interactive to enable alias, because alias is enabled by default only for interactive shell, but script is non-interactive by default.
bash --login -i script.sh
Solution 3:
Similar to above, except alias is enabled explicitly
bash --login -O expand_aliases script.sh
Have you tried:
python=/data/Python-3.6.9/bin/python3 ./[your_bash].sh
In your .sh
Do this
#!/usr/bin/env bash
export PATH=/data/Python-3.6.9/bin:$PATH
exec python neural_network_trainer.py "$#"
Aliases are tricky.
A maybe more nasty solution
mapfile < <(declare -p | grep -m 1 BASH_ALIASES) && bash script.sh "${MAPFILE[#]}"
within your script you will need
shopt -s expand_aliases
eval $1
echo ${BASH_ALIASES[python]}
python --version
How about this:
#!/bin/bash
/data/Python-3.6.9/bin/python3 neural_network_trainer.py [args]
# Repeated with varied args
I need to write a bash script, that executes a python program and i need to output the time of execution and the results in the same file.
I CAN NOT edit the python code.
As there are multiple tests I want to execute them in background.
I've tried this
#!bin/bash
$(time python3 program.py file1 > solAndTimeFile1.txt &)
but it didn't work at all, it only outputs the python program results in the solAndTimeFile1.txt and the time is shown in the terminal.
I've also tried this:
#!bin/bash
$(time python3 program.py file1 > solAndTimeFile1.txt >> solAndTimeFile1.txt &)
Same output and makes even less sense to me.
Put your command into curly braces so it is run in a subshell and you can capture its output. To redirect both stdout and stderr to a file use &>file. See man bash for further information.
{ time python3 program.py file1; } &>solAndTimeFile1.txt &
I have a python script that I want to execute in the background on my unix server. The catch is that I need the python script to wait for the previous step to finish before moving onto the next task, yet I want my job to continue to run after I exit.
I think I can set up as follows but would like confirmation:
An excerpt of the script looks like this where command 2 is dependent on the output from command 1 since it outputs an edited executable file in same directory. I would like to point out that commands 1 and 2 do not have the nohup/& included.
subprocess.call('unix command 1 with options', shell=True)
subprocess.call('unix command 2 with options', shell=True)
If when I initiate my python script like so:
% nohup python python_script.py &
Will my script run in the background since I explicitly did not put nohup/& in my scripted unix commands but instead ran the python script in the background?
yes, by running your python script with nohup (no hangup), your script won't keel over when the network is severed and the trailing & symbol will run your script in the background.
You can still view the output of your script, nohup will pipe the stdout to the nohop.out file. You can babysit the output in real time by tailing that output file:
$ tail -f nohop.out
quick note about the nohup.out file...
nohup.out The output file of the nohup execution if
standard output is a terminal and if the
current directory is writable.
or append the command with & to run the python script as a deamon and tail the logs.
$ nohup python python_script.py > my_output.log &
$ tail -f my_output.log
You can use nohup
chomd +x /path/to/script.py
nohup python /path/to/script.py &
Or
Instead of closing your terminal, use logout It is not SIGHUP when you do logout thus the shell won't send a SIGHUP to any of its children.children.