Develop Reference

Self invocation of interactive shell through Python3 with bash

I am using python3 and subprocess.Popen to spawn a process of bash and invoking the Python3 interpreter again through the standard interpreter.
bash -i states:
-s If the -s option is present, or if no arguments remain after
option processing, then commands are read from the standard
input. This option allows the positional parameters to be
set when invoking an interactive shell.
This is a minimized example but it mainly bakes down to the following code:
import subprocess
import sys
p = subprocess.Popen(["bash", "-s"], stdin=subprocess.PIPE,stderr=sys.stderr, stdout=sys.stdout)
p.stdin.write(b"python3\n")
p.stdin.flush()
print("Done")
The output is simply "Done". Any suggestions how I need to handle the stdin pipes in order to let the interactive shell pop up inside the newly executed python3 interpreter?
Actual output
% python3 test.py
Done
Expected output:
% python3 test.py
Python 3.10.8 (main, Oct 13 2022, 10:17:43) [Clang 14.0.0 (clang-1400.0.29.102)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

How to add header to a file using sed command in python

I'm trying to add the following line to the beginning of file using sed command in python:
ID|SEC_NO|SEC_CD|SEC_DATE|SEC_ID1|SEC_DESC1|SEC_ID2|SEC_DESC2|SEC_ID3|SEC_DESC3
Command: sed -i '1i ID|SEC_NO|SEC_CD|SEC_DATE|SEC_ID1|SEC_DESC1|SEC_ID2|SEC_DESC2|SEC_ID3|SEC_DESC3' file.csv
The above command is working fine from bash. But when i am trying to run the same command from python I am getting error.
cmd =["sed", "-i", "'1i ID|SEC_NO|SEC_CD|SEC_DATE|SEC_ID1|SEC_DESC1|SEC_ID2|SEC_DESC2|SEC_ID3|SEC_DESC3'", "file.csv"]
i got this error message
"sed: -e expression #1, char 1:unkown command: `''\n"
Please assist

Single quotes ('s) are interpreted by shell to treat the enclosed sequence of characters as one argument to the command. When you run a command along with its arguments as a list with subprocess.run in Python, each list item would be passed to the command as-is as an argument, so you should not use single quotes to enclose any of the arguments:
cmd = ["sed", "-i", "1i ID|SEC_NO|SEC_CD|SEC_DATE|SEC_ID1|SEC_DESC1|SEC_ID2|SEC_DESC2|SEC_ID3|SEC_DESC3", "file.csv"]
Otherwise the single quotes themselves would be part of the arguments, which in your case get interpreted by sed.

You should try shlex lib to split your bash command in python.
For example:
$ cat test.txt
abc
$ python3
Python 3.8.10 (default, Jun 2 2021, 10:49:15)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess, shlex
>>> subprocess.run(shlex.split("sed -i '1i ID|SEC_NO|SEC_CD|SEC_DATE|SEC_ID1|SEC_DESC1|SEC_ID2|SEC_DESC2|SEC_ID3|SEC_DESC3' test.txt"))
CompletedProcess(args=['sed', '-i', '1i ID|SEC_NO|SEC_CD|SEC_DATE|SEC_ID1|SEC_DESC1|SEC_ID2|SEC_DESC2|SEC_ID3|SEC_DESC3', 'test.txt'], returncode=0)
>>>
$ cat test.txt
ID|SEC_NO|SEC_CD|SEC_DATE|SEC_ID1|SEC_DESC1|SEC_ID2|SEC_DESC2|SEC_ID3|SEC_DESC3
abc

Running search and replace with sed via python subprocess results in unviewable characters

Given the file /tmp/hi with content: bali=${hi
and running the command on it sed -i -E 's/(^|[^.])hi/\1bi/g' /tmp/hi
results in the following content in bali=${bi as expected.
However, running the sed command inside python3.5 subprocess:
import subprocess
subprocess.run("sed -i -E 's/(^|[^.])hi/\1bi/g' /tmp/hi", shell=True)
results in the following content:
inspected the file in vi and it shows: bali=$^Abi
Why does it happen and how to achieve the same file content using python3.5 subprocess?

That's because the \1 is being interpreted by Python. You need to use raw string syntax (r"some \1 string with escape sequences") if you want to use escape sequences without having to escape them:
Python 3.5.3 (default, Jan 19 2017, 14:11:04)
[GCC 6.3.0 20170118] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print("sed -i -E 's/(^|[^.])hi/\1bi/g' /tmp/hi")
sed -i -E 's/(^|[^.])hi/bi/g' /tmp/hi
>>> print(r"sed -i -E 's/(^|[^.])hi/\1bi/g' /tmp/hi")
sed -i -E 's/(^|[^.])hi/\1bi/g' /tmp/hi

Is it possible to cross reference bash and python variables in python script

I can get a value n when I run a shell command using os.system in the python script, but I also need to sum it up to get a total number for subsequent computation in the python script,
total=0
for i in xrange(1,8):
os.system('n=$(qstat -n1 | grep -o node'+str(i)+' | wc -l) && echo $n')
Is it possible? Also is it possible to use python variable in shell command, something like
os.system('echo $total')

Use the shell's exportcommand:
$ export ABC=1 # Set and export var in shell
$ bash # Start another shell
$ echo $ABC # variable is still here
1
$ python # Start python still in the deeper shell
Python 2.7.2 (default, Oct 11 2012, 20:14:37)
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from os import environ # import environnement
>>> environ['ABC'] # And here's the variable again (it's a string because the shell doesn't have types)
'1'

You can subprocess module's check_output method like this
import subprocess
print sum(int(subprocess.check_output(["/bin/sh", "-c", "n=`expr {} + 1` && echo $n".format(i)])) for i in range(10))
Output
55

python interative mode does not work when reading from std in

Given the following python script....
$ cat readStdin.py
#!/usr/bin/python
import sys
var = "".join(sys.stdin.readlines()).rstrip()
print var
... I get the follwing output:
$ echo hello | python -i readStdin.py
hello
>>>
$
... in other words it does not hang in the python console, but goes back to bash. Does anyone out there know how to make it stay in the python console???

Consider this -
$ echo print 4*2 | python -i
Python 2.7.2 (default, Jun 20 2012, 16:23:33)
Type "help", "copyright", "credits" or "license" for more information.
>>> 8
>>>
$
Echo produces print 4*2. Python even in interactive mode, considers this as input to be interpreted. Hence we see the 8 there. After this, the interpreter encounters an EOF, so it quits. Consider what you press to quit the interpreter - Ctrl+d or ^D. This is just another way to produce EOF on *nix.