Develop Reference

Newline character written to file with echo works in shell, but literal in Python 3

I'm on a Ubuntu 20.04 system, and I'm using Python 3.8 to write a script that does multiple things using configurable lines of bash, but one of them is that it creates desktop shortcuts.
This single-line command creates a desktop shortcut, and works flawlessly when I execute it directly in my terminal:
echo "[Desktop Entry]"$'\n'"Type=Application"$'\n'"Name[en_US]=Documents"$'\n'"Exec=pcmanfm ~/Documents"$'\n'"Icon=system-file-manager" > ~/Desktop/1;
However, when I execute it in Python, like so:
foobar.py
rl = """echo "[Desktop Entry]"$'\n'"Type=Application"$'\n'"Name[en_US]=Documents"$'\n'"Exec=pcmanfm ~/Documents"$'\n'"Icon=system-file-manager" > ~/Desktop/1;"""
subprocess.run(rl, shell=True)
...instead of creating a desktop shortcut with the proper name, icon, and action, it creates an empty file that contains the following text:
0.txt:
[Desktop Entry]$\nType=Application$\nName[en_US]=Sign Out$\nExec=/home/agent/QaSC/login/login.bin$\nIcon=system-switch-user
Is there any particular reason why Python would be handling the newline characters differently than the bash shell does, and if so, how can I resolve this problem?

$'...' is a bash extension, but the default shell used when shell=True is specified is sh. Use the executable option to specify an alternate shell.
subprocess.run(rl, shell=True, executable='/bin/bash')

Since the argument to echo has quotes, it could contain literal newlines at the command line, and therefore also in the Python process. I see no reason to use the Bash extension $'\n' syntax.
$ echo "foo
> bar"
foo
bar
$ python
Python 3.8.10 (default, Mar 15 2022, 12:22:08)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.run('echo "foo\nbar"', shell=True)
foo
bar
CompletedProcess(args='echo "foo\nbar"', returncode=0)
>>>
(It's also unclear why the Python code doesn't just write a file in the normal way, but I assume the command in the OP is essentially a placeholder.)

How to add header to a file using sed command in python

I'm trying to add the following line to the beginning of file using sed command in python:
ID|SEC_NO|SEC_CD|SEC_DATE|SEC_ID1|SEC_DESC1|SEC_ID2|SEC_DESC2|SEC_ID3|SEC_DESC3
Command: sed -i '1i ID|SEC_NO|SEC_CD|SEC_DATE|SEC_ID1|SEC_DESC1|SEC_ID2|SEC_DESC2|SEC_ID3|SEC_DESC3' file.csv
The above command is working fine from bash. But when i am trying to run the same command from python I am getting error.
cmd =["sed", "-i", "'1i ID|SEC_NO|SEC_CD|SEC_DATE|SEC_ID1|SEC_DESC1|SEC_ID2|SEC_DESC2|SEC_ID3|SEC_DESC3'", "file.csv"]
i got this error message
"sed: -e expression #1, char 1:unkown command: `''\n"
Please assist

Single quotes ('s) are interpreted by shell to treat the enclosed sequence of characters as one argument to the command. When you run a command along with its arguments as a list with subprocess.run in Python, each list item would be passed to the command as-is as an argument, so you should not use single quotes to enclose any of the arguments:
cmd = ["sed", "-i", "1i ID|SEC_NO|SEC_CD|SEC_DATE|SEC_ID1|SEC_DESC1|SEC_ID2|SEC_DESC2|SEC_ID3|SEC_DESC3", "file.csv"]
Otherwise the single quotes themselves would be part of the arguments, which in your case get interpreted by sed.

You should try shlex lib to split your bash command in python.
For example:
$ cat test.txt
abc
$ python3
Python 3.8.10 (default, Jun 2 2021, 10:49:15)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess, shlex
>>> subprocess.run(shlex.split("sed -i '1i ID|SEC_NO|SEC_CD|SEC_DATE|SEC_ID1|SEC_DESC1|SEC_ID2|SEC_DESC2|SEC_ID3|SEC_DESC3' test.txt"))
CompletedProcess(args=['sed', '-i', '1i ID|SEC_NO|SEC_CD|SEC_DATE|SEC_ID1|SEC_DESC1|SEC_ID2|SEC_DESC2|SEC_ID3|SEC_DESC3', 'test.txt'], returncode=0)
>>>
$ cat test.txt
ID|SEC_NO|SEC_CD|SEC_DATE|SEC_ID1|SEC_DESC1|SEC_ID2|SEC_DESC2|SEC_ID3|SEC_DESC3
abc

Running search and replace with sed via python subprocess results in unviewable characters

Given the file /tmp/hi with content: bali=${hi
and running the command on it sed -i -E 's/(^|[^.])hi/\1bi/g' /tmp/hi
results in the following content in bali=${bi as expected.
However, running the sed command inside python3.5 subprocess:
import subprocess
subprocess.run("sed -i -E 's/(^|[^.])hi/\1bi/g' /tmp/hi", shell=True)
results in the following content:
inspected the file in vi and it shows: bali=$^Abi
Why does it happen and how to achieve the same file content using python3.5 subprocess?

That's because the \1 is being interpreted by Python. You need to use raw string syntax (r"some \1 string with escape sequences") if you want to use escape sequences without having to escape them:
Python 3.5.3 (default, Jan 19 2017, 14:11:04)
[GCC 6.3.0 20170118] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print("sed -i -E 's/(^|[^.])hi/\1bi/g' /tmp/hi")
sed -i -E 's/(^|[^.])hi/bi/g' /tmp/hi
>>> print(r"sed -i -E 's/(^|[^.])hi/\1bi/g' /tmp/hi")
sed -i -E 's/(^|[^.])hi/\1bi/g' /tmp/hi

Is it possible to cross reference bash and python variables in python script

I can get a value n when I run a shell command using os.system in the python script, but I also need to sum it up to get a total number for subsequent computation in the python script,
total=0
for i in xrange(1,8):
os.system('n=$(qstat -n1 | grep -o node'+str(i)+' | wc -l) && echo $n')
Is it possible? Also is it possible to use python variable in shell command, something like
os.system('echo $total')

Use the shell's exportcommand:
$ export ABC=1 # Set and export var in shell
$ bash # Start another shell
$ echo $ABC # variable is still here
1
$ python # Start python still in the deeper shell
Python 2.7.2 (default, Oct 11 2012, 20:14:37)
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from os import environ # import environnement
>>> environ['ABC'] # And here's the variable again (it's a string because the shell doesn't have types)
'1'

You can subprocess module's check_output method like this
import subprocess
print sum(int(subprocess.check_output(["/bin/sh", "-c", "n=`expr {} + 1` && echo $n".format(i)])) for i in range(10))
Output
55

Using "konsole" command to run python script

I'm trying to, from a command line, open an instance of konsole and run a python script. I'm trying:
konsole -hold -e 'python -i hello.py'
The behaviour I'm getting is that a persistent konsole opens, and I am dropped into python, but the script does not run.
Python 2.7.2+ (default, Oct 4 2011, 20:03:08)
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
What do I need to do to get the python script to run in the konsole window?

jsbueno's solution is the correct one. However, as described here, you can also do something like this ...
konsole --hold -e /bin/sh -c "python -i hello.py"
P.S. you'll need to specify --workdir (before the -e arg), or provide the full path to the python script, if it's not always in the initial working dir of konsole. But, you probably already knew that.

The problem is the way "konsole" uses the parameters after the -e switch - it seems like it simply pass them in a call that does not interpret the space separators as parameter separators.
However, if you don't put your call parameters inside quotes it will work - that is, simply:
konsole --hold -e python -i hello.py
(I just tested it here)