Develop Reference

escape ampersand & in string when send as argument python

I have written two python scripts A.py and B.py So B.py gets called in A.py like this:
config_object = {}
with open(config_filename) as data:
config_object = json.load(data, object_pairs_hook=OrderedDict)
command = './scripts/B.py --config-file={} --token-a={} --token-b={}'.format(promote_config_filename, config_object['username'], config_object['password'])
os.system(command)
In here config_object['password'] contains & in it. Say it is something like this S01S0lQb1T3&BRn2^Qt3
Now when this value get passed to B.py it gets password as S01S0lQb1T3 So after & whatever it is getting ignored.
How to solve this?

os.system runs a shell. You can escape arbitrary strings for the shell with shlex.quote() ... but a much superior solution is to use subprocess instead, like the os.system documentation also recommends.
subprocess.run(
['./scripts/B.py',
'--config-file={}'.format(promote_config_filename),
'--token-a={}'.format(config_object['username']),
'--token-b={}'.format(config_object['password'])])
Because there is no shell=True, the strings are now passed to the subprocess verbatim.
Perhaps see also Actual meaning of shell=True in subprocess

#tripleee has good suggestions. In terms of why this is happening, if you are running Linux/Unix at least, the & would start a background process. You can search "linux job control" for more info on that. The shortest (but not best) solution is to wrap your special characters in single or double quotes in the final command.
See this bash for a simple example:
$ echo foo&bar
[1] 20054
foo
Command 'bar' not found, but can be installed with:
sudo apt install bar
[1]+ Done echo foo
$ echo "foo&bar"
foo&bar

Piping to a python script from cmd shell

I have a python script (not created by me), let's call it myscript, which I call with several parameters.
So I run the script like this in Windows cmd:
Code:
/wherever/myscript --username=whoever /some/other/path/parameter
And then a prompt appears and I can pass arguments to the python script:
Process started successfully, blabla
Python 2.7.2 blabla
(LoggingConsole)
>>>
And I write my stuff, then quit to be back into cmd:
>>> command1()
>>> command2()
>>> quit()
I suspect some errors occurring in this part, but only once for a hundred trials. So I want to do it by a script.
I want to pipe to this script the internal command1 command2, so that I can test this function thousand times and see when it breaks. I have the following piece of code:
echo 'command1()' | py -i /wherever/myscript --username=whoever /some/other/path/parameter
This unfortunately doesn't generate the same behaviour, as if it would be manually entered.
Can I simulate this behaviour with pipes/redirecting output? Why doesn't it work? I expect that the 'command1()' text will be entered when the script waits for the commands, but it seems I'm wrong.
Thanks!
EDIT 16/02/2021 3:33PM :
I was looking for the cmd shell way to solve this, no python stuff
The piece of script
echo 'command1()' | py -i /wherever/myscript --username=whoever /some/other/path/parameter
is almost correct, just remove the '' :
echo command1() | py -i /wherever/myscript --username=whoever /some/other/path/parameter
my issues were coming from myscript. Once I fixed the weird things on this side, this part was all ok. You can even put all commands together:
echo command1();command2();quit(); | py -i /wherever/myscript --username=whoever /some/other/path/parameter
This question is adapted from a question of gplayersv the 23/08/2012 on unix.com, but the original purpose made the question not answered.

Easy to have pipes.
If you want to get the standard input :
import sys
imput = sys.stdin.read()
print(f'the standard imput was\n{imput}')
sys.stderr.write('This is an error message that will be ignored by piping')
If you want to use the standard input as argument:
echo param | xargs myprogram.py

Python's built-in fileinput module makes this simple and concise:
#!/usr/bin/env python3
import fileinput
with fileinput.input() as f:
for line in f:
print(line, end='')
Than you can accept input in whatever mechanism is easier for you:
$ ls | ./filein.py
$ ./filein.py /etc/passwd
$ ./filein.py < $(uname -r)

Variable notation when running python commands with arguments in a bash script

I have a bash script which is running a bunch of python script all with arguments. In order to have a clean code, I wanted to use variables along the scripts
#!/bin/bash
START=0
SCRIPT_PATH="/opt/scripts/"
IP="192.168.1.111"
if [ "$START" = "0" ]; then
printf "%s: Starting\n" "$DATE_TIME"
PORT=1234
TEST_FILE="$SCRIPT_PATH/Test Scripts/test.TXT"
SCRIPT="$SCRIPT_PATH/script1.py"
ARGS="-P $SCRIPT_PATH/script2.py -n 15 -p $PORT -i $IP"
python "$SCRIPT" ${ARGS} -f "${TEST_FILE}" > ./out.log 2>&1 &
fi
This code is actually working but few things I don't understand :
Why, if I add quotes around ${ARGS}, the arguments are not parsed correctly by python ? What would be the best way to write this ?
What is the best method to add -f "${TEST_FILE}" to the ARGS variable without python blocking on the whitespace and throwing the error: "$SCRIPT_PATH/Test " not found

When you wrap quotes around an argument list, the argument vector receives a single argument with everything that is wrapped in quotes and so, the argument parser fails to do its job properly and you have your issue.
Regarding your second question, it is not easy to embed the quotes into the array, because the quotes will be parsed before being stored in the array, and then when you perform the array expansion to run the command, they will be missing and fail. I have tried this several times with no success.
An alternative approach would mean that you modify a little your script to use a custom internal field separator (IFS) to manually tell what should be considered an argument and what not:
#!/bin/bash
START=0
SCRIPT_PATH="/opt/scripts/"
IP="192.168.1.111"
if [ "$START" = "0" ]; then
printf "%s: Starting\n" "$DATE_TIME"
PORT=1234
TEST_FILE="$SCRIPT_PATH/Test Scripts/test.TXT"
SCRIPT="$SCRIPT_PATH/script1.py"
OLD_IFS=$IFS
IFS=';'
ARGS="$SCRIPT;-P;$SCRIPT_PATH/script2.py;-n;15;-p;$PORT;-i;$IP;-f;$TEST_FILE"
python ${ARGS} > ./out.log 2>&1 &
IFS=$OLD_IFS
fi
As you can see, I replace the spaces in ARGS with semicolons. This way, TEST_FILE variable contents will be considered as a single argument for bash and will be properly populated in argument vector. I'm also moving the script to the argument vector for simplicity, otherwise, Python will not get the proper script path and fail, due to this modification we did to IFS.

I was thinking something like this (with some cruft edited out to make it a standalone example):
#!/bin/bash
SCRIPT_PATH="/opt/scripts/"
IP="192.168.1.111"
PORT=1234
TEST_FILE="$SCRIPT_PATH/Test Scripts/test.TXT"
SCRIPT="$SCRIPT_PATH/script1.py"
set -a ARGS
ARGS=(-P "$SCRIPT_PATH/script2.py" -n 15 -p "$PORT" -i "$IP")
ARGS+=(-f "${TEST_FILE}")
python3 -c "import sys; print(*enumerate(sys.argv), sep='\n')" "${ARGS[#]}"

Executing string of python code within bash script

I've come across a situation where it would be convenient to use python within a bash script I'm writing. I call some executables within my script, then want to do a bit of light data processing with python, then carry on. It doesn't seem worth it to me to write a dedicated script for the processing.
So what I want to do is something like the following:
# do some stuff in bash script
# write some data into datafile.d
python_fragment= << EOF
f = open("datafile.d")
// do some stuff with opened file
print(result)
EOF
result=$(execute_python_fragment $python_fragment) # <- what I want to do
# do some stuff with result
Basically all I want to do is execute a string containing python code. I could of course just make another file containing the python code and execute that, but I'd prefer not to do so. I could do something like echo $python_fragment > temp_code_file, then execute temp_code_file, but that seems inelegant. I just want to execute the string directly, if that's possible.
What I want to do seems simple enough, but haven't figured it out or found the solution online.
Thanks!

You can run a python command direct from the command line with -c option
python -c 'from foo import hello; print (hello())'
Then with bash you could do something like
result=$(python -c '$python_fragment')

You only have to redirect that here-string/document to python
python <<< "print('Hello')"
or
python <<EOF
print('Hello')
EOF
and encapsulate that in a function
execute_python_fragment() {
python <<< "$1"
}
and now you can do your
result=$(execute_python_fragment "${python_fragment}")
You should also add some kind of error control, input sanitizing... it's up to you the level of security you need in this function.

If the string contains the exact python code, then this simple eval() function works.
Here's a really basic example:
>>> eval("print(2)")
2
Hope that helps.

maybe something like
result=$(echo $python_fragment | python3)
only problem is the heredoc assignment in the question doesn't work either. But https://stackoverflow.com/a/1167849 suggests a way to do it if that is what you want to do:
python_fragment=$(cat <<EOF
print('test message')
EOF
) ;
result=$(echo $python_fragment | python3)
echo result was $result

Pass variable from Python to Bash

I am writing a bash script in which a small python script is embedded. I want to pass a variable from python to bash. After a few search I only found method based on os.environ.
I just cannot make it work. Here is my simple test.
#!/bin/bash
export myvar='first'
python - <<EOF
import os
os.environ["myvar"] = "second"
EOF
echo $myvar
I expected it to output second, however it still outputs first. What is wrong with my script? Also is there any way to pass variable without export?
summary
Thanks for all answers. Here is my summary.
A python script embedded inside bash will run as child process which by definition is not able to affect parent bash environment.
The solution is to pass assignment strings out from python and eval it subsequently in bash.
An example is
#!/bin/bash
a=0
b=0
assignment_string=$(python -<<EOF
var1=1
var2=2
print('a={};b={}'.format(var1,var2))
EOF
)
eval $assignment_string
echo $a
echo $b

Unless Python is used to do some kind of operation on the original data, there's no need to import anything. The answer could be as lame as:
myvar=$(python - <<< "print 'second'") ; echo "$myvar"
Suppose for some reason Python is needed to spit out a bunch of bash variables and assignments, or (cautiously) compose code on-the-fly. An eval method:
myvar=first
eval "$(python - <<< "print('myvar=second')" )"
echo "$myvar"

Complementing the useful Cyrus's comment in question, you just can't do it. Here is why,
Setting an environment variable sets it only for the current process and any child processes it launches. os.environ will set it only for the shell that is running to execute the command you provided. When that command finishes, the shell goes away, and so does the environment variable.
You can pretty much do that with a shell script itself and just source it to reflect it on the current shell.

There are a few "dirty" ways of getting something like this done. Here is an example:
#!/bin/bash
myvar=$(python - <<EOF
print "second"
EOF
)
echo "$myvar"
The output of the python process is stored in a bash variable. It gets a bit messy if you want to return more complex stuff, though.

You can make python return val and pass it to bash:
pfile.py
print(100)
bfile.sh
var=$(python pfile.py)
echo "$var"
output: 100

Well, this may not be what you want but one option could be running the other batch commands in python using subprocess
import subprocess
x =400
subprocess.call(["echo", str(x)])
But this is more of a temporary work around. The other solutions are more along what you are looking for.
Hope I was able to help!