Develop Reference

Change which python.exe to be used in bash script

Assume that python refers to C:\Program\python.exe as standard and I have a program which should be run with C:\Program\python_2.exe.
If I do
#!/bin/bash/
python=C:\Program\python_2.exe
python -c "print('Hello world!')" >log.txt 2>&1
it still uses the standard python and not python_2

first, let's check what your script actually does:
the first line, assigns a value C:\Program\python_2.exe to a variable named python:
python=C:\Program\python_2.exe
however, the next line doesn't use this variable at all.
it will simply run a program python:
python -c "print('Hello world!')" >log.txt 2>&1
funnily the program has the same name as one of the many variables, but that doesn't really matter.
for the shell, variables are totally unrelated to program-names (which are literals, searched for in ${PATH}).
if you want to use a variable for the program, you must make this explicit:
${python} -c "print('Hello world!')" >log.txt 2>&1
(
this still might not work, as backslashes on un*x systems (and bash comes from that realm) are considered special, so the ${python} variable might not actually hold what you think it does:
$ echo ${python}
C:Programpython_2.exe
so you probably need to escape the backslashes:
python="C:\\Program\\python_2.exe"
)
if you don't want to use a variable for calling your program but the literal python, you could define a function:
#!/bin/sh
# this defines a *shell-function* named `python`, which can be used as if it were a program:
python() {
# call a program (python_2.exe) with all the arguments that were given to the function:
C:\\Program\\python_2.exe "$#"
}
# call the 'python' function with some args:
python -c "print('Hello world!')" >log.txt 2>&1

You can try to use scl enable, to enable python version 2.7:
cd /var/www/python/scripts/
scl enable python27 "python runAllUpserts.py >/dev/null 2>&1"
Then you can use :
python -V

Evaluate "Shell command line with shell variables" from python OR evaluate python string as shell command line

CONTEXT
I am working on a simulation cluster.
In order to make as flexible as possible (working with different simulation soft) , we created a python file that parse a config file defining environment variables, and command line to start the simulation. This command is launched through SLURM sbatch command (shell $COMMAND)
ISSUE
From python, all Environment variables are enrolled reading the config file
I have issue with variable COMMAND that is using other environment variables (displayed as shell variable)
For example
COMMAND = "fluent -3ddp -n$NUMPROCS -hosts=./hosts -file $JOBFILE"
os.environ['COMMAND']=COMMAND
NUMPROCS = "32"
os.environ['NUMPROCS']=NUMPROCS
[...]
exe = Popen(['sbatch','template_document.sbatch'], stdout=PIPE, stderr=PIPE)
sbatch distribute COMMAND to all simulation nodes as COMMAND being a command line
COMMAND recalls other saved env. variables. Shell interprets it strictly as text... Which makes the command line fails. it is strictly as a string using $ not variable for example :
'fluent -3ddp -n$NUMPROCS -hosts=./hosts -file $JOBFILE'
SOLUTION I AM LOOKING FOR
I am looking for a simple solution
Solution 1: A 1 to 3 python command lines to evaluate the COMMAND as shell command to echo
Solution 2: A Shell command to evaluate the variables within the "string" $COMMAND as a variable
At the end the command launched from within sbatch should be
fluent -3ddp -n32 -hosts=./hosts -file /path/to/JOBFILE

You have a few options:
Partial or no support for bash's variable substitution, e.g. implement some python functionality to reproduces bash's $VARIABLE syntax.
Reproduce all of bash's variable substitution facilities which are supported in the config file ($VARIABLE, ${VARIABLE}, ${VARIABLE/x/y}, $(cmd) - whatever.
Let bash do the heavy lifting, for the cost of performance and possibly security, depending on your trust of the content of the config files.
I'll show the third one here, since it's the most resilient (again, security issues notwithstanding). Let's say you have this config file, config.py:
REGULAR = "some-text"
EQUALS = "hello = goodbye" # trap #1: search of '='
SUBST = "decorated $REGULAR"
FANCY = "xoxo${REGULAR}xoxo"
CMDOUT = "$(date)"
BASH_A = "trap" # trap #2: avoid matching variables like BASH_ARGV
QUOTES = "'\"" # trap #3: quoting
Then your python program can run the following incantation:
bash -c 'source <(sed "s/^/export /" config.py | sed "s/[[:space:]]*=[[:space:]]*/=/") && env | grep -f <(cut -d= -f1 config.py | grep -E -o "\w+" | sed "s/.*/^&=/")'
which will produce the following output:
SUBST=decorated some-text
CMDOUT=Thu Nov 28 12:18:50 PST 2019
REGULAR=some-text
QUOTES='"
FANCY=xoxosome-textxoxo
EQUALS=hello = goodbye
BASH_A=trap
Which you can then read with python, but note that the quotes are now gone, so you'll have to account for that.
Explanation of the incantation:
bash -c 'source ...instructions... && env | grep ...expressions...' tells bash to read & interpret the instructions, then grep the environment for the expressions. We're going to turn the config file into instructions which modify bash's environment.
If you try using set instead of env, the output will be inconsistent with respect to quoting. Using env avoids trap #3.
Instructions: We're going to create instructions for the form:
export FANCY="xoxo${REGULAR}xoxo"
so that bash can interpret them and env can read them.
sed "s/^/export /" config.py prefixes the variables with export.
sed "s/[[:space:]]*=[[:space:]]*/=/" converts the assignment format to syntax that bash can read with source. Using s/x/y/ instead of s/x/y/g avoids trap #1.
source <(...command...) causes bash to treat the output of the command as a file and run its lines, one by one.
Of course, one way to avoid this complexity is to have the file use bash syntax to begin with. If that were the case, we would use source config.sh instead of source <(...command...).
Expressions: We want to grep the output of env for patterns like ^FANCY=.
cut -d= -f1 config.py | grep -E -o "\w+" finds the variable names in config.py.
sed "s/.*/^&=/" turns variable names like FANCY to grep search expressions such as ^FANCY=. This is to avoid trap #2.
grep -f <(...command...) gets grep to treat the output of the command as a file containing one search expression in each line, which in this case would be ^FANCY=, ^CMDOUT= etc.
EDIT
Since you actually want to just pass this environment to another bash command rather than use it in python, you can actually just have python run this:
bash -c 'source <(sed "s/^/export /" config.py | sed "s/[[:space:]]*=[[:space:]]*/=/") && $COMMAND'
(assuming that COMMAND is specified in the config file).

It seems I have not explained well enough the issue, but your 3rd solution seems replying to my expectations... though so far I did not manage to adapt it
Based on your 3rd solution BASH, I will make it more straight :
Let's say I have got following after running python, and this that cannot be modified
EXPORT COMMAND='fluent -3ddp -n$NUMPROCS -hosts=./hosts -file $JOBFILE'
EXPORT JOBFILE='/path/to/jobfile'
EXPORT NUMPROCS='32'
EXPORT WHATSOEVER='SPECIFIC VARIABLE TO SIMULATION SOFTWARE'
I wish to execute the following from the slurm batch file (bash), using $COMMAND / $JOBFILE /$NUMPROCS
fluent -3ddp -n32-hosts=./hosts -file /path/to/jobfile
Please note : I have backup solution in python - I managed to substitute $VARIABLE by its value - based on the assumption $VARIABLE is not composed by another $variable... using regex substitution... just it looks so many lines to have what seemed to me simple request

SHA256 of visually similar strings differs in Bash vs Python

I am trying to construct an AWS Signature v4 Auth header to call the STS GetCallerIdentity API as per the documentation at https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html in Bash.
Now, I have the same process working in Python, and after poring minutely over my scripts and outputs in Python and Bash, I see that the SHA256 calculated in Bash for the string is different from the one calculated in Python.
The strings look the same in my text editor - character by character.
But since the SHA256 differs, I am assuming that this much be a problem with encoding of the string.
The Python script uses UTF8, and even though I have tried doing a printf "%s" "${string}" | iconv -t utf-8 | openssl dgst -sha256 in the Bash script, the hash values still differ.
How do I convert bash strings/variables to UTF8 before calculating the SHA256 sum.

It might helpful to see how you're calculating it in Python. From what I can see, it looks like the output is the same.
$ python -c "import hashlib; \
print(hashlib.sha256('test'.encode('utf8')).digest().hex())"
9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
$ printf "%s" test | openssl dgst -sha256
(stdin)= 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08

How do I pipe Unicode into a native application in PowerShell

I have a native program written in Python that expects its input on stdin. As a simple example,
#!python3
import sys
with open('foo.txt', encoding='utf8') as f:
f.write(sys.stdin.read())
I want to be able to pass a (PowerShell) string to this program as standard input. Python expects its standard input in the encoding specified in $env:PYTHONIOENCODING, which I will typically set to UTF8 (so that I don't get any encoding errors).
But no matter what I do, characters get corrupted. I've searched the net and found suggestions to change [Console]::InputEncoding/[Console]::OutputEncoding, or to use chcp, but nothing seems to work.
Here's my basic test:
PS >[Console]::OutputEncoding.EncodingName
Unicode (UTF-8)
PS >[Console]::InputEncoding.EncodingName
Unicode (UTF-8)
PS >$env:PYTHONIOENCODING
utf-8
PS >python -c "print('\N{Euro sign}')" | python -c "import sys; print(sys.stdin.read())"
´╗┐?
PS >chcp 1252
Active code page: 1252
PS >python -c "print('\N{Euro sign}')" | python -c "import sys; print(sys.stdin.read())"
?
PS >chcp 65001
Active code page: 65001
PS >python -c "print('\N{Euro sign}')" | python -c "import sys; print(sys.stdin.read())"
?
How can I fix this problem?
I can't even explain what's going on here. Basically, I want the test (python -c "print('\N{Euro sign}')" | python -c "import sys; print(sys.stdin.read())") to print out a Euro sign. And to understand why, I have to do whatever is needed to get that to work :-) (Because then I can translate that knowledge to my real scenario, which is to be able to write working pipelines of Python programs that don't break when they encounter Unicode characters).

Thanks to mike z, the following works:
$OutputEncoding = [Console]::OutputEncoding = (new-object System.Text.UTF8Encoding $false)
$env:PYTHONIOENCODING = "utf-8"
python -c "print('\N{Euro sign}')" | python -c "import sys; print(sys.stdin.read())"
The new-object is needed to get a UTF-8 encoding without a BOM. The $OutputEncoding variable and [Console]::OutputEncoding both appear to need to be set.
I still don't fully understand the difference between the two encoding values, and why you would ever have them set differently (which appears to be the default).

How can I pass the results of Python's help('modules') output to pbcopy in one line?

Just like the title addresses, how can this be done? I stupidly tried the following, but I will share the stupidity here so you can get an idea as to what I want to happen:
myself$ python help('modules') | pbcopy
Is this a good idea:
fout = open('output.txt', 'w')
fout.write(help('modules'))

On my Ubuntu, and hopefully on your boxen too (as it is a standard python feature), there is the handy pydoc command, thus it is very easy to type
pydoc modules | pbcopy

Use pydoc to look up documentation and print it.
Example:
$ python -c 'import pydoc; print pydoc.getdoc(id)'
id(object) -> integer
Return the identity of an object. This is guaranteed to be unique among
simultaneously existing objects. (Hint: it's the object's memory address.)

I don't know what is pbcopy, but I gouess this woul do the trick:
python -c 'import urllib; help(urllib)' | pbcopy
at least this is definitely works:
python -c 'import urllib; help(urllib)' > file
From man python:
-c command
Specify the command to execute (see next section). This terminates the option list (following options are passed as arguments to the command).
Update:
In order to copy this to clipboard you can add this to ~/.bashrc:
pc() { python -c "import $1; help($1);" | xclip -i -selection clipboard; }
then just call pc logging or pc my_module
Or you can pipe it to pbcopy or what ever works for you.