I have this command as part of a bash script
$(python -c "import urllib, sys; print urllib.unquote(sys.argv[0])", "h%23g")
But when I run it, I get this:
-bash: -c: command not found
As though bash has missed reading the python, and is thinking -c is the name of the command. Exactly the same happens when using backticks.
How can I make bash recognise the python?
the Python command is returning the string "-c" from your $(...) structure, which bash then tries to execute.
for example
python -c "import urllib, sys; print urllib.unquote(sys.argv[0])"
prints "-c", so you are essentially asking bash to interpret $(-c), for which the error is valid.
I think you want something like the following:
$(python -c "import urllib, sys; print urllib.unquote(sys.argv[1])" "h%23g")
This will result in h#g, if this is all you have on a line then it will also attempt to run a command called h#g, so I'm assuming you are actually using this as a part of a larger command.
The issue with your version is that sys.argv[0] is the -c from the command, and urllib.unquote('-c') will just return '-c'.
From the documentation on sys.argv:
If the command was executed using the -c command line option to the interpreter, argv[0] is set to the string '-c'.
Combining that with info from the man page (emphasis mine):
-c command
Specify the command to execute (see next section). This terminates the option list (following options are passed as arguments to the command).
So, when you use -c, sys.argv[0] will be '-c', the argument provided to -c is the script so it will not be included in sys.argv, and any additional arguments are added to sys.argv starting at index 1.
Related
I use Python 3.10.7 and I am trying to get the Python interpreter to run this command:
rg mysearchterm /home/user/stuff
This command, when I run it in bash directly successfully runs ripgrep and searches the directory (recursively) /home/user/stuff for the term mysearchterm. However, I'm trying to do this programmatically with Python's subprocess.Popen() and I am running into issues:
from subprocess import Popen, PIPE
proc1 = Popen(["rg", "term", "/home/user/stuff", "--no-filename"],stdout=PIPE,shell=True)
proc2 = Popen(["wc","-l"],stdin=proc1.stdin,stdout=PIPE,shell=True)
#Note: I've also tried it like below:
proc1 = Popen(f"rg term /home/user/stuff --no-filename",stdout=PIPE,shell=True)
proc2 = Popen("wc -l",stdin=proc1.stdin,stdout=PIPE,shell=True)
result, _ = proc2.communicate()
print(result.decode())
What happens here was bizarre to me; I get an error (from rg itself) which says:
error: The following required arguments were not provided:
<PATTERN>
So, using my debugging/tracing skills, I looked at the process chain and I see that the python interpreter itself is performing:
python3 1921496 953810 0 /usr/bin/python3 ./debug_script.py
sh 1921497 1921496 0 /bin/sh -c rg term /home/user/stuff --no-filename
sh 1921498 1921496 0 /bin/sh -c wc -l
So my next thought is just trying to run that manually in bash, leading to the same error. However, in bash, when I run /bin/sh -c "rg term /home/user/stuff --no-filename" with double quotations, the command works in bash but when I try to do this programmatically in Popen() it again doesn't work even when I try to escape them with \. This time, I get errors about unexpected EOF.
As for the behavior when shell=True is specified,
the python document tells:
If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
Then your command invocation is equivalent to:
/bin/sh -c "rg" "term" "/home/tshiono/stackoverflow/221215" ...
where no arguments are fed to rg.
You need to pass the command as a string (not a list) or just drop shell=True.
I wrote a python function called plot_ts_ex that takes two arguments ts_file and ex_file (and the file name for this function is pism_plot_routine). I want to run this function from a bash script from terminal.
When I don't use variables in the bash script in pass the function arguments (in this case ts_file = ts_g10km_10ka_hy.nc and ex_file = ex_g10km_10ka_hy.nc) directly, like this:
#!/bin/sh
python -c 'import pism_plot_routine; pism_plot_routine.plot_ts_ex("ts_g10km_10ka_hy.nc", "ex_g10km_10ka_hy.nc")'
which is similar as in Run function from the command line, that works.
But when I define variables for the input arguments, it doesn't work:
#!/bin/sh
ts_name="ts_g10km_10ka_hy.nc"
ex_name="ex_g10km_10ka_hy.nc"
python -c 'import pism_plot_routine; pism_plot_routine.plot_ts_ex("$ts_name", "$ex_name")'
It gives the error:
FileNotFoundError: [Errno 2] No such file or directory: b'$ts_name'
Then I found a similar question passing an argument to a python function from bash for a python function with only one argument and I tried
#!/bin/sh
python -c 'import sys, pism_plot_routine; pism_plot_routine.plot_ts_ex(sys.argv[1])' "$ts_name" "$ex_name"
but that doesn't work.
So how can I pass 2 arguments for a python function in a bash script using variables?
When you use single quotes the variables aren’t going to be expanded, you should use double quotes instead:
#!/bin/sh
ts_name="ts_g10km_10ka_hy.nc"
ex_name="ex_g10km_10ka_hy.nc"
python -c "import pism_plot_routine; pism_plot_routine.plot_ts_ex('$ts_name', '$ex_name')"
You can also use sys.argv, arguments are stored in a list, so ts_name is sys.arv[1] and ex_name is sys.argv[2]:
#!/bin/sh
ts_name="ts_g10km_10ka_hy.nc"
ex_name="ex_g10km_10ka_hy.nc"
python -c 'import sys, pism_plot_routine; pism_plot_routine.plot_ts_ex(sys.argv[1], sys.argv[2])' "$ts_name" "$ex_name"
You are giving the value $ts_name to python as string, bash does not do anything with it. You need to close the ', so that it becomes a string in bash, and then open it again for it to become a string in python.
The result will be something like this:
#!/bin/sh
ts_name="ts_g10km_10ka_hy.nc"
ex_name="ex_g10km_10ka_hy.nc"
python -c 'import pism_plot_routine; pism_plot_routine.plot_ts_ex("'$ts_name'", "'$ex_name'")'
For issues like this it is often nice to use some smaller code to test it, I used python3 -c 'print("${test}")' to figure out what it was passing to python, without the bother of the pism_plot.
I have a simple task and want to run command from the line.
E.g.
python3 -c 'print(2*2)'
The issue is when I want to invoke a function and pass parameter to it. E.g I want to lower the string 'ABC'.
I use
python3 -c 'print(x="ABC",x.lower())'
Hence my question: how could I pass string value to function when invoke python from command line?
Use sys.argv to obtain command-line arguments:
$ python3 -c "import sys; print(sys.argv[1].lower())" HELLO
hello
$
I want to get result of a Python function in terminal.
I tried to run the command:
$ python3 -m uuid uuid.uuid4().hex
And I expect to see the output be something like: '78cbf0fadaa34ff7ac3f7b965965e207'
Unfortunately I get error:
-bash: syntax error near unexpected token `('
You were close.
The flag to run a single command is -c and not -m.
You also need to import uuid so you can use it.
You also need to use print() to actually see some output.
Finally the whole passed command has to be in quotes.
$ python3 -c "import uuid; print(uuid.uuid4().hex)"
8e79508445db4aca91bb0990529fdd89
I am running a python3 script which performs the following snippet on Debian 9:
os.environ["PA_DIR"] = "/home/myname/some_folder"
command_template = ("sudo java -Dconfig.file=$PA_DIR/google.conf "
"-jar ~/big/cromwell-42.jar run $PA_DIR/WholeGenomeGermlineSingleSample.wdl "
"-i {} -o $PA_DIR/au_options.json > FDP{}.log 2>&1")
command = command_template.format("test.json, "1")
os.system("screen -dm -S S{} bash -c '{}'".format("1", command))
The use of PA_DIR works as intended. When I tried it on command line:
PA_DIR="/home/myname/some_folder"
screen -dm -S S1 bash -c 'sudo java -Dconfig.file=$PA_DIR/google.conf -jar ~/big/cromwell-42.jar run $PA_DIR/WholeGenomeGermlineSingleSample.wdl -i test.json -o $PA_DIR/au_options.json > FDP1.log 2>&1'
it doesn't do variable substitution due to single quotes and I had to replace them with double quotes (it complains it cannot find the file /google.conf).
What is different when python runs it?
Thanks!
The Python os.system() invokes the underlying system function of the C library, which on POSIX systems is equivalent to do something like
sh -c "your_command and all its arguments"
So the command and all arguments are already surrounded by double-quotes, which does environment variable substitution. Any single quotes inside the string is irrelevant for the variable substitution.
You can test it easily. In a shell do something like
$ foo="bar"
$ echo "foo is '$foo'" # Will print foo is 'bar'
$ echo 'foo is "$foo"' # Will print foo is "$foo"
Waiting for your answer to daltonfury42, I'd bet the problem is, when running in a command line, you are not exporting the PA_DIR environment variable so it is not present in the second bash interpreter. And it behaves different beacuse of what Mihir answered.
If you run
PA_DIR=foo
you only declare a bash variable but it is not an environment variable. Then
bash -c "echo $PA_DIR"
this will output foo because your current interpreter interpolates $PA_DIR and then raises a second bash process with the command echo foo. But
bash -c 'echo $PA_DIR'
this prevents your bash interpreter from interpolating it so it raises a second bash process with the comand echo $PA_DIR. But in this second process the variable PA_DIR does not exist.
If you start your journey running
export PA_DIR=foo
this will become an environment variable that will be accessible to children processes, thus
bash -c 'echo $PA_DIR'
will output foo because the nested bash interpreter has access to the variable even if the parent bash interpreter did not interpolate it.
The same is true for any kind of children process. Try running
PA_DIR=foo
python3 -c 'import os; print(os.environ.get("PA_DIR"))'
python3 -c "import os; print(os.environ.get('PA_DIR'))"
export PA_DIR=foo
python3 -c 'import os; print(os.environ.get("PA_DIR"))'
python3 -c "import os; print(os.environ.get('PA_DIR'))"
in your shell. No quotes are involved here!
When you use the os.environ dictionary in a Python script, Python will export the variables for you. That's why you will see the variable interpolated by either
os.system("bash -c 'echo $PA_DIR'")
or
os.system('bash -c "echo $PA_DIR"')
But beware that in each case it is either the parent or either the children shell process who is interpolating the variable.
You must understand your process tree here:
/bin/bash # but it could be a zsh, fish, sh, ...
|- /usr/bin/python3 # presumably
|- /bin/sh # because os.system uses that
|- /bin/bash
If you want an environment variable to exist in the most nested process, you must export it anywhere in the upper tree. Or in that very process.