Develop Reference

Using sys.argv to pass command line arguments to a python script

I know this question has been asked in various variations but none of them seem to work for my specific case:
(btw all mentioned files are in my PATH)
I have a simple python script called test.py:
import sys
print('Hello world!')
print(sys.argv)
counter = 0
while True:
counter +=1
The counter is just there to keep the command window open so don't worry about that.
When I enter
test.py test test
into cmd I get the following output:
Hello world!
['C:\\Users\\path\\to\\test.py']
For some reason unknown to me the two other commands (sys.argv[1] and sys.argv[2]) are missing.
However when I create a .bat file like this:
#C:\Users\path\to\python.exe C:\Users\path\to\test.py %*
and call it in cmd
test.bat test test
I get my desired output:
Hello world!
['C:\\Users\\path\\to\\test.py', 'test', 'test']
I've read that the
%*
in the .bat file means that all command line arguments are passed to the python script but why are exactly these arguments not passed to the python script when I explicitly call it in cmd?
From what I've read all command line arguments entered after the script name should be passed to said script but for some reason it doesn't work that way.
What am I overlooking here?

I'm guessing that you need to actually run the script through the command line with C:\Users\path\to\python.exe test.py test test.
I'm not sure how Windows handles just test.py test test but from my limited experience it is probably just trying to open all 3 of those files. Since the first one (test.py) has a .py extension, it is opened with the Python Interpreter and is run automatically. The other two are not actually being passed in as an argument.

Running python script from perl, with argument to stdin and saving stdout output

My perl script is at path:
a/perl/perlScript.pl
my python script is at path:
a/python/pythonScript.py
pythonScript.py gets an argument from stdin, and returns result to stdout. From perlScript.pl , I want to run pythonScript.py with the argument hi to stdin, and save the results in some variable. That's what I tried:
my $ret = `../python/pythonScript.py < hi`;
but I got the following error:
The system cannot find the path specified.
Can you explain the path can't be found?

The qx operator (backticks) starts a shell (sh), in which prog < input syntax expects a file named input from which it will read lines and feed them to the program prog. But you want the python script to receive on its STDIN the string hi instead, not lines of a file named hi.
One way is to directly do that, my $ret = qx(echo "hi" | python_script).
But I'd suggest to consider using modules for this. Here is a simple example with IPC::Run3
use warnings;
use strict;
use feature 'say';
use IPC::Run3;
my #cmd = ('program', 'arg1', 'arg2');
my $in = "hi";
run3 \#cmd, \$in, \my $out;
say "script's stdout: $out";
The program is the path to your script if it is executable, or perhaps python script.py. This will be run by system so the output is obtained once that completes, what is consistent with the attempt in the question. See documentation for module's operation.
This module is intended to be simple while "satisfy 99% of the need for using system, qx, and open3 [...]. For far more power and control see IPC::Run.

You're getting this error because you're using shell redirection instead of just passing an argument
../python/pythonScript.py < hi
tells your shell to read input from a file called hi in the current directory, rather than using it as an argument. What you mean to do is
my $ret = `../python/pythonScript.py hi`;
Which correctly executes your python script with the hi argument, and returns the result to the variable $ret.

The Some of the other answers assume that hi must be passed as a command line parameter to the Python script but the asker says it comes from stdin.
Thus:
my $ret = `echo "hi" | ../python/pythonScript.py`;

To launch your external script you can do
system "python ../python/pythonScript.py hi";
and then in your python script
import sys
def yourFct(a, b):
...
if __name__== "__main__":
yourFct(sys.argv[1])
you can have more informations on the python part here

Import bash variables from a python script

I have seen plenty examples of running a python script from inside a bash script and either passing in variables as arguments or using export to give the child shell access, I am trying to do the opposite here though.
I am running a python script and have a separate file, lets call it myGlobalVariables.bash
myGlobalVariables.bash:
foo_1="var1"
foo_2="var2"
foo_3="var3"
My python script needs to use these variables.
For a very simple example:
myPythonScript.py:
print "foo_1: {}".format(foo_1)
Is there a way I can import them directly? Also, I do not want to alter the bash script if possible since it is a common file referenced many times elsewhere.

If your .bash file is formatted as you indicated - you might be able to just import it direct as a Python module via the imp module.
import imp
bash_module = imp.load_source("bash_module, "/path/to/myGlobalVariables.bash")
print bash_module.foo_1

You can also use os.environ:
Bash:
#!/bin/bash
# works without export as well
export testtest=one
Python:
#!/usr/bin/python
import os
os.environ['testtest'] # 'one'

I am very new to python, so I would welcome suggestions for more idiomatic ways to do this, but the following code uses bash itself to tell us which values get set by first calling bash with an empty environment (env -i bash) to tell us what variables are set as a baseline, then I call it again and tell bash to source your "variables" file, and then tell us what variables are now set. After removing some false-positives and an apparently-blank line, I loop through the "additional" output, looking for variables that were not in the baseline. Newly-seen variables get split (carefully) and put into the bash dictionary. I've left here (but commented-out) my previous idea for using exec to set the variables natively in python, but I ran into quoting/escaping issues, so I switched gears to using a dict.
If the exact call (path, etc) to your "variables" file is different than mine, then you'll need to change all of the instances of that value -- in the subprocess.check_output() call, in the list.remove() calls.
Here's the sample variable file I was using, just to demonstrate some of the things that could happen:
foo_1="var1"
foo_2="var2"
foo_3="var3"
if [[ -z $foo_3 ]]; then
foo_4="test"
else
foo_4="testing"
fi
foo_5="O'Neil"
foo_6='I love" quotes'
foo_7="embedded
newline"
... and here's the python script:
#!/usr/bin/env python
import subprocess
output = subprocess.check_output(['env', '-i', 'bash', '-c', 'set'])
baseline = output.split("\n")
output = subprocess.check_output(['env', '-i', 'bash', '-c', '. myGlobalVariables.bash; set'])
additional = output.split("\n")
# these get set when ". myGlobal..." runs and so are false positives
additional.remove("BASH_EXECUTION_STRING='. myGlobalVariables.bash; set'")
additional.remove('PIPESTATUS=([0]="0")')
additional.remove('_=myGlobalVariables.bash')
# I get an empty item at the end (blank line from subprocess?)
additional.remove('')
bash = {}
for assign in additional:
if not assign in baseline:
name, value = assign.split("=", 1)
bash[name]=value
#exec(name + '="' + value + '"')
print "New values:"
for key in bash:
print "Key: ", key, " = ", bash[key]
Another way to do it:
Inspired by Marat's answer, I came up with this two-stage hack. Start with a python program, let's call it "stage 1", which uses subprocess to call bash to source the variable file, as my above answer does, but it then tells bash to export all of the variables, and then exec the rest of your python program, which is in "stage 2".
Stage 1 python program:
#!/usr/bin/env python
import subprocess
status = subprocess.call(
['bash', '-c',
'. myGlobalVariables.bash; export $(compgen -v); exec ./stage2.py'
]);
Stage 2 python program:
#!/usr/bin/env python
# anything you want! for example,
import os
for key in os.environ:
print key, " = ", os.environ[key]

As stated in #theorifice answer, the trick here may be that such formatted file may be interpreted by both as bash and as python code. But his answer is outdated. imp module is deprecated in favour of importlib.
As your file has extension other than ".py", you can use the following approach:
from importlib.util import spec_from_loader, module_from_spec
from importlib.machinery import SourceFileLoader
spec = spec_from_loader("foobar", SourceFileLoader("foobar", "myGlobalVariables.bash"))
foobar = module_from_spec(spec)
spec.loader.exec_module(foobar)
I do not completely understand how this code works (where there are these foobar parameters), however, it worked for me. Found it here.

When using os.execlp, why `python` needs `python` as argv[0]

The code is like this:
os.execlp('python', 'python', 'child.py', #other args#) # this works
os.execlp('python', 'child.py', #other args#) # this doesn't work
I read this question: execlp() in python
But I'm still confused. The answer said:
The first argument is the program to execute (found on the PATH). The
rest are the sys.argv arguments to the program.
However, if I run: python child.py 1 2 3 and the sys.argv of this process would be ["child.py", "1", "2", "3"], where the python doesn't exist. Then why should I add python as the second parameter of os.execlp?

When python is executed, it creates sys.argv for you. The values in that list are based on the arguments passed to it by the operating system, but it leaves off the sys.executable value from that list.
In other words, when Python is invoked, it sets sys.argv to everything but it's own executable.
When you invoke a new executable via os.execlp(), you still need to include Python in that as that is what executable that the OS will run. The first two values of what you a pass to os.execlp() are still required, whatever you find in sys.argv later on.

The second python is a name for python, it can be any string, but it has to be there.
See the second paragraph of http://docs.python.org/3/library/os.html?highlight=os.exec#process-management:
The various exec* functions take a list of arguments for the new program loaded into the process. In each case, the first of these arguments is passed to the new program as its own name rather than as an argument a user may have typed on a command line. For the C programmer, this is the argv[0] passed to a program’s main(). For example, os.execv('/bin/echo', ['foo', 'bar']) will only print bar on standard output; foo will seem to be ignored.

I realize this was answered LONG ago and the answer is basically right, but there are a few things that are misleading in the way it is worded and in the comments to the answer that I would like to address.
First, I think the clearer way to state what is happening is to highlight that the difference is between the Unix argv list that a process gets handed by the OS and the python sys.argv. The python sys.argv is the Unix argv list with the first element (the command name) removed.
The various os.exec* commands use their first argument to be the actual executable to invoke and the remainder of the line is the Unix argv list, which means that the second argument passed to execlp will be interpreted by the executable as the command line name it was invoked as.
Which takes us to the problem with the comment. The reason that the ls example os.execlp('ls','.') "works" is not because ls does anything special to detect it is called with too few arguments. This example code starts the 'ls' executable with the unix argv list being ['.']. That just means that the ls executable gets started while being told (oddly) that it was invoked as '.', and there are no other command line arguments. And what does ls do when it is run with no other command line arguments: it prints the contents of the current directory, or exactly what one mistakenly thought they were doing when the invoked os.execlp('ls', '.').
You can see that this example really isn't "working" by instead trying os.execlp('ls', '/some/non-existant/path'). That also prints out the contents of the current working directory, and would not be mistaken for "working".

execlp() in python

I was reading programming python 4th edition by Mark Luze, Oreilly, by teaching myself.
There's an example on how to fork a child process, which I do not quite understand:
os.execlp('python', 'python', 'child.py', #other args#)
In an interactive shell(like bash), I know I can type python child.py #args# to ask python interpreter to run child.py with args.
Why are there TWO 'python' in the execlp() function? If I put only one python in the function, I would get an error complainting cannot find file or directory, which is the 1st args of child.py

The first argument is the program to execute (found on the PATH). The rest are the sys.argv arguments to the program.
The first such argument is the program name used to invoke it, and the display value used in the OS process list. It is the value of sys.argv[0] in a python script.

First of all, execlp is rarely used today. In most cases, you'd use the subprocess module, like this:
subprocess.call(['python', 'child.py'])
The first argument of execlp is the file you want to execute.
The latter arguments form the argument array to that program (sys.argv in Python). The first argument is then the name the program got invoked with. For example, Python sets the name to '-c' if the program is being run with the -c option. Similarly, grep behaves differently depending on the first argument, so that users can execute rgrep to imply grep -r.