I need to run a python command inside a Popen. The problem is that the command NEEDS to run in python3 and I need it to be portable, which means that I can't really use the python3 alias for every situation...
I have computers where python is already the correct version, and others where the correct one is python3. I tried to insert #!/usr/bin python3 in the beginning of the file and then run as python but it didn't work.
I can't modify environment vars to change python3 to python. I would like to know if there is a way to check which one I need to use or a way to change the python3 to python ONLY inside the Popen command...
The Popen command I am trying to run is very simple and no I can't just import the file and use as a class... it needs to be ran through Popen. Also, virtualenv or similars are not an option.
subprocess.Popen(['python', 'main.py'], shell=True, universal_newlines=True)
The shebang -- the initial line showing which interpreter to use, such as #!/usr/bin/python or #!/usr/bin/python3 -- is only honored if you don't explicitly select an interpreter yourself: If you run python foo.py, the OS is invoking a specific Python interpreter and passing it foo.py as an argument (which it interprets as the name of a script it should run); whereas when you run ./foo.py, you're telling the OS itself to figure out which interpreter to use to run foo.py, which it does by looking at the shebang.
To leave it up to the operating system to select, just explicitly specify the name of your script:
subprocess.Popen(['./main.py'], universal_newlines=True)
Related
I'm attempting to run a Linux script through Python's subprocess module. Below is the subprocess command:
result = subprocess.run(['/dir/scripts/0_texts.sh'], shell=True)
print(result)
Here is the 0_texts.sh script file:
cd /dir/TXTs
pylanguagetool text_0.txt > comments_0.txt
The subprocess command executes the script file, writing a new comments_0.txt in the correct directory. However, there's an error in the execution. The comments_0.txt contains an error of "input file is required", and the subprocess result returns returncode=2. When I run the pylanguagetool text_0.txt > comments_0.txt directly in the terminal the command executes properly, with the comments_0.txt written with the proper input file of text_0.txt.
Any suggestions on what I'm missing?
There is some ambiguity here in that it's not obvious which shell is run each time 0_texts.sh is invoked, and whether it has the values you expect of environment variables like PATH, which could result in a different copy of pylanguagetool running from when you call it at the command line.
First I'd suggest removing the shell=True option in subprocess.run, which is only involving another, potentially different shell here. Next I would change subprocess.run(['/dir/scripts/0_texts.sh']) to subprocess.run(['bash', '/dir/scripts/0_texts.sh']) (or whichever shell you wanted to run, probably bash or dash) to remove that source of ambiguity. Finally, you can try using type pylanguagetool in the script, invoking pylanguagetool with its full path, or calling bash /dir/scripts/0_texts.sh from your terminal to debug the situation further.
A bigger-picture issue is, pyLanguageTool is a Python library, so you're almost certainly going to be better off calling its functions from your original Python script directly instead of using a shell script as an intermediary.
I am trying to run python code on a build server. In order to keep the agent clean, I'm creating a virutal environment which can be deleted after the task. The python script calls python via subprocess. The Questions are:
why does the call to subprocess not use the same python virtual env the actual script was called in?
How can this be achieved?
Miminal example:
tmp.py:
from subprocess import check_output
import sys
# python interpreter used to call this script
print(sys.executable)
# check which python interpreter is used when calling subprocess
print(check_output(f'python -c "import sys\nprint(sys.executable)').decode())
run.bat:
#echo off
python -m venv .\test_venv
call .\test_venv\Scripts\activate.bat
python tmp.py
output, where the second line is the default python installation on my computer:
λ run.bat
D:\tmp\pytest\test_venv\Scripts\python.exe
D:\tools\python\python.exe
desired output:
λ run.bat
D:\tmp\pytest\test_venv\Scripts\python.exe
D:\tmp\pytest\test_venv\Scripts\python.exe
I am on 64 bit Windows 10.
The subprocess you create uses the operating system's general PATH traversal to find and run the commands you specify, and doesn't know anything about the parent process.
You already know the value of sys.executable; if that's specifically what you want to run, say so:
print(check_output([sys.executable, "-c", "import sys\nprint(sys.executable)"]), text=True)
(This also avoids the shell, which was providing no value at all. Without an explicit shell=True, your code would only work on Windows.)
(Conversely, on any sane platform, the environment, including the virtual environment, would be inherited by child processes.)
However, Python calling Python is almost always an antipattern. Instead, you want to refactor the code so you can import it and run it in the same process.
I would like to use a python script anywhere within command prompt. This is possible in unix, but I couldn't find anything for windows. Do I have to convert it to .exe?
Edit: not sure why this is being downvoted, maybe it's a silly question but I can't find any similar threads here and I can't be the first person to want to execute .py scripts from their path...
Edit 2: I think my wording was unclear. I would like to know a method to execute python scripts in Windows without needing to specify python path/to/script.py every time. Here is a solution on Linux where the shebang statement invokes the python interpreter, and the script in question can be easily placed in bin: How do I install a script to run anywhere from the command line? . Does there exist a solution like this for Windows?
Here's a solution for running myScript.py:
add to the myScript.py file a first line #!python (or #!python3 if you want to use Python 3)
For instance:
#!python
import sys
sys.stdout.write("hello from Python %s\n" % (sys.version,))
change the "opens with" property of myScript.py to py.exe (to find where it is use where py-- I have it in C:\Windows\py.exe)
put the script myScript.py somewhere in your Windows path
and now you should be able to type myScript.py anywhere in a command prompt and run the Python script with your chosen Python version.
See: https://docs.python.org/3.6/using/windows.html#from-the-command-line
I have a python program that needs to run a bash command on a server. The program works when I run a bash command in my local directory like this:
import subprocess
from subprocess import call
call(['bash', 'Script_Test.sh'])
However, after SSH'ing to a server and running a similar line of code below with a path on the server to a bash script, I get the error "No such file or directory"
call['bash', path]
This doesn't make sense for a number of reasons. I triple checked that the path is correct. I went on Putty, connected to the server on there, and ran a bash command with the same path and it worked, so it can't be the path. I also ran a number of tests to make sure I was SSH'd into the correct server, and I was. I thought there was a security issue on the server with me running bash, so I tried cat instead. Nope, still unable to locate the path.
I'm not super familiar with python subprocesses, so any pointers to anything I'm missing here with "call" would be very helpful.
Making Sure Your Script is Ready for Execution
Give your script a shebang line
First things first, it is important that you include a shebang line in your script on Unix-like systems. I recommend, for your script's portability, that you use #!/usr/bin/env bash.
A Note on File Extensions:
I would recommend that you remove the .sh extension from your script. If you are using the Bourne Again Shell (bash) to execute your script, then using the .sh extension is misleading. Simply put, the Bourne Shell (sh) is different than the Bourne Again Shell (bash) - so don't use a file extension that suggests you are using a different shell than you actually are!
It's not the end of the world if you don't do change your file extension - your script will still be executed as a bash script if you have the proper bash shebang line. Still, it is good practice to either use no file extension (Script_Test -- strongly preferred) or the .bash file extension (Script_Test.bash).
Give your script the proper file permissions
For your purposes, maybe it is only important to give the current user permissions to read and execute the script. In that case, use chmod u+x Script_Test.sh. What is important here is that the correct user (u+) / group (g+) has permissions to execute the script.
Make sure that your script's path is in the $PATH environment variable
Executing your bash script in a Python script
Once you've followed these steps, your Python script should work as you've called it in your question:
import subprocess
from subprocess import call
your_call = call("Test_Script.sh")
If you would rather not move your script into the $PATH environment variable, just make sure that you refer to the script's full path (which is the current directory, ./, in your case):
import subprocess
from subprocess import call
your_call = call("./Test_Script.sh")
Lastly, if your script does not have a shebang line, you will need to specify an additional parameter in your call function:
import subprocess
from subprocess import call
your_call = call("./Test_Script.sh", shell=True)
However, I would not recommend this last approach. See the Python 2.7.12 documentation for the subprocess package...
Warning: Using shell=True can be a security hazard. See the warning under Frequently Used Arguments for details.
Please check out #zenpoy's explanation to a similar StackOverflow question if you are still having problems.
Happy coding!
In Linux, we usually add a shebang in a script to invoke the respective interpreter. I tried the following example.
I wrote a shell script without a shebang and with executable permission. I was able to execute it using ./. But if I write a similar python program, without shebang, I am not able to execute it.
Why is this so? As far as my understanding, shebang is required to find the interpreter. So how does shell scripts work, but not a python script?
My assumption is that a script without a shebang is executed in the current environment, which at the command line is your default shell, e.g. /bin/bash.
shell scripts will only work if you are in the shell you targeted ... there is not python shell ... as such python will never work without explicity calling python (via shebang or command line)
By default the shell will try to execute the script. The #! notation came later
There’a subtle distinction here. If the target is a binary or begins with a #! shebang line, then the shell calls execv successfully. If the target is a text file without a shebang, then the call to execv will fail, and the shell is free to try launching it under /bin/sh or something else.
http://en.wikipedia.org/wiki/Shebang_(Unix)
Under Unix-like operating systems, when a script with a shebang is run as a program, the program loader parses the rest of the script's initial line as an interpreter directive; the specified interpreter program is run instead, passing to it as an argument the path that was initially used when attempting to run the script.
Now when the #! is not found, every line is interpreted as native shell command. And hence if you write a bash script and run it under bash shell it will work. If you run the same bash script in say a tcsh shell it will not work without the initial #!/usr/bin/tcsh