I am trying to prepare my pythonpath under Ubuntu via subprocess.Popen call, for another script. The call to Python estimateskeleton.py does work fine. However since it needs the python path to be prepared it doesn't work completely correct, since it can not find some other scripts which need to be imported. The export PYTHONPATH command did work with commands.getoutput. However with commands.getoutput the estimateskeleton script still doesn't work / can't find the other files which should be imported. My try to export PYTHONPATH via subprocess.Popen resulted in Error Number 2:
OSError: [Errno 2] No such file or directory
I couldn't find a proper solution with the search function. So I am hoping that one of the more advanced users of this board can help me
Best Regards
import subprocess as sub
import os
import commands
proc = sub.Popen(["export", "PYTHONPATH=\"${PYTHONPATH}:/media/sf_myFolder/Scripts/code/\""],
stdout=sub.PIPE,
stderr=sub.STDOUT)
print proc.communicate()[0]
proc2 = sub.Popen(["python", "estimateskeleton.py"],
stdout=sub.PIPE,
stderr=sub.STDOUT)
print proc2.communicate()[0]
your first Popen command would work without shell=True because export is a shell built-in.
However, that won't fix it, because the second process spawned by Popen is unaware of the previous variable set in a dead process.
So instead of running the first useless Popen, you could add your path to existing PYTHONPATH using os.putenv() like this:
os.putenv("PYTHONPATH",os.pathsep.join([os.getenv("PYTHONPATH",""),"/media/sf_myFolder/Scripts/code"]))
so your next python command is run with the added folder in PYTHONPATH
Related
I'm working on using the subprocess module to send shell commands from Python, specifically, ssh. Below is a barebones sample:
import subprocess
sp = subprocess.run(["ssh"], shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print(f"stdout: {sp.stdout.decode()} \n\nstderr: {sp.stderr.decode()}")
This should return the ssh command help from stdout, and nothing from stderr. However, I get:
stdout:
stderr: 'ssh' is not recognized as an internal or external command,
operable program or batch file.
I've tried other commands, like echo and cd, and those work fine. I am also able to use ssh when manually typing the command into the shell, but it fails when I try to do it through subprocess. The directory C:\Windows\System32\OpenSSH does exist on my computer (and it contains ssh.exe), but for some strange reason I'm unable to cd to it using subprocess.
If it matters, subprocess is using the command prompt, cmd.exe, as it seems to be the default.
Any help is appreciated. Thanks!
-- Edits with tests from comments --
Using the absolute path C:/Windows/System32/OpenSSH/ssh.exe does not work, and gives The system cannot find the path specified via stderr. The OpenSSH folder doesn't seem to be visible to Python through subprocess
os.environ[PATH] contains both C:/Windows/System32/ and C:/Windows/System32/OpenSSH/
Running it with shell=False (either with the absolute path or just with ssh) raises an error in Python: FileNotFoundError: [WinError 2] The system cannot find the file specified
You say C:\Windows\System32\OpenSSH\ssh.exe exists, but that it's not found when running from Python. This is likely a result of having a 32 bit version of Python installed, rather than a 64 bit version.
If the path exists elsewhere, but not for Python, that would tend to implicate the file system redirector. Python is probably seeing C:\Windows\SysWOW64 when you tell it to look in C:\Windows\System32. I'd recommend uninstalling whatever Python you have, and explicitly installing a 64 bit version, so it isn't affected by the redirector, and sees the "real" System32.
I can enter SVN commands successfully in the Windows command line, but when I try to pass them via subprocess with shell=True I get this error:
'svn' is not recognized as an internal or external command, operable program or batch file
When I omit the shell argument I get this:
WindowsError: [Error 2] The system cannot find the file specified
I'm running Python 2.7 on Windows 10, where I also have Python 3.8 installed. I've tried a variety of SVN commands, some complex and some simple, with a variety of arguments, both as a single string and a list of strings, both in IDLE and Spyder (console and script in each), and keep getting the same results. I am able to pass other types of Windows commands via subprocess, just not SVN. I've confirmed that the COMSPEC environment variable is correct. I've also tried moving the path to svn.exe all the way up in the PATH environment variable and rebooting. No dice.
Here's an example of what I'm trying to do:
import subprocess
my_cmd = ['svn', 'propget', 'svn:externals', '-R', '"https://the/rest/of/the/url"']
res = subprocess.check_output(my_cmd, shell=True)
print "the result of the svn command via subprocess is...\n{}".format(res)
After finding that the path to svn.exe was missing from the string returned by os.environ.get('PATH'), I added this path with the following line:
os.environ['PATH'] += r"C:\Program Files\TortoiseSVN\bin;"
And now my subsequent SVN commands are working via subprocess.
Thank you #John Gordon and #Maurice Meyer for the help!
I need to run a python command inside a Popen. The problem is that the command NEEDS to run in python3 and I need it to be portable, which means that I can't really use the python3 alias for every situation...
I have computers where python is already the correct version, and others where the correct one is python3. I tried to insert #!/usr/bin python3 in the beginning of the file and then run as python but it didn't work.
I can't modify environment vars to change python3 to python. I would like to know if there is a way to check which one I need to use or a way to change the python3 to python ONLY inside the Popen command...
The Popen command I am trying to run is very simple and no I can't just import the file and use as a class... it needs to be ran through Popen. Also, virtualenv or similars are not an option.
subprocess.Popen(['python', 'main.py'], shell=True, universal_newlines=True)
The shebang -- the initial line showing which interpreter to use, such as #!/usr/bin/python or #!/usr/bin/python3 -- is only honored if you don't explicitly select an interpreter yourself: If you run python foo.py, the OS is invoking a specific Python interpreter and passing it foo.py as an argument (which it interprets as the name of a script it should run); whereas when you run ./foo.py, you're telling the OS itself to figure out which interpreter to use to run foo.py, which it does by looking at the shebang.
To leave it up to the operating system to select, just explicitly specify the name of your script:
subprocess.Popen(['./main.py'], universal_newlines=True)
I have a tcsh shell script that sets up all the necessary environment including PYTHONPATH, which then run an executable at the end of it. I also have a python script that gets sent to the shell script as an input. So the following works perfectly fine when it is ran from Terminal:
path to shell script path to python script
Now, the problem occurs when I want to do the same thing from a subprocess. The python script fails to be ran since it cannot find many of the modules that's already supposed to be set via the shell script. And clearly, the PYTHONPATH ends up having many missing paths compared to the parent environment the subprocess was ran from or the shell script itself! It seems like the subprocess does not respect the environment the shell script sets up.
I've tried all sorts of things already but none help!
cmd = [shell_script_path, py_script_path]
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=os.environ.copy())
It makes no difference if env is not given either!
Any idea how to fix this?!
Knowing the subprocess inherits all the parent process environment and they are supposed to be ran under same environment, making the shell script to not setup any environment, fixed it.
This solves the environment being retained, but now the problem is, the process just hangs! (it does not happen when it is ran directly from shell)
I have a python program that needs to run a bash command on a server. The program works when I run a bash command in my local directory like this:
import subprocess
from subprocess import call
call(['bash', 'Script_Test.sh'])
However, after SSH'ing to a server and running a similar line of code below with a path on the server to a bash script, I get the error "No such file or directory"
call['bash', path]
This doesn't make sense for a number of reasons. I triple checked that the path is correct. I went on Putty, connected to the server on there, and ran a bash command with the same path and it worked, so it can't be the path. I also ran a number of tests to make sure I was SSH'd into the correct server, and I was. I thought there was a security issue on the server with me running bash, so I tried cat instead. Nope, still unable to locate the path.
I'm not super familiar with python subprocesses, so any pointers to anything I'm missing here with "call" would be very helpful.
Making Sure Your Script is Ready for Execution
Give your script a shebang line
First things first, it is important that you include a shebang line in your script on Unix-like systems. I recommend, for your script's portability, that you use #!/usr/bin/env bash.
A Note on File Extensions:
I would recommend that you remove the .sh extension from your script. If you are using the Bourne Again Shell (bash) to execute your script, then using the .sh extension is misleading. Simply put, the Bourne Shell (sh) is different than the Bourne Again Shell (bash) - so don't use a file extension that suggests you are using a different shell than you actually are!
It's not the end of the world if you don't do change your file extension - your script will still be executed as a bash script if you have the proper bash shebang line. Still, it is good practice to either use no file extension (Script_Test -- strongly preferred) or the .bash file extension (Script_Test.bash).
Give your script the proper file permissions
For your purposes, maybe it is only important to give the current user permissions to read and execute the script. In that case, use chmod u+x Script_Test.sh. What is important here is that the correct user (u+) / group (g+) has permissions to execute the script.
Make sure that your script's path is in the $PATH environment variable
Executing your bash script in a Python script
Once you've followed these steps, your Python script should work as you've called it in your question:
import subprocess
from subprocess import call
your_call = call("Test_Script.sh")
If you would rather not move your script into the $PATH environment variable, just make sure that you refer to the script's full path (which is the current directory, ./, in your case):
import subprocess
from subprocess import call
your_call = call("./Test_Script.sh")
Lastly, if your script does not have a shebang line, you will need to specify an additional parameter in your call function:
import subprocess
from subprocess import call
your_call = call("./Test_Script.sh", shell=True)
However, I would not recommend this last approach. See the Python 2.7.12 documentation for the subprocess package...
Warning: Using shell=True can be a security hazard. See the warning under Frequently Used Arguments for details.
Please check out #zenpoy's explanation to a similar StackOverflow question if you are still having problems.
Happy coding!