Using Java's ProcessBuilder, I try to run a python script with activating virtual environment but i'm getting error like '.\venv\Scripts\activate" "' is not recognized as an internal or external command.
In command window cmd, I can use ".\venv\Scripts\activate" directly with no problem but in my code at below, error occured.
ProcessBuilder processBuilder = new ProcessBuilder("cmd.exe", "/c","cd C:\\Users\\onurc\\PycharmProjects\\OCR", "& .\\venv\\Scripts\\activate", "& python main.py");
That command doesn't look valid to me. You may be confused about what ProcessBuilder does / how windows (or any OS) works.
When you type a command in a dosbox or terminal, you're sending it to the shell - on linux, probably /bin/bash, or zsh or whatnot. On windows, you're sending it to cmd.exe. That program (so, bash or cmd) then decodes what you typed and does all sorts of fancy things to it, and then it in turns asks the underlying kernel to actually go run that app.
Java's ProcessBuilder does not invoke cmd/bash/zsh, it is a sibling to cmd/bash/zsh - it also does some fancy things, but far fewer fancy things. On windows, even the OS (or rather, the tool) does some fancy things whereas on linux it does not.
In general, trying to even guess whether the tool you run does fancy things, and which fancy things java's ProcessBuilder does, is not a sound way to write dependable software. Especially considering the relative hardship of testing this, given that it involves invoking external stuff.
Hence, you should opt out. Of all the fancy things. Thus:
Always call the command with a complete and absolute path.
Always use the multi-string/list version, never rely on java to split on spaces.
Never use *, ? or any other glob. Write out each and every parameter in full.
Do not try to use > or & or any other 'symbol that means something specific'. An OS doesn't support any of this stuff - you just pass [A] the full path to an executable and [B] a bunch of strings that the kernel will directly pass to the process without doing any processing on it. If you want to do stuff to standard out, error out, or standard in - then use Process's methods to do this.
Those &s in your command sure look like shellisms to me.
To fix:
You can tell the process which directory to run in, so there is no need for a cd command.
You should write the absolute path to CMD.exe. You can ask the environment. You're now relying on a combination of a properly configured PATH + the undefined behaviour of the JVM that does a bare minimum of fancy things (turning relative paths to absolute ones by PATH-scanning is a 'fancy thing' you should not be relying on).
I don't know what that & thing is, but you don't want that.
If activate is a script (shouldn't that end in .bat or .js or .vbs or whatnot?), run that, then run python separately. If activate sets things in the local environment table, make a script file that runs both and run the script instead.
Running python is not a good idea. Run C:\whatever\python.exe.
You have the right idea for activate, more or less (you're in a known dir and then write an actual path. If python.exe is guaranteeed to be in that OCR dir, run .\\python.exe, not python.
Convert command as a single string instead of passing separate arguments to the ProcessBuilder.
ProcessBuilder processBuilder = new ProcessBuilder("cmd.exe", "/c","cd C:\\Users\\onurc\\PycharmProjects\\OCR & .\\venv\\Scripts\\activate & python main.py");
Another way could be, if all command has to run under same folder, then change ProcessBuilder working directory using directory method:
Path rootDir = Paths.get("C:\\Users\\onurc\\PycharmProjects\\OCR");
ProcessBuilder processBuilder = new ProcessBuilder("cmd", "/C", ".\\venv\\Scripts\\activate & python main.py");
processBuilder.directory(rootDir.toFile());
Process process = processBuilder.start();
process.waitFor();
Related
In bash or C, exec will terminate the current process and replace it with something new. Can this functionality be accomplished in Python? I don't wish to simply execute some code then continue running the python script (even just to immediately exit), or spawn a child process.
My specific situation is the following. I'm developing a command line application (python + curses) to manage data generation/analysis in the context of scientific computing. It will sometimes be necessary for me to terminate the application and go wrangle with the data in a given subdirectory manually. It would be convenient if I could do something like:
# within python script; d=<some directory>
if exit_and_goto_dir:
exec("pushd {}".format(d)) # C-style exec -- terminate program and execute new command
The following do not work, for example:
# attempt 1
if exit_and_goto_dir:
os.system("pushd {}".format(d))
exit(0) # pushd does not outlast the python script
# attempt 2
if exit_and_goto_dir:
os.chdir(d)
exit(0)
This behavior isn't really critical. There are plenty of work arounds (e.g. print the directory I care about to terminal then cd manually). Mostly I'm curious if it's possible. Thanks!
The os module contains Python wrappers for the various exec* functions in the C standard library:
>>> [method for method in dir(os) if method.startswith("exec")]
['execl', 'execle', 'execlp', 'execlpe', 'execv', 'execve', 'execvp', 'execvpe']
However, pushd is not an executable that you can exec but rather a bash builtin (and the same is true for cd).
What you could do would be to change directory inside the python process and then exec an interactive shell:
import os
os.chdir(d)
os.execvp("bash", ["bash", "-login"])
Python's current directory will be inherited by the shell that you exec. When you later exit from that shell, control will then return to the original login shell from which you invoked python (unless you used that shell's exec command to launch python in the first place).
What you can't do is to modify the current directory of the calling shell from inside python, in order to return to the shell prompt but in a different working directory from when python was invoked. (At least there's no straightforward way. There is a hack involving attaching gdb to it, described here, but which only worked as root when I tried it on Ubuntu.)
I'm trying to change the terminal directory through a python script. I've seen this post and others like it so I know about os.chdir, but it's not working the way I'd like. os.chdir appears to change the directory, but only for the python script. For instance I have this code.
#! /usr/bin/env python
import os
os.chdir("/home/chekid/work2/")
print os.getcwd()
Unfortunately after running I'm still in the directory of the python script (e.g. /home/chekid) rather than the directory I want to be in. See below.
gandalf(pts/42):~> pwd
/home/chekid
gandalf(pts/42):~> ./changedirectory.py
/home/chekid/work2
gandalf(pts/42):~> pwd
/home/chekid
Any thoughts on what I should do?
Edit: Looks like what I'm trying to do doesn't exist in 'normal' python. I did find a work around, although it doesn't look so elegant to me.
cd `./changedirectory.py`
You can't. The shell's current directory belongs to the shell, not to you.
(OK, you could ptrace(2) the shell and make it call chdir(2), but that's probably not a great design, won't work on Windows, and I would not begin to know how to do it in pure Python except that you'd probably have to mess around with ctypes or something similar.)
You could launch a subshell with your current working directory. That might be close enough to what you need:
os.chdir('/path/to/somewhere')
shell = os.environ.get('SHELL', '/bin/sh')
os.execl(shell, shell)
# execl() does not return; it replaces the Python process with a new shell process
The original shell will still be there, so make sure you don't leave it hanging around. If you initially call Python with the exec builtin (e.g. exec python /path/to/script.py), then the original shell will be replaced with the Python process and you won't have to worry about this. But if Python exits without launching the shell, you'll be left with no shell open at all.
You can if you cheat: Make a bash script that calls your python script. The python script returns the path you want to change directory to. Then the bash script does the acctual chdir. Of course you would have to run the bash script in your bash shell using "source".
The current working directory is an attribute of a process. It cannot be changed by another program, such as changing the current working directory in your shell by running a separate Python program. This is why cd is always a shell built-in command.
You can make your python print the directory you want to move to, and then call your script with cd "$(./python-script.py)". In condition your script actually does not print anything else.
So this is an unusual one, and perhaps I am simply missing the obvious, but I have the following python code that creates a powershell script and runs it.
# Create the PowerShell file
f = open("getKey.ps1", "w")
f.write('$c = Get-BitlockerVolume -MountPoint C:\n')
f.write('$c.KeyProtector[1].RecoveryPassword | Out-File C:\\Temp\\recovery.key\n')
# Invoke Script
startPS = subprocess.Popen([r'C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe',
'-ExecutionPolicy', 'Unrestricted', './getKey.ps1'], cwd=os.getcwd())
result = startPS.wait()
When this is run, it gives me the following error:
The term 'Get-BitlockerVolume' is not recognized as the name of a cmdlet, function, script file, or operable program.
However, if I then go and manually run the generated script, it works perfectly. To add to the oddity, if I run the same command exactly as above ie:
C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy Unrestricted ./getKey.ps1
it also works exactly as expected.
Clearly, the above error is a powershell error, so it is successfully running the script. It almost seems like powershell somehow knows that this is being run from python and has some restricted library of commands when a script is run from a particular source. I grant that that idea makes no real sense, but it's certainly how things appear.
I don't think this is a permissions issue, because when you run the same command from an unelevated powershell prompt, you get an Access is denied type error, rather than a command doesn't exist kind of error.
Anyway, any help would be greatly appreciated!
Edits
Edit: New evidence to help figure this out:
It's definitely an issue of cmdlets being loaded properly. If I programmatically run a script to dump the list of all available commands to a text file, it is only about 2/3's as big as if I do so through a powershell prompt directly
I bet Python is running as a 32-bit process on 64-bit Windows. In this case, you'll end up running 32-bit PowerShell, which in practice is a Bad Thing since many PowerShell modules depend on native binaries that may not have 32-bit equivalents. I hit this with IIS Manager commandlets--the commandlets themselves are registered in 32-bit PowerShell, but the underlying COM objects they rely on are not.
If you need to run 64-bit PowerShell from a 32-bit process, specify the path as %SystemRoot%\SysNative\WindowsPowerShell\v1.0\PowerShell.exe instead of System32.
System32 is actually virtualized for 32-bit processes and refers to the 32-bit binaries in %SystemRoot%\SysWow64. This is why your paths (and PSMODULEPATH) will look the same, but aren't. (SysNative is also a virtualized path that only exists in virtualized 32-bit processes.)
Adding to what #jbsmith said in the comment, also check to make sure that the environment variable that PowerShell relies on to know where it's modules are is populated correctly when python starts the process.
%PSMODULEPATH% is the environment variable in question, and it works the same way the %PATH% variable does, multiple directories separated by ;. Based on what you say your observed behavior is, it seems that you are using PowerShell 3.0, and cmdlet autoloading is in effect.
The solution here: Run a powershell script from python that uses Web-Administration module got me the cmdlet I needed, however there are still missing cmdlets even when using this method. I'm still at a loss as to why some are loaded and others are not, but for the time being, my script does what I need it to and I can't spend any more time to figure it out.
For reference here is the code that worked for me
startPS = subprocess.Popen([r'C:\Windows\sysnative\cmd.exe', '/c', 'powershell',
'-ExecutionPolicy', 'Unrestricted', './getKey.ps1'], cwd=os.getcwd())
I had the same issue, and it was simply that the BitLocker feature was not installed, hence the module wasn't present.
I fixed it by installing the Bitlocker feature:
Windows Server:
Install-WindowsFeature BitLocker -IncludeAllSubFeature -IncludeManagementTools -Restart
Windows Desktop:
Enable-WindowsOptionalFeature -Online -FeatureName BitLocker -All
(Note: I’ve Linux in mind, but the problem may apply on other platforms.)
Problem: Linux doesn’t do suid on #! scripts nor does it activate “Linux capabilities” on them.
Why dow we have this problem? Because during the kernel interpreter setup to run the script, an attacker may have replaced that file. How? The formerly trusted suid/capability-enabled script file may be in a directory he has control over (e.g. can delete the not-owned trusted file, or the file is actually a symbolic link he owns).
Proper solution: make the kernel allow suid/cap scripts if: a) it is clear that the caller has no power over the script file -or- like a couple of other operating systems do b) pass the script as /dev/fd/x, referring to the originally kernel-opened trusted file.
Answer I’m looking for: for kernels which can’t do this (all Linux), I need a safe “now” solution.
What do I have in mind? A binary wrapper, which does what the kernel does not, in a safe way.
I would like to
hear from established wrappers for (Python) scripts that pass Linux capabilities and possibly suid from the script file to the interpreter to make them effective.
get comments on my wrapper proposed below
Problems with sudo: sudo is not a good wrapper, because it doesn’t help the kernel to not fall for that just explained “script got replaced” trap (“man sudo” under caveats says so).
Proposed wrapper
actually, I want a little program, which generates the wrapper
command line, e.g.: sudo suid_capability_wrapper ./script.py
script.py has already the suid bit and capabilites set (no function, just information)
the generator suid_capability_wrapper does
generate C(?) source and compile
compile output into: default: basename script.py .py, or argument -o
set the wrapper owner, group, suid like script.py
set the permitted capabilities like script.py, ignore inheritable and effective caps
warn if the interpreter (e.g. /usr/bin/python) does not have the corresponding caps in its inheritable set (this is a system limitation: there is no way to pass on capabilites without suid-root otherwise)
the generated code does:
check if file descriptors 0, 1 and 2 are open, abort otherwise (possibly add more checks for too crazy environment conditions)
if compiled-in target script is compiled-in with relative path, determine self’s location via /proc/self/exe
combine own path with relative path to the script to find it
check if target scripts owner, group, permissions, caps, suid are still like the original (compiled-in) [this is the only non-necessary safety-check I want to include: otherwise I trust that script]
set the set of inherited capabilities equal to the set of permitted capabilities
execve() the interpreter similar to how the kernel does, but use the script-path we know, and the environment we got (the script should take care of the environment)
A bunch of notes and warnings may be printed by suid_capability_wrapper to educate the user about:
make sure nobody can manipulate the script (e.g. world writable)
be aware that suid/capabilities come from the wrapper, nothing cares about suid/xattr mounts for the script file
the interpreter (python) is execve()ed, it will get a dirty environment from here
it will also get the rest of the standard process environment passed through it, which is ... ... ... (read man-pages for exec to begin with)
use #!/usr/bin/python -E to immunize the python interpreter from environment variables
clean the environment yourself in the script or be aware that there is a lot of code you run as side-effect which does care about some of these variables
You don't want to use a shebang at all, on any file - you want to use a binary which invokes the Python interpreter, then tells it to start the script file for which you asked.
It needs to do three things:
Start a Python interpreter (from a trusted path, breaking chroot jails and so on). I suggest statically linking libpython and using the CPython API for this, but it's up to you.
Open the script file FD and atomically check that it is both suid and owned by root. Don't allow the file to be altered between the check and the execution - be careful.
Tell CPython to execute the script from the FD you opened earlier.
This will give you a binary which will execute all owned-by-root-and-suid scripts under Python only. You only need one such program, not one per script. It's your "suidpythonrunner".
As you surmised, you must clear the environment before running Python. LD_LIBRARY_PATH is taken care of by the kernel, but PYTHONPATH could be deadly.
I have created a little pre-commit hook in python. This hook works like a charm under Linux, but in Windows it keeps telling me:
error: cannot spawn .git/hooks/pre-commit: No such file or directory
I know there have been similar questions here about the same issue and the conclusion seams to be the shebang. My script has this on the very first line:
#!F:\PortableApps\PortablePython3.2\App\python.exe
It's also interesting to note that executing the script simply by writing .git/hooks/pre-commit works wonderful, but as soon as I try to commit, git spits out the above message.
Another interesting thing is, when I convert the encoding from ANSI to UTF-8 (using Notepad++), I get the following error when trying to execute the script:
.git/hooks/pre-commit: Cannot execute binary file
I'm using the following tools:
PortablePython 3.2.1.1
msysgit 1.7.6 (Portable)
I used the proxy-approach to make the python script work under windows (with msysgit). The complete script (with description on how I did it) might be found here: https://gist.github.com/1839424
Here is the important part about making it work under Windows
If you're working with Windows (and "msysgit"), it's a little more complicated. Since "msysgit" seems to have a problem handling the SHEBANG, you'll have to use a little trick to make the script executable (further information on this problem can be found here).
In order to make the script work, you'll want to remove the SHEBANG from the Python script ("pre-commit.py") and use a wrapper bash-script to call the interpreter. This script should look something like this:
#!/bin/sh
python .git/hooks/pre-commit.py
Store this script as a file called "pre-commit" (no file-ending). This assumes that you have Python in your PATH. If you don't, you can also specify the full path to your interpreter-executable.
This script will be called by "git commit" and call the python-script to check for the huge files. The path after the SHEBANG should not be changed, as "msysgit" will remap it automatically. You must specify a path relative to the repo-root for the Python script to be executed (because thats from where the script is called).
Afterwards you'll want to copy both the wrapper-file ("pre-commit") and the Python-script ("pre-commit.py") to your repos ".git/hooks"-directory, personalize the Python-script ("max_file_size" and "git_binary_path") and mark the "pre-commit"-file executable.