Develop Reference

Automatization of third party libraries in python with PowerShell or CMD

If I want to run my Python 3.8 program in Windows 10, I have to firstly load below commands in PowerShell:
PS> cmd
CMD> call [somePath]/environmentSettingScript.bat
CMD> set Path=%Path%;D:[someOtherPath]\win64
After that, I am able to run Python in the same window (still CMD running in PS) and load my third party libraries.
CMD> python
>>> from thirdPartyLibrary import something
or
CMD> python programThatUseThirdPartyLibrary.py
Of course, it works also, if I make call and set procedures from first snippet directly in CMD. Nevertheless, I don't like such kind of solutions. I want to automatize this procedure, in programThatUseThirdPartyLibrary.py code directly with Spyder IDE usage. To do this I try write something like that:
import subprocess
pipe = subprocess.Popen(r"cmd \c call [somePath]/environmentSettingScript.bat \c set Path=%Path%;D:[someOtherPath]\win64",
stdin = subprocess.PIPE,
stdout = subprocess.PIPE,
stderr = subprocess.PIPE,
encoding='utf8',
shell=True)
stdout, stderr = pipe.communicate()
print("No stderr" if pipe.returncode == 0 else "Some stderr")
from thirdPartyLibrary import something
Unfortunately, stdout shows only the information about loading first line of cmd command:
Microsoft Windows [Version 10.0.17763.1697]
(c) 2018 Microsoft Corporation. All rights reserved.
D:\[workDir]>
and it looks like, that the rest of commands stuck in the pipe. I am quite confused, because print() and stderr shows that, everything goes ok - but import at the end of the script fails with error
ModuleNotFoundError: No module named thirdPartyLibrary

unable to execute powershell script using python process but the same works in cmd line

i have a powershell script i'm trying to execute via python subprocess. i use check_output imported from subprocess for other cmd line processes, so i am using it for the powershell script. inside my powershell script i'm loading some 3rd party modules and then executing a cmdlet from those modules. here is my python code..
from subprocess import Popen, STDOUT, check_output
cmd = 'C:\\Windows\\SysWOW64\\WindowsPowerShell\\v1.0\\powershell.exe -nologo -file c:\\test.ps1'
cmd_output = check_output(cmd, stderr=STDOUT)
output = cmd_output.decode("utf-8").split('\n')
When the python code runs it takes some time where i assume it's loading the modules via the code i have in the powershell ps1 script , BUT i get an error that the cmdlet i was referencing inside my powershell script is "not recognized as the name of a cmdlet", which happens when the modules don't get loaded properly in the powershell script, so i don't think my modules are loading which is weird.. i also noticed when the ps1 script loads via python it runs for about 10 secs then returns, if i run the ps1 script (see further below) manually from command line it usually takes 60 secs to load the modules inside the ps1 script.
i also tried another method for sub proccess instead of check_output, see below, with no differences
cmd = Popen(['powershell.exe', '-nologo', '-file', 'c:\\test.ps1'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
BUT i can open a cmd window and execute it manually perfectly fine using
C:\Windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe -nologo -file C:\test\test.ps1
which should be the same as i'm already doing in python :( so i'm at a loss why it's not working in python??
also my execution policy is as follows
PS C:\> Get-ExecutionPolicy -list
Scope ExecutionPolicy
----- ---------------
MachinePolicy Undefined
UserPolicy Undefined
Process Undefined
CurrentUser Unrestricted
LocalMachine Unrestricted

Try using unrestricted ExecutionPolicy parameter :
# python 2.7 and 3.4
import subprocess
fname = "C:\\Demo\\test.ps1"
p = subprocess.Popen(["powershell", "-ExecutionPolicy", "unrestricted", "-nologo", "-file", "%s"%(fname)], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
if p.returncode == 0:
print("buffer :")
print(out)
else:
print("error :")
print(err)
The content of 'C:\Demo\test.ps1'
write-host $env:USERPROFILE;
Get-ExecutionPolicy -list;
try{
Get-Module -Name VMware.VimAutomation.Core -ListAvailable | Import-Module;
write-host 'ok import VMware';
}catch{
write-host 'error import VMware';
write-host $_;
}
The output after script execution (note the Process policy)
buffer :
C:\Users\MyUserName
ok import VMware
Scope ExecutionPolicy
----- ---------------
... ...
... ...
Process Unrestricted
... ...
... ...
On system32 Module path is here:
C:\WINDOWS\system32\WindowsPowerShell\v1.0\
Try using:
p = subprocess.Popen(["powershell", "-ExecutionPolicy", "unrestricted", "-nologo", "-command", "&{ Get-Module –ListAvailable }"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for showing all available modules

Open Developer Command Promt from Python and write commands

I tried to open "Developer Command Prompt for VS 2017" from Python, write some command within like cd .., and then run an exe.
However it doesn't let me to write commands at all.
I have tried:
process = subprocess.Popen('C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\Common7\\Tools\\VsDevCmd.bat',
shell=False, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=None)
process.stdin.write('1.txt'.encode(encoding='utf_8', errors='strict'));
output = process.stdout.readlines()
print(output)
When I open the developer command promtpt on my own it is directed to c:\>.
So, I saved c:\ in a text file called 1.txt to know if it's run the command within Python. However, in the Python console I only get the welcome text:
[b'**********************************************************************\r\n',
b'** Visual Studio 2017 Developer Command Prompt v15.5.2\r\n',
b'** Copyright (c) 2017 Microsoft Corporation\r\n',
b'**********************************************************************\r\n']

The issue is with VsDevCmd.bat closing before you can communicate with the generated subprocess as i believe
Your method works but it is better if you use communicate instead of directly sending input to stdin:
output = communicate('cd c: \n')[0]
Check the documentation on communicate
You can run the same code above using communicate and calling CMD instead :
process = subprocess.Popen('cmd',shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=None)
output = process.communicate('dir\n')[0]
print output

Open new gnome-terminal and run command

I'm trying to write a script that opens a new terminal then runs a separate python script from that terminal.
I've tried:
os.system("gnome-terminal 'python f.py'")
and
p = Popen("/usr/bin/gnome-terminal", stdin=PIPE)
p.communicate("python f.py")
but both methods only open a new terminal and do not run f.py. How would I go about opening the terminal AND running a separate script?
Edit:
I would like to open a new terminal window because f.py is a simply server that is running serve_forever(). I'd like the original terminal window to stay "free" to run other commands.

Like most terminals, gnome terminal needs options to execute commands:
gnome-terminal [-e, --command=STRING] [-x, --execute]
You probably need to add -x option:
x, --execute
Execute the remainder of the command line inside the terminal.
so:
os.system("gnome-terminal -x python f.py")
That would not run your process in the background unless you add & to your command line BTW.
The communicate attempt would need a newline for your input but should work too, but complex processes like terminals don't "like" being redirected. It seems like using an interactive tool backwards.
And again, that would block until termination. What could work would be to use p.stdin.write("python f.py\n") to give control to the python script. But in that case it's unlikely to work.
So it seems that you don't even need python do to what you want. You just need to run
python f.py &
in a shell.

As of GNOME Terminal 3.24.2 Using VTE version 0.48.4 +GNUTLS -PCRE2
Option “-x” is deprecated and might be removed in a later version of gnome-terminal.
Use “-- ” to terminate the options and put the command line to execute after it.
Thus the preferred syntax appears to be
gnome-terminal -- echo hello
rather than
gnome-terminal -x echo hello

Here is a complete example of how you would call a executable python file with subprocess.call Using argparse to properly parse the input.
the target process will print your given input.
Your python file to be called:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--file", help="Just A test", dest='myfile')
args = parser.parse_args()
print args.myfile
Your calling python file:
from subprocess import call
#call(["python","/users/dev/python/sandboxArgParse.py", "--file", "abcd.txt"])
call(["gnome-terminal", "-e", "python /users/dev/python/sandboxArgParse.py --file abcd.txt"])
Just for information:
You probably don't need python calling another python script to run a terminal window with a process, but could do as follows:
gnome-terminal -e "python /yourfile.py -f yourTestfile.txt"

The following code will open a new terminal and execute the process:
process = subprocess.Popen(
"sudo gnome-terminal -x python f.py",
stdout=subprocess.PIPE,
stderr=None,
shell=True
)

I am running a uWS server with this.In my case Popen didn't help(Even though it run the executable, still it couldn't communicate with a client -: socket connection is broken).This is working.Also now they recommends to use "--" instead of "-e".
subprocess.call(['gnome-terminal', "--", "python3", "server_deployment.py"])
#server_deployment.py
def run():
execution_cmd = "./my_executable arg1 arg2 dll_1 dll_2"
os.system(execution_cmd)
run()

Getting the entire output from subprocess.Popen

I'm getting a slightly weird result from calling subprocess.Popen that I suspect has a lot to do with me being brand-new to Python.
args = [ 'cscript', '%USERPROFILE%\\tools\\jslint.js','%USERPROFILE%\\tools\\jslint.js' ]
p = Popen(args, stdout=PIPE, shell=True).communicate()[0]
Results in output like the following (the trailing double \r\n is there in case it's important)
Microsoft (R) Windows Script Host Version 5.8
Copyright (C) Microsoft Corporation. All rights reserved.\r\n\r\n
If I run that command from an interactive Python shell it looks like this
>>> args = ['cscript', '%USERPROFILE%\\tools\\jslint.js', '%USERPROFILE%\\tools\jslint.js']
>>> p = subprocess.Popen(args, stdout=subprocess.PIPE, shell=True).communicate()[0]
Lint at line 5631 character 17: Unexpected /*member 'OpenTextFile'.
f = fso.OpenTextFile(WScript.Arguments(0), 1),
...
Lint at line 5649 character 17: Unexpected /*member 'Quit'.
WScript.Quit(1);
So there's all the output I really care about, but if I dump the value of the "p" variable I just set up...
>>> p
'Microsoft (R) Windows Script Host Version 5.8\r\nCopyright (C) Microsoft Corpor
ation. All rights reserved.\r\n\r\n'
>>>
Where'd all that data I want end up going? It definitely didn't end up in "p". Looks like it's going to stdout, but I didn't I explictly tell it not to do that?
I'm running this on Windows 7 x64 with Python 2.6.6

Is it going to stderr? Try redirecting:
p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True).communicate()[0]

It's probably going to stderr, as SimonJ suggested.
Also, the docs say not to use shell=True in Windows for your case:
The executable argument specifies the
program to execute. It is very seldom
needed: Usually, the program to
execute is defined by the args
argument. If shell=True, the
executable argument specifies which
shell to use. On Unix, the default
shell is /bin/sh. On Windows, the
default shell is specified by the
COMSPEC environment variable. The only
reason you would need to specify
shell=True on Windows is where the
command you wish to execute is
actually built in to the shell, eg
dir, copy. You don’t need shell=True
to run a batch file, nor to run a
console-based executable.
Later: oh wait. Are you using the shell to get those environment variables expanded? Okay, I take it back: you do need the shell=True.