Develop Reference

When using subprocess.Popen to create tmux sessions is it possible to change the envionment variables for each new tmux session spawned from Popen?

I'm trying to spawn multiple tmux sessions with different environment variables from the same python3 script.
I have been arguing {**os.environ, "CUDA_VISIBLE_DEVICES":str(device_id)} to the env key word argument to subprocess.Popen.
for device_id in device_ids:
new_env = {**os.environ, "CUDA_VISIBLE_DEVICES":str(device_id)}
p = subprocess.Popen([
'tmux', 'new', '-d', "-c", "./", '-s',
sesh_name,
"python3",
path_to_script
], env=new_env)
I'm finding that the CUDA_VISIBLE_DEVICES parameter, however, is equal to the first device_id that I argue across all processes. What is the meaning of this!?
Is this an inherent issue with Popen and the subprocess module? If so, how do I fix it?
I've tried to argue the device id to the script of the new process, but sadly torch won't allow me to update the environment variable after it's been imported and it would be way more trouble than it's worth to rework the code for that.
EDIT: Providing minimal example
Save this script as test.py (or whatever else you fancy):
import subprocess
import os
def sesh(name):
procs = []
for device_id in [4,5,6]:
proc_env = {**os.environ, "CUDA_VISIBLE_DEVICES": str(device_id)}
p = subprocess.Popen(['tmux', 'new', '-d', "-c", "./", '-s', name+str(device_id), "python3", "deleteme.py"], env=proc_env)
procs.append(p)
return procs
if __name__=="__main__":
sesh("foo")
Save this script as deleteme.py within the same directory:
import time
import os
if __name__=="__main__":
print(os.environ)
for i in range(11):
print("running")
if "CUDA_VISIBLE_DEVICES" in os.environ:
print(os.environ["CUDA_VISIBLE_DEVICES"])
else:
print("CUDA_VISIBLE_DEVICES not found")
time.sleep(5)
Then run test.py from the terminal.
$ python3 test.py
Then switch to the tmux sessions to figure out what environment is being created.

For anyone else running into this problem, you can use os.system instead of subprocess.Popen in the following way.
import os
def sesh(name, device_id, script):
command = "tmux new -d -s \"{}{}\" \'export CUDA_VISIBLE_DEVICES={}; python3 {} \'"
command = command.format(
name,
device_id,
device_id,
script
)
os.system(command)
if __name__=="__main__":
sesh("foo", 4, "deleteme.py")

Execute linux command in Python Flask

How can I execute linux command inside a Python function? I will run the python file in a linux-based server, and in some functions I want to have something like,
def function():
#execute some commands on the linux system, eg. python /path1/path2/file.py
# Or execute a shell script, eg. /path1/path2/file.sh
What python module do I need to achieve this?
Thanks in advance.

This code will create a flask server and allow you to run commands. You can also capture the output.
import subprocess
from flask import Flask
app = Flask(__name__)
def run_command(command):
return subprocess.Popen(command, shell=True, stdout=subprocess.PIPE).stdout.read()
#app.route('/<command>')
def command_server(command):
return run_command(command)
You can run it by saving above text in server.py
$ export FLASK_APP=server.py
$ flask run

Try the following:
import os, subprocess
# if you do not need to parse the result
def function ():
os.system('ls')
# collect result
def function(command):
out = subprocess.run(
command.split(" "),
stdout=subprocess.PIPE)
return out.stdout

rundeck unable to execute powershell script with import-module

I am running several python scripts via rundeck (in-line) on windows 2012 target node. These scripts used to run locally but now in the process of moving to rundeck.
One of the python scripts opens subprocess to invoke a powershell script and read the output.
import subprocess
CMD = [r'C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe ', '-File', r'C:\Users\Osman\Code\mop.ps1']
cmd = CMD[:]
cmd.append('arg1')
cmd.append('arg2')
cmd.append('arg3')
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
r = p.communicate()
print(r)
The mop.ps1 is
Import-Module MSOnline
$domain = $args[0]
$login = $args[1]
$pass = $args[2]
$EPass = $pass | ConvertTo-SecureString -AsPlainText -Force
$Cred = New-Object System.Management.Automation.PsCredential($login, $EPass)
Connect-MsolService -Credential $Cred
$TenantId = Get-MsolPartnerContract -Domain $domain | Select-Object -ExpandProperty TenantId
Get-MsolAccountSKU -TenantId $TenantId | select SkuPartNumber,ActiveUnits,ConsumedUnits | ConvertTo-Csv -NoTypeInformation
This part of the code always fails to execute and if I check stderr it says:
Connect-MsolService : Exception of type 'Microsoft.Online.Administration.Automation.MicrosoftOnlineException' was thrown.
At C:\Users\Osman\Code\mop.ps1:7 char:1
+ Connect-MsolService -Credential $Cred
I'm not sure why it fails. I tried
Import-Module MSOnline -Verbose
And I can see Cmdlets being loaded. I tried creating profile.ps1 file in C:\WINDOWS\system32\WindowsPowerShell\v1.0\ location.
Everything works fine if I execute the code locally. I tried running a regular test .ps1 file 'disk.ps1' instead of my code, and that works fine because it doesn't load any modules:
get-WmiObject win32_logicaldisk -Computername $env:computername
What's the workaround to get a script with module to run properly? stdout is always empty.
Node is registered as 64 bit so I tried changing the cmd to
C:\Windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe
I tried to copy profile.ps1 there, copied the Module there but still doesn't work via rundeck.

From your description, since you are able to get a valid output when running the script directly from the server, it may be possible that you error could be related to the “Second Hop” that you use to login into MS-Online. Currently, the Rundeck Python-Winrm plugin supports Basic, ntlm or CredSSP Authentication, and CredSSP authentication allows you to perform the second hop successfully.

How to run an AppleScript from within a Python script?

How to run an AppleScript from within a Python script?
The questions says it all..
(On a Mac obviously)

this nice article suggests the simple solution
cmd = """osascript -e 'tell app "Finder" to sleep'"""
def stupidtrick():
os.system(cmd)
though today you'd use the subprocess module instead of os.system, of course.
Be sure to also check page 2 of the article for many more info and options, including appscript.

A subprocess version which allows running an original apple script as-is, without having to escape quotes and other characters which can be tricky. It is a simplified version of the script found here which also does parametrization and proper escaping (Python 2.x).
import subprocess
script = '''tell application "System Events"
activate
display dialog "Hello Cocoa!" with title "Sample Cocoa Dialog" default button 2
end tell
'''
proc = subprocess.Popen(['osascript', '-'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE)
stdout_output = proc.communicate(script)[0]
print stdout_output
NOTE: If you need to execute more than one script with the same Popen instance then you'll need to write explicitly with proc.stdin.write(script) and read with proc.stdout.read() because communicate() will close the input and output streams.

I got the Output folks... Here it's following:
import subprocess
import sys
for i in range(int(sys.argv[1])):
ip = str(sys.argv[2])
username = str(sys.argv[3])
pwd = str(sys.argv[4])
script = '''tell application "Terminal"
activate
do script with command "cd Desktop && python test_switch.py {ip} {username} {pwd}"
delay 15
end tell
'''
proc = subprocess.Popen(['osascript', '-'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE)
stdout_output = proc.communicate(script.format(ip=ip, username=username, pwd=pwd))[0]

I was pretty frustrated at the lack of detail in Apple's own documentation regarding how to do this AND to also pass in arguments. I had to send the desired arg (in this case a zoom id) as a string otherwise the argument didn't come through to the applescript app
Here's my code running from python:
f = script if os.path.exists(script) else _tempfile()
if not os.path.exists(script):
open(f,'w').write(script)
args = ["osascript", f, str(zoom_id)]
kwargs = {'stdout':open(os.devnull, 'wb'),'stderr':open(os.devnull, 'wb')}
#kwargs.update(params)
proc = subprocess.Popen(args,**kwargs)
and here is my applescript:
on run argv
set zoom_id to 0
zoom_id = item 1 in argv
tell application "zoom.us"
--do stuff
end tell
end run

Python problem executing popen in cron

I use popen to execute commands in a Python script, and I call it via cron.
Cron calls out this script but the behavior isn't the same if I call it by hand.
Source:
from subprocess import Popen, PIPE
pp = Popen('/usr/bin/which iptables', shell=True, stdout=PIPE)
data = ''
for ln in pp.stdout:
data = data+ln
if data == '':
print 'ko'
else:
print 'ok : '+data
By hand:
# python /home/user/test.py
> : /sbin/iptables
By cron (in /tmp/err_cron):
* * * * * /usr/bin/python /home/user/test.py >> /tmp/err_cron
ko
ko
ko
Why does cron not run this script normally?

Normally when processes are run from cron, the PATH is set to a very restrictive value (the man page for my crontab says /usr/bin:/bin). You may need to add:
PATH=/usr/bin:/bin:/sbin
to the top of your crontab file.