Develop Reference

Executing Linux command in background from a python script

I have an ubuntu machine where kubernetes cluster is running. I am basically trying to run a Kubernetes command in background from python3 script but it's not working, please help...
Below is a part of code from a larger script, I am creating cmd4 using a formatted string and then passing cmd4 to os.system as os.system(cmd4). But as soon as I execute the script it starts showing logs in cmdline. I tried this using nohup as well as mentioned below but it starts populating logs in nohup.out.
for podname in pod_names:
if "smf" in podname or "pcf" in podname or "udm" in podname:
containername = 'worker'
else:
containername = 'cppe'
**cmd4 = f"kubectl logs -f -n core {podname} --container={containername} > {podname}_{containername}_log </dev/null &>/dev/null &&"**
cmd5 = f"echo $! >> pid.txt"
os.system(cmd4)
os.system(cmd5)
pid_file = open('pid.txt', 'a+')
pid_file.write("\n")
pid_file.close
=================================
tried with nohup as:
cmd4 = f"nohup kubectl logs -f -n core {podname} --container={containername} > {podname}_{containername}_log </dev/null &>/dev/null &&"
But it gives this o/p
updatedReplicas: 1
deployment.apps/core-pcf configured
nohup: ignoring input and appending output to 'nohup.out'

Have you tried using the subprocess library?
I think the solution for your script will be the following.
import subprocess
for podname in pod_names:
if "smf" in podname or "pcf" in podname or "udm" in podname:
containername = 'worker'
else:
containername = 'cppe'
**cmd4 = f"kubectl logs -f -n core {podname} --container={containername} > {podname}_{containername}_log </dev/null &>/dev/null &&"**
cmd5 = f"echo $! >> pid.txt"
cmd4_bash = subprocess.Popen(cmd4.split(), stdout=subprocess.PIPE)
cmd5_bash = subprocess.Popen(cmd5.split(), stdout=subprocess.PIPE)
pid_file = open('pid.txt', 'a+')
pid_file.write("\n")
pid_file.close

Syntax error when trying to create and print out a series of command-line commands

I'm trying to write a Python 3 script that will print out a series of command-line commands using variables.
Here's an example of the commands I'm trying to replicate:
filemorph cp testBitmap_1 gs://mapper-bitmap/TestBitmaps
filemorph cp gs://mapper-bitmap/TestBitmaps/testBitmap_1.svs /mnt/pixels/bitmaps
mkdir -p /mnt/pixels/1024/testBitmap_1
image_rotate --image_rotate-progress bitsave "/mnt/pixels/bitmaps/testBitmap_1.svs" /mnt/pixels/1024/testBitmap_1/ --pixel-size 1024
filemorph -m rsync -d -r /mnt/pixels/1024/testBitmap_1 gs://mapper-pixels/1024/testBitmap_1
Whenever I run my script, I get this error:
G:\Projects\Python\BitmapBot
λ python bitmapbot.py
File "bitmapbot.py", line 26
commands = """\
^
SyntaxError: invalid syntax
I checked all my intentations and they all seem correct so I'm not sure why it's giving me an error.
I'm not quite sure what I'm doing wrong.
If anyone sees anything, please let me know.
Thanks!
Oh here's the script:
import os
# define variables
data = dict(
Bitmap_Name = 'testBitmap_1.svs',
Bitmap_Title = 'testBitmap_1',
Bitmap_Folder_Name = 'TestBitmaps',
Cloud_Bitmap_Directory = 'gs://mapper-bitmap/',
Pixel_Bitmap_Engine = '/mnt/pixels/bitmaps',
Local_Bitmap_Directory = '',
Local_Pixel_Directory = '/mnt/pixels/1024/',
Cloud_Pixel_Directory = 'gs://mapper-pixels/1024/'
# create commands with Python:
commands = """\
filemorph cp {Bitmap_Name} {Cloud_Bitmap_Directory}/{Bitmap_Folder_Name}
filemorph cp {Cloud_Bitmap_Directory}/{Bitmap_Folder_Name}/{Bitmap_Name} {Pixel_Bitmap_Engine}
mkdir -p {Local_Pixel_Directory}/{Bitmap_Title}
image_rotate --image_rotate-progress bitsave {Pixel_Bitmap_Engine}/{Bitmap_Name} {Local_Pixel_Directory}/{Bitmap_Title}/ --pixel-size 1024
filemorph -m rsync -d -r {Local_Pixel_Directory}/{Bitmap_Title} {Cloud_Pixel_Directory}/{Bitmap_Title}
"""
# loop through commands and print
for command in commands.splitlines():
command = command.format(**data) # populate command
# os.system(command) # execute command
print(command)

You never closed the bracket after data = dict(…

pexpect not executing command by steps

I have this Python3 code which use Pexpect.
import pexpect
import getpass
import sys
def ssh(username,password,host,port,command,writeline):
child = pexpect.spawn("ssh -p {} {}#{} '{}'".format(port,username,host,command))
child.expect("password: ")
child.sendline(password)
if(writeline):
print(child.read())
def scp(username,password,host,port,file,dest):
child = pexpect.spawn("scp -P {} {} {}#{}:{}".format(port,file,username,host,dest))
child.expect("password: ")
child.sendline(password)
try:
filename = sys.argv[1]
print("=== sendhw remote commander ===")
username = input("Username: ")
password = getpass.getpass("Password: ")
ssh(username,password,"some.host.net","22","mkdir ~/srakrnSRV",False)
scp(username,password,"some.host.net","22",filename,"~/srakrnSRV")
ssh(username,password,"some.host.net","22","cd srakrnSRV && sendhw {}".format(filename),True)
except IndexError:
print("No homework name specified.")
My aim is to:
SSH into the host with the ssh function, create the directory srakrnSRV, then
upload a file into the srakrnSRV directory, which is previously created
cd into srakrnSRV, and execute the sendhw <filename> command. The filename variable is defined by command line parameteres, and print the result out.
After running the entire code, Python prints out
b'\r\nbash: line 0: cd: srakrnSRV: No such file or directory\r\n'
which is not expected, as the directory should be previously created.
Also, I tried manually creating the srakrnSRV folder in my remote host. After running the command again, it appears that scp function is also not running. The only runnning pexpect coomand was the last ssh function.
How to make it execute in order? Thanks in advance!

You may lack permission for executing commands through ssh. Also there is possibility that your program sends scp before prompt occurs.

freebsd pw adduser command and python file descriptors

I'm trying to write a system management script in Python 2.7 on FreeBSD and I'm stuck trying to programmatically set the user's password when adding them. I'm using the FreeBSD pw command which has a -h flag which accepts a file descriptor as an argument.
The route I was taking is using Python's subprocess module, but I
seem to be getting stuck in that Python treats everything as strings
and the pw -h option is expecting a fd (file descriptor) back.
The command I'm trying to run is:
/usr/sbin/pw useradd foobar2 -C /usr/local/etc/bsdmanage/etc/pw.conf -m -c "BSDmanage foobar2 user" -G foobar2-www -h
I'm doing this via:
objTempPassFile = open(strTempDir + 'foobar.txt', 'w+')
objTempPassFile.write(strTempPass)
objTempPassFile.seek(0)
listCmdArgs = shlex.split(strPwUserCmd)
processUser = subprocess.Popen(listCmdArgs,stdin=objTempPassFile.fileno(),stdout=subprocess.PIPE,stderr=subprocess.PIPE)
strOutPut, strErrorValue = processUser.communicate()
where strPwUserCmd is the above pw command and strTempPass is just a string.
I also tried passing the password string as an option to Popen.communicate() and changing stdin to stdin=subprocess.PIPE
I also tried using a StringIO object. However, passing that either gets errors about it not being a valid I/O object or the pw commands fails and doesn't see any arguments passed to the -h switch.
FreeBSD pw manpage
Any ideas? Thanks.

So, if you use the -h 0 flag to pw it prompts for stdin pipe and then you just use process.communicate(string) to pass the password in.
So, like this:
/usr/sbin/pw useradd foobar2 -C /usr/local/etc/bsdmanage/etc/pw.conf -m -c "BSDmanage foobar2 user" -G foobar2-www -h 0
as the command string. Then call that via:
listCmdArgs = shlex.split(strPwUserCmd)
processUser = subprocess.Popen(listCmdArgs,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
strOutPut, strErrorValue = processUser.communicate(strTempPass)
and have strTempPass be the password string. strPwUserCmd is the above 'pw' command string.

python sub-process

I usually execute a Fortran file in Linux (manually) as:
Connect to the server
Go to the specific folder
ifort xxx.for -o xxx && ./xxx (where 'xxx.for' is my Fortran file and 'xxx' is Fortran executable file)
But I need to call my fortran file (xxx.for) from python (I'm a beginner), so I used subprocess with the following command:
cmd = ["ssh", sshConnect, "cd %s;"%(workDir), Fortrancmd %s jobname "%s -o %s" exeFilename "%s && %s ./ %s%s"%(exeFilename)]
But I get an error, and I'm not sure what's wrong. Here's the full code:
import string
import subprocess as subProc
from subprocess import Popen as ProcOpen
from subprocess import PIPE
import numpy
import subprocess
userID = "pear"
serverName = "say4"
workDir = "/home/pear/2/W/fortran/"
Fortrancmd = "ifort"
jobname = "rad.for"
exeFilename = "rad"
sshConnect=userID+"#"+servername
cmd=["ssh", sshConnect, "cd %s;"%(workDir), Fortrancmd %s jobname "%s -o %s" exeFilename "%s && %s ./ %s%s"%(exeFilename)]
**#command to execute fortran files in Linux
**#ifort <filename>.for -o <filename> && ./<filename> (press enter)
**#example:ifort xxx.for -o xxx && ./xxx (press enter)
print cmd
How can I write a python program that performs all 3 steps described above and avoids the error I'm getting?

there are some syntax errors...
original:
cmd=["ssh", sshConnect, "cd %s;"%(workDir), Fortrancmd %s jobname "%s -o %s" exeFilename "%s && %s ./ %s%s"%(exeFilename)]
I think you mean:
cmd = [
"ssh",
sshConnect,
"cd %s;" % (workDir,),
"%s %s -o %s && ./%s" % (Fortrancmd, jobname, exeFilename, exeFilename)
]
A few notes:
a tuple with one element requires a comma at the end of the first argument see (workDir,) to be interpreted as a tuple (vs. simple order-of-operations parens)
it is probably easier to contruct your fortan command with a single string format operation
PS - For readability it is often a good idea to break long lists into multiple lines :)
my advice
I would recommend looking at this stackoverflow thread for ssh instead of using subprocess

For the manual part you may want to look into pexpect or for windows wexpect. These allow you to perform subprocesses and pass input under interactive conditions.
However most of what you're doing sounds like it would work well in a shell script. For simplicity, you could make a shell script on the server side for your server side operations, and then plug in the path in the ssh statement:
ssh user#host "/path/to/script.sh"

one error:
you have an unquoted %s in your list of args, so your string formatting will fail.

Here is a complete example of using the subprocess module to run a remote command via ssh (a simple echo in this case) and grab the results, hope it helps:
>>> import subprocess
>>> proc = subprocess.Popen(("ssh", "remoteuser#host", "echo", "1"), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> stdout, stderr = proc.communicate()
Which in this case returns: ('1\n', '')
Note that to get this to work without requiring a password you will likely have to add your local user's public key to ~remoteuser/.ssh/authorized_keys on the remote machine.

You could use fabric for steps 1 and 2.
This is the basic idea:
from fabric.api import *
env.hosts = ['host']
dir = '/home/...'
def compile(file):
with cd(dir):
run("ifort %s.for -o %s" %(file,file))
run("./%s > stdout.txt" % file)
Create fabfile.py
And you run fab compile:filename

do you have to use python?
ssh user#host "command"