Develop Reference

Not able to run command via wsl.exe

I'm trying to run this commands from a python script:
def raw(path_avd_py, path_avd, snp_name, out_file):
if OS == 'Windows':
cmd_raw = f"wsl.exe -e sh -c 'python3 {path_avd_py} -a {path_avd}
-s {snp_name} -o {out_file}'"
else:
cmd_raw = f'python3 {path_avd_py} -a {path_avd} -s {snp_name} -o {out_file}'
subprocess.Popen(cmd_raw, shell=True)
time.sleep(25)
return None
def idiffer(i_path, raw_1, raw_2, path, state):
if OS == 'Windows':
cmd_idiff = f"wsl.exe -e sh -c 'python3 {i_path} {raw_1} {raw_2}'"
[...]
file = os.path.join(path, f'{state}.idiff')
with open(file, 'w') as f:
subprocess.Popen(cmd_idiff, stdout=f, text=True)
If im executing cmd_raw with subprocess.run from a python-shell (Powershell), things are working. If im try running this via script, this exception occurs, using different shells:
-e sh: avdecrypt-master\avdecrypt.py: 1: Syntax error: Unterminated quoted string
-e bash: avdecrypt-master\avdecrypt.py: -c: line 0: unexpected EOF while looking for matching `''
avdecrypt-master\avdecrypt.py: -c: line 1: syntax error: unexpected end of file
I already tried os.system, os.run([list]) no change.
Thanks for the help!

For those who have a similar question, I found a solution, which is working for me:
Apparently calling scripts with some argv has to be in one single quotation mark and can be executed via run (in my case important, because the process has to be terminated). This leads to a form like:
cmd = ['wsl.exe', '-e', 'bash', '-c', '-a foo -b bar [...]']
subprocess.run(cmd, shell=True)
Lib shlex is helping here and formatting the strings like subprocess is needing it:
cmd_finished = shlex.split(cmd)
https://docs.python.org/3/library/shlex.html

Store output of gunzip/tar -t in Python

I try to store output of command : gunzip -t / tar -t in Python but I dont know how. Indeed, in shell termianl I have no problem to ctch result with echo $? but in Python it's impossible with os.popen() or os.system().
My current script is below :
os.system("gunzip -t Path_to_tar.gz")
gzip_corrupt = os.popen("echo $?").read().replace('\n','')
os.system("gunzip -c Path_to_tar.gz | tar -t > /dev/null")
tar_corrup = os.popen("echo $?").read().replace('\n','')
print(tar_corrup)
print(gzip_corrupt)
Do you have an idea how to store output of gunzip -t in python please ?

I'm no python wiz, but I'd say you need to change your output for os.system from:
os.system("gunzip -c Path_to_tar.gz | tar -t > /dev/null")
to something like:
os.system("gunzip -c Path_to_tar.gz | tar -t > /tmp/myfile.out")
Then, turn around, open up /tmp/myfile.out, and read it back in, etc. (I'd suggest generating a unique name to avoid multiple runs that would collide and cause errors - also include a date/time stamp to keep separate runs - separate)
This line:
tar_corrup = os.popen("echo $?").read().replace('\n','')
is only going to give you the exit code of the gunzip command - NOT the output of gunzip itself (see "What does echo $? do?" 1.)
This is a "brute-force" method - but easy to read, and edit later, and should work.

This is a solution of my question, i test it on differents tar file and it's look like to work :
# On check si la backup est corrompue
gzip_corrupt = False
tar_corrup = False
if const.weekday < 6:
incr_backup.incremental_backup()
else:
full_backup.full_backup()
# On vérifie l'intégrité du gzip
if os.system("gunzip -t " + const.target_directory + const.backup_file_name):
gzip_corrupt = True
# On vérifie l'intégrité du tar
if os.system("gunzip -c " + const.target_directory + const.backup_file_name + " | tar -t"):
tar_corrup = True
# Si la backup est corrompue --> on envoie un mail à la BAL PIC
if gzip_corrupt or tar_corrup:
mail_notice.send_email()
So, apparently. os.system() know is tar -t or gunzip -t output something and I test with an if block.
If os.system("tar -t ...") return something , it means tar is corrupted or it's not a tar file so my boolean takes True
When tar is ok, os.system() return nothing --> read as False by Python
I hope it will help other on this specific command in python
Thank you for help all

Using a python script to run a shell script (containing a make file and an executable for fortran)

I am trying to execute a shell script using python. The shell script contains variables (which I want to set in the python script itself), a make file and an executable file for fortran. Is it possible to be done using python?
I tried with the following python code but unfortunately it does not work:
import subprocess import os import shutil
os.environ['FLUID'] ="MDM"
os.environ['TTYPE'] = str(0)
os.environ['FLIBRARY'] = "RefProp"
HSPACE=[0.01, 0.009, 0.008, 0.007]
os.chdir(tmesh_dir)
subprocess.call(['./testfile3', str(FLUID), str(TTYPE), str(FLIBRARY)]
+ [str(n) for n in HSPACE])
testfile3 is a shell script which has the following contents:
fluid=$1; shift
ttype=$2; shift
flibrary=$3; shift
hspace=( "$#" )
make clean
make -j2
./vls.exe <<<"$fluid, $flibrary"
for i; do
cd Db
awk -v a="$i" 'NR==8 {$1=" " a }1' spacingcontrol.vls > tmp.vls && mv tmp.vls spacingcontrol.vls
awk -v b="$i" 'NR==8 {$2=" " b " "}1' spacingcontrol.vls > tmp.vls && mv tmp.vls spacingcontrol.vls
done
cd ..
umg/mcrv.exe
umg/bgrid.exe
umg/umg2d.exe
#make clean

You can use subprocess.Popen, which allows you to pass in a set of environment variables to the process. You can also specify the working directory for the subprocess instead of changing the current one.
env = dict(os.environ)
env['FLUID'] ="MDM"
env['TTYPE'] = str(0)
env['FLIBRARY'] = "RefProp"
HSPACE=[0.01, 0.009, 0.008, 0.007]
cmd = ['./testfile3', env['FLUID'], env['TTYPE'], env[FLIBRARY]]
cmd.extend([str(n) for n in HSPACE])
p = subprocess.Popen(cmd, env=env, cwd=tmesh_dir)

Change output logfile name inside python subprocess to make it unique

I am using python subprocess to email logfile to a user who runs python script. However each time user runs the script logfile gets overwritten. Here is the unix subprocess command I am using inside python code:
subprocess.Popen("mail -s 'logfile.log attached' -r az12#abc.com -a logfile.log $USER#abc.com &> /dev/null",shell=True)
How could I make logfile name unique? Maybe incremant logfile name as logfile1.log, logfile2.log and so on?
Trick is how do I achieve this inside subprocess?

Also you can do this with datetime module:
import datetime
filename = "logfile-%s.log" % datetime.datetime.today().isoformat()
command = "mail -s '{0} attached' -r az12#abc.com -a {0} $USER#abc.com &> /dev/null".format(filename)
subprocess.Popen(command, shell=True)
The name of log file will look like logfile-2015-03-13T21:37:14.927095.log.

Try using timestamp to generate a logfile name. About using that one in subprocess, command is nothing but a string. So it is as simple as
import time
fileName = "logfile." + str(time.time()) + ".log" # use your logic to generate logFile name.
command = "mail -s '%s attached' -r az12#abc.com -a %s $USER#abc.com &> /dev/null" %(fileName, fileName)
subprocess.Popen(command,shell=True)

python sub-process

I usually execute a Fortran file in Linux (manually) as:
Connect to the server
Go to the specific folder
ifort xxx.for -o xxx && ./xxx (where 'xxx.for' is my Fortran file and 'xxx' is Fortran executable file)
But I need to call my fortran file (xxx.for) from python (I'm a beginner), so I used subprocess with the following command:
cmd = ["ssh", sshConnect, "cd %s;"%(workDir), Fortrancmd %s jobname "%s -o %s" exeFilename "%s && %s ./ %s%s"%(exeFilename)]
But I get an error, and I'm not sure what's wrong. Here's the full code:
import string
import subprocess as subProc
from subprocess import Popen as ProcOpen
from subprocess import PIPE
import numpy
import subprocess
userID = "pear"
serverName = "say4"
workDir = "/home/pear/2/W/fortran/"
Fortrancmd = "ifort"
jobname = "rad.for"
exeFilename = "rad"
sshConnect=userID+"#"+servername
cmd=["ssh", sshConnect, "cd %s;"%(workDir), Fortrancmd %s jobname "%s -o %s" exeFilename "%s && %s ./ %s%s"%(exeFilename)]
**#command to execute fortran files in Linux
**#ifort <filename>.for -o <filename> && ./<filename> (press enter)
**#example:ifort xxx.for -o xxx && ./xxx (press enter)
print cmd
How can I write a python program that performs all 3 steps described above and avoids the error I'm getting?

there are some syntax errors...
original:
cmd=["ssh", sshConnect, "cd %s;"%(workDir), Fortrancmd %s jobname "%s -o %s" exeFilename "%s && %s ./ %s%s"%(exeFilename)]
I think you mean:
cmd = [
"ssh",
sshConnect,
"cd %s;" % (workDir,),
"%s %s -o %s && ./%s" % (Fortrancmd, jobname, exeFilename, exeFilename)
]
A few notes:
a tuple with one element requires a comma at the end of the first argument see (workDir,) to be interpreted as a tuple (vs. simple order-of-operations parens)
it is probably easier to contruct your fortan command with a single string format operation
PS - For readability it is often a good idea to break long lists into multiple lines :)
my advice
I would recommend looking at this stackoverflow thread for ssh instead of using subprocess

For the manual part you may want to look into pexpect or for windows wexpect. These allow you to perform subprocesses and pass input under interactive conditions.
However most of what you're doing sounds like it would work well in a shell script. For simplicity, you could make a shell script on the server side for your server side operations, and then plug in the path in the ssh statement:
ssh user#host "/path/to/script.sh"

one error:
you have an unquoted %s in your list of args, so your string formatting will fail.

Here is a complete example of using the subprocess module to run a remote command via ssh (a simple echo in this case) and grab the results, hope it helps:
>>> import subprocess
>>> proc = subprocess.Popen(("ssh", "remoteuser#host", "echo", "1"), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> stdout, stderr = proc.communicate()
Which in this case returns: ('1\n', '')
Note that to get this to work without requiring a password you will likely have to add your local user's public key to ~remoteuser/.ssh/authorized_keys on the remote machine.

You could use fabric for steps 1 and 2.
This is the basic idea:
from fabric.api import *
env.hosts = ['host']
dir = '/home/...'
def compile(file):
with cd(dir):
run("ifort %s.for -o %s" %(file,file))
run("./%s > stdout.txt" % file)
Create fabfile.py
And you run fab compile:filename

do you have to use python?
ssh user#host "command"