I'm programming on Windows 7 and in one of my Python projects I need to call bedtools, which only works with Cygwin on Windows. I'm new to Cygwin, installed the default version + everything needed for bedtools and then used Cygwin to install bedtools by using make as described in the installation instructions.
$ tar -zxvf BEDTools.tar.gz
$ cd BEDTools-<version>
$ make
When I use the Cygwin terminal to call it manually like below, it works without problem and the output file contains the correct result.
bedtools_exe_path intersect -a gene_bed_file -b snp_bed_file -wa -wb > output_file
But when I use subprocess.call in my program it seems to use Windows cmd instead of Cygwin, which doesn't work.
arguments = [bedtools_exe_path, 'intersect', '-a', gene_bed_file, '-b',
snp_bed_file, '-wa', '-wb', '>', output_file]
return_code = suprocess.call(arguments)
Results in no output file and a return code of 3221225781.
arguments = [bedtools_exe_path, 'intersect', '-a', gene_bed_file, '-b',
snp_bed_file, '-wa', '-wb', '>', output_file]
return_code = suprocess.call(arguments, shell=True)
Results in an empty output file and a return code of 3221225781.
cygwin_bash_path = 'D:/Cygwin/bin/bash.exe'
arguments = [cygwin_bash_path, bedtools_exe_path, 'intersect', '-a', gene_bed_file, '-b',
snp_bed_file, '-wa', '-wb', '>', output_file]
return_code = suprocess.call(arguments)
Results in no output file, a return code of 126 and
D:/BEDTools/bin/bedtools.exe: D:/BEDTools/bin/bedtools.exe: cannot execute binary file
arguments = [cygwin_bash_path, bedtools_exe_path, 'intersect', '-a', gene_bed_file, '-b',
snp_bed_file, '-wa', '-wb', '>', output_file]
return_code = suprocess.call(arguments, shell=True)
Results in an empty output file, a return code of 126 and
D:/BEDTools/bin/bedtools.exe: D:/BEDTools/bin/bedtools.exe: cannot execute binary file
Any ideas how I can get it to work?
Imagine you want to run a Linux command from Windows. You could install Linux into a VM and run commands via ssh (Putty/plink on Windows):
#!/usr/bin/env python
import subprocess
cmd = [r'C:\path\to\plink.exe', '-ssh', 'user#vm_host', '/path/to/bedtools']
with open('output', 'wb', 0) as file:
subprocess.check_call(cmd, stdout=file)
Cygwin provides run command that allows to run commands directly:
cmd = [r'C:\cygwin\path\to\run.exe', '-p', '/path/to/', 'bedtools',
'-wait', 'arg1', 'arg2']
Note: Python script is run from Windows in both cases. bedtools is Linux or Cygwin (non-Windows) command here and therefore you should provide POSIX paths.
The following works without a problem. The " does not need to be escaped.
argument = 'sh -c \"' + bedtools_exe_path + ' intersect -a ' + gene_bed_file +
' -b ' + snp_bed_file + ' -wa -wb\"'
with open(output_file, 'w') as file:
subprocess.call(argument, stdout=file)
Using the following works as well:
argument = 'bash -c \"' + bedtools_exe_path + ' intersect -a ' + gene_bed_file +
' -b ' + snp_bed_file + ' -wa -wb\"'
with open(output_file, 'w') as file:
subprocess.call(argument, stdout=file)
With:
bedtools_exe_path = 'D:/BEDTools/bin/bedtools.exe'
gene_bed_file = 'output/gene.csv'
snp_bed_file = 'output/snps.csv'
output_file = 'output/intersect_gene_snp.bed'
Using the path to the cygwin bash.exe (D:/Cygwin/bin/bash.exe) instead of bash or sh does not work.
Thank you, eryksun, Padraic Cunningham and J.F. Sebastian.
Related
Python 3.10.6
Windows 10
I have a python function that executes a DXL script using subsystem.run() or os.system() (whichever works best I guess). The problem is that when I run a custom command using python it does not work, but when I paste the same command in the command prompt, it works. I should also clarify that command prompt is not the ms store windows terminal (cannot run ibm doors commands there for some reason). It is the OG prompt
I need to use both python and IBM Doors for the solution.
Here is a summer version of my code (Obviously, the access values are not real):
#staticmethod
def run_dxl_importRTF():
dquotes = chr(0x22) # ASCII --> "
module_name = "TEST_TEMP"
script_path = "importRTF.dxl"
script_do_nothing_path = "doNothing.dxl"
user = "user"
password = "pass"
database_config = "11111#11.11.1111.0"
doors_path = dquotes + r"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe" + dquotes
file_name = "LIBC_String.rtf"
# Based On:
# "C:\Program Files\IBM\Rational\DOORS\9.7\\bin\doors.exe" -dxl "string pModuleName = \"%~1\";string pFilename = \"%~2\";#include <importRTF.dxl>" -f "%TEMP%" -b "doNothing.dxl" -d 11111#11.11.1111.0 -user USER -password PASSWORD
script_arguments = f"{dquotes}string pModuleName=\{dquotes}{module_name}\{dquotes};string pFileName=\{dquotes}{file_name}\{dquotes};#include <{script_path}>{dquotes}"
command = [doors_path, "-dxl", script_arguments, "-f", "%TEMP%", "-b", script_do_nothing_path, '-d', database_config, '-user', user, '-password', password]
res = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(f"COMMAND:\n{' '.join(res.args)}")
print(f"STDERR: {repr(res.stderr)}")
print(f'STDOUT: {res.stdout}')
print(f'RETURN CODE: {res.returncode}')
return
PYTHON SCRIPT OUTPUT:
COMMAND:
"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe" -dxl "string pModuleName=\"TEST_TEMP\";string pFileName=\"LIBC_String.rtf\";#include <importRTF.dxl>" -f %TEMP% -b doNothing.dxl -d 11111#11.11.1111.0 -user USER_TEMP -password PASS_TEMP
STDERR: 'The system cannot find the path specified.\n'
STDOUT:
RETURN CODE: 1
When I run the same command in the command prompt, it works (dxl script is compiled).
I identified the problem which is the script_argument variable. Meaning that, when I try to just enter the IBM Doors server without compiling a DXL script, it works on python and the command prompt.
The python script needs to be dynamic meaning that all of the initial declared variables can change value and have a path string in it. I am also trying to avoid .bat files. They also did not work with dynamic path values
Thanks for your time
I tried:
Changing CurrentDirectory (cwd) to IBM Doors
os.system()
Multiple workarounds
Tried IBM Doors path without double quotes (it doesnt work because of the whitespaces)
.bat files
When calling subprocess.run with a command list and shell=True, python will expand the command list to a string, adding more quoting along the way. The details are OS dependent (on Windows, you always have to expand the list to a command) but you can see the result via the subprocess.list2cmdline() function.
Your problem is these extra escapes. Instead of using a list, build a shell command string that already contains the escaping you want. You can also use ' for quoting strings so that internal " needed for shell quoting can be entered literally.
Putting it all together (and likely messing something up here), you would get
#staticmethod
def run_dxl_importRTF():
module_name = "TEST_TEMP"
script_path = "importRTF.dxl"
script_do_nothing_path = "doNothing.dxl"
user = "user"
password = "pass"
database_config = "11111#11.11.1111.0"
doors_path = r"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe"
file_name = "LIBC_String.rtf"
script_arguments = (rf'string pModuleName=\"{module_name}\";'
'string pFileName=\"{file_name}\";'
'#include <{script_path}>')
command = (f'"{doors_path}" -dxl "{script_arguments}" -f "%TEMP%"'
' -b "{script_do_nothing_path}" -d {database_config}'
' -user {user} -password {pass}')
res = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(f"COMMAND:\n{' '.join(res.args)}")
print(f"STDERR: {repr(res.stderr)}")
print(f'STDOUT: {res.stdout}')
print(f'RETURN CODE: {res.returncode}')
I'm trying to run this commands from a python script:
def raw(path_avd_py, path_avd, snp_name, out_file):
if OS == 'Windows':
cmd_raw = f"wsl.exe -e sh -c 'python3 {path_avd_py} -a {path_avd}
-s {snp_name} -o {out_file}'"
else:
cmd_raw = f'python3 {path_avd_py} -a {path_avd} -s {snp_name} -o {out_file}'
subprocess.Popen(cmd_raw, shell=True)
time.sleep(25)
return None
def idiffer(i_path, raw_1, raw_2, path, state):
if OS == 'Windows':
cmd_idiff = f"wsl.exe -e sh -c 'python3 {i_path} {raw_1} {raw_2}'"
[...]
file = os.path.join(path, f'{state}.idiff')
with open(file, 'w') as f:
subprocess.Popen(cmd_idiff, stdout=f, text=True)
If im executing cmd_raw with subprocess.run from a python-shell (Powershell), things are working. If im try running this via script, this exception occurs, using different shells:
-e sh: avdecrypt-master\avdecrypt.py: 1: Syntax error: Unterminated quoted string
-e bash: avdecrypt-master\avdecrypt.py: -c: line 0: unexpected EOF while looking for matching `''
avdecrypt-master\avdecrypt.py: -c: line 1: syntax error: unexpected end of file
I already tried os.system, os.run([list]) no change.
Thanks for the help!
For those who have a similar question, I found a solution, which is working for me:
Apparently calling scripts with some argv has to be in one single quotation mark and can be executed via run (in my case important, because the process has to be terminated). This leads to a form like:
cmd = ['wsl.exe', '-e', 'bash', '-c', '-a foo -b bar [...]']
subprocess.run(cmd, shell=True)
Lib shlex is helping here and formatting the strings like subprocess is needing it:
cmd_finished = shlex.split(cmd)
https://docs.python.org/3/library/shlex.html
I'm trying to call a program from within Python that creates output and I want to work with this output when the external program has finished.
The programstring is
"sudo fing -r 1 > fingoutput.txt".
I managed to call the program with
from subprocess import call
cmd = ['sudo', 'fing', '-r 1']
call(cmd)
but I can't direct it's output to a file.
cmd = ['sudo', 'fing', '-r 1 > fingoutput.txt']
or
cmd = ['sudo', 'fing', '-r 1', '> fingoutput.txt']
produce
Error: multiple occurrences
I want to write the output to a file because it might be thousands of lines.
Thank you for your help,
Herbert.
You can use the stdout argument to redirect the output of your command to a file:
from subprocess import call
cmd = ['sudo', 'fing', '-r 1']
file_ = open('your_file.txt', 'w')
call(cmd, stdout=file_)
If you want to redirect to file from the shell-script itself you can always go for this
cmd = 'sudo fing -r 1 > fingoutput.txt'
call(cmd, shell=True)
Or
cmd = 'sudo fing -r 1 > fingoutput.txt'
p = Popen(cmd, shell=True, stdout=PIPE, stdin=PIPE)
Keeping shell=True may lead to security issues.
I'm using Python 2.7.3.
I have a function that runs tesseract as a command line. Everything is working fine and now I would like to add a new parameter to the command -l rus (signifying russian language). Eventhough this works on my commandline, it doesn't seem to work from Python.
Command line:
$ /usr/local/bin/tesseract /Users/anthony/Downloads/rus.png outfile -l rus && more outfile.txt
Tesseract Open Source OCR Engine v3.02.02 with Leptonica
Полу-Милорд, полу-купец,
Полу-мудрец, полу-невежда,
Полу-подлец, но есть надежда,
Что будет полным наконец.
Python function
def ocr(self,path):
path = "/Users/anthony/Downloads/rus.png"
process = subprocess.Popen(['/usr/local/bin/tesseract', path,'outfile','-l rus'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
out, err = process.communicate()
print err
print out
with open('outfile.txt', 'r') as handle:
contents = handle.read()
os.remove(temp.name + '.txt')
os.remove(temp.name)
return contents, out
the above returns "HOIIY nony HOIIY nony Hony no ecTb HHJICXQRI 6y11e" which suggests that the -l rus flag is being ignored.
Question
How can I execute the following command as a python subprocess?
/usr/local/bin/tesseract /Users/anthony/Downloads/rus.png outfile -l rus
You need to split the '-l rus' argument to two separate ones to make sure it's parsed correctly by the program:
process = subprocess.Popen(
['/usr/local/bin/tesseract', path, 'outfile', '-l', 'rus'],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)
It might be handy to use str.split() or shlex.split() for this:
cmd = '/usr/local/bin/tesseract /Users/anthony/Downloads/rus.png outfile -l rus'
process = subprocess.Popen(
cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)
process = subprocess.Popen('/usr/local/bin/tesseract '+path+' outfile -l rus', stdout=subprocess.PIPE, stderr=subprocess.STDOUT,shell=True)
You can run it with shell=True.
I'm using the subprocess.Popen call, and in another question I found out that I had been misunderstanding how Python was generating arguments for the command line.
My Question
Is there a way to find out what the actual command line was?
Example Code :-
proc = subprocess.popen(....)
print "the commandline is %s" % proc.getCommandLine()
How would you write getCommandLine ?
It depends on the version of Python you are using. In Python3.3, the arg is saved in proc.args:
proc = subprocess.Popen(....)
print("the commandline is {}".format(proc.args))
In Python2.7, the args not saved, it is just passed on to other functions like _execute_child. So, in that case, the best way to get the command line is to save it when you have it:
proc = subprocess.Popen(shlex.split(cmd))
print "the commandline is %s" % cmd
Note that if you have the list of arguments (such as the type of thing returned by shlex.split(cmd), then you can recover the command-line string, cmd using the undocumented function subprocess.list2cmdline:
In [14]: import subprocess
In [15]: import shlex
In [16]: cmd = 'foo -a -b --bar baz'
In [17]: shlex.split(cmd)
Out[17]: ['foo', '-a', '-b', '--bar', 'baz']
In [18]: subprocess.list2cmdline(['foo', '-a', '-b', '--bar', 'baz'])
Out[19]: 'foo -a -b --bar baz'
The correct answer to my question is actually that there IS no command line. The point of subprocess is that it does everything through IPC. The list2cmdline does as close as can be expected, but in reality the best thing to do is look at the "args" list, and just know that that will be argv in the called program.
Beautiful and scalable method
I have been using something like this:
#!/usr/bin/env python3
import os
import shlex
import subprocess
import sys
def run_cmd(cmd, cwd=None, extra_env=None, extra_paths=None, dry_run=False):
if extra_env is None:
extra_env = {}
newline_separator = ' \\\n'
out = []
kwargs = {}
env = os.environ.copy()
# cwd
if 'cwd' is not None:
kwargs['cwd'] = cwd
# extra_env
env.update(extra_env)
for key in extra_env:
out.append('{}={}'.format(shlex.quote(key), shlex.quote(extra_env[key])) + newline_separator)
# extra_paths
if extra_paths is not None:
path = ':'.join(extra_paths)
if 'PATH' in env:
path += ':' + env['PATH']
env['PATH'] = path
out.append('PATH="{}:${{PATH}}"'.format(':'.join(extra_paths)) + newline_separator)
# Command itself.
for arg in cmd:
out.append(shlex.quote(arg) + newline_separator)
# Print and run.
kwargs['env'] = env
print('+ ' + ' '.join(out) + ';')
if not dry_run:
subprocess.check_call(cmd, **kwargs)
run_cmd(
sys.argv[1:],
cwd='/bin',
extra_env={'ASDF': 'QW ER'},
extra_paths=['/some/path1', '/some/path2']
)
Sample run:
./a.py echo 'a b' 'c d'
Output:
+ ASDF='QW ER' \
PATH="/some/path1:/some/path2:${PATH}" \
echo \
'a b' \
'c d' \
;
a b c d
Feature summary:
makes huge command lines readable with one option per line
add a + to commands like sh -x so users can differentiate commands from their output easily
show cd, and extra environment variables if they are given to the command. These only printed if given, generating a minimal shell command.
All of this allows users to easily copy the commands manually to run them if something fails, or to see what is going on.
Tested on Python 3.5.2, Ubuntu 16.04. GitHub upstream.
You can see it by passing the process id to ps command, if you are on POSIX OS:
import subprocess
proc = subprocess.Popen(["ls", "-la"])
subprocess.Popen(["ps", "-p", str(proc.pid)])
Output (see the CMD column):
PID TTY TIME CMD
7778 ttys004 0:00.01 ls -la
On windows, I used #catwith 's trick (thanks, btw):
wmic process where "name like '%mycmd%'" get processid,commandline
where "mycmd" is a part of the cmd unique to your command (used to filter irrelevant system commands)
That's how I revealed another bug in the suprocess vs windows saga. One of the arguments I had had its double-quotes escaped a-la unix! \"asdasd\"