I am trying to run through subprocess a command line command that receives as arguments files. However, these files might have characters like "&" and those can be interpreted as CMD commands if they are not between quotes (").
It usually worked and I had the command passed broken in a list.
Example:
from subprocess import run
file = r'broken&difficult.txt'
command = ['convert', file]
run(command)
However it will return an stdErr:
StdErr: 'diffcult.txt' is not recognized as an internal or external command, operable program or batch file
The returncode is 1.
I have tried to change the file name variable to:
file =r'"broken&difficult.txt"'
The result is that it is not able to find any file. With a returncode of 0
You need to use the CMD escape character - the carrot ^ - before the ampersand.
Try:
import subprocess
file = 'broken^&difficult.txt'
command = ['convert', file]
subprocess.run(command, shell=True)
Example of how this works:
import subprocess
# create a file
with open('broken&difficult.txt', 'w') as fp:
fp.write('hello\nworld')
# use `more` to have OS read contents
subprocess.run(['more', 'broken^&difficult.txt'], shell=True)
# prints:
hello
world
# returns:
CompletedProcess(args=['more', 'broken^&difficult.txt'], returncode=0)
Related
I am under Python 3.8.10 in Ubuntu 20.04 trying to execute a multiline bash command and get its output. For this I am trying to combine this and this. My bash command is this:
/home/foo/.drsosc/drs-5.0.6/drscl << ASD
info
exit
ASD
and it works as I want. Now in Python I have this:
from pathlib import Path
import subprocess
PATH_TO_drscl = Path.home()/Path('.drsosc/drs-5.0.6/drscl')
def send_command(cmd: str):
execute_this = f'''{PATH_TO_drscl} << ASD
{cmd}
exit
ASD'''
return subprocess.run([execute_this], stdout=subprocess.PIPE)
print(send_command('info'))
but I get
FileNotFoundError: [Errno 2] No such file or directory: '/home/foo/.drsosc/drs-5.0.6/drscl << ASD\ninfo\nexit\nASD'
It seems that the problem is with the '\n' not being properly interpreted?
I found that this works as I want:
result = subprocess.run(
str(PATH_TO_drscl),
input = f'{cmd}\nexit',
text = True,
stdout = subprocess.PIPE
)
No, the problem is that you're trying to run small shell script but
you're calling an executable that has a name composed of all commands
in the script. Try with shell=True:
return subprocess.run([execute_this], stdout=subprocess.PIPE, shell=True)
I have a batch file which is running a python script and in the python script, I have a subprocess function which is being ran.
I have tried subprocess.check_output, subprocess.run, subprocess.Popen, all of them returns me an empty string only when running it using a batch file.
If I run it manually or using an IDE, I get the response correctly. Below is the code for subprocess.run:
response = subprocess.run(fileCommand, shell=True, cwd=pSetTableauExeDirectory, capture_output=True)
self.writeInLog(' Command Response: \t' + str(response))
Response is in stdout=b''
When ran in batch file and from task scheduler:
Command Response: CompletedProcess(args='tableau refreshextract
--config-file "Z:\XXX\tableau_config\SampleSuperStore.txt"',
returncode=0, stdout=b'', stderr=b'')
When ran manually or in IDE:
Command Response: CompletedProcess(args='tableau refreshextract
--config-file "Z:\XXX\tableau_config\SampleSuperStore.txt"',
returncode=0, stdout=b'Data source refresh completed.\r\n0 rows uploaded.\r\n', stderr=b'')
Batch file which runs the python program. Parameters are parsed to the python application
SET config=SampleSuperStore.txt
CALL C:\XXX\AppData\Local\Continuum\anaconda3\Scripts\activate.bat
C:\XXX\AppData\Local\Continuum\anaconda3\python.exe Z:\XXX\pMainManual.py "%config%"
Why is that??
--Complete python code---
try:
from pWrapper import wrapper
import sys
except Exception as e:
print(str(e))
class main:
def __init__(self):
self.tableauPath = 'C:\\Program Files\\Tableau\\Tableau 2018.3\\bin\\'
self.tableauCommand = 'tableau refreshextract --config-file'
def runJob(self,argv):
self.manual_sProcess(argv[1])
def manual_sProcess(self,tableauConfigFile):
new_wrapper = wrapper()
new_wrapper.tableauSetup(self.tableauPath,self.tableauCommand)
if new_wrapper.tableauConfigExists(tableauConfigFile):
new_wrapper.tableauCommand(tableauConfigFile)
if __name__ == "__main__":
new_main = main()
new_main.runJob(sys.argv)
Wrapper class:
def tableauCommand(self,tableauConfigFile):
command = self.setTableauExeDirectory + ' ' + self.refreshConfigCommand + ' "' + tableauConfigFile + '"'
self.new_automateTableauExtract.runCommand(tableauConfigFile,command,self.refreshConfigCommand,self.tableauFilePath,self.setTableauExeDirectory)
Automate Class:
def runCommand(self,pConfig,pCommand,pRefreshConfigCommand,pFilePath,pSetTableauExeDirectory):
try:
fileCommand = pRefreshConfigCommand + ' "' + pFilePath + '"'
response = subprocess.run(fileCommand, shell=True, cwd=pSetTableauExeDirectory, capture_output=True)
self.writeInLog(' Command Response: \t' + str(response))
except Exception as e:
self.writeInLog('Exception in function runCommand: ' + str(e))
UPDATE: I initially thought that the bat file was causing this issue but it looks like it works when running manually a batch file but not when it is set on task scheduler
Updated
First of all, if there is a need to run anaconda-prompt by calling activate.bat file, you can simply do as follows:
import subprocess
def call_anaconda_venv():
subprocess.call('python -m venv virtual.env')
subprocess.call('cmd.exe /k /path/venv/Scripts/activate.bat')
if __name__ == "__main__":
call_anaconda_venv()
The result of the above code would be a running instance of anaconda-prompt as required.
Now as Problem Seems Like:
I have a batch file which is running a python script and in the python script, I have a subprocess function which is being run.
I have implemented the same program as required; Suppose we have
Batch File ---> script.bat **** includes a command to run python script i.e test.py. ****
Python Script File ---> test.py **** includes a method to run commands using subprocess. ****
Batch File ---> sys_info.bat **** includes a command which would give the system information of my computer. ****
Now First, script.bat includes a command that will run the required python script as given below;
python \file_path\test.py
pause
Here, pause command is used to prevent auto-closing console after execution. Now we have test.py, python script which includes subprocess method to run required commands and get their output.
from subprocess import check_output
class BatchCommands:
#staticmethod
def run_commands_using_subprocess(commands):
print("Running commands from File: {}".format(commands))
value = check_output(commands, shell=True).decode()
return value
#staticmethod
def run():
commands_from_file = "\file-path\sys_info.bat"
print('##############################################################')
print("Shell Commands using >>> subprocess-module <<<")
print('##############################################################')
values = BatchCommands.run_commands_using_subprocess(commands_from_file)
print(values)
if __name__ == '__main__':
BatchCommands.run()
Now, in the end, I have a sys_info.bat file which includes commands to renew the IP-Adress of my computer. Commands in sys_info.bat file are as follows;
systeminfo
Place multiple commands in sys_info.bat file, then you can also run multiple commands at a time like:
ipconfig/all
ipconfig/release
ipconfig/reset
ipconfig/renew
ipconfig
Before to use the file, set all files directory paths, and run the batch file i.e script.py in command-prompt as follows;
Run command-prompt or terminal as an administrator.
run \file_path\script.py
Here is the result after running the batch file in the terminal.
This is happening because your ide is not running in a shell that works in the way that open subprocess is expecting.
If you set SHELL=False and specify the absolute path to the batch file it will run.
you might still need the cwd if the batch file requires it.
This question already has answers here:
Running shell command and capturing the output
(21 answers)
Closed 4 years ago.
I'm using python 3.7 on Windows. I'm trying to execute a simple scan command and get its output as a string.
When I execute the command in python I only get the first line:
import subprocess
def execute(command):
proc = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
output = proc.stdout if proc.stdout else proc.stderr
path = "Somepath"
command = ['ecls.exe', '/files', path]
print(execute(command))
output:
WARNING! The scanner was run in the account of a limited user.
But when I run it in the CMD:
$ ecls.exe /files "SomePath"
WARNING! The scanner was run in the account of a limited user.
ECLS Command-line scanner ...
Command line: /files SomePath
Scan started at: 11/24/18 14:18:11
Scan completed at: 11/24/18 14:18:11 Scan time: 0 sec (0:00:00)
Total: files - 1, objects 1 Infected: files - 0, objects 0 Cleaned: files - 0, objects 0
I think that the command spawn a child process and it produces the scan output. I also tried to iterate over stdout but got the same output.
EDIT:
I tried other methods like check_output, Popen, etc with using PIPE but I only get the first line of output. I also tried to use shell=True but didn't make any difference. As I already said the command spawn a child process and I need to capture its output which seems that subprocess can't do it directly.
As I couldn't find a direct way to solve this problem, with help of this reference, the output can be redirected to a text file and then read it back.
import subprocess
import os
import tempfile
def execute_to_file(command):
"""
This function execute the command
and pass its output to a tempfile then read it back
It is usefull for process that deploy child process
"""
temp_file = tempfile.NamedTemporaryFile(delete=False)
temp_file.close()
path = temp_file.name
command = command + " > " + path
proc = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
if proc.stderr:
# if command failed return
os.unlink(path)
return
with open(path, 'r') as f:
data = f.read()
os.unlink(path)
return data
if __name__ == "__main__":
path = "Somepath"
command = 'ecls.exe /files ' + path
print(execute(command))
I am trying to run pdftotext using python subprocess module.
import subprocess
pdf = r"path\to\file.pdf"
txt = r"path\to\out.txt"
pdftotext = r"path\to\pdftotext.exe"
cmd = [pdftotext, pdf, txt, '-enc UTF-8']
response = subprocess.check_output(cmd,
shell=True,
stderr=subprocess.STDOUT)
TB
CalledProcessError: Command '['path\\to\\pdftotext.exe',
'path\\to\\file.pdf', 'path\\to\\out.txt', '-enc UTF-8']'
returned non-zero exit status 99
When I remove last argument '-enc UTF-8' from cmd, it works OK in python.
When I run pdftotext pdf txt -enc UTF-8 in cmd, it works ok.
What I am missing?
Thanks.
subprocess has some complicated rules for handling commands. From the docs:
The shell argument (which defaults to False) specifies whether to use
the shell as the program to execute. If shell is True, it is
recommended to pass args as a string rather than as a sequence.
More details explained in this answer here.
So, as the docs explain, you should convert your command to a string:
cmd = r"""{} "{}" "{}" -enc UTF-8""".format('pdftotext', pdf, txt)
Now, call subprocess as:
subprocess.call(cmd, shell=True, stderr=subprocess.STDOUT)
I've been trying to run a Java program and capture it's STDOUT output to a file from the Python script. The idea is to run test files through my program and check if it matches the answers.
Per this and this SO questions, using subprocess.call is the way to go. In the code below, I am doing subprocess.call(command, stdout=f) where f is the file I opened.
The resulted file is empty and I can't quite understand why.
import glob
test_path = '/path/to/my/testfiles/'
class_path = '/path/to/classfiles/'
jar_path = '/path/to/external_jar/'
test_pattern = 'test_case*'
temp_file = 'res'
tests = glob.glob(test_path + test_pattern) # find all test files
for i, tc in enumerate(tests):
with open(test_path+temp_file, 'w') as f:
# cd into directory where the class files are and run the program
command = 'cd {p} ; java -cp {cp} package.MyProgram {tc_p}'
.format(p=class_path,
cp=jar_path,
tc_p=test_path + tc)
# execute the command and direct all STDOUT to file
subprocess.call(command.split(), stdout=f, stderr=subprocess.STDOUT)
# diff is just a lambda func that uses os.system('diff')
exec_code = diff(answers[i], test_path + temp_file)
if exec_code == BAD:
scream(':(')
I checked the docs for subprocess and they recommended using subprocess.run (added in Python 3.5). The run method returns the instance of CompletedProcess, which has a stdout field. I inspected it and the stdout was an empty string. This explained why the file f I tried to create was empty.
Even though the exit code was 0 (success) from the subprocess.call, it didn't mean that my Java program actually got executed. I ended up fixing this bug by breaking down command into two parts.
If you notice, I initially tried to cd into correct directory and then execute the Java file -- all in one command. I ended up removing cd from command and did the os.chdir(class_path) instead. The command now contained only the string to run the Java program. This did the trick.
So, the code looked like this:
good_code = 0
# Assume the same variables defined as in the original question
os.chdir(class_path) # get into the class files directory first
for i, tc in enumerate(tests):
with open(test_path+temp_file, 'w') as f:
# run the program
command = 'java -cp {cp} package.MyProgram {tc_p}'
.format(cp=jar_path,
tc_p=test_path + tc)
# runs the command and redirects it into the file f
# stores the instance of CompletedProcess
out = subprocess.run(command.split(), stdout=f)
# you can access useful info now
assert out.returncode == good_code