I have a C file say, myfile.c.
Now to compile I am doing : gcc myfile.c -o myfile
So now to run this I need to do : ./myfile inputFileName > outputFileName
Where inputFileName and outputFileName are 2 command line inputs.
Now I am trying to execute this within a python program and I am trying this below approach but it is not working properly may be due to the >
import subprocess
import sys
inputFileName = sys.argv[1];
outputFileName = sys.argv[2];
subprocess.run(['/home/dev/Desktop/myfile', inputFileName, outputFileName])
Where /home/dev/Desktop is the name of my directory and myfile is the name of the executable file.
What should I do?
The > that you use in your command is a shell-specific syntax for output redirection. If you want to do the same through Python, you will have to invoke the shell to do it for you, with shell=True and with a single command line (not a list).
Like this:
subprocess.run(f'/home/dev/Desktop/myfile "{inputFileName}" > "{outputFileName}"', shell=True)
If you want to do this through Python only without invoking the shell (which is what shell=True does) take a look at this other Q&A: How to redirect output with subprocess in Python?
You can open the output file in Python, and pass the file object to subprocess.run().
import subprocess
import sys
inputFileName = sys.argv[1];
outputFileName = sys.argv[2];
with open(outputFileName, "w") as out:
subprocess.run(['/home/dev/Desktop/myfile', inputFileName], stdout=out)
Related
I am trying to run a bash script multiple times on a cluster. The issue however is I need to grab certain file names to fill the command which I only know how to do via python.
Note:I want to run the the last line (the line that calls the script) in parallel in groups of like two. How can I do this?
I have thought of: outputting all commands to a .txt and catting that in parallel. However, I feel that it is not the most efficient.
Thank you for any help
The script looks like this:
#!/usr/bin/python
import os
import sys
cwd = os.getcwd()
for filename in os.listdir(cwd):
if "_2_" in filename:
continue
elif "_1_" in filename:
in1 = os.path.join(cwd, filename)
secondread = filename.replace("_1.fastq_1_trimmed.fq","_1.fastq_2_trimmed.fq")
in2 = os.path.join(cwd, secondread)
outrename = filename.replace("_1.fastq_1_trimmed.fq",".bam")
out = "/home/blmatsum/working/bamout/" + outrename
cmd = "bbmap.sh ref=/home/blmatsum/working/datafiles/sequence.phages_clustered.fna in={} in2={} out={}".format(in1,in2,out)
os.system(cmd)
an example of the command I want to run would be:
bbmap.sh ref=/home/working/datafiles/sequence.phages_clustered.fna in=/home/working/trimmed/SRR7077355_1.fastq_1_trimmed.fq in2=/home/working/trimmed/SRR7077355_1.fastq_2_trimmed.fq out=/home/working/bamout/SRR7077355.bam'
bbmap.sh ref=/home/working/datafiles/sequence.phages_clustered.fna in=/home/working/trimmed/SRR7077366_1.fastq_1_trimmed.fq in2=/home/working/trimmed/SRR7077366_1.fastq_2_trimmed.fq out=/home/working/bamout/SRR7077366.bam
Consider the following python (3.9) script, called test.py:
import sys
import os
import subprocess
from pathlib import Path
# This is for pyinstaller (as main_dir = sys.path[0] won't work)
if getattr(sys, 'frozen', False):
main_dir = os.path.dirname(sys.executable)
else:
main_dir = os.path.dirname(os.path.abspath(__file__))
processes_dir = Path(main_dir, "processes")
outfile = Path(main_dir, "output.txt")
# Initialize the output text file
with open(outfile, 'w') as f:
f.write('')
# This calls A1.py (see below)
result = subprocess.run([sys.executable, Path(processes_dir, "A1.py")], input="1\n2", capture_output=True, text=True)
# If an error is raised, it's written to output.txt; stdout is written to output.txt
if result.stderr:
with open(outfile, 'a') as f:
f.write("{0}\n\n".format(result.stderr))
else:
with open(outfile, 'a') as f:
f.write("{0}\n\n".format(result.stdout))
The subprocess.run invokes the following simple script:
x1 = int(input())
x2 = int(input())
print(x1+x2)
This runs just fine. I'm trying to work out how to convert this into an executable (.exe) using Pyinstaller. In the appropriate directory, I run:
pyinstaller --onefile test.py
This builds test.exe successfully. When I run test.exe (either from cmd or double-clicking on the file) it opens with no errors, produces an empty output.txt and then simply hangs indefinitely. It appears subprocess.run doesn't work properly with pyinstaller. Any ideas/suggestions to get test.exe to work with pyinstaller?
What happens here is that, when the script is not compiled, sys.executable returns the python executable file (C:\Users\randomuser\AppData\Local\Programs\Python38), but when the code is compiled, sys.executable returns the .exe file you havce made. So, your .exe file calls itself, calling itself again infinite times, hainging.
You could solve this in two different (easy) ways:
(Less recommended if you want to distribute your exe file, because it dependes on a python installation):
Replace sys.executable with 'python'. This will ensure the script is executed with Python rather than with your own .exe file if compiled:
result = subprocess.run(['python', Path(processes_dir, "A1.py")], input="1\n2", capture_output=True, text=True)
You can import the A1.py script (ensuring the script is in the same folder as the executable, and making the main code to be in a function called main returning the result as a string):
Then, you'll be able to run import A1, and call the main procedure running A1.main():
import sys
import os
import subprocess
import A1
from pathlib import Path
# This is for pyinstaller (as main_dir = sys.path[0] won't work)
if getattr(sys, 'frozen', False):
main_dir = os.path.dirname(sys.executable)
else:
main_dir = os.path.dirname(os.path.abspath(__file__))
processes_dir = Path(main_dir, "processes")
outfile = Path(main_dir, "output.txt")
# Initialize the output text file
with open(outfile, 'w') as f:
f.write('')
# This calls A1.py (see below)
result = A1.main()
# The output is written into output.txt.
with open(outfile, 'a') as f:
f.write("{0}\n\n".format(result))
You could consider capturing error tracebacks with try-except clause and with the traceback module
The pyinstaller command then should be: pyinstaller --onefile test.py --add-data "A1.py;."
So I have written a piece of code which first runs a powershell command to generate a UTF-8 version of a DAT file (have been having special character issues with the original file, hence the step). Following which I try to open the newly created file. But the issue is, I keep getting 'FileNotFoundError: [Errno 2]' Initially I was only trying with the file name since the newly created file was in the same folder, but then i tried to generate the absolute path as well.
import os
import subprocess
subprocess.Popen('powershell.exe -Command "Get-Content .\Own.DAT | Set-Content -Encoding utf8 Own1.dat"')
filepath = __file__
filepath = filepath[:-7]
with open(filepath+"Own1.dat", "r") as f:
I can confirm that filepath+"Own1.dat" is fetching the correct filepath. Yet can't figure out what the issue could be.
Edit: Someone asked for confirmation, here is the message i am getting:
C:\Users\Debojit\MiniConda3\python.exe "E:/My Documents/Work/essbase/ownership/test.py"
Traceback (most recent call last):
File "E:/My Documents/Work/essbase/ownership/test.py", line 18, in <module>
with open(filepath+"Own1.dat", "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'E:/My Documents/Work/essbase/ownership/Own1.dat'
Process finished with exit code 1
Note: Curiously enough if i put the powershell command into a separate batch file, write a code in the python script to run it, the works without any issues. Here is the code i am talking about:
import os
import subprocess
from subprocess import Popen
p = Popen("conversion.bat", cwd=r"E:\My Documents\Work\essbase\ownership")
stdout, stderr = p.communicate()
filepath = __file__
filepath = filepath[:-7]
with open(filepath+"Own1.dat", "r") as f:
The conversion.bat file contains the following
powershell.exe -Command "Get-Content .\Own.DAT | Set-Content -Encoding utf8 Own1.DAT"
But I don't want to include a separate batch file to go with the python script.
Any idea what might be causing the issue?
Your error is unrelated to powershell. Popen runs asynchronously. In one command, you are using communicate(), but in the other, you are not.
You're using Popen() incorrectly.
If you want run a command and also pass arguments to it, you have to pass them as a list, like so:
subprocess.Popen(['powershell.exe', '-Command', ...])
In your code, popen tries to run a command literally named powershell.exe -Command "Get-Content ... which of course doesn't exist.
To use a simpler example, this code won't work:
subprocess.Popen('ls -l')
because it's trying to run a command literally named ls -l.
But this does work:
subprocess.Popen(['ls', '-l'])
I still couldn't figure out why the error was happening. But I found a workaround
with open("conversion.bat","w") as f:
f.writelines("powershell.exe -Command \"Get-Content '" +fileName+ "' | Set-Content -Encoding utf8 Own1.dat\"")
from subprocess import Popen
p = Popen("conversion.bat", cwd=os.path.dirname(os.path.realpath(__file__)))
stdout, stderr = p.communicate()
os.remove("conversion.bat")
Basically I would create the batch file, run it and then delete it once the file has been created. Don't why I have to use this route, but it works.
I would like to loop over files using subprocess.run(), something like:
import os
import subprocess
path = os.chdir("/test")
files = []
for file in os.listdir(path):
if file.endswith(".bam"):
files.append(file)
for file in files:
process = subprocess.run("java -jar picard.jar CollectHsMetrics I=file", shell=True)
How do I correctly call the files?
shell=True is insecure if you are including user input in it. #eatmeimadanish's answer allows anybody who can write a file in /test to execute arbitrary code on your machine. This is a huge security vulnerability!
Instead, supply a list of command-line arguments to the subprocess.run call. You likely also want to pass in check=True – otherwise, your program would finish without an exception if the java commands fails!
import os
import subprocess
os.chdir("/test")
for file in os.listdir("."):
if file.endswith(".bam"):
subprocess.run(
["java", "-jar", "picard.jar", "CollectHsMetrics", "I=" + file], check=True)
Seems like you might be over complicating it.
import os
import subprocess
path = os.chdir("/test")
for file in os.listdir(path):
if file.endswith(".bam"):
subprocess.run("java -jar picard.jar CollectHsMetrics I={}".format(file), shell=True)
In my Python Program, I have lines to open an input file:
f = open('/home/han/fasta.txt',"r")
and to write an output file:
with open("output.txt", "w") as text_file:
text_file.write ("{:<16}{:<16}{:<16}{:<16}{:<16}".format('','A','C','G','T')+'\n')
However, every time I want to run the Python program in Linux Command Prompt with different, I have to change the input and output file names in codes.
I would like to know how to achieve running the program at Linux command prompt exactly as below: (in which I need to enter "-i" and "-o" for input and output file names respectively)
$ python codon.py -i fasta.txt -o output.txt
I have tried for the input file name
from sys import argv
script, filename = argv
f = open(filename,"r")
but I felt it doesn't necessarily need the "-i" in command prompt.
Sorry if the question is incredibly obvious...I am new to Python
Use python standard argparse library
import argparse
parser = argparse.ArgumentParser(description='My app description')
parser.add_argument('-i', '--input', help='Path to input file')
parser.add_argument('-o', '--output', help='Path to output file')
args = parser.parse_args()
f = open(args.input,"r")
....etc
If you don't want to use the -i, which isn't strictly obligatory, you can do as follows:
python codon.py fasta.txt output.txt
And you can get the names by using argv[1] and argv[2]:
from sys import argv
inputfile = argv[1]
outputfile = argv[2]
f = open(inputfile ,"r")
...