I am writing a python script using subprocess. I would like to get the following bash behaviour...
cat input_file.txt | awk '/pattern/' > output_file.txt
I wrote this code, but it does not work...
cat = subprocess.Popen(shlex.split("cat "+input_file_name),stdout=subprocess.PIPE)
output_file: = open('output_file.txt','wb')
awk_search = subprocess.run(shlex.split("awk '/pattern/'"),stdin=cat.stdout,stdout=output_file)
When I try this step by step in the python terminal things work until I attempt to output to a file.
Could you please help me? Thanks!
Related
How come this command python test.py <(cat file1.txt) does not work accordingly. I could've sworn I had this working previously. Basically, I would like to get the output of that cat command as an input to the python script.
This command
cat file1.txt | python test.py
works okay, which outputs:
reading: file11
reading: file12
Which are based on the following scripts/files below.
The reason I want this to work is because I really want to feed in 2 input files like
python test.py <(cat file1.txt) <(cat file2.txt)
And would like some python output like:
reading: file11 file21
reading: file12 file22
I know this is a very simple example, and I can just read in or open() both files inside the python script and iterate accordingly. This is a simplified version of my current screnario, the cat command is technically another executable doing other things, so its not as easy as just reading/opening the file to read.
Sample script/files:
test.py:
import sys
for line in sys.stdin:
print("reading: ", line.strip())
sys.stdin.close()
file1.txt:
file11
file12
file2.txt:
file21
file22
changing test.py to:
import sys
input1 = open(sys.argv[1], "r")
input2 = open(sys.argv[2], "r")
for line1, line2 in zip(input1, input2):
print("reading: ", line1.strip(), line2.strip())
input1.close()
input2.close()
will enable python test.py <(cat file1.txt) <(cat file2.txt) to work
Actually it depends on shell you are using.
I guess you use bash which unfortunately can't have it working as only last redirection from specific descriptor is taken. You could create temporary file, redirect output of scripts to it and then feed your main script with tmp file.
Or if you don't mind you can switch e.g to zsh, which has such feature enabled by default.
I have been working with this piece of code for some time in Unix systems and it works just fine while running on a normal command line. However, for the sake of a project and learning how to execute Unix commands through Python, I am trying to run it using the os.system() command in Python.
Overall data structure has 5 columns and 1500 rows and the goal is to replace values greater than 2.706 in column 4 ($4) and then proceed to save them into file2.txt while keeping all other values in the file the same.
os.system("awk '{print $1,$2,$3,$5,($4>=2.706)? -999 : $4}' file1.txt > file2.txt")
From this code, I receive the message Invalid syntax after attempting the execution from a python script.
As I am new to Python, I believe I must just missing something in the syntax from that side of the code, but I cannot for the life of me figure it out. Any help would be greatly appreciated.
A new attempt with the same code, but using the subprocess module instead of os.system:
arg1 = "awk '{print $1,$2,$3,$5,($4>=2.7059553)\? -999 \: $4}' phenotypes.txt > replacetest.txt"
subprocess.run(arg1, shell=TRUE)
This code also gives the Invalid syntax response to the creation of the arg1 command.
(Code is being run in Python 2.7.5 on Linux2.)
Hello I am new to Python and would like to know different execution modes of the python program, say for below my program runs fine in first mode but second mode gives me error.
cat inputfile | ./pythonprogram.py - works
./pythonprogram.py | inputfile -- doesn't work
Also what are the all the best practices in executing the program through parsing input files.
FYI.. i am on the google python babyname exercise and below is my program:-
PS: I haven't coded a decent complete code, but this is more like a draft before attempting to execute full program.
Did you perhaps mean:
./pythonprogram.py < inputfile
This takes the contents of inputfile and pipes it to your program.
On the other hand:
./pythonprogram.py | inputfile
Will take the output from your python program, try to execute inputfile (it can't), and then give it the output from the python program.
I would like to start out by saying any help is greatly appreciated. I'm new to Python and scripting in general. I am trying to use a program called samtools view to convert a file from .sam to a .bam I need to be able do what this BASH command is doing in Python:
samtools view -bS aln.sam > aln.bam
I understand that BASH commands like | > < are done using the subprocess stdin, stdout and stderr in Python. I have tried a few different methods and still can't get my BASH script converted correctly. I have tried:
cmd = subprocess.call(["samtools view","-bS"], stdin=open(aln.sam,'r'), stdout=open(aln.bam,'w'), shell=True)
and
from subprocess import Popen
with open(SAMPLE+ "."+ TARGET+ ".sam",'wb',0) as input_file:
with open(SAMPLE+ "."+ TARGET+ ".bam",'wb',0) as output_file:
cmd = Popen([Dir+ "samtools-1.1/samtools view",'-bS'],
stdin=(input_file), stdout=(output_file), shell=True)
in Python and am still not getting samtools to convert a .sam to a .bam file. What am I doing wrong?
Abukamel is right, but in case you (or others) are wondering about your specific examples....
You're not too far off with your first attempt, just a few minor items:
Filenames should be in quotes
samtools reads from a named input file, not from stdin
You don't need "shell=True" since you're not using shell tricks like redirection
So you can do:
import subprocess
subprocess.call(["samtools", "view", "-bS", "aln.sam"],
stdout=open('aln.bam','w'))
Your second example has more or less the same issues, so would need to be changed to something like:
from subprocess import Popen
with open('aln.bam', 'wb',0) as output_file:
cmd = Popen(["samtools", "view",'-bS','aln.sam'],
stdout=(output_file))
You can pass execution to the shell by kwarg 'shell=True'
subprocess.call('samtools view -bS aln.sam > aln.bam', shell=True)
In the Linux kernel, I can send a file to the printer using the following command
cat file.txt > /dev/usb/lp0
From what I understand, this redirects the contents in file.txt into the printing location. I tried using the following command
>>os.system('cat file.txt > /dev/usb/lp0')
I thought this command would achieve the same thing, but it gave me a "Permission Denied" error. In the command line, I would run the following command prior to concatenating.
sudo chown root:lpadmin /dev/usb/lp0
Is there a better way to do this?
While there's no reason your code shouldn't work, this probably isn't the way you want to do this. If you just want to run shell commands, bash is much better than python. On the other hand, if you want to use Python, there are better ways to copy files than shell redirection.
The simplest way to copy one file to another is to use shutil:
shutil.copyfile('file.txt', '/dev/usb/lp0')
(Of course if you have permissions problems that prevent redirect from working, you'll have the same permissions problems with copying.)
You want a program that reads input from the keyboard, and when it gets a certain input, it prints a certain file. That's easy:
import shutil
while True:
line = raw_input() # or just input() if you're on Python 3.x
if line == 'certain input':
shutil.copyfile('file.txt', '/dev/usb/lp0')
Obviously a real program will be a bit more complex—it'll do different things with different commands, and maybe take arguments that tell it which file to print, and so on. If you want to go that way, the cmd module is a great help.
Remember, in UNIX - everything is a file. Even devices.
So, you can just use basic (or anything else, e.g. shutil.copyfile) files methods (http://docs.python.org/2/tutorial/inputoutput.html#reading-and-writing-files).
In your case code may (just a way) be like that:
# Read file.txt
with open('file.txt', 'r') as content_file:
content = content_file.read()
with open('/dev/usb/lp0', 'w') as target_device:
target_device.write(content)
P. S. Please, don't use system() call (or similar) to solve your issue.
under windows OS there is no cat command you should usetype instead of cat under windows
(**if you want to run cat command under windows please look at: https://stackoverflow.com/a/71998867/2723298 )
import os
os.system('type a.txt > copy.txt')
..or if your OS is linux and cat command didn't work anyway here are other methods to copy file..
with grep:
import os
os.system('grep "" a.txt > b.txt')
*' ' are important!
copy file with sed:
os.system('sed "" a.txt > sed.txt')
copy file with awk:
os.system('awk "{print $0}" a.txt > awk.txt')