I have a python script which uses a text file and manipulate the data from the file and output to another file. Basically I want it to work for any text file input. Right now I readline from the file and then print the output to screen. I want the output in a file.
So user can type the following and test for any file:
cat input_file.txt | python script.py > output_file.txt.
How can I implement this in my script? Thank You.
cat is command in linux. I don't know how it works.
The best way to do this is probably to call the input and output files as arguments for the python script:
import sys
inFile = sys.argv[1]
outFile = sys.argv[2]
Then you can read in all your data, do your manipulations, and write out the results:
with open(inFile,'r') as i:
lines = i.readlines()
processedLines = manipulateData(lines)
with open(outFile,'w') as o:
for line in processedLines:
o.write(line)
You can call this program by running python script.py input_file.txt output_file.txt
If you absolutely must pipe the data to python (which is really not recommended), use sys.stdin.readlines()
This method (your question) describes reading data from STDIN:
cat input_file.txt | python script.py
Solution: script.py:
import sys
for line in sys.stdin:
print line
The method in above solutions describes taking argument parameters with your python call:
python script.py input_file.txt
Solution: script.py:
import sys
with open(sys.argv[1], 'r') as file:
for line in file:
print line
Hope this helps!
cat input_file.txt | python script.py > output_file.txt.
You can passing a big string that has all the data inside input_file.txt instead of an actual file so in order to implement your python script, just take that it as a string argument and split the strings by new line characters, for example you can use "\n" as a delimiter to split that big string and to write to an outputfile, just do it in the normal way
i.e. open file, write to the file and close file
Sending output to a file is very similar to taking input from a file.
You open a file for writing the same way you do for reading, except with a 'w' mode instead of an 'r' mode.
You write to a file by calling write on it the same way you read by calling read or readline.
This is all explained in the Reading and Writing Files section of the tutorial.
So, if your existing code looks like this:
with open('input.txt', 'r') as f:
while True:
line = f.readline()
if not line:
break
print(line)
You just need to do this:
with open('input.txt', 'r') as fin, open('output.txt', 'w') as fout:
while True:
line = fin.readline()
if not line:
break
fout.write(line)
If you're looking to allow the user to pass the filenames on the command line, use sys.argv to get the filenames, or use argparse for more complicated command-line argument parsing.
For example, you can change the first line to this:
import sys
with open(sys.argv[1], 'r') as fin, open(sys.argv[2], 'w') as fout:
Now, you can run the program like this:
python script.py input_file.txt outputfile.txt
cat input_file.txt | python script.py > output_file.txt
Basically, python script needs to read the input file and write to the standard output.
import sys
with open('input_file.txt', 'r') as f:
while True:
line = f.readline()
if not line:
break
sys.stdout.write(line)
Related
My Python program requests some user input:
n = int(input())
for i in range(n):
s = input()
# ...
For debugging purposes, I would like to create a test input.txt file like this:
3
string1
string2
string3
so that I need not to manually enter data from the keyboard every time I launch it for debug.
Is there any way to get input() from text file, not from keyboard (without changing program code).
I am using Wing Personal as a dev tool.
Update
I know for sure, that there exist bots which are able to test python programs passing various inputs and comparing output with known answers. I guess these bots do not hit the keyboard with iron fingers. Maybe they use command-line tools to pass input.txt to the Python so that it reads input() from that file, not from keyboard. I would like to learn how to do it.
Input reads from Standard input so if you use bash you can redirect stdin to a file
without changing your code
in bash you would run something like
cat textfile | programm.py
or
< textfile programm.py
I've noticed that none of these solutions redirects the stdin from Python. To do that, you can use this snippet:
import sys
sys.stdin = open('input.txt', 'r')
Hopefully, that helps you!
You can read the file into a list like this:
with open('filename.txt', 'r') as file:
input_lines = [line.strip() for line in file]
s would be data in index i.
n = int(input())
for i in range(n):
s = input_lines[i]
# ...
Make sure your py file is in the same dir as your txt file.
You can create a test.py file which is used to run your test. In the test.py, you will read the content of input.txt and import the programming.
For example, in test.py file.
import main_code
with open('input.txt', 'r') as file:
lines = file.readlines()
for line in lines:
s = line
"""
working with the input.
"""
I am writing a script where I first read a file using
with open('file', 'r') as file,
do some operation, and then write it to a new file using
with open('newfile', 'w') as newfile.
My question is, what do I need to change in the script to make it general for a number of files, so that I can just call the script with the file name from the terminal like python3 script.py file.ext? Also, is there a way to write the output back into the original file using this method?
One way of doing this is:
import sys
file=open(sys.argv[1], "r")
newfile=open("example.txt", "w")
content=file.read()
#do stuff to content here such as content=content.upper()
newfile.write(content)
If you inputted into the command line:
python script.py aaaaaa.txt
Then the file you set as the output would have the edited content of aaaaaa.txt
Hope this helps :)
How do I export the output of my code to it's own text file? When I run my code I get a large set of data from it. How do I export this so that I can read all lines of data in it's own text file.
You can write file in python like
with open("out.txt", "w") as f:
f.write("OUTPUT")
Or you can use io redirection to redirect output to a file
$ python code.py > out.txt
https://docs.python.org/3/tutorial/inputoutput.html#methods-of-file-objects
https://en.wikipedia.org/wiki/Redirection_(computing)
Assuming you will read the results in another application, you can use redirections, usually something like this:
./myprogram >results.txt
You may want to take a look at file objects, which would allow you to write all the data you want to a file.
For example :
file = open('output.txt', 'w')
file.write('Here is some data')
file.close()
One way to do it is this:
import csv
with open("name_of_file_to_be_created") as f:
writer = csv.writer(f)
for i in range(n):
writer.writerow('stuff to write')
# writes a single line in each iteration, e.g. assuming you are computing something inside the loop
Another way to do this is:
with open("name_of_file_to_be_created") as f:
print("here you can type freely", file = f)
# or
f.write('whatever it is that you have to write')
Sometimes when I open a file for reading or writing in Python
f = open('workfile', 'r')
or
f = open('workfile', 'w')
I read/write the file, and then at the end I forget to do f.close(). Is there a way to automatically close after all the reading/writing is done, or after the code finishes processing?
with open('file.txt','r') as f:
#file is opened and accessible via f
pass
#file will be closed before here
You could always use the with...as statement
with open('workfile') as f:
"""Do something with file"""
or you could also use a try...finally block
f = open('workfile', 'r')
try:
"""Do something with file"""
finally:
f.close()
Although since you say that you forget to add f.close(), I guess the with...as statement will be the best for you and given it's simplicity, it's hard to see the reason for not using it!
Whatever you do with your file, after you read it in, this is how you should read and write it back:
$ python myscript.py sample.txt sample1.txt
Then the first argument (sample.txt) is our "oldfile" and the second argument (sample1.txt) is our "newfile". You can then do the following code into a file called "myscript.py"
from sys import argv
script_name,oldfile,newfile = argv
content = open(oldfile,"r").read()
# now, you can rearrange your content here
t = open(newfile,"w")
t.write(content)
t.close()
How to read a whole file in Python? I would like my script to work however it is called
script.py log.txt
script.py < log2.txt
python script.py < log2.txt
python -i script.py logs/yesterday.txt
You get the idea.
I tried
import fileinput
from bs4 import BeautifulSoup
f = fileinput.input()
soup = BeautifulSoup(f.read())
But I get
Traceback (most recent call last):
File "visual-studio-extension-load-times.py", line 5, in <module>
soup = BeautifulSoup(f.read())
AttributeError: FileInput instance has no attribute 'read'
Instead of using fileinput, open the file directly yourself:
import sys
try:
fileobj = open(sys.argv[1], 'r')
except IndexError:
fileobj = sys.stdin
with fileobj:
data = fileobj.read()
f = open('file.txt', 'r')
data = f.read()
f.close()
further more, to open a file passed from the command line you can do:
(also, this is a smarter way to open files, instead of f = open(...) you can do with ...)
import sys
with open(sys.argv[1], 'r') as f:
data = f.read()
The reason for with being a smarter way to open files, is because it will automaticlly close the file after you leave the indented with block.
This means you don't have to "worry" about files being open or forgotten for to long (that can cause "to many open filehandles" from your OS)
Then to sys.argv
sys.argv[1] will be the first parameter on the command line after your python file.
sys.argv[0] will be your scripts name. for instance:
python myscript.py heeyooo will be:
sys.argv[0] == "myscript.py"
sys.argv[1] == "heeyooo" :)
Then there's all sorts of modules that will be interesting to you when working with files.
For one, os.path is a good start because you will most likely want to do as much cross-platform as possible and this gives you the option to convert \ into / on Linux and vice versa.
A few good ones are:
os.path.abspath
os.path.isfile
os.path.isdir
You also have os.getcwd() which might be good :)
argparse to the rescue!:
>>> import sys
>>> import argparse
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('infile', nargs='?', type=argparse.FileType('r'),
... default=sys.stdin)
>>> args = parser.parse_args()
>>> file_data = args.infile.read()
script.py log.txt
script.py < log2.txt
These two are very different invocations of your script! In the first, the shell passes the filename log.txt to the script; in the second, the shell connects the script's standard input to the file log2.txt, and the script never actually sees the filename.
It is possible to handle both of these in the same script. One way to do it is to read from standard input if no files are passed on the command line. Another way is to read from standard input if it's not a terminal and then also read files listed on the command line, if any (I do like fileinput for this if you are interested in reading the lines but don't care what file they come from). You can use sys.stdin.isatty() which returns True if the standard input is a terminal. So something like this:
import sys, fileinput
if not sys.stdin.isatty():
for line in sys.stdin:
process(line)
for line in fileinput.input():
process(line)
But if you are looking to process each file as a whole, as it appears, then fileinput won't do. Instead, read each filename from the command line individually, read the indicated file, and process it:
import sys
if not sys.stdin.isatty():
stdin = sys.stdin.read()
if stdin:
process(stdin)
for filename in sys.argv[1:]:
with open(filename) as f:
process(f.read())
Now with regard to these invocations:
python script.py < log2.txt
python -i script.py logs/yesterday.txt
These are the same as though you had just invoked script.py directly as far as the script can tell, so you don't need to handle them specially. Using the -i option with input indirection (<) could cause some unexpected behavior, but I haven't tried it (and there wouldn't be any way to work around it anyway).
It doesn't sound like you really wanted fileinput in the first place, since you're not trying to concatenate multiple files, handle the name - as "put stdin here", etc.
But if you do want fileinput, instead of trying to reproduce all of its behavior, just wrap it up.
You want to read all of the input into one string, but all it provides is functions that give you one line or one file at a time. So, what can you do? Join them together:
soup = BeautifulSoup(''.join(fileinput.input()))
That's it.
Went with this.
import sys
from bs4 import BeautifulSoup
f = open(sys.argv[1] ) if sys.argv[1:] else sys.stdin
soup = BeautifulSoup(f)