This is a simple ask but I can't find any information on how to do it: I have a python script that is designed to take in a text file of a specific format and perform functions on it--how do I pipe a test file into the python script such that it is recognized as input()? More specifically, the Python is derived from skeleton code I was given that looks like this:
def main():
N = int(input())
lst = [[int(i) for i in input().split()] for _ in range(N)]
intervals = solve(N, lst)
print_solution(intervals)
if __name__ == '__main__':
main()
I just need to understand how to, from the terminal, input one of my test files to this script (and see the print_solution output)
Use the fileinput module
input.txt
...input.txt contents
script.py
#!/usr/bin/python3
import fileinput
def main():
for line in fileinput.input():
print(line)
if __name__ == '__main__':
main()
pipe / input examples:
$ cat input.txt | ./script.py
...input.txt contents
$ ./script.py < input.txt
...input.txt contents
You can take absolute or relative path in your input() function and then open this path via open()
filename = input('Please input absolute filename: ')
with open(filename, 'r') as file:
# Do your stuff
Please let me know if I misunderstood your question.
You can either:
A) Use sys.stdin (import sys at the top of course)
or
B) Use the ArgumentParser (from argparse import ArgumentParser) and pass the file as an argument.
Assuming A it would look something like this:
python script.py < file.extension
Then in the script it would look like:
fData = []
for line in sys.stdin.readLines():
fData.append(line)
# manipulate fData
There are a number of ways to achieve what you want. This is what I came up with off the top of my head. It may not be the best / efficient way, but it should work. I do a lot of file I/O with python at work and this is one of the ways I've achieved it in the past.
Note: If you want to write the manipulated lines back to the file use the argparse library.
Edit:
from argparse import ArgumentParser
def parseInput():
parser = ArgumentParser(description = "Takes input file to read")
parser.add_argument('-f', type = str, default = None, required =
True, help = "File to perform I/O on")
args = parser.parse_args()
return args
def main():
args = parseInput()
fData = []
# perform rb
with open(args.f, 'r') as f:
for line in f.readlines():
fData.append(line)
# Perform data manipulations
# perform wb
with open(args.f, 'w') as f:
for line in fData:
f.write(line)
if __name__ == "__main__":
main()
Then on command line it would look like:
python yourScript.py -f fileToInput.extension
Related
I have a file with N rows and M columns. I would like to use the command line (python) to specify the file and then store its contents into a matrix in this form [0,0,0],[0,0,0]...
I have this so far, but when I run in through terminal, it does not ask for the text file.
import sys
def read_data(filename):
with open(filename, 'r') as f:
data = [[int(num) for num in line.split(',')] for line in f]
return data
def main():
if len(sys.argv) < 2:
print("Usage: {0} <Data Points>".format(sys.argv[0]))
sys.exit(1)
file1 = sys.argv[1]
data_points = read_data(file1)
print(data_points)
You should pass the file path as an argument when calling your program.
python script.py /myfile
Or whatever arguments your script.py file needs.
Your script does not call the main() function. We define the main() function, but it will not run until called. I added a call to the main() function at the bottom of the script.
import sys
def read_data(filename):
with open(filename, 'r') as f:
data = [[int(num) for num in line.split(',')] for line in f]
return data
def main():
if len(sys.argv) < 2:
print("Usage: {0} <Data Points>".format(sys.argv[0]))
sys.exit(1)
file1 = sys.argv[1]
data_points = read_data(file1)
print(data_points)
main()
When you run the script, you will want to run it on the command line using a command that looks similar to this:
$ python3 myscript.py mydata.csv
Where:
myscript.py is the name of your script
mydata.csv is the name of your data file
I want to filter a log file to keep all lines matching a certain pattern. I want to do this with Python.
Here's my first attempt:
#!/usr/bin/env python
from sys import argv
script, filename = argv
with open(filename) as f:
for line in f:
try:
e = line.index("some_term_I_want_to_match")
except:
pass
else:
print(line)
How can I improve this to:
save the result to a new file of similar name (i.e., a different extension)
use regex to make it more flexible/powerful.
(I'm just learning Python. This question is as much about learning Python as it is about accomplishing this particular result.)
OK, here's what I came up with so far... But how do you do the equivalent of prepending an r as in the following line
re.compile(r"\s*")
where the string is not a string literal, as in the next line?
re.compile(a_string_variable)
Other than that, I think this updated version does the job:
#!/usr/bin/env python
from sys import argv
import re
import os
import argparse #requires Python 2.7 or above
parser = argparse.ArgumentParser(description='filters a text file on the search phrase')
parser.add_argument('-s','--search', help='search phrase or keyword to match',required=True)
parser.add_argument('-f','--filename', help='input file name',required=True)
parser.add_argument('-v','--verbose', help='display output to the screen too', required=False, action="store_true")
args = parser.parse_args()
keyword = args.search
original_file = args.filename
verbose = args.verbose
base_file, ext = os.path.splitext(original_file)
new_file = base_file + ".filtered" + ext
regex_c = re.compile(keyword)
with open(original_file) as fi:
with open(new_file, 'w') as fo:
for line in fi:
result = regex_c.search(line)
if(result):
fo.write(line)
if(verbose):
print(line)
Can this be easily improved?
Well, you know, you have answered most of your questions yourself already :)
For regular expression matching use re module (the doc has pretty explanatory examples).
You already have made use open() function for opening a file. Use the same function for open files for writing, just provide a corresponding mode parameter ("w" or "a" combined with "+" if you need, see help(open) in the Python interactive shell). That's it.
I'm experimenting with file I/O. I have a small practice program that creates a text file when run. I packaged it with pyinstaller so that double clicking on the exe creates a new folder and places a text file with "hello world" inside of it. Easy peasy.
Then I started wondering about main(). This is just a function like any other, right? So does that mean I can pass arguments to it at runtime?
I was thinking about the Steam client and how you can put stuff like '-dev' and '-console' in the shortcut. Is there a way to do this to a python exe that I have made?
I may be explaining terribly, so here's an example:
def makeFile(string):
if string:
f = open('mytext.txt', 'w') #create text file in local dir
print >> f, 'hello, ' + string + '! \nHow are ya?'
f.close()
else:
f = open('mytext.txt', 'w') #create text file in local dir
print >> f, 'hello, person! \nHow are ya?'
f.close()
def main(string = None):
makeFile(string)
So if I take this code and make it an exe, would I be able to add my optional arguments somehow.
I tried the above code, and the running test.exe --"myname" but that didn't work.
Is there a way to do this?
What you're looking for is either the sys module, or the optparse module.
sys will give you very basic control over command line args.
For example:
import sys
if __name__ == "__main__":
if len(sys.argv)>1:
print sys.argv[1]
In the above example, if you were to open up a shell and type -
test.exe "myname"
The resultant output would be:
myname
Note that sys.argv[0] is the name of the script you are currently running. Each subsequent argument is defined by a space, so in your example above
test.exe -- myname
argv[0] = "test.exe"
argv[1] = "--"
argv[2] = "myname"
Optparse gives a much more robust solution that allows you to define command line switches with multiple options and defines variables that will store the appropriate options that can be accessed at runtime.
Re-writing your example:
from optparse import OptionParser
def makeFile(options = None):
if options:
f = open('mytext.txt', 'w') #create text file in local dir
print >> f, 'hello, ' + options.name + '! \nHow are ya?'
f.close()
else:
f = open('mytext.txt', 'w') #create text file in local dir
print >> f, 'hello, person! \nHow are ya?'
f.close()
if __name__ == "__main__":
parser = OptionParser()
parser.add_option('-n','--name',dest = 'name',
help='username to be printed out')
(options,args) = parser.parse_args()
makeFile(options)
You would run your program with :
test.exe -n myname
and the output (in myfile.txt) would be the expected:
Hello, myname!
How are ya?
Hope that helps!
Yes, you can do it with sys.argv. Check out this link: http://docs.python.org/library/sys.html#sys.argv. But remember not to forget import sys, and then you can use it.
import sys
# If there is an argument passed to your file
if len(sys.argv) > 1:
# argv[1] has your filename
filename = sys.argv[1]
print (filename)
# Output...
# new-host:~ yanwchan$ python3.2 test.py text.txt
# text.txt
argv[0] has test.py
argv[1] has text.txt
Edit: However, I do some more research on this topic and found out this: https://stackoverflow.com/a/4188500/1276534
As katrielalex points out, maybe you can look into argparse as well.? It provides a lot more functionality as well as safety check. Interesting information.
And here is a great tutorial: http://www.doughellmann.com/PyMOTW/argparse/
What you are looking for is something like the python argparse module
Or you can read the values directly using sys.argv
import sys
sys.argv[0] # the name of the command that was called
sys.argv[1] # the first argument, eg '--dev'
sys.argv[2] # the second...
Just a note for completeness: there is docopt now, which makes it really easy to write even complex command line interfaces by describing it in a simple language. Documenting and parsing the interface actually becomes the same task with docopt.
I've got a Python program that reads from sys.stdin, so I can call it with ./foo.py < bar.png. How do I test this code from within another Python module? That is, how do I set stdin to point to the contents of a file while running the test script? I don't want to do something like ./test.py < test.png. I don't think I can use fileinput, because the input is binary, and I only want to handle a single file. The file is opened using Image.open(sys.stdin) from PIL.
You should generalise your script so that it can be invoked from the test script, in addition to being used as a standalone program. Here's an example script that does this:
#! /usr/bin/python
import sys
def read_input_from(file):
print file.read(),
if __name__ == "__main__":
if len(sys.argv) > 1:
# filename supplied, so read input from that
filename = sys.argv[1]
file = open(filename)
else:
# no filename supplied, so read from stdin
file = sys.stdin
read_input_from(file)
If this is called with a filename, the contents of that file will be displayed. Otherwise, input read from stdin will be displayed. (Being able to pass a filename on the command line might be a useful improvement for your foo.py script.)
In the test script you can now invoke the function in foo.py with a file, for example:
#! /usr/bin/python
import foo
file = open("testfile", "rb")
foo.read_input_from(file)
Your function or class should accept a stream instead of choosing which stream to use.
Your main function will choose sys.stdin.
Your test method will probably choose a StringIO instance or a test file.
The program:
# foo.py
import sys
from PIL import Image
def foo(stream):
im = Image.open(stream)
# ...
def main():
foo(sys.stdin)
if __name__ == "__main__":
main()
The test:
# test.py
import StringIO, unittest
import foo
class FooTest(unittest.TestCase):
def test_foo(self):
input_data = "...."
input_stream = StringIO.StringIO(input_data)
# can use a test file instead:
# input_stream = open("test_file", "rb")
result = foo.foo(input_stream)
# asserts on result
if __name__ == "__main__":
unittest.main()
A comp.lang.python post showed the way: Substitute a StringIO() object for sys.stdout, and then get the output with getvalue():
def setUp(self):
"""Set stdin and stdout."""
self.stdin_backup = sys.stdin
self.stdout_backup = sys.stdout
self.output_stream = StringIO()
sys.stdout = self.output_stream
self.output_file = None
def test_standard_file(self):
sys.stdin = open(EXAMPLE_PATH)
foo.main()
self.assertNotEqual(
self.output_stream.getvalue(),
'')
def tearDown(self):
"""Restore stdin and stdout."""
sys.stdin = self.stdin_backup
sys.stdout = self.stdout_backup
You can always monkey patch Your stdin. But it is quite ugly way. So better is to generalize Your script as Richard suggested.
import sys
import StringIO
mockin = StringIO.StringIO()
mockin.write("foo")
mockin.flush()
mockin.seek(0)
setattr(sys, 'stdin', mockin)
def read_stdin():
f = sys.stdin
result = f.read()
f.close()
return result
print read_stdin()
Also, do not forget to restore stdin when tearing down your test.
I'am new to Python 3 and could really use a little help. I have a txt file containing:
InstallPrompt=
DisplayLicense=
FinishMessage=
TargetName=D:\somewhere
FriendlyName=something
I have a python script that in the end, should change just two lines to:
TargetName=D:\new
FriendlyName=Big
Could anyone help me, please? I have tried to search for it, but I didnt find something I could use. The text that should be replaced could have different length.
import fileinput
for line in fileinput.FileInput("file",inplace=1):
sline=line.strip().split("=")
if sline[0].startswith("TargetName"):
sline[1]="new.txt"
elif sline[0].startswith("FriendlyName"):
sline[1]="big"
line='='.join(sline)
print(line)
A very simple solution for what you're doing:
#!/usr/bin/python
import re
import sys
for line in open(sys.argv[1],'r').readlines():
line = re.sub(r'TargetName=.+',r'TargetName=D:\\new', line)
line = re.sub(r'FriendlyName=.+',r'FriendlyName=big', line)
print line,
You would invoke this from the command line as ./test.py myfile.txt > output.txt
Writing to a temporary file and the renaming is the best way to make sure you won't get a damaged file if something goes wrong
import os
from tempfile import NamedTemporaryFile
fname = "lines.txt"
with open(fname) as fin, NamedTemporaryFile(dir='.', delete=False) as fout:
for line in fin:
if line.startswith("TargetName="):
line = "TargetName=D:\\new\n"
elif line.startswith("FriendlyName"):
line = "FriendlyName=Big\n"
fout.write(line.encode('utf8'))
os.rename(fout.name, fname)
Is this a config (.ini) file you're trying to parse? The format looks suspiciously similar, except without a header section. You can use configparser, though it may add extra space around the "=" sign (i.e. "TargetName=D:\new" vs. "TargetName = D:\new"), but if those changes don't matter to you, using configparser is way easier and less error-prone than trying to parse it by hand every time.
txt (ini) file:
[section name]
FinishMessage=
TargetName=D:\something
FriendlyName=something
Code:
import sys
from configparser import SafeConfigParser
def main():
cp = SafeConfigParser()
cp.optionxform = str # Preserves case sensitivity
cp.readfp(open(sys.argv[1], 'r'))
section = 'section name'
options = {'TargetName': r'D:\new',
'FriendlyName': 'Big'}
for option, value in options.items():
cp.set(section, option, value)
cp.write(open(sys.argv[1], 'w'))
if __name__ == '__main__':
main()
txt (ini) file (after):
[section name]
FinishMessage =
TargetName = D:\new
FriendlyName = Big
subs_names.py script works both Python 2.6+ and Python 3.x:
#!/usr/bin/env python
from __future__ import print_function
import sys, fileinput
# here goes new values
substitions = dict(TargetName=r"D:\new", FriendlyName="Big")
inplace = '-i' in sys.argv # make substitions inplace
if inplace:
sys.argv.remove('-i')
for line in fileinput.input(inplace=inplace):
name, sep, value = line.partition("=")
if name in substitions:
print(name, sep, substitions[name], sep='')
else:
print(line, end='')
Example:
$ python3.1 subs_names.py input.txt
InstallPrompt=
DisplayLicense=
FinishMessage=
TargetName=D:\new
FriendlyName=Big
If you are satisfied with the output then add -i parameter to make changes inplace:
$ python3.1 subs_names.py -i input.txt