Develop Reference

Applying python script to file

This is a simple ask but I can't find any information on how to do it: I have a python script that is designed to take in a text file of a specific format and perform functions on it--how do I pipe a test file into the python script such that it is recognized as input()? More specifically, the Python is derived from skeleton code I was given that looks like this:
def main():
N = int(input())
lst = [[int(i) for i in input().split()] for _ in range(N)]
intervals = solve(N, lst)
print_solution(intervals)
if __name__ == '__main__':
main()
I just need to understand how to, from the terminal, input one of my test files to this script (and see the print_solution output)

Use the fileinput module
input.txt
...input.txt contents
script.py
#!/usr/bin/python3
import fileinput
def main():
for line in fileinput.input():
print(line)
if __name__ == '__main__':
main()
pipe / input examples:
$ cat input.txt | ./script.py
...input.txt contents
$ ./script.py < input.txt
...input.txt contents

You can take absolute or relative path in your input() function and then open this path via open()
filename = input('Please input absolute filename: ')
with open(filename, 'r') as file:
# Do your stuff
Please let me know if I misunderstood your question.

You can either:
A) Use sys.stdin (import sys at the top of course)
or
B) Use the ArgumentParser (from argparse import ArgumentParser) and pass the file as an argument.
Assuming A it would look something like this:
python script.py < file.extension
Then in the script it would look like:
fData = []
for line in sys.stdin.readLines():
fData.append(line)
# manipulate fData
There are a number of ways to achieve what you want. This is what I came up with off the top of my head. It may not be the best / efficient way, but it should work. I do a lot of file I/O with python at work and this is one of the ways I've achieved it in the past.
Note: If you want to write the manipulated lines back to the file use the argparse library.
Edit:
from argparse import ArgumentParser
def parseInput():
parser = ArgumentParser(description = "Takes input file to read")
parser.add_argument('-f', type = str, default = None, required =
True, help = "File to perform I/O on")
args = parser.parse_args()
return args
def main():
args = parseInput()
fData = []
# perform rb
with open(args.f, 'r') as f:
for line in f.readlines():
fData.append(line)
# Perform data manipulations
# perform wb
with open(args.f, 'w') as f:
for line in fData:
f.write(line)
if __name__ == "__main__":
main()
Then on command line it would look like:
python yourScript.py -f fileToInput.extension

Read Value from Config File Python

I have a file .env file contain 5 lines
DB_HOST=http://localhost/
DB_DATABASE=bheng-local
DB_USERNAME=root
DB_PASSWORD=1234567890
UNIX_SOCKET=/tmp/mysql.sock
I want to write python to grab the value of DB_DATABASE
I want this bheng-local
I would have use
import linecache
print linecache.getline('.env', 2)
But some people might change the order of the cofigs, that's why linecache is not my option.
I am not sure how to check for only some strings match but all the entire line, and grab the value after the =.
I'm kind of stuck here :
file = open('.env', "r")
read = file.read()
my_line = ""
for line in read.splitlines():
if line == "DB_DATABASE=":
my_line = line
break
print my_line
Can someone please give me a little push here ?

Have a look at the config parser:
https://docs.python.org/3/library/configparser.html
It's more elegant than a self-made solution

Modify your .env to
[DB]
DB_HOST=http://localhost/
DB_DATABASE=bheng-local
DB_USERNAME=root
DB_PASSWORD=1234567890
UNIX_SOCKET=/tmp/mysql.sock
Python code
#!/usr/local/bin/python
import configparser
config = configparser.ConfigParser()
config.read('test.env')
print config.get('DB','DB_DATABASE')
Output:
bheng-local
Read https://docs.python.org/3/library/configparser.html

This should work for you
#!/usr/local/bin/python
file = open('test.env', "r")
read = file.read()
for line in read.splitlines():
if 'DB_DATABASE=' in line:
print line.split('=',1)[1]

#!/usr/local/bin/python
from configobj import ConfigObj
conf = ConfigObj('test.env')
print conf['DB_DATABASE']

from os import environ, path
from dotenv import load_dotenv
basedir = path.abspath(path.dirname(__file__))
load_dotenv(path.join(basedir, '.env'))
DB_DATABASE = environ.get('DB_DATABASE')
print(DB_DATABASE)
This could be another option

How can I call Vim from Python?

I would like a Python script to prompt me for a string, but I would like to use Vim to enter that string (because the string might be long and I want to use Vim's editing capability while entering it).

You can call vim with a file path of your choice:
from subprocess import call
call(["vim","hello.txt"])
Now you can use this file as your string:
file = open("hello.txt", "r")
aString = file.read()

Solution:
#!/usr/bin/env python
from __future__ import print_function
from os import unlink
from tempfile import mkstemp
from subprocess import Popen
def callvim():
fd, filename = mkstemp()
p = Popen(["/usr/bin/vim", filename])
p.wait()
try:
return open(filename, "r").read()
finally:
unlink(filename)
data = callvim()
print(data)
Example:
$ python foo.py
This is a big string.
This is another line in the string.
Bye!

filter a Linux log file using Python

I want to filter a log file to keep all lines matching a certain pattern. I want to do this with Python.
Here's my first attempt:
#!/usr/bin/env python
from sys import argv
script, filename = argv
with open(filename) as f:
for line in f:
try:
e = line.index("some_term_I_want_to_match")
except:
pass
else:
print(line)
How can I improve this to:
save the result to a new file of similar name (i.e., a different extension)
use regex to make it more flexible/powerful.
(I'm just learning Python. This question is as much about learning Python as it is about accomplishing this particular result.)
OK, here's what I came up with so far... But how do you do the equivalent of prepending an r as in the following line
re.compile(r"\s*")
where the string is not a string literal, as in the next line?
re.compile(a_string_variable)
Other than that, I think this updated version does the job:
#!/usr/bin/env python
from sys import argv
import re
import os
import argparse #requires Python 2.7 or above
parser = argparse.ArgumentParser(description='filters a text file on the search phrase')
parser.add_argument('-s','--search', help='search phrase or keyword to match',required=True)
parser.add_argument('-f','--filename', help='input file name',required=True)
parser.add_argument('-v','--verbose', help='display output to the screen too', required=False, action="store_true")
args = parser.parse_args()
keyword = args.search
original_file = args.filename
verbose = args.verbose
base_file, ext = os.path.splitext(original_file)
new_file = base_file + ".filtered" + ext
regex_c = re.compile(keyword)
with open(original_file) as fi:
with open(new_file, 'w') as fo:
for line in fi:
result = regex_c.search(line)
if(result):
fo.write(line)
if(verbose):
print(line)
Can this be easily improved?

Well, you know, you have answered most of your questions yourself already :)
For regular expression matching use re module (the doc has pretty explanatory examples).
You already have made use open() function for opening a file. Use the same function for open files for writing, just provide a corresponding mode parameter ("w" or "a" combined with "+" if you need, see help(open) in the Python interactive shell). That's it.

python search for a string and append to it using regular expression

I need to search for a certain parameter known as jvm_args in a configuration file known as config.ini
**contents of config.ini:
first_paramter=some_value1
second_parameter=some_value2
jvm_args=some_value3**
I need to know how to find this parameter in my file and append something to its value, (i.e append a string to the string some_value3).

If you "just" want to find keys and values in an ini file, I think the configparser module is a better bet than using regexps. The configparser asserts that the file has "sections", though.
Documentation for configparser is here: http://docs.python.org/library/configparser.html - useful examples at the bottom. The configparser can also be used for setting values and writing out a new .ini-file.
Input file:
$ cat /tmp/foo.ini
[some_section]
first_paramter = some_value1
second_parameter = some_value2
jvm_args = some_value3
Code:
#!/usr/bin/python3
import configparser
config = configparser.ConfigParser()
config.read("/tmp/foo.ini")
jvm_args = config.get('some_section', 'jvm_args')
print("jvm_args was: %s" % jvm_args)
config.set('some_section', 'jvm_args', jvm_args + ' some_value4')
with open("/tmp/foo.ini", "w") as fp:
config.write(fp)
Output file:
$ cat /tmp/foo.ini
[some_section]
first_paramter = some_value1
second_parameter = some_value2
jvm_args = some_value3 some_value4

You can use re.sub
import re
import os
file = open('config.ini')
new_file = open('new_config.ini', 'w')
for line in file:
new_file.write(re.sub(r'(jvm_args)\s*=\s*(\w+)', r'\1=\2hello', line))
file.close()
new_file.close()
os.remove('config.ini')
os.rename('new_config.ini', 'config.ini')
also check ConfigParser

As both avasal and tobixen have suggested, you can use the python ConfigParser module to do this. For example, I took this "config.ini" file:
[section]
framter = some_value1
second_parameter = some_value2
jvm_args = some_value3**
and ran this python script:
import ConfigParser
p = ConfigParser.ConfigParser()
p.read("config.ini")
p.set("section", "jvm_args", p.get("section", "jvm_args") + "stuff")
with open("config.ini", "w") as f:
p.write(f)
and the contents of the "config.ini" file after running the script was:
[section]
framter = some_value1
second_parameter = some_value2
jvm_args = some_value3**stuff

without regex you can try:
with open('data1.txt','r') as f:
x,replace=f.read(),'new_entry'
ind=x.index('jvm_args=')+len('jvm_args=')
end=x.find('\n',ind) if x.find('\n',ind)!=-1 else x.rfind('',ind)
x=x.replace(x[ind:end],replace)
with open('data1.txt','w') as f:
f.write(x)