Develop Reference

How to edit text file in server using python

I want to edit a line in a text file in a Linux server using python. The process involves following steps
Telnet to the server (using telnetlib)
Go to the required directory
open the text file in the directory
set or unset the flag (YES or NO) of the variable in the text file based on the requirement
save the file and exit
I'm able to automate until step 2. However, I'm stuck at step 3 through 5.
I tried to mimic the steps I follow manually (using vim editor). But I'm not able to perform the 'ESC', replace and ':wq!' steps. Is there an alternative procedure to edit the file or any ways to improve upon mimicking the manual process
I have added my code here
host = input("Enter the IP address:")
port = input("Enter the port:")
tn = telnetlib.Telnet(host,port)
tn.write(b'\n')
tn.read_until(b"login:")
tn.write(b"admin" + b'\n')
tn.read_until(b"Password:")
tn.write(b"admin" + b'\n')
tn.write(b"version" + b'\n')
tn.write(b"path/to/file/" + b'\n')
# OPEN THE FILE and SET or RESET THE FLAG and CLOSE
with in_place.InPlace('filename.txt') as file:
for line in file:
line = line.replace('line_to_change', 'changed_data')
file.write(line)
print('Task executed')
I tried using the in-place library to set the flag but the programme is looking for the file in my local machine rather in the server. So it throws an error message indicating that the file is not present.

If you are able to connect to your remote server, the rest should work as follows:
with open('path/to/file','r') as fr:
data = fr.readlines() # returns list of lines
changed_data = ["changed_data\n" if line=="line_to_change\n" else line
for line in data]
with open('path/to/file','w') as fw:
for line in changed_data:
fw.write(line) # write the lines back to the back

How to check if a block of lines has a particular keyword using python?

I am checking a text file with blocks of commands as following -
File start -
!
interface Vlan100
description XYZ
ip vrf forwarding XYZ
ip address 10.208.56.62 255.255.255.192
!
interface Vlan101
description ABC
ip vrf forwarding ABC
ip address 10.208.55.126 255.255.255.192
no ip redirects
no ip unreachables
no ip proxy-arp
!
File End
and I want to create a txt file where if in source file I am getting a pattern vrf forwarding ABC output should be interface Vlan101
as of now what I have done following script but it showing only the line which contains the pattern.
import re
f = open("output_file.txt","w") #output file to be generated
shakes = open("input_file.txt","r") #input file to read
for lines in shakes:
if re.match("(.*)ABC(.*)",lines):
f.write(lines)
f.close()

Easiest: read the file, cut where ! is, then for each of those, if there's the desired text, get the first line:
with open("input_file.txt") as r, open("output_file.txt", "w") as w:
txt = r.read()
result = [block.strip().split("\n")[0]
for block in txt.split('!')
if 'vrf forwarding ABC' in block]
w.write("\n".join(result))

Just to be clear, I imagine that you want to replace any instances of "interface Vlan101" with "vrf forwarding ABC". In this case, I had test.txt as the input file and out.txt as the output file with all the replaced instances as was needed. I used a list comprehension--with a list string method-- to replace the substrings of "interface Vlan101" with "vrf forwarding ABC".
with open("test.txt") as f:
lines = f.readlines()
new_lines = [line.replace("interface Vlan101", "vrf forwarding ABC" for line in lines]
with open("out.txt", "w") as f1:
f1.writelines(new_lines)
Hope this helps.

If you are just interested in the interface, you can do following as well.
#Read File
with open('sample.txt', 'r') as f:
lines = f.readlines()
#Capture 'interfaces'
interfaces = [i for i in lines if i.strip().startswith('inter')]
#Write it to a file
with open('output.txt', 'w') as f:
f.writelines(interfaces)

With your code you are going through the document line by line.
If you want to parse blocks (between "!"-signs) you could split the blocks into lines first (though if it's a really large document, you may need to consider something else as this will read the entire document into memory)
import re
f = open("output_file.txt","w") #output file to be generated
source = open("input_file.txt","r") #input file to read
lines = "".join(source) #creates a string from the document
shakes = lines.replace("\n","").replace("! ","\n")
# remove all newlines and create new ones from "!"-block delimiter
# retrieve all text before "vrf forwarding ABC"
finds = re.findall("(.*)vrf forwarding ABC",shakes)
# return start of line
# if the part you want is the same length in all,
# then you could use find[:17] instead of
# find to get only the beginning. otherwise you need to modify your
# regex to only take the first 2 words of the line.
for find in finds:
f.write(find)
f.close()
Alternatively, if you want to use match per line, you can do the same as above, however instead of replacing "!" with new line, you can just split it, and then use the previous code and go line by line.
Hope this helps!

Python counter to text file

So im trying to analyse a log file and extract information from it.
One of the things im trying to do is extract a list of IP addresses that have more than 30 failed attempts. In this a failed attempt is one that starts with the line failed password for.
I have an idea for this that i wanted to try as i wasn't sure whether it will work.
If i use python to create a counter that looks for the keyword failed that i total and print out
This is what i have so far
failed_line=0
with open('blacklisttips.txt') as f2:
lines= f1.readlines()
for i, line in enumerate (lines):
if line.startswith(failed_line):
f2.write(line)
f2.write(lines[i+1])

So let's say your file looks like this:
failed password for 192.168.1.1
failed password for 192.168.1.2
...
more similar lines
import collections
prefix = failed password for
with open('path/to/file') as infile:
counts = collections.Counter(line.rsplit(" ",1)[1] for line in infile)

Unable to separate file names with file contents in Python socket programming

I'm trying to send data from the client's side to the server's side using TCP socket programming.
What I did is I read in the file names in the client's directory and then send the file name to the server's side by sending clientSocket.send("FILE "+fileToTransfer + "\n"). Then on the server's side, I use regex to get the file name out.
However, the client will always send the "FILE fileName.txt" and the file's contents together. So I suppose at the server's side, I will have to use regex to separate the file name with the file's contents.
So what I did at the server's side is to use getFileName = re.match(r'FILE (.*)(\n)(.*)',data) to get the file name and its contents separately. Unfortunately, (.*) does not include line breaks.
In that case, how do I separate the file contents with the file name? Is there a way to get the client's side to send the file name first then wait for server's side to get the file name before the file contents can be sent over? Or is there a regex which I can use so that I can separate the file name and the file contents?

You can send the file size along with the filename. This allows the server side to know how many bytes it should read. And in this case you don't need to read the entire file content into memory, you can read it chunk-by-chunk until the file size is exhausted (zero-ed out) and write chunks to disk. Something like this:
## client side
# get file size here, for example:
# filesize = os.path.getfilesize(filepath)
sock.sendall("FILE %s %d\n" % (filename, filesize))
sock.sendall(fd.read())
...
## server side
# error handling is left out
header = ""
while True:
d = sock.recv(1)
if d == '\n':
break
header += d
filesize = int(header.split()[-1])
# or search for the last space in header
# and get a substring of header as filename
filename = "".join(header.split()[1:-1])
data = ""
while filesize > 0:
chunk = sock.recv(1024) # or any amount of data
filesize -= chunk
data += chunk
Or you can give your regex up and just find the first \n:
## client side
sock.sendall("FILE %s\n")
sock.sendall(fd.read())
...
## server side
# data = read data here
newline = data.find('\n')
assert newline != -1 # some error handling here
header = data[newline]
filename = header[len("FILE "):]
content = data[newline+1:]

If you just want to fix the regex, change the line to:
getFileName = re.match(r'FILE (.*?)(\n)(.*)', data, re.DOTALL)
The DOTALL flag makes the . math newlines as well. The extra ? I added makes the * multiplier non-greedy, i.e., it'll stop at the first newline it sees (I assume newlines cannot be a part of the file name).
What you should probably do is send the file name as part of a header or something.

process large text file in python

I have a very large file (3.8G) that is an extract of users from a system at my school. I need to reprocess that file so that it just contains their ID and email address, comma separated.
I have very little experience with this and would like to use it as a learning exercise for Python.
The file has entries that look like this:
dn: uid=123456789012345,ou=Students,o=system.edu,o=system
LoginId: 0099886
mail: fflintstone#system.edu
dn: uid=543210987654321,ou=Students,o=system.edu,o=system
LoginId: 0083156
mail: brubble#system.edu
I am trying to get a file that looks like:
0099886,fflintstone#system.edu
0083156,brubble#system.edu
Any tips or code?

That actually looks like an LDIF file to me. The python-ldap library has a pure-Python LDIF handling library that could help if your file possesses some of the nasty gotchas possible in LDIF, e.g. Base64-encoded values, entry folding, etc.
You could use it like so:
import csv
import ldif
class ParseRecords(ldif.LDIFParser):
def __init__(self, csv_writer):
self.csv_writer = csv_writer
def handle(self, dn, entry):
self.csv_writer.writerow([entry['LoginId'], entry['mail']])
with open('/path/to/large_file') as input, with open('output_file', 'wb') as output:
csv_writer = csv.writer(output)
csv_writer.writerow(['LoginId', 'Mail'])
ParseRecords(input, csv_writer).parse()
Edit
So to extract from a live LDAP directory, using the python-ldap library you would want to do something like this:
import csv
import ldap
con = ldap.initialize('ldap://server.fqdn.system.edu')
# if you're LDAP directory requires authentication
# con.bind_s(username, password)
try:
with open('output_file', 'wb') as output:
csv_writer = csv.writer(output)
csv_writer.writerow(['LoginId', 'Mail'])
for dn, attrs in con.search_s('ou=Students,o=system.edu,o=system', ldap.SCOPE_SUBTREE, attrlist = ['LoginId','mail']:
csv_writer.writerow([attrs['LoginId'], attrs['mail']])
finally:
# even if you don't have credentials, it's usually good to unbind
con.unbind_s()
It's probably worthwhile reading through the documentation for the ldap module, especially the example.
Note that in the example above, I completely skipped supplying a filter, which you would probably want to do in production. A filter in LDAP is similar to the WHERE clause in a SQL statement; it restricts what objects are returned. Microsoft actually has a good guide on LDAP filters. The canonical reference for LDAP filters is RFC 4515.
Similarly, if there are potentially several thousand entries even after applying an appropriate filter, you may need to look into the LDAP paging control, though using that would, again, make the example more complex. Hopefully that's enough to get you started, but if anything comes up, feel free to ask or open a new question.
Good luck.

Assuming that the structure of each entry will always be the same, just do something like this:
import csv
# Open the file
f = open("/path/to/large.file", "r")
# Create an output file
output_file = open("/desired/path/to/final/file", "w")
# Use the CSV module to make use of existing functionality.
final_file = csv.writer(output_file)
# Write the header row - can be skipped if headers not needed.
final_file.writerow(["LoginID","EmailAddress"])
# Set up our temporary cache for a user
current_user = []
# Iterate over the large file
# Note that we are avoiding loading the entire file into memory
for line in f:
if line.startswith("LoginID"):
current_user.append(line[9:].strip())
# If more information is desired, simply add it to the conditions here
# (additional elif's should do)
# and add it to the current user.
elif line.startswith("mail"):
current_user.append(line[6:].strip())
# Once you know you have reached the end of a user entry
# write the row to the final file
# and clear your temporary list.
final_file.writerow(current_user)
current_user = []
# Skip lines that aren't interesting.
else:
continue

Again assuming your file is well-formed:
with open(inputfilename) as inputfile, with open(outputfilename) as outputfile:
mail = loginid = ''
for line in inputfile:
line = inputfile.split(':')
if line[0] not in ('LoginId', 'mail'):
continue
if line[0] == 'LoginId':
loginid = line[1].strip()
if line[0] == 'mail':
mail = line[1].strip()
if mail and loginid:
output.write(loginid + ',' + mail + '\n')
mail = loginid = ''
Essentially equivalent to the other methods.

To open the file you'll want to use something like the with keyword to ensure it closes properly even if something goes wrong:
with open(<your_file>, "r") as f:
# Do stuff
As for actually parsing out that information, I'd recommend building a dictionary of ID email pairs. You'll also need a variable for the uid and the email.
data = {}
uid = 0
email = ""
To actually parse through the file (the stuff run while your file is open) you can do something like this:
for line in f:
if "uid=" in line:
# Parse the user id out by grabbing the substring between the first = and ,
uid = line[line.find("=")+1:line.find(",")]
elif "mail:" in line:
# Parse the email out by grabbing everything from the : to the end (removing the newline character)
email = line[line.find(": ")+2:-1]
# Given the formatting you've provided, this comes second so we can make an entry into the dict here
data[uid] = email
Using the CSV writer (remember to import csv at the beginning of the file) we can output like this:
writer = csv.writer(<filename>)
writer.writerow("User, Email")
for id, mail in data.iteritems:
writer.writerow(id + "," + mail)
Another option is to open the writer before the file, write the header, then read the lines from the file at the same time as writing to the CSV. This avoids dumping the information into memory, which might be highly desirable. So putting it all together we get
writer = csv.writer(<filename>)
writer.writerow("User, Email")
with open(<your_file>, "r") as f:
for line in f:
if "uid=" in line:
# Parse the user id out by grabbing the substring between the first = and ,
uid = line[line.find("=")+1:line.find(",")]
elif "mail:" in line:
# Parse the email out by grabbing everything from the : to the end (removing the newline character)
email = line[line.find(": ")+2:-1]
# Given the formatting you've provided, this comes second so we can make an entry into the dict here
writer.writerow(iid + "," + email)