I am fresh meat in the coding world and am diving head-first into python as we speak, so I apologize for such a rookie question in advance. But basically, I got assigned this log parsing project for school and found a seemingly nice parsing script/template online which I thought could be great for me to use. However, after looking at some of the syntax, it became apparent that it was written in an older version of python. Since I only started learning python a few weeks ago, I am not confident in understanding what syntax needs to change, if any, and if it is properly constructed like I initially assumed. My main question is, after viewing the script below, what statements, methods, variables, functions, ect., do you think must be (or should be) changed in order for me to get the ball rolling?
I'm running Win10 and using Spyder (from Conda) with python version 3.7.
I'd post the website link to this code but I don't know the rules on posting links in questions! However, if you search for 'pythonicways' you'll find it, as well as a walk-through guide for why it was made the way it was by the creator.
import time
from time import strftime
def main():
log_file_path = r"C:\ios logs\sfbios.log"
export_file_path = r"C:\ios logs\filtered"
time_now = str(strftime("%Y-%m-%d %H-%M-%S", time.localtime()))
file = "\\" + "Parser Output " + time_now + ".txt"
export_file = export_file_path + file
regex = '(<property name="(.*?)">(.*?)<\/property>)'
parseData(log_file_path, export_file, regex, read_line=True)
def parseData(log_file_path, export_file, regex, read_line=True):
with open(log_file_path, "r") as file:
match_list = []
if read_line == True:
for line in file:
for match in re.finditer(regex, line, re.S):
match_text = match.group()
match_list.append(match_text)
print match_text
else:
data = file.read()
for match in re.finditer(regex, data, re.S):
match_text = match.group();
match_list.append(match_text)
file.close()
with open(export_file, "w+") as file:
file.write("EXPORTED DATA:\n")
match_list_clean = list(set(match_list))
for item in xrange(0, len(match_list_clean)):
print match_list_clean[item]
file.write(match_list_clean[item] + "\n")
file.close()
if __name__ == '__main__':
main()
I'm not expecting anything in particular, per se. I just want to know what changes must be or should be made to this template/script. My overall goal is to open & read a .log file, parse it for several various capture groups using regex, store them all in some variables or some sequence type, and then extract that information onto a new .txt file (preferably in CSV format, or some other tabular delimited format)
You should be able to just change the print lines to get this going.
# print match_text
print(match_text)
#####
#print match_list_clean[item]
print(match_list_clean[item])
Also, try running it first next time. Get used to reading traceback (error messages). You can learn a lot from them.
This is Python 2. Use the standard library module lib2to3 to convert it to Python 3. That will be far better than trying to update the code manually, particularly as you aren't proficient in either version of Python.
I hope that I can ask this in a clear way, im very much a beginner to python and forums in general so I apologise if i've got anything wrong from the start!
My issue is that I am currently trying to use os.system() to enable a program to run on every file within a directory (this is a directory of ASCII tables which I am crossing with a series of other tables to find matches.
import os
for filename in os.listdir('.'):
os.system('stilts tmatch2 ifmt1=ascii ifmt2=ascii in1=intern in2= %s matcher=2d values1='col1 col2' values2='col1 col2' params=5 out= %s-table.fits'%(filename,filename))
So what im hoping this would do is for every 'filename' it would operate this program known as stilts. Im guessing this gets interrupted/doesn't work because of the presence of apostrophes ' in the line of code itself, which must disrupt the syntax? (please correct me if I am wrong)
I then replaced the ' in os.system() with "" instead. This, however, stops me using the %s notation to refer to filenames throughout the code (at least I am pretty sure anyway).
import os
for filename in os.listdir('.'):
os.system("stilts tmatch2 ifmt1=ascii ifmt2=ascii in1=intern in2= %s matcher=2d values1='col1 col2' values2='col1 col2' params=5 out= %s-table.fits"%(filename,filename))
This now runs but obviously doesn't work, as it inteferes with the %s input.
Any ideas how I can go about fixing this? are there any alternative ways to refer to all of the other files given by 'filename' without using %s?
Thanks in advance and again, sorry for my inexperience with both coding and using this forum!
I am not familiar with os.system() but maybe if you try do some changes about the string you are sending to that method before it could behave differently.
You must know that in python you can "sum" strings so you can save your commands in a variable and add the filenames as in:
os.system(commands+filename+othercommands+filename)
other problem that could be working is that when using:
for file in os.listdir()
you may be recievin file types instead of the strings of their names. Try using a method such as filename.name to check if this is a different type of thing.
Sorry I cant test my answers for you but the computer I am using is too slow for me to try downloading python.
I am making a program that records what a user does in a user friendly Log File, which is stored as a .txt. So far, it just records the beginning of the session and the end, which is the exact same time. However, currently when I run the program, it removes the prior content of the file. I believe it is due to the fact that the cursor is seeking to point (0, 0). Is there a way to get to the end of the file? The code is as follows.
import atexit
import datetime
def logSession():
sessionRecord = open(r'C:\Users\Aanand Kainth\Desktop\Python Files\Login Program\SessionRecords.txt', "r+")
sessionRecord.seek(-1,-1)
sessionRecord.write("Session Began: " '{:%Y-%m-%d | %H:%M:%S}'.format(datetime.datetime.now()) + "\n")
def runOnQuit():
sessionRecord = open(r'C:\Users\Aanand Kainth\Desktop\Python Files\Login Program\SessionRecords.txt', "r+")
sessionRecord.seek(-1,-1)
sessionRecord.write("Session Ended: " '{:%Y-%m-%d | %H:%M:%S}'.format(datetime.datetime.now()) + """\n-------------------------------------------------------------------------------------------\n""")
logSession()
atexit.register(runOnQuit)
If you create the necessary files on your computer, and run this .PY file, you will find that sometimes it works as desired, and other times it doesn't. Can anyone tell me how to seek to the end, or if there is a better solution to help me? This is a self-assigned challenge, so I don't appreciate giveaways, and just hints.
Your parameters to seek are incorrect, seek takes 2 parameters, an offset (how many bytes) and whence which describes from where to seek (start:0, current:1, end:2). So what you need is:
import io
sessionRecord.seek(0, io.SEEK_END)
Or:
sessionRecord.seek(0, 2) # Py2
But you can avoid doing this by opening the file in append mode:
with open('<filename>', 'a'):
sessionRecord.write(...)
Just open the file in the proper mode. If you want to add to a file without deleting its previous contents, use 'a' (for "append") instead of 'r+', which is a truncating read/write. You will then no longer need to use seek().
I have written a few lines of code in Python to see if I can make it read a text file, make a list out of it where the lines are lists themselves, and then turn everything back into a string and write it as output on a different file. This may sound silly, but the idea is to shuffle the items once they are listed, and I need to make sure I can do the reading and writing correctly first. This is the code:
import csv,StringIO
datalist = open('tmp/lista.txt', 'r')
leyendo = datalist.read()
separando = csv.reader(StringIO.StringIO(leyendo), delimiter = '\t')
macrolist = list(separando)
almosthere = ('\t'.join(i) for i in macrolist)
justonemore = list(almosthere)
arewedoneyet = '\n'.join(justonemore)
with open('tmp/randolista.txt', 'w') as newdoc:
newdoc.write(arewedoneyet)
newdoc.close()
datalist.close()
This seems to work just fine when I run it line by line on the interpreter, but when I save it as a separate Python script and run it (myscript.py) nothing happens. The output file is not even created. After having a look at similar issues raised here, I have introduced the 'with' parameter (before I opened the output file through output = open()), I have tried flushing as well as closing the file... Nothing seems to work. The standalone script does not seem to do much, but the code can't be too wrong if it works on the interpreter, right?
Thanks in advance!
P.S.: I'm new to Python and fairly new to programming, so I apologise if this is due to a shallow understanding of a basic issue.
Where are the input file and where do you want to save the output file. For this kind of scripts i think that it's better use absolute paths
Use:
open('/tmp/lista.txt', 'r')
instead of:
open('tmp/lista.txt', 'r')
I think that the error can be related to this
It may have something to do with where you start your interpreter.
Try use a absolute path /tmp/randolista.txt instead of relative path tmp/randolista.txt to isolate the problem.
I'm doing Zed Shaw's fantastic Learn Python The Hard Way, but an extra question has me stumped: Line 9--10 could be written in one line, how? I've tried some different thoughts, but to no avail. I could move on, but what would the fun in that be?
from sys import argv
from os.path import exists
script, from_file, to_file = argv
print "Copying from %s to %s" % (from_file, to_file)
# we could do these two on one line too, how?
input = open(from_file)
indata = input.read()
print "The input file is %d bytes long" % len(indata)
print "Does the output file exist? %r" % exists(to_file)
print "Ready, hit RETURN to continue, CTRL-C to abort."
raw_input()
output = open(to_file, 'w')
output.write(indata)
print "Alright, all done."
Zed also writes that he could do the whole script in one line. I'm not exactly sure what he means by that.
Feel free to help me however you want: by giving the answer or merely hinting---and perhaps including a collapsed or hidden answer to the question.
indata = open(from_file).read()
shutil is the way to do one-liner file copies in Python:
shutil.copy(sys.argv[1], sys.argv[2])
Putting the import shutil, sys on the same line as this one (with a semicolon in-between, of course) would however be stylistically goofy;-).
Well you can just do "algebraic substitution," right? ...assuming you don't care about the "UI"...
open(to_file, 'w').write(open(from_file).read())
from sys import argv
open(argv[2], 'w').write(open(argv[1]).read())
I agree with the algebraic substitution mentioned by #dash-tom-bang.
My functioning Exercise 17 extra credit has 5 lines. The operation is being conducted on one line.
open(to_file, 'w').write(open(from_file).read())
followed by a simple 'print' for verification feedback
print "File %s copied to %s" % (from_file, to_file)
I should have a 6th line that replaces the original ''output.close'' but I am confused about how to do this without the ''output'' variable? Ahh, since I now have no output variable there is nothing to close.
btw- It is a little spooky for me to see the same exact line typed here that I have worked out and entered myself in gedit. Great stuff, I am really enjoying the mental challenge and community support.
Edit:answered my own question
try the following code:
import shutil, sys; shutil.copy(sys.argv[0], sys.argv[2])
output = open(to_file, 'w')
output.write(indata)
can be written as
open(to_file, 'w').write(indata)
Had some fun with this one. In answering this I was looking for a solution that preserved the functionality, used the file commands the exercise was about and did not use ";" to combine lines. Closest I could come is
open(input("Out?: "), 'w').write(open(input("In?: ")).read())
Not quite the same functionality as it prompts for input rather than taking command line. But a single line that gets the job done while using the file commands in the exercise and avoiding concatenating lines using semi colon.
He answers this below in the section "Common Student Questions":
No way you can make this one line!
That ; depends ; on ; how ; you ; define ; one ; line ; of ; code.
Hey Kiwi (and whomever else finds this!),
I'm on the same exercise and I believe I've cracked it.
There are two possible readings of Shaw's "I could make this one line long" tease.
He could make the Python script one line long, upon importing all the necessary commands from the modules, e.g from sys import argv, etc.
He could copy the contents of one file to another in one line using the command line.
I personally think he means the latter, but I will explain both solutions for the sake of learning!
The first (Long) solution:
You must acknowledge that you require the importx from y lines in the Python file, otherwise argv and exist won't work because they will only have been implicitly referenced, i.e. you haven't made it clear to Python that you want to use these functions.
The next thing to do is delete all irrelevant code, with irrelevant being code that is written for the benefit of the user, i.e. print, raw_input(), len(), etc.
If you do this, you will be left with:
from sys import argv
from os.path import exists
script, from_file, to_file = argv
indata = open(from_file).read()
out_file = open(to_file, 'w')
out_file.write(indata)
out_file.close()
in_file.close()
To make this even shorter, you can begin nesting the variables and function in one another. This is the same principle as in maths when you could define a function and then substitute the variable representing that function into another function.
For example:
y = x + 3
z = y, which is essentially z = (x + 3)
If you work this through, you can simplify the code down to:
from sys import argv
from os.path import exists
script, from_file, to_file = argv
(open(to_file, 'w').write(open(from_file).read()))
You can then use lots of ; to link up all the lines of code and vio-la you're done.
Note: You don't need to close the files, as you did in the original, as Python will automatically close them upon executing the script.
The second (Short) solution:
If you look at his 'What You Should See' section, he uses cat in the terminal. This is short for concatenation, which is a means of connecting strings together. If you combine it with > you can overwrite the contents of one file with another in one line:
cat from_file.txt > to_file.txt
That's it. One line that will take the contents of one file and put it into another.
Of course, both solutions aren't perfect, as the first isn't truly one line and the second doesn't even use Python!
Feedback appreciated, I only started doing this two days ago...
Cutting away everything you don't need, all the 'features you don't need' as Zed puts it, and you get one line. It's even less than 80 characters in length, can't get much more 'pythonic' than that!
from sys import argv; s, f, t = argv; (open(t, 'w').write(open(f).read()))
All he is saying is that you can use a semicolon to put two lines onto one line and have it run
in_file = open(from_file); indata = in_file.read()
You could do that with the whole piece of code if you wanted to
I'm also doing the same book online. I tried this, and it worked:
open(to_file, 'w').write(open(from_file).read())
In other words, I opened the file I was copying to in the write mode and then inserted the contents of the file I was copying from, after opening and reading it, into the write function. I checked my files and it worked. Yay!
The below line worked for me:
open(to_file, 'w').write(open(from_file).read())
There was a hint within the Common Student Questions section:
"No way you can make this one line!
That ; depends ; on ; how ; you ; defi ne ; one ; line ; of ; code."
Note: This is clearly not best practice and difficult to read.
from sys import argv
script, from_file, to_file = argv
in_file = open(from_file); indata = in_file.read(); out_file = open(to_file, 'w'); out_file.write(indata); out_file.close(); in_file.close()
This the most reduced code that preserves the ui maybe there is other better solutions.
From 21 lines down to 8 lines of code.
Before:
from sys import argv
from os.path import exists
script, from_file, to_file = argv
print("Copying from %s to %s" % (from_file, to_file))
# we could do these two on one line too, how?
input = open(from_file)
indata = input.read()
print("The input file is %d bytes long" % len(indata))
print("Does the output file exist? %r" % exists(to_file))
print("Ready, hit RETURN to continue, CTRL-C to abort.")
#raw_input()
output = open(to_file, 'w')
output.write(indata)
print("Alright, all done.")
To:
from sys import argv
from os.path import exists
script, from_file, to_file = argv
output = open(to_file, 'w').write(open(from_file).read())
print("""Copying from %s to %s\nThe input file is %d bytes long\nDoes the output file exist? %r\nReady, hit RETURN to continue, CTRL-C to abort.\nAlright, all done.""" % (from_file, to_file, len(open(from_file).read()), exists(to_file)))