So, I'll explain briefly my idea, then, what I've tried and errors that I've got so far.
I want to make a Python script that will:
Search for files in a directory, example: /home/mystuff/logs
If he found it, he will execute a command like print('Errors found'), and then stop.
If not, he will keep it executing on and on.
But other logs will be there, so, my intention is to make Python read logs in /home/mystuff/logs filtering by the current date/time only.. since I want it to be executed every 2 minutes.
Here is my code:
import time
import os
from time import sleep
infile = r"/home/mystuff/logs`date +%Y-%m-%d`*"
keep_phrases = ["Error",
"Lost Connection"]
while True:
with open(infile) as f:
f = f.readlines()
if phrase in f:
cmd = ['#print something']
erro = 1
else:
sleep(1)
I've searched for few regex cases for current date, but nothing related to files that will keep changing names according by the date/time.. do you have any ideas?
You can't use shell features like command substitutions in file names. To the OS, and to Python, a file name is just a string. But you can easily create a string which contains the current date and time.
from datetime import datetime
infile = r"/home/mystuff/logs%s" % datetime.now().strftime('%Y-%m-%d')
(The raw string doesn't do anything useful, because the string doesn't contain any backslashes. But it's harmless, so I left it in.)
You also can't open a wildcard; but you can expand it to a list of actual file names with glob.glob(), and loop over the result.
from glob import glob
for file in glob(infile + '*'):
with open(file, 'r') as f:
# ...
If you are using a while True: loop you need to calculate today's date inside the loop; otherwise you will be perpetually checking for files from the time when the script was started.
In summary, your changed script could look something like this. I have changed the infile variable name here because it isn't actually a file or a file name, and fixed a few other errors in your code.
# Unused imports
# import time
# import os
from datetime import datetime
from glob import glob
from time import sleep
keep_phrases = ["Error",
"Lost Connection"]
while True:
pattern = "/home/mystuff/logs%s*" % datetime.now().strftime('%Y-%m-%d')
for file in glob(pattern):
with open(file) as f:
for line in f:
if any(phrase in line for phrase in keep_phrases):
cmd = ['#print something']
erro = 1
break
sleep(120)
Related
I've got a script that is intended to sort my photo/video collection (Windows). The photos work fine as they are sortable by EXIF which is easily accessed.
Videos are harder because I have to get the file's "Media Creation Date" which is readable by only pywin32, to my understanding. However, once I've accessed the media creation date, shutil.move() does not work. It throws no error, it just runs indefinitely without progress until I manually kill the script:
Here's the snippet in question:
from datetime import datetime
import exifread
import os
from pathlib import Path
import shutil
from win32com.propsys import propsys, pscon
# get the file list, do stuff with photos, etc
# f is the file
# cr is the path root to which it will be moved
elif str(f).lower().endswith(("mp4", "mov")):
props = propsys.SHGetPropertyStoreFromParsingName(f)
dt = props.GetValue(pscon.PKEY_Media_DateEncoded).GetValue()
year, month = str(dt.year), str(dt.month).zfill(2)
new_fn = dt.strftime("%Y-%m-%d_%H%M%S")
new_fn = f"{new_fn}{os.path.splitext(f)[1]}"
move_path = os.path.join(cr, year, month, new_fn)
print(f"SRC: {f}")
print(f"DESTINATION: {move_path}")
print("----------------------------------")
shutil.move(f, move_path)
It prints the source correctly, and the destination correctly, but does not move the file. I have also tried os.rename() and os.replace() with the same result, which suggests that perhaps the propsys method still has a lock on the file? How do I free up this file for moving?
Yes, propsys is blocking the file (you can check in Process Explorer), try just deleting it:
fpath = r'c:\temp\user\t\test.mp4'
move_path = r'c:\temp\user\t\test moved.mp4'
props = propsys.SHGetPropertyStoreFromParsingName(fpath)
print( props.GetValue(pscon.PKEY_Media_DateEncoded).GetValue() )
del props
shutil.move(fpath , move_path)
I'm trying to have my Python code write everything it does to a log, with a timestamp. But it doesn't seem to work.
this is my current code:
filePath= Path('.')
time=datetime.datetime.now()
bot_log = ["","Set up the file path thingy"]
with open ('bot.log', 'a') as f:
f.write('\n'.join(bot_log)%
datetime.datetime.now().strftime("%d-%b-%Y (%H:%M:%S.%f)"))
print(bot_log[0])
but when I run it it says:
Traceback (most recent call last):
File "c:\Users\Name\Yuna-Discord-Bot\Yuna Discord Bot.py", line 15, in <module>
f.write('\n'.join(bot_log)%
TypeError: not all arguments converted during string formatting
I have tried multiple things to fix it, and this is the latest one. is there something I'm doing wrong or missing? I also want the time to be in front of the log message, but I don't think it would do that (if it worked).
You need to put "%s" somewhere in the input string before string formatting. Here's more detailed explanation.
Try this:
filePath= Path('.')
time=datetime.datetime.now()
bot_log = "%s Set up the file path thingy\n"
with open ('bot.log', 'a') as f:
f.write(bot_log % datetime.datetime.now().strftime("%d-%b-%Y (%H:%M:%S.%f)"))
print(bot_log)
It looks like you want to write three strings to your file as separate lines. I've rearranged your code to create a single list to pass to writelines, which expects an iterable:
filePath= Path('.')
time=datetime.datetime.now()
bot_log = ["","Set up the file path thingy"]
with open ('bot.log', 'a') as f:
bot_log.append(datetime.datetime.now().strftime("%d-%b-%Y (%H:%M:%S.%f)"))
f.writelines('\n'.join(bot_log))
print(bot_log[0])
EDIT: From the comments the desire is to prepend the timestamp to the message and keep it on the same line. I've used f-strings as I prefer the clarity they provide:
import datetime
from pathlib import Path
filePath = Path('.')
with open('bot.log', 'a') as f:
time = datetime.datetime.now()
msg = "Set up the file path thingy"
f.write(f"""{time.strftime("%d-%b-%Y (%H:%M:%S.%f)")} {msg}\n""")
You could also look at the logging module which does a lot of this for you.
I'm trying to monitor a CSV file that is being written to by a separate program. Around every 10 seconds, the CSV file is updated with a couple more lines. Each time the file is updated, I want to be able to detect the file has been changed (will always be the same file), take the new lines, and write them to console (just for a test).
I have looked around the website, and have found numerous ways of watching a file to see if its updated (like so http://thepythoncorner.com/dev/how-to-create-a-watchdog-in-python-to-look-for-filesystem-changes/), but I can't seem to find anything that will allow me to get to the changes made in the file to print out to console.
Current code:
import time
from watchdog.observers import Observer
from watchdog.events import PatternMatchingEventHandler
def on_created(event):
print(f"hey, {event.src_path} has been created!")
def on_deleted(event):
print(f"Someone deleted {event.src_path}!")
def on_modified(event):
print(f"{event.src_path} has been modified")
def on_moved(event):
print(f"ok ok ok, someone moved {event.src_path} to {event.dest_path}")
if __name__ == "__main__":
patterns = "*"
ignore_patterns = ""
ignore_directories = False
case_sensitive = True
my_event_handler = PatternMatchingEventHandler(patterns, ignore_patterns, ignore_directories, case_sensitive)
my_event_handler.on_created = on_created
my_event_handler.on_deleted = on_deleted
my_event_handler.on_modified = on_modified
my_event_handler.on_moved = on_moved
path = "."
go_recursively = True
my_observer = Observer()
my_observer.schedule(my_event_handler, path, recursive=go_recursively)
my_observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
my_observer.stop()
my_observer.join()
This runs, but looks for changes in files all over the place. How do I make it listen for changes from one single file?
If you're more or less happy with the script other than it tracking a bunch of files then you could change the patterns = "*" part which is a wildcard matching string which tells the PatternMatchingEventHandler to look for any file. You could change that to paterns = 'my_file.csv' and also change the path variable to the directory that the file is in to save some time recursively scanning all the directories in '.'. Then you don't need recursive set to True for a single file either.
Print new lines to console part (one option):
import pandas as pd
...
def on_modified(event):
print(f"{event.src_path} has been modified")
# You said "a couple more lines" I'm going to take that
# as two:
df = pd.read_csv(event.src_path)
print("Newest 2 lines:")
print(df[-2:])
If it's not two lines you'll want to track the length of the file and pass that to the function which opens the CSV so it knows how many lines are new.
I believe since this is a CSV file, reading file using pandas and checking the file size can help. You can use df.tail(2) to print last two rows after reading the csv using pandas
I am scraping info to a text file and am trying to write the date at the top. I have the method to grab the date but have no clue how I can use the write function to place at top. Below is a stripped down version of what I am working on.
import re
import urllib2
import json
from datetime import datetime
import time
now = datetime.now()
InputDate = now.strftime("%Y-%m-%d")
Today = now.strftime("%B %d")
header = ("Today").split()
newfile = open("File.txt", "w")
### Irrelevant Info Here ###
string = title"\n"+info+"\n"
#newfile.write(header)
newfile.write(string)
print title+" written to file"
newfile.close()
You can't insert something at the beginning of a file. You need to write a new file, starting with the line you want to insert, then finish with the contents of the old file. Unlike appending to the end, writing to the start of the file is really, really inefficient
The key to this problem is to use a NamedTemporaryFile. After you finish constructing it, you then rename it on top of the old file.
Code:
def insert_timestamp_in_file(filename):
with open(filename) as src, tempfile.NamedTemporaryFile(
'w', dir=os.path.dirname(filename), delete=False) as dst:
# Save the new first line
dst.write(dt.datetime.now().strftime("%Y-%m-%d\n"))
# Copy the rest of the file
shutil.copyfileobj(src, dst)
# remove old version
os.unlink(filename)
# rename new version
os.rename(dst.name, filename)
Test Code:
import datetime as dt
import tempfile
import shutil
insert_timestamp_in_file("file1")
file1
I am scraping info to a text file and am trying to write the date at
the top. I have the method to grab the date but have no clue how I can
use the write function to place at top. Been trying for 2 days and all.
Results:
2018-02-15
I am scraping info to a text file and am trying to write the date at
the top. I have the method to grab the date but have no clue how I can
use the write function to place at top. Been trying for 2 days and all.
To write the date to the 'top' of the file you would want to put:
newfile.write(InputDate)
newfile.write(Today)
after where you open the file and before anything else.
Just to give you idea
Try this:-
import re
import urllib2
import json
from datetime import datetime
import time
now = datetime.now()
InputDate = now.strftime("%Y-%m-%d")
Today = now.strftime("%B %d")
#start writing from here
newfile = open("File.txt", "a")
newfile.write(InputDate+"\n")
newfile.write("hello Buddy")
newfile.close()
Simple One will be, if you will not call it as a str then it will throw an error TypeError: write() argument must be str, not list
I have rfreshed teh code to be more precise and effective use..
import re
from datetime import datetime
import time
now = datetime.now()
InputDate = now.strftime("%B"+" "+"%Y-%m-%d")
newfile = open("File.txt", "a")
string = "Hi trying to add a datetime at the top of the file"+"\n"
newfile.write(str(InputDate+"\n"))
newfile.write(string)
newfile.close()
Result will be:
February 152018-02-15
Hi trying to add a datetime at the top of the file
Using python 2.7..
I am using below to send all print output to a file called output.log. How can i have this send to a different file each time it runs...In bash we could declare a variable called date or something and have that part of the file name...so how can i achieve the same with python ??
So my question is..
every time i run the below script, my file should have a naming convention of output_date/time.log
Also how can i delete file that are older than X days that have a file naming convention of output_*.log
import sys
f = open('output.log', 'w')
sys.stdout = f
print "test"
f.close()
with some personal preference of formatting this is generally what I do.
import time
moment=time.strftime("%Y-%b-%d__%H_%M_%S",time.localtime())
f = open('output'+moment+'.log', 'w')
as far as automated deleting, do you want it deleted on run of the test?
os.remove(fileStringName)
works, you just have to do the arithmetic and string conversion. I would use os.listdir(pathToDirWithOutputLogs) iterate through the file names and do the math on them and call os.remove on the old ones.
To get date/time:
from time import gmtime, strftime
outputFileName = "output_#.log"
outputFileName = outputFileName.replace("#", strftime("%Y-%m-%d_%H:%M:%S", gmtime()))
For numerical incrementing:
outputFileName = "output #.log"
outputVersion = 1
while os.path.isfile(outputFileName.replace("#", str(outputVersion))):
outputVersion += 1
outputFileName = outputFileName.replace("#", str(outputVersion))
To delete files older than a certain date, you can iterate through all the files in the directory with ``, and delete them with os.remove(). You can compare the file names after parsing them.
lastTime = "2015-08-03_19:04:41"
for fn in filter(os.path.isfile, os.listdir()):
strtime = fn[fn.find("_"):fn.rfind(".")]
if strtime < lastTime:
os.remove(fn)