I am writing some code that will go through a file, edit it as a temp file, and than copy the temp file over the new file so as to make the edit. However when using the move method from shutil I keep getting this error :
IOError: [Errno 22] Invalid Argument
I've tried using copy, copy2, and copyfile. Here is a copy of the code :
def writePPS(seekValue,newData):
PPSFiles = findPPS("/pps")
for file in PPSFiles:
#create a temp file
holder,temp = mkstemp(".pps")
print holder, temp
pps = open(file,"r+")
newpps = open(temp,"w")
info = pps.readlines()
#parse through the pps file and find seekvalue, replace with newdata
for data in info:
valueBoundry = data.find(":")
if seekValue == data[0:(valueBoundry)]:
print "writing new value"
newValue = data[0:(valueBoundry+1)] + str(newData)
#write to our temp file
newpps.write(newValue)
else: newpps.write(data)
pps.close()
close(holder)
newpps.close()
#remove original file
remove(file)
#move temp file to pps
copy(temp,"/pps/ohm.pps")
I am not exactly sure why you are getting that error, but to start you could try cleaning up your code a bit and fixing all those import statements. Its hard to see where the functions are coming from and for all you know you could have a namespace collision eventually.
Lets start here with some actually runnable code:
import shutil
import os
import tempfile
def writePPS(seekValue,newData):
PPSFiles = findPPS("/pps")
for file_ in PPSFiles:
#create a temp file
newpps = tempfile.NamedTemporaryFile(suffix=".pps")
print newpps.name
with open(file_,"r+") as pps:
#parse through the pps file and find seekvalue, replace with newdata
for data in pps:
valueBoundry = data.find(":")
if seekValue == data[0:(valueBoundry)]:
print "writing new value"
newValue = data[0:(valueBoundry+1)] + str(newData)
#write to our temp file
newpps.write(newValue)
else:
newpps.write(data)
#move temp file to pps
newpps.flush()
shutil.copy(newpps.name,"/pps/ohm.pps")
You don't need to read all your lines into memory. You can just loop over each line. You also don't need to manage all those open/close file operations. Just use a with context and also a NamedTemporaryFile which will clean itself up when it is garbage collected.
Important note, that in your example and above, you are overwriting the same destination file each time for every source file. I left it that way for you to address. But if you start here, we can then begin to figure out why you are getting errors.
Related
I am writing a Python program where I need to write to a file. I need an if condition to determine if I need to keep writing to same file or open a new file. How do I declare the file so that I can access it with both the if and else? Right now I'm making a test file before the loop just so I have access to the variable. How to avoid opening a TEST.txt file while still having a variable f that I can operate on?
f = open(outputFolder + "TEST.txt", 'w') # how to avoid opening TEST.txt here
while row:
#print(str(row[0]) + '|' + str(row[4]))
currentFileName = getFileName(str(row[0]))
# If coming up on new date open new file
if currentFileName != fileName:
f.close()
fileName = currentFileName
print("Processing: " + fileName)
f = open(outputFolder + fileName, 'w')
f.write(getLine(row))
# else write to current file
else:
f.write(getLine(row))
row = cursor.fetchone()
You didn't work out your logic before writing the program. What you describe in words does not match what you wrote. Start with the words, draw a flowchart, and then write your code.
In your posted program, you're trying to open currentFile multiple times, you don't initialize or change row, and it's not clear what you intend the program to do.
if [condition]:
filename = currentFileName
else:
filename = "TEST.txt"
f = open(filename)
for ... # whatever you're trying to do with your input and output,
# look up the proper method in a file-handling tutorial.
I'm trying to monitor a CSV file that is being written to by a separate program. Around every 10 seconds, the CSV file is updated with a couple more lines. Each time the file is updated, I want to be able to detect the file has been changed (will always be the same file), take the new lines, and write them to console (just for a test).
I have looked around the website, and have found numerous ways of watching a file to see if its updated (like so http://thepythoncorner.com/dev/how-to-create-a-watchdog-in-python-to-look-for-filesystem-changes/), but I can't seem to find anything that will allow me to get to the changes made in the file to print out to console.
Current code:
import time
from watchdog.observers import Observer
from watchdog.events import PatternMatchingEventHandler
def on_created(event):
print(f"hey, {event.src_path} has been created!")
def on_deleted(event):
print(f"Someone deleted {event.src_path}!")
def on_modified(event):
print(f"{event.src_path} has been modified")
def on_moved(event):
print(f"ok ok ok, someone moved {event.src_path} to {event.dest_path}")
if __name__ == "__main__":
patterns = "*"
ignore_patterns = ""
ignore_directories = False
case_sensitive = True
my_event_handler = PatternMatchingEventHandler(patterns, ignore_patterns, ignore_directories, case_sensitive)
my_event_handler.on_created = on_created
my_event_handler.on_deleted = on_deleted
my_event_handler.on_modified = on_modified
my_event_handler.on_moved = on_moved
path = "."
go_recursively = True
my_observer = Observer()
my_observer.schedule(my_event_handler, path, recursive=go_recursively)
my_observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
my_observer.stop()
my_observer.join()
This runs, but looks for changes in files all over the place. How do I make it listen for changes from one single file?
If you're more or less happy with the script other than it tracking a bunch of files then you could change the patterns = "*" part which is a wildcard matching string which tells the PatternMatchingEventHandler to look for any file. You could change that to paterns = 'my_file.csv' and also change the path variable to the directory that the file is in to save some time recursively scanning all the directories in '.'. Then you don't need recursive set to True for a single file either.
Print new lines to console part (one option):
import pandas as pd
...
def on_modified(event):
print(f"{event.src_path} has been modified")
# You said "a couple more lines" I'm going to take that
# as two:
df = pd.read_csv(event.src_path)
print("Newest 2 lines:")
print(df[-2:])
If it's not two lines you'll want to track the length of the file and pass that to the function which opens the CSV so it knows how many lines are new.
I believe since this is a CSV file, reading file using pandas and checking the file size can help. You can use df.tail(2) to print last two rows after reading the csv using pandas
In my code, user uploads file which is saved on server and read using the server path. I'm trying to delete the file from that path after I'm done reading it. But it gives me following error instead:
An error occurred while reading file. [WinError 32] The process cannot access the file because it is being used by another process
I'm reading file using with, and I've tried f.close() and also f.closed but its the same error every time.
This is my code:
f = open(filePath)
with f:
line = f.readline().strip()
tempLst = line.split(fileSeparator)
if(len(lstHeader) != len(tempLst)):
headerErrorMsg = "invalid headers"
hjsonObj["Line No."] = 1
hjsonObj["Error Detail"] = headerErrorMsg
data['lstErrorData'].append(hjsonObj)
data["status"] = True
f.closed
return data
f.closed
after this code I call the remove function:
os.remove(filePath)
Edit: using with open(filePath) as f: and then trying to remove the file gives the same error.
Instead of:
f.closed
You need to say:
f.close()
closed is just a boolean property on the file object to indicate if the file is actually closed.
close() is method on the file object that actually closes the file.
Side note: attempting a file delete after closing a file handle is not 100% reliable. The file might still be getting scanned by the virus scanner or indexer. Or some other system hook is holding on to the file reference, etc... If the delete fails, wait a second and try again.
Use below code:
import os
os.startfile('your_file.py')
To delete after completion:
os.remove('your_file.py')
This
import os
path = 'path/to/file'
with open(path) as f:
for l in f:
print l,
os.remove(path)
should work, with statement will automatically close the file after the nested block of code
if it fails, File could be in use by some external factor. you can use Redo pattern.
while True:
try:
os.remove(path)
break
except:
time.sleep(1)
There is probably an application that is opening the file; check and close the application before executing your code:
os.remove(file_path)
Delete files that are not used by another application.
I modified the code based on the comments from experts in this thread. Now the script reads and writes all the individual files. The script reiterates, highlight and write the output. The current issue is, after highlighting the last instance of the search item, the script removes all the remaining contents after the last search instance in the output of each file.
Here is the modified code:
import os
import sys
import re
source = raw_input("Enter the source files path:")
listfiles = os.listdir(source)
for f in listfiles:
filepath = source+'\\'+f
infile = open(filepath, 'r+')
source_content = infile.read()
color = ('red')
regex = re.compile(r"(\b be \b)|(\b by \b)|(\b user \b)|(\bmay\b)|(\bmight\b)|(\bwill\b)|(\b's\b)|(\bdon't\b)|(\bdoesn't\b)|(\bwon't\b)|(\bsupport\b)|(\bcan't\b)|(\bkill\b)|(\betc\b)|(\b NA \b)|(\bfollow\b)|(\bhang\b)|(\bbelow\b)", re.I)
i = 0; output = ""
for m in regex.finditer(source_content):
output += "".join([source_content[i:m.start()],
"<strong><span style='color:%s'>" % color[0:],
source_content[m.start():m.end()],
"</span></strong>"])
i = m.end()
outfile = open(filepath, 'w+')
outfile.seek(0)
outfile.write(output)
print "\nProcess Completed!\n"
infile.close()
outfile.close()
raw_input()
The error message tells you what the error is:
No such file or directory: 'sample1.html'
Make sure the file exists. Or do a try statement to give it a default behavior.
The reason why you get that error is because the python script doesn't have any knowledge about where the files are located that you want to open.
You have to provide the file path to open it as I have done below. I have simply concatenated the source file path+'\\'+filename and saved the result in a variable named as filepath. Now simply use this variable to open a file in open().
import os
import sys
source = raw_input("Enter the source files path:")
listfiles = os.listdir(source)
for f in listfiles:
filepath = source+'\\'+f # This is the file path
infile = open(filepath, 'r')
Also there are couple of other problems with your code, if you want to open the file for both reading and writing then you have to use r+ mode. More over in case of Windows if you open a file using r+ mode then you may have to use file.seek() before file.write() to avoid an other issue. You can read the reason for using the file.seek() here.
I'm new to python and the following piece of code is driving me crazy. It lists the files in a directory and for each file does some stuff. I get a IOError: [Errno2] No such file or directory: my_file_that_is_actually_there!
def loadFile(aFile):
f_gz = gzip.open(aFile, 'rb')
data = f_gz.read()
#do some stuff...
f_gz.close()
return data
def main():
inputFolder = '../myFolder/'
for aFile in os.listdir(inputFolder):
data = loadFile(aFile)
#do some more stuff
The file exists and it's not corrupted. I do not understand how it's possible that python first finds the file when it checks the content of myFolder, and then it cannot find itanymore... This happens on the second iteration of my for loop only with any files.
NOTE: Why does this exception happen ONLY at the second iteration of the loop?? The first file in the folder is found and opened without any issues...
This is because open receives the local name (returned from os.listdir). It doesn't know that you mean that it should look in ../myFolder. So it receives a relative path and applies it to the current dir. To fix it, try:
data = loadFile(os.path.join(inputFolder, aFile))