My problem is that logging stops for a python program when the log is rotated.
I have tracked it down to the stream itself. I don't see any way to tell if the stream is broken from python. After the file is deleted it still accepts writes without any issue.
import os
FILE = 'testing.txt'
fs = open(FILE, 'a')
fs.write('word')
os.remove(FILE)
fs.write('Nothing....') # Nothing breaks
print(fs.errors) # No errors
So, how can I find out if the file stream is still valid?
And checking to see if the file exists will not help since the file will always exist regardless of whether or not the stream is still valid.
Upon much more inspection, I found the solution. It is an OS specific problem. When the file is deleted in Linux (or Macintosh) it just unlinks it. (I was not aware of this)
So if you run lsof on the machine, it still shows the file as open.
[user#machine]$ lsof | grep --color -i "testing.txt"
python26 26495 user 8w REG 8,33 23474 671920 /home/user/temp/testing.txt (deleted)
The solution is to stat the stream in python.
stat = os.fstat(fs.fileno())
Which will give you the number of links it has.
if stat.st_nlink < 1:
#has been deleted
And there you go. Now you know if you should reload it or not. Hopefully this helps someone else.
Try Exception handling:
import os
FILE = 'testing.txt'
try:
fs = open(FILE, 'a')
fs.write('word')
os.remove(FILE)
fs.write('Nothing....') # Nothing breaks
except Exception, e:
print "Error:", e
print(fs.errors) # No errors
There are python bindings for ionotify if you need more intelligence than just an try: except: clause. But I think its only pertinent to Linux (im not sure of your platform)
Another solution I found is to add the "copytruncate" flag into the logrotate config.
See "man logrotate" for more info.
Related
Made a quick script on my windows pc to query an api and write to a spreadsheet. The issue is that the person running this will be on mac, so Im trying to port it to mac os but I believe I've run into a file path error? Everything looks right though so I'm confused. Thank you in advance for any help. I'm very unfamiliar with mac.
The excel module I'm using isnt able to open the workbook I have. It's openpyxl.
root = Tkinter.Tk()
root.withdraw()
file_path = tkFileDialog.askopenfilename()
save =tkFileDialog.asksaveasfilename(defaultextension=".xlsx",initialfile=file_path)
main(file_path,save)
def main(load,save):
try:
wb = load_workbook(load)
except:
print load,save
return
I expect the workbook to be opened but instead it's printing the file path and ending main().
Here are the file paths it's printing out:
/Users/edwin/Downloads/190117 CA Device Roster.xlsx
/Users/edwin/Downloads/:Users:edwin:Downloads:test.xlsx
except:
. Why are you expclicitly catching all exceptions and then not exposing them to the user? Ignoring exceptions doesn't make the problem go away - it just stops you from seeing it. Which means your code doesn't work and you don't know why.
If you need additional debugging in your except, it's fine, just raise the original exception afterwards:
except Exception as e:
print(load,save)
raise e
Anything other than exposing the exception you got is going to hide your errors, leaving you with no option but to guess what's going wrong. Good engineers don't guess - they increase logging.
I want to open .wav file in default program. But it doesn´t work. This is my code:
audiofile=(myFile[index]+".wav") # I have all files in array (without ".wav")
try:
try:
os.system('xdg-open audiofile')
except:
os.system('start audiofile')
except:
print "error"
I don´t get any error, but it doesn´t work. How can I solve it? Thank you.
You aren't substituting the name of the audio file into your OS commands, so it can't possibly work.
You'd need something like:
os.system('xdg-open ' + audiofile)
This assumes that you have a default application associated with .wav files, which of course you can test by trying your command manually.
You might also want to check the return value of os.system for an error code, rather than relying on exceptions.
First of all, you should fill the variable audiofile into the command, not the string 'audiofile' itself
os.system('xdg-open %s' % audiofile)
Second,
os.system will NOT throw an exception when xdg-open or start doesn't exist in system.
Determine the type of system first by platform.system
>>> import platform
>>> platform.system()
'Linux'
Quite simply, I am cycling through all sub folders in a specific location, and collecting a few numbers from three different files.
def GrepData():
import glob as glob
import os as os
os.chdir('RUNS')
RUNSDir = os.getcwd()
Directories = glob.glob('*.*')
ObjVal = []
ParVal = []
AADVal = []
for dir in Directories:
os.chdir(dir)
(X,Y) = dir.split(sep='+')
AADPath = glob.glob('Aad.out')
ObjPath = glob.glob('fobj.out')
ParPath = glob.glob('Par.out')
try:
with open(os.path.join(os.getcwd(),ObjPath[0])) as ObjFile:
for line in ObjFile:
ObjVal.append(list([X,Y,line.split()[0]]))
ObjFile.close()
except(IndexError):
ObjFile.close()
try:
with open(os.path.join(os.getcwd(),ParPath[0])) as ParFile:
for line in ParFile:
ParVal.append(list([X,Y,line.split()[0]]))
ParFile.close()
except(IndexError):
ParFile.close()
try:
with open(os.path.join(os.getcwd(),AADPath[0])) as AADFile:
for line in AADFile:
AADVal.append(list([X,Y,line.split()[0]]))
AADFile.close()
except(IndexError):
AADFile.close()
os.chdir(RUNSDir)
Each file open command is placed in a try - except block, as in a few cases the file that is opened will be empty, and thus appending the line.split() will lead to an index error since the list is empty.
However when running this script i get the following error: "OSError: [Errno 24] Too Many open files"
I was under the impression that the idea of the "with open..." statement was that it took care of closing the file after use? Clearly that is not happening.
So what I am asking for is two things:
The answer to: "Is my understanding of with open correct?"
How can I correct whatever error is inducing this problem?
(And yes i know the code is not exactly elegant. The whole try - except ought to be a single object that is reused - but I will fix that after figuring out this error)
Try moving your try-except inside the with like so:
with open(os.path.join(os.getcwd(),ObjPath[0])) as ObjFile:
for line in ObjFile:
try:
ObjVal.append(list([X,Y,line.split()[0]]))
except(IndexError):
pass
Notes: there is no need to close your file manually, this is what with is for. Also, there is no need to use as os in your imports if you are using the same name.
"Too many open files" has nothing to do with writing semantically incorrect python code, and you are using with correctly. The key is the part of your error that says "OSError," which refers to the underlying operating system.
When you call open(), the python interpreter will execute a system call. The details of the system call vary a bit by which OS you are using, but on linux this call is open(2). The operating system kernel will handle the system call. While the file is open, it has an entry in the system file table and takes up OS resources -- this means effectively it is "taking up space" whilst it is open. As such the OS has a limit to the number of files that can be opened at any one time.
Your problem is that while you call open(), you don't call close() quickly enough. In the event that your directory structure requires you to have many thousands files open at once that might approach this cap, it can be temporarily changed (at least on linux, I'm less familiar with other OSes so I don't want to go into too many details about how to do this across platforms).
edit:OK, I could swear that the way I'd tested it showed that the getcwd was also causing the exception, but now it appears it's just the file creation. When I move the try-except blocks to their it actually does catch it like you'd think it would. So chalk that up to user error.
Original Question:
I have a script I'm working on that I want to be able to drop a file on it to have it run with that file as an argument. I checked in this question, and I already have the mentioned registry keys (apparently the Python 2.6 installer takes care of it.) However, it's throwing an exception that I can't catch. Running it from the console works correctly, but when I drop a file on it, it throws an exception then closes the console window. I tried to have it redirect standard error to a file, but it threw the exception before the redirection occurred in the script. With a little testing, and some quick eyesight I saw that it was throwing an IOError when I tried to create the file to write the error to.
import sys
import os
#os.chdir("C:/Python26/Projects/arguments")
try:
print sys.argv
raw_input()
os.getcwd()
except Exception,e:
print sys.argv + '\n'
print e
f = open("./myfile.txt", "w")
If I run this from the console with any or no arguments, it behaves as one would expect. If I run it by dropping a file on it, for instance test.txt, it runs, prints the arguments correctly, then when os.getcwd() is called, it throws the exception, and does not perform any of the stuff from the except: block, making it difficult for me to find any way to actually get the exception text to stay on screen. If I uncomment the os.chdir(), the script doesn't fail. If I move that line to within the except block, it's never executed.
I'm guessing running by dropping the file on it, which according to the other linked question, uses the WSH, is somehow messing up its permissions or the cwd, but I don't know how to work around it.
Seeing as this is probably not Python related, but a Windows problem (I for one could not reproduce the error given your code), I'd suggest attaching a debugger to the Python interpreter when it is started. Since you start the interpreter implicitly by a drag&drop action, you need to configure Windows to auto-attach a debugger each time Python starts. If I remember correctly, this article has the needed information to do that (you can substitute another debugger if you are not using Visual Studio).
Apart from that, I would take a snapshot with ProcMon while dragging a file onto your script, to get an idea of what is going on.
As pointed out in my edit above, the errors were caused by the working directory changing to C:\windows\system32, where the script isn't allowed to create files. I don't know how to get it to not change the working directory when started that way, but was able to work around it like this.
if len(sys.argv) == 1:
files = [filename for filename in os.listdir(os.getcwd())
if filename.endswith(".txt")]
else:
files = [filename for filename in sys.argv[1:]]
Fixing the working directory can be managed this way I guess.
exepath = sys.argv[0]
os.chdir(exepath[:exepath.rfind('\\')])
I am on windows with Python 2.5. I have an open file for writing. I write some data. Call file close. When I try to delete the file from the folder using Windows Explorer, it errors, saying that a process still holds a handle to the file.
If I shutdown python, and try again, it succeeds.
It does close them.
Are you sure f.close() is getting called?
I just tested the same scenario and windows deletes the file for me.
Are you handling any exceptions around the file object? If so, make sure the error handling looks something like this:
f = open("hello.txt")
try:
for line in f:
print line
finally:
f.close()
In considering why you should do this, consider the following lines of code:
f = open('hello.txt')
try:
perform_an_operation_that_causes_f_to_raise_an_exception()
f.close()
except IOError:
pass
As you can see, f.close will never be called in the above code. The problem is that the above code will also cause f to not get garbage collected. The reason is that f will still be referenced in sys.traceback, in which case the only solution is to manually call close on f in a finally block or set sys.traceback to None (and I strongly recommend the former).
Explained in the tutorial:
with open('/tmp/workfile', 'r') as f:
read_data = f.read()
It works when you writing or pickling/unpickling, too
It's not really necessary that try finally block: Java way of doing things, not Python
I was looking for this, because the same thing happened to me. The question didn't help me, but I think I figured out what happened.
In the original version of the script I wrote, I neglected to add in a 'finally' clause to the file in case of an exception.
I was testing the script from the interactive prompt and got an exception while the file was open. What I didn't realize was that the file object wasn't immediately garbage-collected. After that, when I ran the script (still from the same interactive session), even though the new file objects were being closed, the first one still hadn't been, and so the file handle was still in use, from the perspective of the operating system.
Once I closed the interactive prompt, the problem went away, at which I remembered that exception occurring while the file was open and realized what had been going on. (Moral: Don't try to program on insufficient sleep. : ) )
Naturally, I have no idea if this is what happened in the case of the original poster, and even if the original poster is still around, they may not remember the specific circumstances, but the symptoms are similar, so I thought I'd add this as something to check for, for anyone caught in the same situation and looking for an answer.
I did it using intermediate file:
import os
f = open("report.tmp","w")
f.write("{}".format("Hello"))
f.close()
os.system("move report.tmp report.html") #this line is for Windows users