Develop Reference

Throw an error when writing to a CSV file if file is in-use in python? [duplicate]

I my application, i have below requests:
1. There has one thread will regularly record some logs in file. The log file will be rollovered in certain interval. for keeping the log files small.
2. There has another thread also will regularly to process these log files. ex: Move the log files to other place, parse the log's content to generate some log reports.
But, there has a condition is the second thread can not process the log file that's using to record the log. in code side, the pseudocode similars like below:
#code in second thread to process the log files
for logFile in os.listdir(logFolder):
if not file_is_open(logFile) or file_is_use(logFile):
ProcessLogFile(logFile) # move log file to other place, and generate log report....
So, how do i check is a file is already open or is used by other process?
I did some research in internet. And have some results:
try:
myfile = open(filename, "r+") # or "a+", whatever you need
except IOError:
print "Could not open file! Please close Excel!"
I tried this code, but it doesn't work, no matter i use "r+" or "a+" flag
try:
os.remove(filename) # try to remove it directly
except OSError as e:
if e.errno == errno.ENOENT: # file doesn't exist
break
This code can work, but it can not reach my request, since i don't want to delete the file to check if it is open.

An issue with trying to find out if a file is being used by another process is the possibility of a race condition. You could check a file, decide that it is not in use, then just before you open it another process (or thread) leaps in and grabs it (or even deletes it).
Ok, let's say you decide to live with that possibility and hope it does not occur. To check files in use by other processes is operating system dependant.
On Linux it is fairly easy, just iterate through the PIDs in /proc. Here is a generator that iterates over files in use for a specific PID:
def iterate_fds(pid):
dir = '/proc/'+str(pid)+'/fd'
if not os.access(dir,os.R_OK|os.X_OK): return
for fds in os.listdir(dir):
for fd in fds:
full_name = os.path.join(dir, fd)
try:
file = os.readlink(full_name)
if file == '/dev/null' or \
re.match(r'pipe:\[\d+\]',file) or \
re.match(r'socket:\[\d+\]',file):
file = None
except OSError as err:
if err.errno == 2:
file = None
else:
raise(err)
yield (fd,file)
On Windows it is not quite so straightforward, the APIs are not published. There is a sysinternals tool (handle.exe) that can be used, but I recommend the PyPi module psutil, which is portable (i.e., it runs on Linux as well, and probably on other OS):
import psutil
for proc in psutil.process_iter():
try:
# this returns the list of opened files by the current process
flist = proc.open_files()
if flist:
print(proc.pid,proc.name)
for nt in flist:
print("\t",nt.path)
# This catches a race condition where a process ends
# before we can examine its files
except psutil.NoSuchProcess as err:
print("****",err)

I like Daniel's answer, but for Windows users, I realized that it's safer and simpler to rename the file to the name it already has. That solves the problems brought up in the comments to his answer. Here's the code:
import os
f = 'C:/test.xlsx'
if os.path.exists(f):
try:
os.rename(f, f)
print 'Access on file "' + f +'" is available!'
except OSError as e:
print 'Access-error on file "' + f + '"! \n' + str(e)

You can check if a file has a handle on it using the next function (remember to pass the full path to that file):
import psutil
def has_handle(fpath):
for proc in psutil.process_iter():
try:
for item in proc.open_files():
if fpath == item.path:
return True
except Exception:
pass
return False

I know I'm late to the party but I also had this problem and I used the lsof command to solve it (which I think is new from the approaches mentioned above). With lsof we can basically check for the processes that are using this particular file.
Here is how I did it:
from subprocess import check_output,Popen, PIPE
try:
lsout=Popen(['lsof',filename],stdout=PIPE, shell=False)
check_output(["grep",filename], stdin=lsout.stdout, shell=False)
except:
#check_output will throw an exception here if it won't find any process using that file
just write your log processing code in the except part and you are good to go.

Instead on using os.remove() you may use the following workaround on Windows:
import os
file = "D:\\temp\\test.pdf"
if os.path.exists(file):
try:
os.rename(file,file+"_")
print "Access on file \"" + str(file) +"\" is available!"
os.rename(file+"_",file)
except OSError as e:
message = "Access-error on file \"" + str(file) + "\"!!! \n" + str(e)
print message

You can use inotify to watch for activity in file system. You can watch for file close events, indicating that a roll-over has happened. You should also add additional condition on file-size. Make sure you filter out file close events from the second thread.

A slightly more polished version of one of the answers from above.
from pathlib import Path
def is_file_in_use(file_path):
path = Path(file_path)
if not path.exists():
raise FileNotFoundError
try:
path.rename(path)
except PermissionError:
return True
else:
return False

On Windows, you can also directly retrieve the information by leveraging on the NTDLL/KERNEL32 Windows API. The following code returns a list of PIDs, in case the file is still opened/used by a process (including your own, if you have an open handle on the file):
import ctypes
from ctypes import wintypes
path = r"C:\temp\test.txt"
# -----------------------------------------------------------------------------
# generic strings and constants
# -----------------------------------------------------------------------------
ntdll = ctypes.WinDLL('ntdll')
kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
NTSTATUS = wintypes.LONG
INVALID_HANDLE_VALUE = wintypes.HANDLE(-1).value
FILE_READ_ATTRIBUTES = 0x80
FILE_SHARE_READ = 1
OPEN_EXISTING = 3
FILE_FLAG_BACKUP_SEMANTICS = 0x02000000
FILE_INFORMATION_CLASS = wintypes.ULONG
FileProcessIdsUsingFileInformation = 47
LPSECURITY_ATTRIBUTES = wintypes.LPVOID
ULONG_PTR = wintypes.WPARAM
# -----------------------------------------------------------------------------
# create handle on concerned file with dwDesiredAccess == FILE_READ_ATTRIBUTES
# -----------------------------------------------------------------------------
kernel32.CreateFileW.restype = wintypes.HANDLE
kernel32.CreateFileW.argtypes = (
wintypes.LPCWSTR, # In lpFileName
wintypes.DWORD, # In dwDesiredAccess
wintypes.DWORD, # In dwShareMode
LPSECURITY_ATTRIBUTES, # In_opt lpSecurityAttributes
wintypes.DWORD, # In dwCreationDisposition
wintypes.DWORD, # In dwFlagsAndAttributes
wintypes.HANDLE) # In_opt hTemplateFile
hFile = kernel32.CreateFileW(
path, FILE_READ_ATTRIBUTES, FILE_SHARE_READ, None, OPEN_EXISTING,
FILE_FLAG_BACKUP_SEMANTICS, None)
if hFile == INVALID_HANDLE_VALUE:
raise ctypes.WinError(ctypes.get_last_error())
# -----------------------------------------------------------------------------
# prepare data types for system call
# -----------------------------------------------------------------------------
class IO_STATUS_BLOCK(ctypes.Structure):
class _STATUS(ctypes.Union):
_fields_ = (('Status', NTSTATUS),
('Pointer', wintypes.LPVOID))
_anonymous_ = '_Status',
_fields_ = (('_Status', _STATUS),
('Information', ULONG_PTR))
iosb = IO_STATUS_BLOCK()
class FILE_PROCESS_IDS_USING_FILE_INFORMATION(ctypes.Structure):
_fields_ = (('NumberOfProcessIdsInList', wintypes.LARGE_INTEGER),
('ProcessIdList', wintypes.LARGE_INTEGER * 64))
info = FILE_PROCESS_IDS_USING_FILE_INFORMATION()
PIO_STATUS_BLOCK = ctypes.POINTER(IO_STATUS_BLOCK)
ntdll.NtQueryInformationFile.restype = NTSTATUS
ntdll.NtQueryInformationFile.argtypes = (
wintypes.HANDLE, # In FileHandle
PIO_STATUS_BLOCK, # Out IoStatusBlock
wintypes.LPVOID, # Out FileInformation
wintypes.ULONG, # In Length
FILE_INFORMATION_CLASS) # In FileInformationClass
# -----------------------------------------------------------------------------
# system call to retrieve list of PIDs currently using the file
# -----------------------------------------------------------------------------
status = ntdll.NtQueryInformationFile(hFile, ctypes.byref(iosb),
ctypes.byref(info),
ctypes.sizeof(info),
FileProcessIdsUsingFileInformation)
pidList = info.ProcessIdList[0:info.NumberOfProcessIdsInList]
print(pidList)

I provided one solution. please see the following code.
def isFileinUsed(ifile):
widlcard = "/proc/*/fd/*"
lfds = glob.glob(widlcard)
for fds in lfds:
try:
file = os.readlink(fds)
if file == ifile:
return True
except OSError as err:
if err.errno == 2:
file = None
else:
raise(err)
return False
You can use this function to check if a file is in used.
Note:
This solution only can be used for Linux system.

How to exit function with signal on Windows?

I have the following code written in Python 2.7 on Windows. I want to check for updates for the current python script and update it, if there is an update, with a new version through ftp server preserving the filename and then executing the new python script after terminating the current through the os.kill with SIGNTERM.
I went with the exit function approach but I read that in Windows this only works with the atexit library and default python exit methods. So I used a combination of the atexit.register() and the signal handler.
***necessary libraries***
filematch = 'test.py'
version = '0.0'
checkdir = os.path.abspath(".")
dircontent = os.listdir(checkdir)
r = StringIO()
def exithandler():
try:
try:
if filematch in dircontent:
os.remove(checkdir + '\\' + filematch)
except Exception as e:
print e
ftp = FTP(ip address)
ftp.login(username, password)
ftp.cwd('/Test')
for filename in ftp.nlst(filematch):
fhandle = open(filename, 'wb')
ftp.retrbinary('RETR ' + filename, fhandle.write)
fhandle.close()
subprocess.Popen([sys.executable, "test.py"])
print 'Test file successfully updated.'
except Exception as e:
print e
ftp = FTP(ip address)
ftp.login(username, password)
ftp.cwd('/Test')
ftp.retrbinary('RETR version.txt', r.write)
if(r.getvalue() != version):
atexit.register(exithandler)
somepid = os.getpid()
signal.signal(SIGTERM, lambda signum, stack_frame: exit(1))
os.kill(somepid, signal.SIGTERM)
print 'Successfully replaced and started the file'
Using the:
signal.signal(SIGTERM, lambda signum, stack_frame: exit(1))
I get:
Traceback (most recent call last):
File "C:\Users\STiX\Desktop\Python Keylogger\test.py", line 50, in <module>
signal.signal(SIGTERM, lambda signum, stack_frame: exit(1))
NameError: name 'SIGTERM' is not defined
But I get the job done without a problem except if I use the current code in a more complex script where the script give me the same error but terminates right away for some reason.
On the other hand though, if I use it the correct way, signal.SIGTERM, the process goes straight to termination and the exit function never executed. Why is that?
How can I make this work on Windows and get the outcome that I described above successfully?

What you are trying to do seems a bit complicated (and dangerous from an infosec-perspective ;-). I would suggest to handle the reload-file-when-updated part of the functionality be adding a controller class that imports the python script you have now as a module and, starts it and the reloads it when it is updated (based on a function return or other technique) - look this way for inspiration - https://stackoverflow.com/a/1517072/1010991
Edit - what about exe?
Another hacky technique for manipulating the file of the currently running program would be the shell ping trick. It can be used from all programming languages. The trick is to send a shell command that is not executed before after the calling process has terminated. Use ping to cause the delay and chain the other commands with &. For your use case it could be something like this:
import subprocess
subprocess.Popen("ping -n 2 -w 2000 1.1.1.1 > Nul & del hack.py & rename hack_temp.py hack.py & hack.py ", shell=True)
Edit 2 - Alternative solution to original question
Since python does not block write access to the currently running script an alternative concept to solve the original question would be:
import subprocess
print "hello"
a = open(__file__,"r")
running_script_as_string = a.read()
b = open(__file__,"w")
b.write(running_script_as_string)
b.write("\nprint 'updated version of hack'")
b.close()
subprocess.Popen("python hack.py")

Print newline if error occurs in python

I have some relatively big program and I used to print its progress to the console in the manner that each function prints only one line of output (it displays "Doing something..." while the function is running (sometimes it displays percentage bar) and turns to "Doing something... Done" when the function finishes successfully. I apply '\r', line clearance, etc. for my progress bar to look nice. However, when error occurs, the message continues the same line, which I want to avoid. For example I have a code:
import os, sys, subprocess
def some_function(filename):
print('Doing something... ', end = '')
sys.stdout.flush()
with open(os.devnull, 'wb') as devnull:
check = subprocess.call(['ls', filename], stdout = devnull)
if check != 0:
sys.exit(1)
print('Done')
some_function('some.file')
It produces the following output (depending on presence of an error):
Doing something... Done
or
Doing something... ls: some.file: No such file or directory
And what I want to see in the case of error:
Doing something...
ls: some.file: No such file or directory
Is there some general way to introduce newline in output if error occurs (it can be some internal or user-defined exception as well)?

call does not raise an exception so you can catch the error with subprocess using stderr:
import os, sys, subprocess
def some_function(filename):
print('Doing something... ', end='')
sys.stdout.flush()
with open(os.devnull, 'wb') as devnull:
check = subprocess.Popen(['ls', filename], stdout = devnull, stderr=subprocess.PIPE)
stdout, stderr = check.communicate()
if stderr:
print("\n{}".format(stderr.decode("utf-8")))
sys.exit(1)
print('Done')

try:
do stuff that might cause an error
except:
print()
raise
If an error is raised inside the try block, this will print a new line, and then re-raise the caught exception.
Edit
As has been pointed out in the comments, in this case the error message is not being generated by a raised exception, so I defer to Padraic's answer.

Error handling when opening applications in python

I need to open a file in a specific application using python. I'm going to default to opening the file using the default app location / filename; however should the app be unable to be opened, I'd like to handle that error and give the user some other options.
I've understood so far that subprocess.call is the best way to do this, as opposed to system.os
Application: /Applications/GreatApp.app
This works fine
subprocess.call(['open','/Applications/GreatApp.app','placeholder.gap'])
However, when I start adding the try / except loops, they seem to do nothing. (note the space in the application name - I'm using the incorrect name to force an exception)
try:
subprocess.call(['open','/Applications/Great App.app','placeholder.gap'])
except:
print 'this is an error placeholder'
I'll still see the following error displayed in python
The file /Applications/Great App.app does not exist.
The closest I've found to some form of error handling is the following. Is looking at the value of retcode the right way to go about this?
try:
retcode = subprocess.call("open " + filename, shell=True)
if retcode < 0:
print >>sys.stderr, "Child was terminated by signal", -retcode
else:
print >>sys.stderr, "Child returned", retcode
except OSError, e:
print >>sys.stderr, "Execution failed:", e
Turns out retcode isn't the way, as both correct and incorrect names give a value greater than 0.

What does this show?
subprocess.call (['ls', '-l', '/Applications'])
The error message you got says that the application that you are trying to open does not exist.
And you won't get an exception if it doesn't as you have found.

Try with this. It will open file with its default editor, if it is installed.
ss=subprocess.Popen(FileName,shell=True)
ss.communicate()

I have no control over the application that the user has associated with the file I'm opening.
I need to be able to specify the application to be used and the file to be opened.
subprocess.call doesn't return exceptions should the application/file be unable to be opened.
It's friend subprocess.check_call, does.
From the docs: http://docs.python.org/2/library/subprocess.html#subprocess.check_call
If the return code was zero then return, otherwise raise
CalledProcessError. The CalledProcessError object will have the return
code in the returncode attribute.
I've provide my usage examples below for future reference
For OSX
FNULL = open(os.devnull, 'w') # Used in combination with the stdout variable
# to prevent output from being displayed while
# the program launches.
try:
# First try the default install location of the application
subprocess.check_call(['open','/Applications/Application.app',filename], stdout=FNULL)
except subprocess.CalledProcessError:
# Then ask user to manually enter in the filepath to Application.app
print 'unable to find /Applications/Application.app'
# Now ask user to manually enter filepath
For Windows, change this line
subprocess.check_call(['C:\file\to\program',filename], stdout=FNULL)

Exception while handling generator.send() in try...except block

filename = 'tempfile'
def tail(filename):
fd = open(filename)
while True:
line = fd.readline()
if not line:
continue
else:
if filename != 'uh':
yield line
else:
print 'Returning f to close the file'
yield fd
try:
genObj = tail(filename)
valfromgen= genObj.next()
while valfromgen:
print valfromgen
valfromgen= genObj.next()
except:
traceback.print_exc()
try:
fd_Got_Back = genObj.send('uh')
fd_Got_Back.close()
except:
traceback.print_exc()
Intention of the code: I have opened the file in the generator function only and not outside it, but, I want to close that file outside the generator function by using 'send' probably.
What I am trying to do: Replicating tail -f from unix.
How I am trying to do:
Open a tempfile in read mode.
If the tempfile has 1 new line written in it (which I'll keep on writing manually and saving tempfile using notepad), yield the newly written line.
Problem:
The problem is that I'm trying to check how I can close the opened tempfile from this python code if I press Ctrl + C (i.e. SIGTERM) when this Python code runs in command prompt. In order to emulate this, I have opened the tempfile in the tail function, and whenever there is an exception (which will be raised by the system when I press Ctrl + C), the control should go in the 1st except. Then, from here, I'm trying to send a value uh to the generator function tail, so that it should yield the file descriptor of the opened file which I can use to close the opened tempfile.
PS: I expect a solution where I have opened the file in the generator function only and not outside it.

I think You're misunderstanding how "send" works. Send simply causes a generator to yield that value on its next iteration. It does not change the value of the original parameters. You can then use that yielded value for some purpose. So you could make your code:
filename = 'tempfile'
def tail(filename):
fd = open(filename)
while True:
line = fd.readline()
if not line:
continue
else:
x = (yield line)
if (x == 'uh'):
print 'Returning f to close the file'
yield fd
try:
genObj = tail(filename)
valfromgen= genObj.next()
while valfromgen:
print valfromgen
valfromgen= genObj.next()
except:
traceback.print_exc()
try:
genObj.send('uh').close()
except:
traceback.print_exc()

I have figured out the problem where I was stuck and I have come up with this solution:-
When I press Ctrl + C (on Windows), the KeyboardInterrupt actually happens in fd.readline(). So, I just placed a try...except there, so that the generator function yields the file descriptor whenever Ctrl + C is hit. If there is no KeyBoardInterrupt , then, just print a newly read line from tempfile
This file descriptor is checked using isinstance() in the main body, and if it is found to be a file, then, I'm closing the file as well as the generator
PS: (this KeyboardInterrupt might vary on Linux..probably SigTerm will be raised, but, please check. So, in order to make the code generic, just remove KeyBoard Interrupt and use just normal except)
import sys, traceback
filename = 'tempfile'
def tail(filename):
fd = open(filename)
while True:
try:
line = fd.readline()
except KeyboardInterrupt:
print 'keyboard interrupt here'
yield fd
if not line:
continue
else:
yield line
try:
genObj = tail(filename)
valfromgen= genObj.next()
while valfromgen:
if isinstance(valfromgen, file):
print 'Closing this file now as `tail` yielded a file descriptor'
valfromgen.close()
genObj.close()
break
print 'Yielded line: ', valfromgen
valfromgen= genObj.next()
print 'Just in order to check that things are in order, the following line will raise StopIteration. If it raises, it means we are good.'
print genObj.next()
except:
traceback.print_exc()