I have developed code that deletes all the files in a folder. I want it to be only able to run concurrently. Meaning, if another user tries to run the script while it is running, the user will get a message saying that the script is already running as to avoid errors in the process.
I tried using threading.Lock(), acquire() and release() but it's not doing any good.
This is my code:
def delete_files(ini_path, days_num):
config = read_config_file(ini_path)
try:
lock.acquire()
log_processes(str(config[1]))
if os.listdir(str(config[0])) == []:
print("Empty list")
else:
print("List loaded")
for file in os.listdir(str(config[0])):
file_path = config[0] + '/' + file
file_datestamp = datetime.fromtimestamp(os.path.getmtime(file_path))
start_of_deletion_date = date.today() - timedelta(days = int(days_num))
if date.today() > file_datestamp.date() and file_datestamp.date() >= start_of_deletion_date:
if not os.path.isdir(file_path):
os.remove(str(file_path))
else:
if not os.path.isdir(file_path):
compress_file(str(file_path))
lock.release()
except Exception as e:
log_processes(config[1], e)
print(e)
When I tried to run it simultaneously, I get this error:
"ERROR: [Errno 2] No such file or directory"
because the file has already been deleted by the first run.
Does anyone have any idea how to solve this?
Related
Good day all,
I am having one of the developers' classics: before the weekend my code worked fine, and now it suddenly doesn't, and honestly I can't figure out why.
I have made a small executable with PyInstaller that, amongst other things, let's you upload a file, after which it writes the file path to a json file. This last step is done so that you only have to pick the file once, and the next time you start the executable it remembers what you picked last.
The problem is imple: "FileNotFoundError: [Errno 2] No such file or directory: 'Data/config.json'"
However, that same bit of code is run twice. The first time Python has no problem opening the file, the second time he does. I cannot figure out why. Here is my code:
dataFolder = 'Data'
config = '/config.json'
def bootScript(): #Checks if files and directories exist. If not, makes them.
print('bootScript starts...')
if os.path.isdir(dataFolder):
print('dataFolder exists.')
else:
try:
os.mkdir(dataFolder)
print('Created dataFolder.')
except:
traceback.print_exc()
if os.path.isfile(dataFolder+ config):
print('Inifile exists.')
else:
try:
open(dataFolder+ config, 'w')
print('Created inifile.')
except:
traceback.print_exc
if os.path.getsize(dataFolder + config) == 0:
try:
defaultPaths = {"outputDestination": NULL, "sourceFile": NULL, "codes": NULL}
with open(dataFolder+ config, 'w') as defaultsJson:
json.dump(defaultPaths, defaultsJson, indent=1)
print('Inifile was empty. Created new json.')
except:
traceback.print_exc
try:
default = loadDefaultJson()
print(default)
if default["outputDestination"] != NULL:
global outputDest
outputDest = default["outputDestination"]
if default["sourceFile"] != NULL:
global sourceFile
global fileString
sourceFile = default["sourceFile"]
fileString = os.path.basename(sourceFile)
if default["codes"] != NULL:
global codeFile
global codeText
codeFile= default["codes"]
codeText = os.path.basename(codeFile)
except:
print('Error while setting one of the default paths.')
traceback.print_exc
As you can see the data folder is first created in the same directory as the executable itself. So it exists.
In the last 'try' there is a call to loadDefaultJson():
def loadDefaultJson():
print('loadDefaultJson starts...')
print(dataFolder + config)
defaultValues = NULL
with open(dataFolder + config, 'r') as defaultsJson:
print('Opened config file.')
try:
defaultValues = json.load(defaultsJson)
except:
print("Could not execute 'json.load()'.")
print('Closed config file.')
return defaultValues
This works fine.
But then, when I use the above function again, but from within a different function, it errors.
def setDefaultPath(key, value): #Sets a new default path
print('setDefaultPaths starts...')
defaultValues = loadDefaultJson()
try:
print('Try to open', dataFolder+ config)
with open(dataFolder+ config, 'w') as defaultsJson:
print('Opened config file.')
defaultValues[key] = value
json.dump(defaultValues, defaultsJson, indent=1)
print('Closed config file.')
except:
print("Cannot save '", value, "' as", key)
traceback.print_exc
It crashes right at the defaultValues = loadDefaultJson(). The whole try and except are not executed.
I do not understand this, as the first time the function is executed, the file is found and opened without issue.
Also, the problem only happens in runtime, not when I run the program with VS Code.
BONUS QUESTION:
Somehow traceback.print_exec does not print anything. Why is this? Only when I remove the try, except and traceback.print_exec, I get to see the actual traceback.
As a "clean up" after my script's main purpose is complete, a function is called to recursively look through each folder and remove all files that end in a pre-determined set of extensions.
I during my testing, I discovered that some files with a file extension in the list of ones to delete actually throw an error: [Errno 1] Operation not permitted: '/location/of/locked/file.png. Looking at the file itself, it appears to be Locked (on mac).
How would I go about removing the locked attribute (should it exist) from every file/folder using Python, then delete the file if it ends in the extension?
Preferably this can all be done in the same function below, as it takes a long time to traverse the input directory - handling each only once is the way to go.
How does this affect the script's integrity on Windows?
I have taken care of programming it in a way that makes it compatible between the OSs but (to my knowledge) the locked attribute does not exist on Windows the way it does on mac and could cause unknown side-effects.
REMOVE_FILETYPES = ('.png', '.jpg', '.jpeg', '.pdf')
def cleaner(currentPath):
if not os.path.isdir(currentPath):
if currentPath.endswith(REMOVE_FILETYPES) or os.path.basename(currentPath).startswith('.'):
try:
os.remove(currentPath)
print('REMOVED: \"{removed}\"'.format(removed = currentPath))
except BaseException as e:
print('ERROR: Could not remove: \"{failed}\"'.format(failed = str(e)))
finally:
return True
return False
if all([cleaner(os.path.join(currentPath, file)) for file in os.listdir(currentPath)]):
try:
os.rmdir(currentPath)
print('REMOVED: \"{removed}\"'.format(removed = currentPath))
except:
print('ERROR: Could not remove: \"{failed}\"'.format(failed = currentPath))
finally:
return True
return False
cleaner(r'/path/to/parent/dir')
I would really appreciate if somebody could show me how to integrate such functionality into the sub-routine. Cheers.
EDIT: Removed error handling as per request
def cleaner(currentPath):
if sys.platform == 'darwin':
os.system('chflags nouchg {}'.format(currentPath))
if not os.path.isdir(currentPath):
if currentPath.endswith(REMOVE_FILETYPES) or os.path.basename(currentPath).startswith('.'):
try:
os.remove(currentPath)
print('REMOVED: \"{removed}\"'.format(removed=currentPath))
except PermissionError:
if sys.platform == 'darwin':
os.system('chflags nouchg {}'.format(currentPath))
os.remove(currentPath)
if all([cleaner(os.path.join(currentPath, file)) for file in os.listdir(currentPath)]) and not currentPath == SOURCE_DIR:
os.rmdir(currentPath)
print('REMOVED: \"{removed}\"'.format(removed=currentPath))
You can unlock the file with the chflags command:
os.system('chflags nouchg {}'.format(filename))
(There is a function os.chflags, but the flag associated with the locked status is not a regular one, but what the os module documentation calls a "user-defined" flag, as you can see by looking at os.stat(locked_filename).st_flags.)
To solve your problem I'd add the chflags command above to a specific except: for the error you get trying to remove a locked file, along with a platform check:
try:
os.remove(currentPath)
print('REMOVED: \"{removed}\"'.format(removed = currentPath))
except PermissionError:
if sys.platform == 'darwin':
os.system('chflags nouchg {}'.format(currentPath))
os.remove(currentPath)
else:
raise
except BaseException as e:
...
Solved: Adding an os.chdir(myArg) resolved the issue.
I'm getting an error when trying to run the following code on anything other than my home directory or files/direcs that I own.
FileNotFoundError: [Errno 2] No such file or directory:
I created a file in root and changed ownership on the file to pi:pi (user running the script). If I specify that file directly, it works, however if I run the script on "/", it will not read that or any other file/direc. I also created a directory /tempdir_delete/ and changed ownership to pi:pi.. If I run the script specifically on "/tempdir_delete/*", it works, but if I leave off the * it fails.
Why does it fail on all except /home/pi/ or files that I explicitly specify and own? It's running the stat as user pi, which is granted by sudo to perform the stat. Also, why do I have to specify the file that I own explicitly? Shouldn't it see that file in root and work because I own it?
import os
import re
import sys
import pwd
myReg = re.compile(r'^\.')
myUID = os.getuid()
myArg = sys.argv[1]
print(os.getuid())
print(pwd.getpwuid(int(myUID)))
print(myArg)
def getsize(direct):
if os.path.isfile(direct) == True:
statinfo = os.stat(myArg)
print(str(statinfo.st_size))
else:
for i in os.listdir(direct):
try:
statinfo = os.stat(i)
if myReg.search(i):
continue
else:
print(i + ' Size: ' + str(statinfo.st_size))
except:
print('Exception occurred, can't read.')
continue
getsize(myArg)
Solved. Adding an os.chdir(myArg) worked to resolve the issue.
I am having an issue with the else statement of this program... I have checked my spacing and it seems to be correct. I keep getting syntax error on the else statement. The program creates and file then attempts to upload it to a ftp server but if it fails to not say anything to the user and just continue It will try again when the program loops. Any help you could provide would be greatly appreciated.
#IMPORTS
import ConfigParser
import os
import random
import ftplib
from ftplib import FTP
#LOOP PART 1
from time import sleep
while True:
#READ THE CONFIG FILE SETUP.INI
config = ConfigParser.ConfigParser()
config.readfp(open(r'setup.ini'))
path = config.get('config', 'path')
name = config.get('config', 'name')
#CREATE THE KEYFILE
filepath = os.path.join((path), (name))
if not os.path.exists((path)):
os.makedirs((path))
file = open(filepath,'w')
file.write('text here')
file.close()
#Create Full Path
fullpath = path + name
#Random Sleep to Accomidate FTP Server
sleeptimer = random.randrange(1,30+1)
sleep((sleeptimer))
#Upload File to FTP Server
try:
host = '0.0.0.0'
port = 3700
ftp = FTP()
ftp.connect(host, port)
ftp.login('user', 'pass')
file = open(fullpath, "rb")
ftp.cwd('/')
ftp.storbinary('STOR ' + name, file)
ftp.quit()
file.close()
else:
print 'Something is Wrong'
#LOOP PART 2
sleep(180.00)
else is valid as part of an exception block, but it is only run if an exception is not raised and there must be a except defined before it.
(edit) Most people skip the else clause and just write code after exiting (dedenting) from the try/except clauses.
The quick tutorial is:
try:
# some statements that are executed until an exception is raised
...
except SomeExceptionType, e:
# if some type of exception is raised
...
except SomeOtherExceptionType, e:
# if another type of exception is raised
...
except Exception, e:
# if *any* exception is raised - but this is usually evil because it hides
# programming errors as well as the errors you want to handle. You can get
# a feel for what went wrong with:
traceback.print_exc()
...
else:
# if no exception is raised
...
finally:
# run regardless of whether exception was raised
...
I am using a python script to transfer the contents of three files to a different three files. The original files are data from three thermometers I have connected to an RPI running raspian. All the script is supposed to do is take the contents of the files and move them so that I can have another program (ComScript) read and parse them.
My problem is that if one or more of the thermometers is disconnected before the script starts, it freezes. It doesn't freeze if I disconnect a thermometer while the script is running.
Here is the code
import time
a = 1
while a == 1:
try:
tfile = open("/sys/bus/w1/devices/28-000004d2ca5e/w1_slave")
text = tfile.read()
tfile.close()
temperature = text
tfile2 = open("/sys/bus/w1/devices/28-000004d2fb20/w1_slave")
text2 = tfile2.read()
tfile2.close()
temperature2 = text2
tfile3 = open("/sys/bus/w1/devices/28-000004d30568/w1_slave")
text3 = tfile3.read()
tfile3.close()
temperature3 = text3
textfile = open("/home/pi/ComScriptPi/profiles/Temperature_parse/w1_slave1", "w ")
textfile2 = open("/home/pi/ComScriptPi/profiles/Temperature_parse/w1_slave2", "w ")
textfile3 = open("/home/pi/ComScriptPi/profiles/Temperature_parse/w1_slave3", "w ")
temperature = str(temperature)
temperature2 = str(temperature2)
temperature3 = str(temperature3)
textfile.write(temperature)
textfile2.write(temperature2)
textfile3.write(temperature3)
textfile.close()
textfile2.close()
textfile3.close()
print temperature
print temperature2
print temperature3
time.sleep(3)
except:
pass
I added the exception pass because I need it to keep running even if it gets bad values. WHen one of the thermometers is disconnected the file python is trying to read is blank, but still there.
Remove the blanket except.
Your script is not freezing, but any error you get is being ignored in an endless loop. Because you use a blanket except: you catch all exceptions, including the keyboard interrupt exception KeyboardInterrupt.
At the very least log the exception, and catch only Exception:
except Exception:
import logging
logging.exception('Oops: error occurred')
KeyboardInterrupt is a subclass of BaseException, not Exception and won't be caught by this except handler.
Take a look at the shutil module for copying files, you doing way too much work:
import time
import shutil
import os.path
paths = ('28-000004d2ca5e', '28-000004d2fb20', '28-000004d30568')
while True:
for i, name in enumerate(paths, 1):
src = os.path.join('/sys/bus/w1/devices', name, 'w1_slave')
dst = '/home/pi/ComScriptPi/profiles/Temperature_parse/w1_slave{}'.format(i)
try:
shutil.copyfile(src, dst)
except EnvironmentError:
import logging
logging.exception('Oops: error occurred')
time.sleep(3)
Handling files should only ever raise EnvironmentError or it's subclasses, there is no need to catch everything here.
The open of the unplugged device is most likely blocking because the device driver won't open if the device is not present.
You'll need to use os.open which is the equivalent of the Unix system call "open" and specify the flag O_NONBLOCK and check the return code. You can then use os.fdopen to turn the return value of os.open into a normal Python file object.