In my program I use shutil to copy files to several computers in a list. I am wondering what's the best way to give an error and move onto next computer if one of the computers happens to be turned off.
My original code:
def copyfiles(servername):
# copy config to remote server
source = os.listdir("C:/Users/myname/Desktop/PythonUpdate/") # directory where original configs are located
destination = '//' + servername + '/c$/test/' # destination server directory
for files in source:
if files.endswith(".config"):
shutil.copy(files,destination)
os.system('cls' if os.name == 'nt' else 'clear')
array = []
with open("C:/Users/myname/Desktop/PythonUpdate/serverlist.txt", "r") as f:
for servername in f:
copyfiles(servername.strip())
What I am trying to do:
def copyfiles(servername):
# copy config to remote server
source = os.listdir("C:/Users/myname/Desktop/PythonUpdate/") # directory where original configs are located
destination = '//' + servername + '/c$/test/' # destination server directory
for files in source:
if files.endswith(".config"):
try:
shutil.copy(files,destination)
except:
print (" //////////////////////////////////////////")
print (" Cannot connect to " + servername + ".")
print (" //////////////////////////////////////////")
os.system('cls' if os.name == 'nt' else 'clear')
array = []
with open("C:/Users/myname/Desktop/PythonUpdate/serverlist.txt", "r") as f:
for servername in f:
copyfiles(servername.strip())
Does this seem like it is implemented well?
While your idea of using a try-catch block is correct, you should definitely be much more precise about which conditions you consider to be forgivable. For example, you would not want to continue if you somehow got a MemoryError or KeyboardInterrupt.
As a first step, you should trap only OSError, as this is what shutil.copy uses to indicate all sorts of read/write errors (at least
that is what the documentation for shutil.copyfile says). According to this post, your error is most likely to be a WindowsError (which is a subclass of OSError):
try:
shutil.copy(files, destination)
except WindowsError:
print('...')
If there are likely to be other causes for the exception, you can further narrow down the exact cause by testing the error object itself using the extended form of the else clause. For example, WindowsError, has an attribute winerror, that contains the system-level error code for the exception. Likely candidates that you might want to forgive:
ERROR_BAD_NETPATH (53)
ERROR_NETWORK_BUSY (54)
ERROR_DEV_NOT_EXIST (55)
ERROR_ADAP_HDW_ERR (57)
ERROR_UNEXP_NET_ERR (59)
ERROR_NETNAME_DELETED (64)
ERROR_SHARING_PAUSED (70)
ERROR_NET_WRITE_FAULT (88)
Your code could then look something like this:
try:
shutil.copy(files, destination)
except WindowsError as ex:
if ex.winerror in (53, 54, 55, 57, 59, 64, 70, 88):
print('...')
else:
raise
You may also want to check the filename2 attribute of the exception. This will tell you if the file you were expecting caused this exception. This check is not related to the one on winerror and can be done along with or in exclusion to the one shown above:
try:
shutil.copy(files, destination)
except WindowsError as ex:
if ex.filename2.startswith('//' + filename2):
print('...')
else:
raise
Related
I am somewhat new to programming. I did not go to school to learn about it, but do read a lot online to learn more about coding.
In general, if I know a concept then I can usually figure it out with google.
Obvious from the post, I like to play games and so I thought one day while configuring some settings for a mod, I realized that the information was extremely difficult to dig up. I thought that I could try to solve it by writing a python script (no real success writing scripts before this point).
The following is the general workflow that I wrote up before tackling it in python:
Replace base path up to 'Mods' with '/Game/Mods'
Find all paths to files in Mods folder
Rename all mod files ending in uasset. E.g. filname.uasset -> filename.filename
Parse Mod's name from binary file in Mod subdirectory, replace integer name with text name for all related paths. E.g. mod name 899987403 -> Primal_Fear_Bosses
Example before (ubuntu path):
/mnt/c/Program Files (x86)/Steam/steamapps/common/ARK/ShooterGame/Content/Mods/899987403/Vanilla/Projectiles/Manticore/Proj/PFBProjManticoreQuill.uasset
& after with script: /Game/Mods/Primal_Fear_Bosses/Vanilla/Projectiles/Manticore/Proj/PFBProjManticoreQuill.PFBProjManticoreQuill,
*Note: align the "/Mods/" section for both lines to see the key difference.
Anyways, I was familiar with a lot of concepts at that point, so I did a lot of googling and wrote the following script (first ever), "Generate_Resource_Pulling_List_for_S+.py":
import os
import sys
from fnmatch import fnmatch
import re
# define the directory that mods are installed in
# ubuntu path
# modDir = "C:/Program Files (x86)/Steam/steamapps/common/ARK/ShooterGame/Content/Mods"
# windows path
modDir = "C:\Program Files (x86)\Steam\steamapps\common\ARK\ShooterGame\Content\Mods\\"
stringPattern = r"\w+(.mod)"
regexPattern = re.compile(stringPattern)
modFolders = [mod for mod in os.listdir(modDir) if not regexPattern.match(mod)]
# loop through mod list and append directory to mod installation path
for items in modFolders:
modPath = "%s%s" % (modDir, items)
# parse mod name from meta file, store for later
modMetaFilePath = "%s\%s" % (modPath, "modmeta.info")
validPath = os.path.exists(modMetaFilePath)
if not validPath:
print('x is:', validPath)
pass
try:
modFile = open(modMetaFilePath, "rb")
binaryContent = modFile.read(-1)
modFile.close()
# if len == 0:
# print('length:', len(binaryContent))
# break
# print(type(binaryContent))
text = binaryContent.decode('UTF-8')
try:
parsedModName = text.split('Mods/')[1].split('/')[0]
if not parsedModName:
break
except ValueError as e:
pass
except Exception as e:
pass
for path, subdirs, files in os.walk(modPath):
# for i in range(len(subdirs)):
# print('Number of Sub Directories: ', len(subdirs))
# print('Current Directory Number in ', path, ': ', subdirs[i])
for name in files:
pattern = "*.uasset"
if fnmatch(name, pattern):
try:
path = os.path.normpath(path) + '\\'
if not path:
continue
try:
basePath = os.path.join('\\Game\\Mods\\', parsedModName)
except Exception as e:
pass
print('failed here:', str(e))
strippedBasePath = os.path.dirname(path).split(items)[1]
if not strippedBasePath:
print('failed at this point stripped path', strippedBasePath, '\n\t', 'path', path)
continue
revisedFileName = os.path.join(
os.path.basename(name).split('.')[0] + '.' + os.path.basename(name).split('.')[0] + ',')
finalPath = "%s%s\%s" % (basePath, strippedBasePath, revisedFileName)
flippedSlashFinalPath = finalPath.replace("\\", "/")
print(flippedSlashFinalPath)
with open("out.txt", "a") as external_file:
add_text = flippedSlashFinalPath
external_file.write(add_text)
external_file.close()
except Exception as e:
print('Something happened', str(e))
I initially installed an Ubuntu environment on Windows because I am unfamiliar with command line/bash/scripting in Windows as seen in the mod path (slashes are reversed and commands are different). I am much more familiar with MacOS and generally work in Linux environments and terminals.
I thought that this script would be usable by others with a bit of programming knowledge, but it is not the most user friendly.
In any case, this was my first attempt at writing something and trying to use good practices.
As another side project to practice more coding, I rewrote it and implemented more complex concepts.
Here is the refactored version:
import os
import sys
from fnmatch import fnmatch
import re
import winreg
stringPattern = r"\w+(.mod)"
regexPattern = re.compile(stringPattern)
class Mod(object):
def __init__(self):
pass
self.steam_path = None
self.mod_folders = []
self.mod_path = None
self.mod_path_list = []
self.mod_meta_path_list = []
self.full_mod_path = None
self.mod_meta = None
self.resource_path = None
#property
def GetSteamPath(self):
try:
steam_key = "SOFTWARE\WOW6432Node\Valve\Steam"
hkey = winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, steam_key)
except ValueError as e:
hkey = None
print(e.sys.exc_info())
try:
steam_path = winreg.QueryValueEx(hkey, "InstallPath")
except ValueError as e:
steam_path = None
print(e.sys.exc_info())
winreg.CloseKey(hkey)
self.steam_path = steam_path[0]
return self.steam_path
def GetArkModPath(self):
mod_dir = self.GetSteamPath[:] + "\steamapps\common\ARK\ShooterGame\Content\Mods\\"
self.mod_folders = [mod for mod in os.listdir(mod_dir) if not regexPattern.match(mod)]
mod_path_list = []
count = 1
while count < len(self.mod_folders):
for mod in self.mod_folders:
self.full_mod_path = ("%s%s\\" % (mod_dir, mod))
mod_path_list.append(self.full_mod_path)
self.mod_path_list = mod_path_list
if not mod:
break
# else:
# print('Test1', format(mod_path_list))
return self.mod_path_list
count += 1
def GetModResourceList(self, mod_path_list):
mod_folder_index = 0
for mod_path in mod_path_list:
mod_meta_path = "%s%s" % (mod_path, "modmeta.info")
self.mod_meta_path_list.append(mod_meta_path)
validPath = os.path.exists(mod_meta_path)
if not validPath:
# print('No Mod Meta File found at: ', mod_meta_path)
continue
try:
mod_file = open(mod_meta_path, "rb")
binary_content = mod_file.read(-1)
mod_file.close()
text = binary_content.decode('UTF-8')
try:
parsed_mod_name = text.split('Mods/')[1].split('/')[0]
if not parsed_mod_name:
break
except ValueError as e:
pass
except Exception as e:
pass
for path, subdirs, files in os.walk(mod_path):
# for i in range(len(subdirs)):
# print('Number of Sub Directories: ', len(subdirs))
# print('Current Directory Number in ', path, ': ', subdirs[i])
for uasset_file in files:
pattern = "*.uasset"
if fnmatch(uasset_file, pattern):
try:
path = os.path.normpath(path) + '\\'
if not path:
continue
try:
base_path = os.path.join('\\Game\\Mods\\', parsed_mod_name)
except Exception as e:
pass
print('failed here:', str(e))
stripped_base_path = os.path.dirname(path).split(self.mod_folders[mod_folder_index])[1]
resource_name = os.path.join(
os.path.basename(uasset_file).split('.')[0] + '.' + os.path.basename(uasset_file).split('.')[0] + ',')
full_path = "%s%s\%s" % (base_path, stripped_base_path, resource_name)
resource_path = full_path.replace("\\", "/")
self.resource_path = resource_path
# to see what text is written to the file, uncomment print here
print(self.resource_path)
with open("test_out.txt", "a") as external_file:
add_text = self.resource_path
external_file.write(add_text)
external_file.close()
except Exception as e:
print('Error: ', str(e))
# return self.resource_path[]
mod_folder_index += 1
ark = Mod()
ark_mods = ark.GetArkModPath()
ark.GetModResourceList(ark_mods)
This revised one is much more user friendly and does not necessarily require input or modification of variables. I tried to avoid requiring arguments passed in a method call because I wanted to automate the script as necessary without any input from user.
If you have Ark installed with some mods, then you may be able to actually use or test the scripts. I'm curious if they work for others. FYI, not all paths are usable. It might be best to delete/remove some or just pick out the ones you want. Can't really figure out the best way to identify this.
I struggled with creating an instance of my class, what attributes to set, what properties to set, setting(?) values for attributes for my class, calling those stored values, and just about everything else you can imagine.
Some things are obvious to me, such as it is pointless to create classes and methods in a script that is not necessarily being re-used. It was still fun and the point was to learn and practice.
In any case, I am open to any feedback because I do not know what is common sense for everything or best practices that others have seen. As a side note, I do versioning with local repos. I have worked with git and bitbucket before.
Preferred topics if not sure:
defensive programming
proper exception handling
SOLID (I'm def lacking here)
Classes (I barely got mine working if its not apparrent)
Methods (seems straight forward to me - if code is going to be copied, put it in a method)
Preferred nomenclature for everything, classes, methods, variables (I googled it, but still don't like how it looks especially when they get long)
Readability (I understand that if code requires comments, its not good)
Cleanliness (I guess this is an art, but I see spaghetti when I look at what I wrote)
Links to resources are welcome! I really appreciate any feedback and look forward to all comments.
As a "clean up" after my script's main purpose is complete, a function is called to recursively look through each folder and remove all files that end in a pre-determined set of extensions.
I during my testing, I discovered that some files with a file extension in the list of ones to delete actually throw an error: [Errno 1] Operation not permitted: '/location/of/locked/file.png. Looking at the file itself, it appears to be Locked (on mac).
How would I go about removing the locked attribute (should it exist) from every file/folder using Python, then delete the file if it ends in the extension?
Preferably this can all be done in the same function below, as it takes a long time to traverse the input directory - handling each only once is the way to go.
How does this affect the script's integrity on Windows?
I have taken care of programming it in a way that makes it compatible between the OSs but (to my knowledge) the locked attribute does not exist on Windows the way it does on mac and could cause unknown side-effects.
REMOVE_FILETYPES = ('.png', '.jpg', '.jpeg', '.pdf')
def cleaner(currentPath):
if not os.path.isdir(currentPath):
if currentPath.endswith(REMOVE_FILETYPES) or os.path.basename(currentPath).startswith('.'):
try:
os.remove(currentPath)
print('REMOVED: \"{removed}\"'.format(removed = currentPath))
except BaseException as e:
print('ERROR: Could not remove: \"{failed}\"'.format(failed = str(e)))
finally:
return True
return False
if all([cleaner(os.path.join(currentPath, file)) for file in os.listdir(currentPath)]):
try:
os.rmdir(currentPath)
print('REMOVED: \"{removed}\"'.format(removed = currentPath))
except:
print('ERROR: Could not remove: \"{failed}\"'.format(failed = currentPath))
finally:
return True
return False
cleaner(r'/path/to/parent/dir')
I would really appreciate if somebody could show me how to integrate such functionality into the sub-routine. Cheers.
EDIT: Removed error handling as per request
def cleaner(currentPath):
if sys.platform == 'darwin':
os.system('chflags nouchg {}'.format(currentPath))
if not os.path.isdir(currentPath):
if currentPath.endswith(REMOVE_FILETYPES) or os.path.basename(currentPath).startswith('.'):
try:
os.remove(currentPath)
print('REMOVED: \"{removed}\"'.format(removed=currentPath))
except PermissionError:
if sys.platform == 'darwin':
os.system('chflags nouchg {}'.format(currentPath))
os.remove(currentPath)
if all([cleaner(os.path.join(currentPath, file)) for file in os.listdir(currentPath)]) and not currentPath == SOURCE_DIR:
os.rmdir(currentPath)
print('REMOVED: \"{removed}\"'.format(removed=currentPath))
You can unlock the file with the chflags command:
os.system('chflags nouchg {}'.format(filename))
(There is a function os.chflags, but the flag associated with the locked status is not a regular one, but what the os module documentation calls a "user-defined" flag, as you can see by looking at os.stat(locked_filename).st_flags.)
To solve your problem I'd add the chflags command above to a specific except: for the error you get trying to remove a locked file, along with a platform check:
try:
os.remove(currentPath)
print('REMOVED: \"{removed}\"'.format(removed = currentPath))
except PermissionError:
if sys.platform == 'darwin':
os.system('chflags nouchg {}'.format(currentPath))
os.remove(currentPath)
else:
raise
except BaseException as e:
...
In the following piece of code, some_path is a string that corresponds to a path (may be a relative path)
def editable(some_path):
"""Safely check whether a file is editable."""
delete = not os.path.exists(some_path)
try:
with open(some_path, 'ab'):
return True
except:
return False
finally:
# If the file didn't exist before, remove the created version
if delete:
try:
os.remove(some_path)
except:
pass
In open's docs we read:
If the file cannot be opened, IOError is raised
Is IOError the only possible error I can get (UnicodeError comes to mind, or OSError etc) ?
os.remove docs are even more vague:
If path is a directory, OSError is raised
So what if the file is in use, or protected or...
UPDATE: How about shutil.move ? This seems to raise yet another shutil.Error(StandardError) - if I read the source right
I am having an issue with the else statement of this program... I have checked my spacing and it seems to be correct. I keep getting syntax error on the else statement. The program creates and file then attempts to upload it to a ftp server but if it fails to not say anything to the user and just continue It will try again when the program loops. Any help you could provide would be greatly appreciated.
#IMPORTS
import ConfigParser
import os
import random
import ftplib
from ftplib import FTP
#LOOP PART 1
from time import sleep
while True:
#READ THE CONFIG FILE SETUP.INI
config = ConfigParser.ConfigParser()
config.readfp(open(r'setup.ini'))
path = config.get('config', 'path')
name = config.get('config', 'name')
#CREATE THE KEYFILE
filepath = os.path.join((path), (name))
if not os.path.exists((path)):
os.makedirs((path))
file = open(filepath,'w')
file.write('text here')
file.close()
#Create Full Path
fullpath = path + name
#Random Sleep to Accomidate FTP Server
sleeptimer = random.randrange(1,30+1)
sleep((sleeptimer))
#Upload File to FTP Server
try:
host = '0.0.0.0'
port = 3700
ftp = FTP()
ftp.connect(host, port)
ftp.login('user', 'pass')
file = open(fullpath, "rb")
ftp.cwd('/')
ftp.storbinary('STOR ' + name, file)
ftp.quit()
file.close()
else:
print 'Something is Wrong'
#LOOP PART 2
sleep(180.00)
else is valid as part of an exception block, but it is only run if an exception is not raised and there must be a except defined before it.
(edit) Most people skip the else clause and just write code after exiting (dedenting) from the try/except clauses.
The quick tutorial is:
try:
# some statements that are executed until an exception is raised
...
except SomeExceptionType, e:
# if some type of exception is raised
...
except SomeOtherExceptionType, e:
# if another type of exception is raised
...
except Exception, e:
# if *any* exception is raised - but this is usually evil because it hides
# programming errors as well as the errors you want to handle. You can get
# a feel for what went wrong with:
traceback.print_exc()
...
else:
# if no exception is raised
...
finally:
# run regardless of whether exception was raised
...
I am using a python script to transfer the contents of three files to a different three files. The original files are data from three thermometers I have connected to an RPI running raspian. All the script is supposed to do is take the contents of the files and move them so that I can have another program (ComScript) read and parse them.
My problem is that if one or more of the thermometers is disconnected before the script starts, it freezes. It doesn't freeze if I disconnect a thermometer while the script is running.
Here is the code
import time
a = 1
while a == 1:
try:
tfile = open("/sys/bus/w1/devices/28-000004d2ca5e/w1_slave")
text = tfile.read()
tfile.close()
temperature = text
tfile2 = open("/sys/bus/w1/devices/28-000004d2fb20/w1_slave")
text2 = tfile2.read()
tfile2.close()
temperature2 = text2
tfile3 = open("/sys/bus/w1/devices/28-000004d30568/w1_slave")
text3 = tfile3.read()
tfile3.close()
temperature3 = text3
textfile = open("/home/pi/ComScriptPi/profiles/Temperature_parse/w1_slave1", "w ")
textfile2 = open("/home/pi/ComScriptPi/profiles/Temperature_parse/w1_slave2", "w ")
textfile3 = open("/home/pi/ComScriptPi/profiles/Temperature_parse/w1_slave3", "w ")
temperature = str(temperature)
temperature2 = str(temperature2)
temperature3 = str(temperature3)
textfile.write(temperature)
textfile2.write(temperature2)
textfile3.write(temperature3)
textfile.close()
textfile2.close()
textfile3.close()
print temperature
print temperature2
print temperature3
time.sleep(3)
except:
pass
I added the exception pass because I need it to keep running even if it gets bad values. WHen one of the thermometers is disconnected the file python is trying to read is blank, but still there.
Remove the blanket except.
Your script is not freezing, but any error you get is being ignored in an endless loop. Because you use a blanket except: you catch all exceptions, including the keyboard interrupt exception KeyboardInterrupt.
At the very least log the exception, and catch only Exception:
except Exception:
import logging
logging.exception('Oops: error occurred')
KeyboardInterrupt is a subclass of BaseException, not Exception and won't be caught by this except handler.
Take a look at the shutil module for copying files, you doing way too much work:
import time
import shutil
import os.path
paths = ('28-000004d2ca5e', '28-000004d2fb20', '28-000004d30568')
while True:
for i, name in enumerate(paths, 1):
src = os.path.join('/sys/bus/w1/devices', name, 'w1_slave')
dst = '/home/pi/ComScriptPi/profiles/Temperature_parse/w1_slave{}'.format(i)
try:
shutil.copyfile(src, dst)
except EnvironmentError:
import logging
logging.exception('Oops: error occurred')
time.sleep(3)
Handling files should only ever raise EnvironmentError or it's subclasses, there is no need to catch everything here.
The open of the unplugged device is most likely blocking because the device driver won't open if the device is not present.
You'll need to use os.open which is the equivalent of the Unix system call "open" and specify the flag O_NONBLOCK and check the return code. You can then use os.fdopen to turn the return value of os.open into a normal Python file object.