I have an issue putting files to a server that contains hyphens ("-"), and I think that it may be because of how Linux is treating the file, but I am in no way sure. The script is scanning a folder for pictures/items, puts them in a list and then transferring all items to the server.
This is a part of the script:
def _transferContent(locale):
## Transferring images to server
now = datetime.datetime.now()
localImages = '/home/bcns/Pictures/upload/'
localList = os.listdir(localImages)
print("Found local items: ")
print(localList)
fname = "/tmp/backup_images_file_list"
f = open(fname, 'r')
remoteList = f.read()
remoteImageLocation = "/var/www/bcns-site/pics/photos/backup_" + locale + "-" + `now.year` + `now.month` + `now.day` + "/"
print("Server image location: " + remoteImageLocation)
## Checking local list against remote list (from the server)
for localItem in localList:
localItem_fullpath = localImages + localItem
if os.path.exists(localItem_fullpath):
if localItem in remoteList:
print("Already exists: " + localItem)
else:
put(localItem_fullpath, remoteImageLocation)
else:
print("File not found: " + localItem)
And this is the out put:
Directory created successfully
/tmp/bcns_deploy/backup_images_file_list
[<server>] download: /tmp/backup_images_file_list <- /tmp/bcns_deploy/backup_images_file_list
Warning: Local file /tmp/backup_images_file_list already exists and is being overwritten.
Found local items:
['darth-vader-mug.jpg', 'gun-dog-leash.jpg', 'think-safety-first-sign.jpg', 'hzmp.jpg', 'cy-happ-short-arms.gif', 'Hogwarts-Crest-Pumpkin.jpg']
Server image location: /var/www/bcns-site/pics/photos/backup_fujitsu-20131031/
[<server>] put: /home/bcns/Pictures/upload/darth-vader-mug.jpg -> /var/www/bcns-site/pics/photos/backup_fujitsu-20131031/
Fatal error: put() encountered an exception while uploading '/home/bcns/Pictures/upload/darth-vader-mug.jpg'
Underlying exception:
Failure
I have tried to remove the hyphons, and then the transfer works just fine.
Server runs Ubuntu 12.04 and client runs Debian 7.1 on ext3 disks.
Irritating error, but anyone out here that has a clue on what might make this error?
Dashes in command line options in Linux matter, but dashes in the middle of filenames are file.
Check file permissions -- it's possible that in transferring one file manually, the perms are set differently than if Fabric transfers.
I suggest using put() to transfer a directory at a time. This will help to make sure all the files (and permissions) are what they should be.
Example (untested):
def _transferContent(locale):
## Transferring images to server
now = datetime.datetime.now()
localImageDir = '/home/bcns/Pictures/upload/'
remoteImageDir = "/var/www/bcns-site/pics/photos/backup_" + locale + "-" + `now.year` + `now.month` + `now.day` + "/"
print("Server image location: " + remoteImageDir)
put( localImagesDir, remoteImageDir)
Related
I'm trying to get the vendorId, the productCode and the SerialNumber of a usb device using python.
Therefore I created the following script which runs quite well in a linux environment.
import sys
import usb
devs = usb.core.find(find_all=True)
nCount=0
for dev in devs:
for cfg in dev:
intf = usb.util.find_descriptor(cfg,bInterfaceClass=0x8)
if intf is not None:
nCount += 1
try:
sys.stdout.write("USB device " + usb.util.get_string(dev,dev.iProduct,None) + '\n')
except:
sys.stdout.write("USB device " + str(nCount) + '\n')
sys.stdout.write("------" + '\n')
sys.stdout.write("VendorId = " + hex(dev.idVendor) + '\n')
sys.stdout.write("ProductId = " + hex(dev.idProduct) + '\n')
if not dev.iSerialNumber == 0:
try:
sys.stdout.write("SerialNbr = " + usb.util.get_string(dev,dev.iSerialNumber ) + '\n')
except NotImplementedError:
sys.stdout.write("Sreial-Number not readable")
else:
sys.stdout.write("SerialNbr = none" + '\n')
sys.stdout.write('\n')
As soon as a start the script in windows, the script returns an error "NotImplementedError" when calling the function usb.util.get_string(dev,dev.iSerialNumber)
I found that there is a problem with the windows drivers and the libusb.
The solution wI0m heading to, does not allow to change system drivers with Zadig first. IIt must be callable without changing drivers first.
So my question is, if it's possible to get the serial number with python on a windows system without changing the drivers first.
I have a python script that makes backups to my QNAP.
QNAP from time to time changes the default name (visible via ftp) of the external drive.
In the new firmware it changed to Seagate Expansion Drive.
Now I need to correct the path in the config file:
serverlocation = Seagate Expansion Drive/www03/config
I tried:
serverlocation = "Seagate Expansion Drive"/www03/config
and
serverlocation = 'Seagate Expansion Drive'/www03/config
but still it says that the directory was not found on the ftp server.
The script uses that variable in this way:
try: ftp.mkd(self.config.get(jn, "serverlocation"))
except: pass
try: ftp.mkd("%s/%s" % (self.config.get(jn, "serverlocation"), self.date))
except: pass
syslog.syslog("Uploading %s..." % os.path.basename(location))
try:
th = FTPKeepalive(ftp)
th.start()
ftp.storbinary("STOR %s/%s/%s" % (self.config.get(jn, "serverlocation"), self.date, os.p>
th.alive = False
I am not a python developer, I am wondering how I can fix that path with spaces.
I have a script that combines around 1000 files by using an external program. It does so by copying the first file to combined.sym and then combining the this file with all other files, one by one. Code is as follows:
# gets all .sym files
symfiles = []
onlyfiles = [f for f in listdir(myPath) if isfile(join(myPath, f))]
for f in onlyfiles:
if '.sym' == splitext(f)[1]:
symfiles.append(f)
# copies one initial combined file (point to start from)
combinedFileName = "combined.sym"
copyfile(myPath + symfiles[0], myPath + combinedFileName)
for sym in symfiles:
print("processing " + str(sym))
cmdString = combinedFileName + " "
cmdString += sym
cmdString += " -a " + combinedFileName
finalCmdString = r'cmd.exe /C fuseFiles.bat "' + myPath + '" "' + cmdString + '"'
call(finalCmdString)
print("done")
The batchfile I'm calling looks like this, but that is not the issue:
#set PATH=\path\to\executable\;%PATH%
#cd %1
#FusingProgram "%2"
Now, when I run this script, it runs for like 15 Minutes before it just stops and returns to the CMD.
processing onefile.sym
processing anotherfile.sym
processing evenmorefiles.sym
processing yougetit.sym
C:\Users\test\>
As you can see, it misses the "done" print, and it didn't process all files. The combined.sym is created and expanded by the other .sym files
When checking the Windows events, I can see, that it logs python as crashed and lists ntdll.dll as faulty module(sorry for german log):
Name der fehlerhaften Anwendung: python.exe, Version: 0.0.0.0, Zeitstempel: 0x53787196
Name des fehlerhaften Moduls: ntdll.dll, Version: ****, Zeitstempel: 0x5b6db230
Ausnahmecode: 0xc0000005
Fehleroffset: 0x0002e2eb
ID des fehlerhaften Prozesses: 0x6224
Startzeit der fehlerhaften Anwendung: 0x01d454e0ff20377c
Pfad der fehlerhaften Anwendung: C:\pat\to\python\python.exe
Pfad des fehlerhaften Moduls: C:\WINDOWS\SysWOW64\ntdll.dll
Berichtskennung: 8b4254f0-c0d5-11e8-9a6f-901b0e5a6206
I've done some research and it seems like I'd have to resolve this by updating the ntdll.dll file, but my system is all up to date, so there is nothing I can do about that (except for downloading another version from some untrustworthy site, which I'm not gonna do)
All this being said, have you ever encountered a similar issue or is this just a stupid mistake I made?
I am a little confused about the way Popen works and I am hoping this is something silly. I am never getting a completion, and poll seems to be returning something odd (log attached)
This is backing up a triplet of schemas (Tablespace) using a utility (CSSBACKUP) supplied to do this.
for i in range(len(schematype)):
schema_base = schemaname + '_' + schematype[i] # we need this without the trailing space.
outputstring = base_folder + '\\' + schemaname + '\\' + schema_base + '_' + timestr + '_cssbackup '
rc = os.unlink(outputstring) # wont run if there is a backup already
logstring = base_folder + '\\' + schemaname + '\\' + schema_base + '_' + timestr + '.log'
exString = "cssbackup " + phys_string + '-schema '+ schema_base + ' ' + '-file ' + outputstring + '-log '+ logstring
logging.debug(exString)
processlist.append(subprocess.Popen(exString)) # start a seperate thread for each one, but we don't want to proceed until processlist[].poll == None (thread is complete)
procdone[i] = False
Now that I have all the processes spawn, I need to sync them up
while finishit < len(schematype):
time.sleep(CSTU_CFG.get(hostname).get("logintv")) # need a delay to keep this program from thrashing
for i in range(len(schematype)): # check each of the procs
if procdone[i] is not True: # if it completed, skip it
if processlist[i].poll is not None: # if it returns something other than "none" it's still running
logging.debug(' Running '+ schematype[i] + ' ' + str(processlist[i])+ ' '+ str(time.time() - start_time))
procdone[i] = False
else:
procdone[i] = True # None was returned so it's finished
logging.debug(' Ended '+ schematype[i]) # log it
finishit = finishit + 1 # update the count
processlist[i].kill # kill the process that was running ( Saves memory )
logging.debug('Dump functions complete')
When I run this, I don't get what I am expecting. I was expecting a pid in the return but I dont see it. So what I get back isnt useful for the .poll command.
So the program runs forever even after the shell that it spawned are gone.
I'm missing something basic.
THanks
11:26:26,133 root, DEBUG Running local 30.014784812927246
11:26:26,133 root, DEBUG Running central 30.014784812927246
11:26:26,133 root, DEBUG Running mngt 30.014784812927246
11:26:56,148 root, DEBUG Running local 60.02956962585449
11:26:56,148 root, DEBUG Running central 60.02956962585449
11:26:56,148 root, DEBUG Running mngt 60.02956962585449
11:27:26,162 root, DEBUG Running local 90.04435467720032
11:27:26,162 root, DEBUG Running central 90.04435467720032
11:27:26,162 root, DEBUG Running mngt 90.04435467720032
11:27:56,177 root, DEBUG Running local 120.05913925170898
11:27:56,177 root, DEBUG Running central 120.05913925170898
11:27:56,177 root, DEBUG Running mngt 120.05913925170898
11:28:26,192 root, DEBUG Running local 150.07392406463623
You should call poll. if processlist[i].poll is not None will always evaluate to True, because processlist[i].poll is the function object, not the result of processlist[i].poll().
Edit:
This looks an quite complicated way to do something like
p = multiprocessing.Pool(n)
p.map_async(subprocess.call, commands)
As a suggestion, you may want to check the multiprocessing module.
I'm currently an Intern in a IT service and I've been asked to build a web based app using Python that will run on a Linux environment. This web app has to be WSGI-compliant and I cannot use any framework.
My issue currently is that I want to have a variable set as a list of files in the said directory. Therefore I can then proceed to list those files by printing a table having each row being a file.
I am aware of os.listdir() but can't find a way to use it on a remote server (which is supposed not to be the case considering what google searches showed me...).
I tried an os.system(ssh root#someip:/path/to/dir/) but as python doc states, I cant get the output I want as it returns some integers...
Below is a piece of my script.
#ip is = to the ip of the server I want to list.
ip = 192..............
directory = "/var/lib/libvirt/images/"
command = "ssh root#"+ip+" ls "+directory
dirs = os.system(command)
files = ""
table_open = "<table>"
table_close = "</table>"
table_title_open = "<th>Server: "
table_title_close = "</th>"
tr_open = "<tr>"
tr_close = "</tr>"
td_open = "<td>"
td_close = "</td>"
input_open = "<input type='checkbox' name='choice' value='"
input_close = "'>"
#If i don't put dirs in brackets it raises an error (dirs not being iterable)
for file in [dirs]:
files = files + tr_open+td_open+file+td_close+td_open+input_open+file+input_close+td_close+tr_close
table = table_open+table_title_open+str(num_server)+table_title_close+files+table_close
I've tried this with a local directory (with os.listdir) and it works perfectly. I am having troubles only with remote directory listing...
I do hope that my question is crystal clear, if not I'll do my best to be more accurate.
Thanks in advance,
-Karink.
You can use subprocess module, here is an example:
import subprocess
ls = subprocess.Popen(['ssh','user#xx.xx.xx.xx', 'ls'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = ls.communicate()
print out
print err
You may also use pysftp
first install it using pip install pysftp then the below code can list the files on remote linux machine from windows as well
import pysftp
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
with pysftp.Connection('ipLinuxmachine', username='username', password='passwd',cnopts=cnopts) as sftp:
out=sftp.execute('cd path of directory ; ls')
print out