I'm currently an Intern in a IT service and I've been asked to build a web based app using Python that will run on a Linux environment. This web app has to be WSGI-compliant and I cannot use any framework.
My issue currently is that I want to have a variable set as a list of files in the said directory. Therefore I can then proceed to list those files by printing a table having each row being a file.
I am aware of os.listdir() but can't find a way to use it on a remote server (which is supposed not to be the case considering what google searches showed me...).
I tried an os.system(ssh root#someip:/path/to/dir/) but as python doc states, I cant get the output I want as it returns some integers...
Below is a piece of my script.
#ip is = to the ip of the server I want to list.
ip = 192..............
directory = "/var/lib/libvirt/images/"
command = "ssh root#"+ip+" ls "+directory
dirs = os.system(command)
files = ""
table_open = "<table>"
table_close = "</table>"
table_title_open = "<th>Server: "
table_title_close = "</th>"
tr_open = "<tr>"
tr_close = "</tr>"
td_open = "<td>"
td_close = "</td>"
input_open = "<input type='checkbox' name='choice' value='"
input_close = "'>"
#If i don't put dirs in brackets it raises an error (dirs not being iterable)
for file in [dirs]:
files = files + tr_open+td_open+file+td_close+td_open+input_open+file+input_close+td_close+tr_close
table = table_open+table_title_open+str(num_server)+table_title_close+files+table_close
I've tried this with a local directory (with os.listdir) and it works perfectly. I am having troubles only with remote directory listing...
I do hope that my question is crystal clear, if not I'll do my best to be more accurate.
Thanks in advance,
-Karink.
You can use subprocess module, here is an example:
import subprocess
ls = subprocess.Popen(['ssh','user#xx.xx.xx.xx', 'ls'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = ls.communicate()
print out
print err
You may also use pysftp
first install it using pip install pysftp then the below code can list the files on remote linux machine from windows as well
import pysftp
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
with pysftp.Connection('ipLinuxmachine', username='username', password='passwd',cnopts=cnopts) as sftp:
out=sftp.execute('cd path of directory ; ls')
print out
Related
I have a python script that makes backups to my QNAP.
QNAP from time to time changes the default name (visible via ftp) of the external drive.
In the new firmware it changed to Seagate Expansion Drive.
Now I need to correct the path in the config file:
serverlocation = Seagate Expansion Drive/www03/config
I tried:
serverlocation = "Seagate Expansion Drive"/www03/config
and
serverlocation = 'Seagate Expansion Drive'/www03/config
but still it says that the directory was not found on the ftp server.
The script uses that variable in this way:
try: ftp.mkd(self.config.get(jn, "serverlocation"))
except: pass
try: ftp.mkd("%s/%s" % (self.config.get(jn, "serverlocation"), self.date))
except: pass
syslog.syslog("Uploading %s..." % os.path.basename(location))
try:
th = FTPKeepalive(ftp)
th.start()
ftp.storbinary("STOR %s/%s/%s" % (self.config.get(jn, "serverlocation"), self.date, os.p>
th.alive = False
I am not a python developer, I am wondering how I can fix that path with spaces.
I have the following program to 7-Zip a directory of files via python subprocess:
import subprocess
import os
appPath = r'C:\\Users\Person\Desktop\7-Zip'
zApp = r'7z.exe'
zAction = r'a'
zipFileName = r'C:\\Users\Person\Desktop\test.zip'
zPass = r'password'
zAnswer = '-y'
zDir = r'C:\\Users\Person\Desktop\test'
progDir = os.path.join(appPath, zApp)
cmd = [zApp, zAction, zipFileName, zPass, zAnswer, zDir]
subprocess.Popen(cmd, executable=progDir, stderr=subprocess.STDOUT, stdout=subprocess.PIPE)
However, I keep running into the following error:
FileNotFoundError: [WinError 2] The system cannot find the file specified
I am using Python 3.6 on a windows machine.
I would presume it is because you don't have C:\\Users\Person\Desktop\7-Zip in the PATH environment variable for either the current user or the machine.
In your example you already seem to be aware of this...
Change:
cmd = [zApp, zAction, zipFileName, zPass, zAnswer, zDir]
To:
cmd = [progDir, zAction, zipFileName, zPass, zAnswer, zDir]
Or else edit the PATH environment variable for the user or machine and put the path to the 7zip executable on it so that it can be called via simply 7z
I had a script that was working. I made one small change and now it stopped working. The top version works, while the bottom one fails.
def makelocalconfig(file="TEXT"):
host = env.host_string
filename = file
conf = open('/home/myuser/verify_yslog_conf/%s/%s' % (host, filename), 'r')
comment = open('/home/myuser/verify_yslog_conf/%s/localconfig.txt' % host, 'w')
for line in conf:
comment.write(line)
comment.close()
conf.close()
def makelocalconfig(file="TEXT"):
host = env.host_string
filename = file
path = host + "/" + filename
pwd = local("pwd")
conf = open('%s/%s' % (pwd, path), 'r')
comment = open('%s/%s/localconfig.txt' % (pwd, host), 'w')
for line in conf:
comment.write(line)
comment.close()
conf.close()
For troubleshooting purposes I added a print pwd and print path line to make sure the variables were getting filled correctly. pwd comes up empty. Why isn't this variable being set correctly? I use this same format of
var = sudo("cmd")
all the time. Is local different than sudo and run?
In short, you may need to add capture=True:
pwd = local("pwd", capture=True)
local runs a command locally:
a convenience wrapper around the use of the builtin Python subprocess
module with shell=True activated.
run runs a command on a remote server and sudo runs a remote command as super-user.
There is also a note in the documentation:
local is not currently capable of simultaneously printing and capturing output, as run/sudo do. The capture kwarg allows you to switch between printing and capturing as necessary, and defaults to False.
When capture=False, the local subprocess’ stdout and stderr streams are hooked up directly to your terminal, though you may use the global output controls output.stdout and output.stderr to hide one or both if desired. In this mode, the return value’s stdout/stderr values are always empty.
I have an issue putting files to a server that contains hyphens ("-"), and I think that it may be because of how Linux is treating the file, but I am in no way sure. The script is scanning a folder for pictures/items, puts them in a list and then transferring all items to the server.
This is a part of the script:
def _transferContent(locale):
## Transferring images to server
now = datetime.datetime.now()
localImages = '/home/bcns/Pictures/upload/'
localList = os.listdir(localImages)
print("Found local items: ")
print(localList)
fname = "/tmp/backup_images_file_list"
f = open(fname, 'r')
remoteList = f.read()
remoteImageLocation = "/var/www/bcns-site/pics/photos/backup_" + locale + "-" + `now.year` + `now.month` + `now.day` + "/"
print("Server image location: " + remoteImageLocation)
## Checking local list against remote list (from the server)
for localItem in localList:
localItem_fullpath = localImages + localItem
if os.path.exists(localItem_fullpath):
if localItem in remoteList:
print("Already exists: " + localItem)
else:
put(localItem_fullpath, remoteImageLocation)
else:
print("File not found: " + localItem)
And this is the out put:
Directory created successfully
/tmp/bcns_deploy/backup_images_file_list
[<server>] download: /tmp/backup_images_file_list <- /tmp/bcns_deploy/backup_images_file_list
Warning: Local file /tmp/backup_images_file_list already exists and is being overwritten.
Found local items:
['darth-vader-mug.jpg', 'gun-dog-leash.jpg', 'think-safety-first-sign.jpg', 'hzmp.jpg', 'cy-happ-short-arms.gif', 'Hogwarts-Crest-Pumpkin.jpg']
Server image location: /var/www/bcns-site/pics/photos/backup_fujitsu-20131031/
[<server>] put: /home/bcns/Pictures/upload/darth-vader-mug.jpg -> /var/www/bcns-site/pics/photos/backup_fujitsu-20131031/
Fatal error: put() encountered an exception while uploading '/home/bcns/Pictures/upload/darth-vader-mug.jpg'
Underlying exception:
Failure
I have tried to remove the hyphons, and then the transfer works just fine.
Server runs Ubuntu 12.04 and client runs Debian 7.1 on ext3 disks.
Irritating error, but anyone out here that has a clue on what might make this error?
Dashes in command line options in Linux matter, but dashes in the middle of filenames are file.
Check file permissions -- it's possible that in transferring one file manually, the perms are set differently than if Fabric transfers.
I suggest using put() to transfer a directory at a time. This will help to make sure all the files (and permissions) are what they should be.
Example (untested):
def _transferContent(locale):
## Transferring images to server
now = datetime.datetime.now()
localImageDir = '/home/bcns/Pictures/upload/'
remoteImageDir = "/var/www/bcns-site/pics/photos/backup_" + locale + "-" + `now.year` + `now.month` + `now.day` + "/"
print("Server image location: " + remoteImageDir)
put( localImagesDir, remoteImageDir)
I've been trying to trouble this for days now, and would appreciate some help --
Basically, I wrote the following Python script
import os, sys
# =__=__=__=__=__=__=__ START MAIN =__=__=__=__=__=__=__
if __name__ == '__main__':
# initialize variables
all_files = []
# directory to download data siphon files to
dDir = '/path/to/download/directory/'
# my S3 bucket
s3bucket = "com.mybucket/"
foldername = "test"
# get a list of available feeds
feeds = <huge JSON object with URLs to feeds>
for item in range(feeds['count']):
# ...check if the directory exists, and if not, create the directory...
if not os.path.exists(folderName):
os.makedirs(folderName)
... ... ...
# Loop through all the splits
for s in dsSplits:
... ... ...
location = requestFeedLocation(name, timestamp)
... ... ...
downloadFeed(location[0], folderName, nameNotGZ)
# THIS IS WHERE I AM HAVING PROBLEMS!!!!!!!!!!!
cmd = 's3cmd sync 'dDir+folderName+'/ s3://'+s3bucket+'/'
os.system(cmd)
Everything in my code works...when I run this straight from the command line, everything runs as expected...however, when I have it executed via cron -- the following DOES NOT execute (everything else does)
# THIS IS WHERE I AM HAVING PROBLEMS!!!!!!!!!!!
cmd = 's3cmd sync 'dDir+folderName+'/ s3://'+s3bucket+'/'
os.system(cmd)
To answer a few questions, I am running the cron as root, s3cmd is configured for the root user, OS is Ubuntu 12.04, python version is 2.7, all of the necessary directories have Read / Write permissions...
What am I missing?
First check variable 'foldername' in command you have used N in variable.
In command syntax you need to add plus (+) sign in prefix like +dDir+folderName+
So I hope command would be like below..
cmd = 's3cmd sync '+dDir+foldername+'/ s3://'+s3bucket+'/'
os.system(cmd)
Here I found some more s3cmd sync help: http://tecadmin.net/s3cmd-file-sync-with-s3bucket/