I'm trying to copy thousands files to a remote server. These files are generated in real-time within the script. I'm working on a Windows system and need to copy the files to a Linux server (hence the escaping).
I currently have:
import os
os.system("winscp.exe /console /command \"option batch on\" \"option confirm off\" \"open user:pass#host\" \"put f1.txt /remote/dest/\"")
I'm using Python to generate the files but need a way to persist the remote connection so that I can copy each file, to the server, as it is generated (as opposed to creating a new connection each time). That way, I'll only need to change the field in the put option thus:
"put f2 /remote/dest"
"put f3 /remote/dest"
etc.
I needed to do this and found that code similar to this worked well:
from subprocess import Popen, PIPE
WINSCP = r'c:\<path to>\winscp.com'
class UploadFailed(Exception):
pass
def upload_files(host, user, passwd, files):
cmds = ['option batch abort', 'option confirm off']
cmds.append('open sftp://{user}:{passwd}#{host}/'.format(host=host, user=user, passwd=passwd))
cmds.append('put {} ./'.format(' '.join(files)))
cmds.append('exit\n')
with Popen(WINSCP, stdin=PIPE, stdout=PIPE, stderr=PIPE,
universal_newlines=True) as winscp: #might need shell = True here
stdout, stderr = winscp.communicate('\n'.join(cmds))
if winscp.returncode:
# WinSCP returns 0 for success, so upload failed
raise UploadFailed
This is simplified (and using Python 3), but you get the idea.
Instead of using an external program (winscp) you could also use an python ssh-library like pyssh.
You would have to start persistent WinSCP sub-process in Python and feed the put commands to its standard input continuously.
I do not have Python example for this, but there's an equivalent JScript example:
https://winscp.net/eng/docs/guide_automation_advanced#inout
or C# example:
https://winscp.net/eng/docs/guide_dotnet#input
Though using WinSCP .NET assembly via its COM interface for Python would be a way easier:
https://winscp.net/eng/docs/library
Related
Iv'e been using the following shell command to read the image off a scanner named scanner_name and save it in a file named file_name
scanimage -d <scanner_name> --resolution=300 --format=tiff --mode=Color 2>&1 > <file_name>
This has worked fine for my purposes.
I'm now trying to embed this in a python script. What I need is to save the scanned image, as before, into a file and also capture any std output (say error messages) to a string
I've tried
scan_result = os.system('scanimage -d {} --resolution=300 --format=tiff --mode=Color 2>&1 > {} '.format(scanner, file_name))
But when I run this in a loop (with different scanners), there is an unreasonably long lag between scans and the images aren't saved until the next scan starts (the file is created as an empty file and is not filled until the next scanning command). All this with scan_result=0, i.e. indicating no error
The subprocess method run() has been suggested to me, and I have tried
with open(file_name, 'w') as scanfile:
input_params = '-d {} --resolution=300 --format=tiff --mode=Color 2>&1 > {} '.format(scanner, file_name)
scan_result = subprocess.run(["scanimage", input_params], stdout=scanfile, shell=True)
but this saved the image in some kind of an unreadable file format
Any ideas as to what may be going wrong? Or what else I can try that will allow me to both save the file and check the success status?
subprocess.run() is definitely preferred over os.system() but neither of them as such provides support for running multiple jobs in parallel. You will need to use something like Python's multiprocessing library to run several tasks in parallel (or painfully reimplement it yourself on top of the basic subprocess.Popen() API).
You also have a basic misunderstanding about how to run subprocess.run(). You can pass in either a string and shell=True or a list of tokens and shell=False (or no shell keyword at all; False is the default).
with_shell = subprocess.run(
"scanimage -d {} --resolution=300 --format=tiff --mode=Color 2>&1 > {} ".format(
scanner, file_name), shell=True)
with open(file_name) as write_handle:
no_shell = subprocess.run([
"scanimage", "-d", scanner, "--resolution=300", "--format=tiff",
"--mode=Color"], stdout=write_handle)
You'll notice that the latter does not support redirection (because that's a shell feature) but this is reasonably easy to implement in Python. (I took out the redirection of standard error -- you really want error messages to remain on stderr!)
If you have a larger working Python program this should not be awfully hard to integrate with a multiprocessing.Pool(). If this is a small isolated program, I would suggest you peel off the Python layer entirely and go with something like xargs or GNU parallel to run a capped number of parallel subprocesses.
I suspect the issue is you're opening the output file, and then running the subprocess.run() within it. This isn't necessary. The end result is, you're opening the file via Python, then having the command open the file again via the OS, and then closing the file via Python.
JUST run the subprocess, and let the scanimage 2>&1> filename command create the file (just as it would if you ran the scanimage at the command line directly.)
I think subprocess.check_output() is now the preferred method of capturing the output.
I.e.
from subprocess import check_output
# Command must be a list, with all parameters as separate list items
command = ['scanimage',
'-d{}'.format(scanner),
'--resolution=300',
'--format=tiff',
'--mode=Color',
'2>&1>{}'.format(file_name)]
scan_result = check_output(command)
print(scan_result)
However, (with both run and check_output) that shell=True is a big security risk ... especially if the input_params come into the Python script externally. People can pass in unwanted commands, and have them run in the shell with the permissions of the script.
Sometimes, the shell=True is necessary for the OS command to run properly, in which case the best recommendation is to use an actual Python module to interface with the scanner - versus having Python pass an OS command to the OS.
I want to be able to store the output of a command I run (say top) on a remote host, from within a Python script using SSH, into a file.
I know how to use SSH( I am currently using Paramiko to connect to the remote device). I need to run the command, and then store the output in a text file.
If you use paramiko's exec_command method to execute the command, you could do something like this:
stdin, stdout, stderr = your_ssh_client_object.exec_command("top")
with open("out.txt", "w") as f:
f.writelines(stdout.readlines())
If you care about error output too, you need to append it to the same file or store it in a separate one.
(NOTE: The above code is neither tested nor working on it's own, since the OP didn't provide a minimal, workable example himself.
What have you tried something?
Depend on your command you can do something like this:
ssh user#machine command > log
The log will be saved to your machine.
A real example:
ssh root#192.168.x.x ls > log
If your command does not support outputs to stdout, run it like this:
ssh root#192.168.x.x "command -o output; cat output" > log
With python, you can use subprocess and do the respective command, something like:
import subprocess
def yourfunction():
subprocess.call("ssh user#machine command > log",shell=False)
yourfunction()
So i have a script from Python that connects to the client servers then get some data that i need.
Now it will work in this way, my bash script from the client side needs input like the one below and its working this way.
client.exec_command('/apps./tempo.sh' 2016 10 01 02 03))
Now im trying to get the user input from my python script then transfer it to my remotely called bash script and thats where i get my problem. This is what i tried below.
Below is the method i tried that i have no luck working.
import sys
client.exec_command('/apps./tempo.sh', str(sys.argv))
I believe you are using Paramiko - which you should tag or include that info in your question.
The basic problem I think you're having is that you need to include those arguments inside the string, i.e.
client.exec_command('/apps./tempo.sh %s' % str(sys.argv))
otherwise they get applied to the other arguments of exec_command. I think your original example is not quite accurate in how it works;
Just out of interest, have you looked at "fabric" (http://www.fabfile.org ) - this has lots of very handy funcitons like "run" which will run a command on a remote server (or lots of remote servers!) and return you the response.
It also gives you lots of protection by wrapping around popen and paramiko for hte ssh login etcs, so it can be much more secure then trying to make web services or other things.
You should always be wary of injection attacks - Im unclear how you are injecting your variables, but if a user calls your script with something like python runscript "; rm -rf /" that would have very bad problems for you It would instead be better to have 'options' on the command, which are programmed in, limiting the users input drastically, or at least a lot of protection around the input variables. Of course if this is only for you (or trained people), then its a little easier.
I recommend using paramiko for the ssh connection.
import paramiko
ssh_client = paramiko.SSHClient()
ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh_client.connect(server, username=user,password=password)
...
ssh_client.close()
And If you want to simulate a terminal, as if a user was typing:
chan=ssh_client.invoke_shell()
chan.send('PS1="python-ssh:"\n')
def exec_command(cmd):
"""Gets ssh command(s), execute them, and returns the output"""
prompt='python-ssh:' # the command line prompt in the ssh terminal
buff=''
chan.send(str(cmd)+'\n')
while not chan.recv_ready():
time.sleep(1)
while not buff.endswith(prompt):
buff+=ssh_client.chan.recv(1024)
return buff[:len(prompt)]
Example usage: exec_command('pwd')
And the result would even be returned to you via ssh
Assuming that you are using paramiko you need to send the command as a string. It seems that you want to pass the command line arguments passed to your Python script as arguments for the remote command, so try this:
import sys
command = '/apps./tempo.sh'
args = ' '.join(sys.argv[1:]) # all args except the script's name!
client.exec_command('{} {}'.format(command, args))
This will collect all the command line arguments passed to the Python script, except the first argument which is the script's file name, and build a space separated string. This argument string is them concatenated with the bash script command and executed remotely.
I have composed an ArcPy script which is run via a windows scheduler.The same script is loaded into a script tool so a user can run the process manually. I've used: get parameters as text, with or's and not's, to hard-wire the standard variables if they are not speicifed.
ReportFolder = arcpy.GetParameterAsText(0)
if ReportFolder == '#' or not ReportFolder:
ReportFolder = "C:\\Data\\GIS"
The process runs and during so writes to a text file log, for example:
txtFile.write("= For ArcGIS 10.3.1: Date: "+str(timed)),txtFile.write ('\n')
I'd like to record what method was used to execute the script; was it via the windows scheduler, or by the script tool via ArcGIS, or by a python client like PyScripter.
Is anyone aware of some form of os environment thingy that can be called by Python?
I have a magnet link from a torrent site that should open this program called transmission.How do i open it with Python?
i'm on ubuntu btw.
I heard this thing called xdg-open can do the trick but how do i use it for opening the magnet link?
and if that's not the code i'm looking for, what should i use to run the magnet link?
Taking a look at the command line arguments of transmission-gtk helps:
$ transmission-gtk --help
Usage: transmission-gtk [OPTION...] [torrent files or urls]
A quick and dirty approach for a python solution is using the os module:
import os
os.system("transmission-gtk urlhere")
A much better and more sophisticated way to do such calls to external programms is using the subprocess module. A few more examples can be found under python - how to create a subprocess?.
xdg-open works pretty much the same way. But instead of calling the transmission client directly, it chooses the preferred Torrent application (preferred in this case means default application, which can be set by using the default applications menu in the Ubuntu system settings). Repeatedly pointing you at the help texts given by calling the programm from commandline, it might be interesting to inspect the exit codes of xdg-open:
$ xdg-open --manual
...
1 Error in command line syntax.
2 One of the files passed on the command line did not exist.
3 A required tool could not be found.
4 The action failed.
The below code sums up the method to download on all the operating systems.
import subprocess , os , sys
def open_magnet(magnet):
"""Open magnet according to os."""
if sys.platform.startswith('linux'):
subprocess.Popen(['xdg-open', magnet],
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
elif sys.platform.startswith('win32'):
os.startfile(magnet)
elif sys.platform.startswith('cygwin'):
os.startfile(magnet)
elif sys.platform.startswith('darwin'):
subprocess.Popen(['open', magnet],
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
else:
subprocess.Popen(['xdg-open', magnet],
stdout=subprocess.PIPE, stderr=subprocess.PIPE)