I'm using Python to transfer (via scp) and database a large number of files. One of the servers I transfer files to has odd ssh config rules to stop too many ssh requests from a single location. The upshot of this is that my python script, currently looping through files and copying via os.system, hangs after a few files have been transferred.
Is there a way in which Python could open up an ssh or other connection to the server, so that each file being transferred does not require an instance of ssh login?
Thanks,
You probably want to look into the paramiko module.
There's a Copy files over SSH using paramiko recipe using it that might be helpful.
Check out SFTP in the Python module Paramiko. You can do multiple file transfers in a single session.
This is not really python specific, but it probably depends on what libraries you can use.
What you need is a way to send files through a single connection.
(This is probably better suited to superuser or severfault.com though.)
Create tarfile locally, upload it and unpack at target?
Maybe you could even run 'tar xz' remotely and upload the file on stdin over SSH? (As MichaelDillon says in the comment, Python can create the tarfile on the fly...)
Is SFTP an option?
Rsync over SSH?
Twisted is an async library that can handle many sockets/connections at once. Is probably overkill for your solution though,
Hope it helps.
Related
I want to send files from my remote server to (a cloud linux machine) a local client (my machine). I have tried to use paramiko, but I just don't find it to have enough functionality. Please, if you have anything could help me then I would love to here it.
There are a few ways to do this, but the easiest way is to use the scp command.
Here is an example:
scp user#remote.server.com:/path/to/file /local/path/to/save/file
You will need to replace the user#remote.server.com and /path/to/file with the appropriate values for your situation.
I normally use a bash script to grab all the files onto local machine and use glob to process all the files. Just wondering what would be the best way to use python (instead of another bash script) to ssh into each server and process those files?
My current program runs as
for filename in glob.glob('*-err.txt'):
input_open = open (filename, 'rb')
for line in input_open:
do something
My files all have the ending -err.txt and the directories where they reside in the remote server have the same name /documents/err/. I am not able to install third party libraries as I don't have the permission.
UPDATE
I am trying to not to scp the files from the server but to read it on the remote server instead..
I want to use a local python script LOCALLY to read in files on remote server.
The simplest way to do it is to use paramico_scp to use ssh copy from the remote server (How to scp in python?)
If you are not allowed to download any libraries, you can create SSH key pair so that connecting to server does not require a password (https://www.debian.org/devel/passwordlessssh). You then can for each file do
import os
os.system('scp user#host:/path/to/file/on/remote/machine /path/to/local/file')
Note that using system is usually considered less portable than using libraries. If you give the script that use system('scp ...') to copy the files and they do not have SSH key pair set up, they will experience problems
Looks like you want to use a local Python script remotely. This has been answered here.
Fist of all, due to Company Policy, Paramiko, or installing anything that requires administrative access to local machine it right out; otherwise I would have just done that.
All I have to work with is python with standard libraries & putty.
I am attempting to automate some tedious work that involves logging into a network device (usually Cisco, occasionally Alcatel-Lucent, or Juniper), running some show commands, and saving the data. (I am planning on using some other scripts to pull data from this file, parse it, and do other things, but that should be irrelevant to the task of retrieving the data.) I know this can be done with telnet, however I need to do this via ssh.
My thought is to use putty's logging ability to record output from a session to a file. I would like to use Python to establish a putty session, send scripted log-in and show commands, and then close the session. Before I set out on this crusade, does anyone know of any way to do this? The closest answers I have found to this all suggest to use Paramiko, or other python ssh library; I am looking for a way to do this given the constraints I am under.
The end-result would ideal be able to be used as a function, so that I can iterate through hundreds of devices from a list of ip addresses.
Thank you for your time and consideration.
If you can't use paramiko, and Putty is all you get so the correct tool is actually not Putty - it's his little brother Plink - you can download it here
Plink is the command line tool for Putty and you can your python script to call it using os.system("plink.exe [options] username#server.com [command])
See MAN Page here
Hope it will help,
Liron
I am using a Mac.
I have a Python file on a server. Let's say it's here: http://jake.com/python_file.py. I want to make a script that I can run in Terminal by double-clicking and will run the Python file in Terminal on the local machine. Any ideas?
Would I have to use SSH to connect to the server and download the file to a temporary location on the hard drive (/tmp?) and then delete it when I'm done? But there's another problem with this. It would have to download the Python file to a location in the user's home folder because I don't think users have the necessary permissions to write to the folder /tmp or /var or something like that.
I was looking around for a solution to my problem and found this. It talks about how to execute a remote script using SSH on unix but I tried doing this with my Python file and it didn't work.
In case you didn't realize, the main reason why I am looking to do this is so that a user can run the file locally but they are unable to read/edit the actual Python file which is stored on the server.
If anyone has any ideas on how to accomplish this (whether using the ideas mentioned above or not) please let me know I'd really appreciate it!
Thanks,
Jake
How about a python script to download and execute the code like so?
import requests
py1 = requests.get('https://example.com/file.py').content
exec(py1, globals(), locals())
note: I'm using the requests library, but you could just as easily use the built in httplib's HTTPSConnection. It's just more verbose.
note 2: When you say you don't want the user to be able to "read/edit the actual Python file", they will be able to read it if they open the url themselves and view the content. they are just less likely to edit the file locally and mess something up. You can also deploy updates to your python script to the URL rather than having to copy them locally to every machine. This could be a security risk depending on the scope of what you are using it for.
Assuming that the remote Python script is a single file with no dependencies other than those of the Python standard library, and that a compatible version of Python is installed on the user's local machine, there are a few obvious ways to do it.
Use ssh:
#!/usr/bin/env sh
ssh user#host cat /path/to/python/script.py | python
Or use scp:
#!/usr/bin/env sh
TMPFILE=/tmp/$$
scp user#host:/path/to/python/script.py $TMPFILE
python $TMPFILE
rm $TMPFILE
The first one has the obvious advantage of not requiring any mucking about with copying files and cleaning up afterwards.
If you wanted to execute the python script on the remote server, this can also be done with ssh:
ssh user#host python /path/to/python/script.py
Standard input and output will be a terminal on the user's local machine.
With Python, I need to read a file into a script similar to open(file,"rb"). However, the file is on a server that I can access through SSH. Any suggestions on how I can easily do this? I am trying to avoid paramiko and am using pexpect to log into the SSH server, so a method using pexpect would be ideal.
Thanks,
Eric
You can mount the remote file system to local by using sshfs, then you can use is like normal file. The fuse module is needed by the sshfs.
If it's a short file you can get output of ssh command using subprocess.Popen
ssh root#ip_address_of_the_server 'cat /path/to/your/file'
Note: Password less setup using keys should be configured in order for it to work.