Develop Reference

Using python to ssh into multiple servers and grab the file with the same postfix

I normally use a bash script to grab all the files onto local machine and use glob to process all the files. Just wondering what would be the best way to use python (instead of another bash script) to ssh into each server and process those files?
My current program runs as
for filename in glob.glob('*-err.txt'):
input_open = open (filename, 'rb')
for line in input_open:
do something
My files all have the ending -err.txt and the directories where they reside in the remote server have the same name /documents/err/. I am not able to install third party libraries as I don't have the permission.
UPDATE
I am trying to not to scp the files from the server but to read it on the remote server instead..
I want to use a local python script LOCALLY to read in files on remote server.

The simplest way to do it is to use paramico_scp to use ssh copy from the remote server (How to scp in python?)
If you are not allowed to download any libraries, you can create SSH key pair so that connecting to server does not require a password (https://www.debian.org/devel/passwordlessssh). You then can for each file do
import os
os.system('scp user#host:/path/to/file/on/remote/machine /path/to/local/file')
Note that using system is usually considered less portable than using libraries. If you give the script that use system('scp ...') to copy the files and they do not have SSH key pair set up, they will experience problems

Looks like you want to use a local Python script remotely. This has been answered here.

How do I execute a remote Python file from a .sh executable?

I am using a Mac.
I have a Python file on a server. Let's say it's here: http://jake.com/python_file.py. I want to make a script that I can run in Terminal by double-clicking and will run the Python file in Terminal on the local machine. Any ideas?
Would I have to use SSH to connect to the server and download the file to a temporary location on the hard drive (/tmp?) and then delete it when I'm done? But there's another problem with this. It would have to download the Python file to a location in the user's home folder because I don't think users have the necessary permissions to write to the folder /tmp or /var or something like that.
I was looking around for a solution to my problem and found this. It talks about how to execute a remote script using SSH on unix but I tried doing this with my Python file and it didn't work.
In case you didn't realize, the main reason why I am looking to do this is so that a user can run the file locally but they are unable to read/edit the actual Python file which is stored on the server.
If anyone has any ideas on how to accomplish this (whether using the ideas mentioned above or not) please let me know I'd really appreciate it!
Thanks,
Jake

How about a python script to download and execute the code like so?
import requests
py1 = requests.get('https://example.com/file.py').content
exec(py1, globals(), locals())
note: I'm using the requests library, but you could just as easily use the built in httplib's HTTPSConnection. It's just more verbose.
note 2: When you say you don't want the user to be able to "read/edit the actual Python file", they will be able to read it if they open the url themselves and view the content. they are just less likely to edit the file locally and mess something up. You can also deploy updates to your python script to the URL rather than having to copy them locally to every machine. This could be a security risk depending on the scope of what you are using it for.

Assuming that the remote Python script is a single file with no dependencies other than those of the Python standard library, and that a compatible version of Python is installed on the user's local machine, there are a few obvious ways to do it.
Use ssh:
#!/usr/bin/env sh
ssh user#host cat /path/to/python/script.py | python
Or use scp:
#!/usr/bin/env sh
TMPFILE=/tmp/$$
scp user#host:/path/to/python/script.py $TMPFILE
python $TMPFILE
rm $TMPFILE
The first one has the obvious advantage of not requiring any mucking about with copying files and cleaning up afterwards.
If you wanted to execute the python script on the remote server, this can also be done with ssh:
ssh user#host python /path/to/python/script.py
Standard input and output will be a terminal on the user's local machine.

python on linux - use 'pwd' module to read passwd file of remote machine

I am executing a python script on one server and needing to read the contents of the passwd file from a remote machine. Does anyone know of a way to do this? Normally I would do:
import pwd
pwlist = pwd.getpwall()
#perform operations
This only works for the current system of course, and I'm needing a way to access another machine (like you would via ssh). Any help is appreciated.

The pwd module will only work for the current machine. It uses the C library functions defined in <pwd.h>, which do not provide any parameters for a remote machine. However, this does not prevent you from using ssh tools to run a script on the remote machine.

You can use a utility such as scp to copy the passwd file locally. Its easy to split the lines and get the information.

How to call .ksh file as part of Unix command through ssh in Python

I would like to achieve the following things:
Given file contains a job list which I need to execute one by one in a remote server using SSH APIs and store results.
When I try to call the following command directly on remote server using putty it executes successfully but when I try to execute it through python SSH programming it says cant find autosys.ksh.
autosys.ksh autorep -J JOB_NAME
Any ideas? Please help. Thanks in advance.

Fabric is a good bet. From the home page,
Fabric is a Python (2.5 or higher) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.
A quick example,
>>> from fabric.api import run, env, cd, settings, hide, show
>>> env.host_string='xxx.xxx.com'
>>> env.user='user'
>>> env.password='password'
>>> run('ls -lart')

After reading your comment on the first answer, you might want to create a bash script with bash path as the interpreter line and then the autosys commands.
This will create a bash shell and run the commands from the script in the shell.
Again, if you are using autosys commands in the shell you better set autosys environment up for the user before running any autosys commands.

open (and maintain) remote connection with python

I'm using Python to transfer (via scp) and database a large number of files. One of the servers I transfer files to has odd ssh config rules to stop too many ssh requests from a single location. The upshot of this is that my python script, currently looping through files and copying via os.system, hangs after a few files have been transferred.
Is there a way in which Python could open up an ssh or other connection to the server, so that each file being transferred does not require an instance of ssh login?
Thanks,

You probably want to look into the paramiko module.
There's a Copy files over SSH using paramiko recipe using it that might be helpful.

Check out SFTP in the Python module Paramiko. You can do multiple file transfers in a single session.

This is not really python specific, but it probably depends on what libraries you can use.
What you need is a way to send files through a single connection.
(This is probably better suited to superuser or severfault.com though.)
Create tarfile locally, upload it and unpack at target?
Maybe you could even run 'tar xz' remotely and upload the file on stdin over SSH? (As MichaelDillon says in the comment, Python can create the tarfile on the fly...)
Is SFTP an option?
Rsync over SSH?
Twisted is an async library that can handle many sockets/connections at once. Is probably overkill for your solution though,
Hope it helps.