i am writing a python script that helps me download files from sftp server to my local folder.
when i run the script it just downloads the blank document
i keep on trying but i am failing
i made a sftp server to test the code and the path specified in remote path is server root directory
i giving code below
import pysftp
myHostname = "192.168.56.1"
myUsername = "new45"
myPassword = "146515"
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
with pysftp.Connection(host=myHostname, username=myUsername, password=myPassword,cnopts=cnopts) as sftp:
# Define the file that you want to download from the remote directory
remoteFilePath = 'C:\\Users\\Simon\\Desktop\\ftp\\testfile.txt'
# Define the local path where the file will be saved
# or absolute "C:\Users\sdkca\Desktop\TUTORIAL.txt"
localFilePath = 'C:\\Users\\Simon\\Desktop\\ftp2\\textfile.txt'
sftp.get(remoteFilePath, localFilePath)
# connection closed automatically at the end of the with-block'''
please tell me whats the error why is blank file is being downloaded
Login to that server using any GUI SFTP client. And observe what path syntax your server is using. It's definitely not C:\Users\Simon\Desktop\ftp\testfile.txt. SFTP uses forward slashes. So it can be like /C:/Users/Simon/Desktop/ftp/testfile.txt. But even something completely different, particularly if the server or your account are chrooted.
You are probably trying to use the path syntax, which you would have used, were you directly on that target Windows system. But your SFTP server presents the files using a different path syntax. You need to adhere to that syntax.
Related
I'm attempting to download a set of files from an SFTP server. I can download the files successfully, but now I want to change the script so that as the files download, they go to a specified directory on my local server.
I attempted doing this with the sftp.getfo() command, but am getting an error:
getfo() got an unexpected keyword argument 'fl'
#SET LOCAL DIRECTORY
directory1 = r'C:\Users\Me\Documents\test sftp'
#CONNECT TO SFTP
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
sftp = pysftp.Connection(host = hostname, username = usereu, password = passeu, cnopts = cnopts)
print("Connection successfully established ... ")
#DOWNLOAD ALL FILES IN FOLDER
files = sftp.listdir()
for file in files:
sftp.getfo(remotepath = file, fl = directory1)
print(file,' downloaded successfully ')
Am I using the right command here or is there a better way to do this?
Thanks!
To download a file to local path, use Connection.get (not getfo):
sftp.get(remotepath=file, localpath=os.path.join(directory1, file))
Some notes:
remotepath takes path to the remote file, not just filename. Your code (which passes filename only) works only because you are downloading from the home folder.
The pysftp is dead. Do not use it. Use Paramiko. See pysftp vs. Paramiko
For an example, see Download file from SFTP server in Python Paramiko
Do not set cnopts.hostkeys = None, unless you do not care about security. For the correct solution see Verify host key with pysftp.
I'm currently utilising Paramiko and SCPClient in Python to transfer a directory from one server to another as a means of backup. This works well however I do not want it to copy hidden files (.file_name) or symbolic links. Is this possible?
Unfortunately rsync isn't an option for me as it's not available on either of the remote servers I connect to. My script is below (sensitive info replaced with dummy data). Note I need to connect to a jump host before being able to connect to target_1 or target_2.
import os
import shutil
import time
import paramiko
from scp import SCPClient
#set up ssh variables
j_host = '00.00.00.00'
target_host_1 = '00.00.00.001'
target_host_2 = '00.00.00.002'
port_no = 22
username = ''
passw = ''
#set up temporary folder on local machine to store files
path = "/local_path/"
os.mkdir(path)
#create SSH Client for jump server
jump_host=paramiko.SSHClient()
jump_host.set_missing_host_key_policy(paramiko.AutoAddPolicy())
jump_host.connect(j_host, username=username, password=passw)
#set up channel to connect to 1 via jump server
jump_host_transport_1 = jump_host.get_transport()
src_addr = (j_host, port_no)
dest_addr_1 = (target_host_1, port_no)
jump_host_channel_1 = jump_host_transport_1.open_channel("direct-tcpip", dest_addr_1, src_addr)
#set up channel to connect to 2 via jump server
jump_host_transport_2 = jump_host.get_transport()
dest_addr_2 = (target_host_2, port_no)
jump_host_channel_2 = jump_host_transport_2.open_channel("direct-tcpip", dest_addr_2, src_addr)
#function which sets up target server, either 1 or 2
def create_SSHClient(server, port, user, password, sock):
target=paramiko.SSHClient()
target.set_missing_host_key_policy(paramiko.AutoAddPolicy())
target.connect(server, port, user, password, sock=sock)
return target
#invoke above function to set up connections for 1 & 2
ssh_1 = create_SSHClient(target_host_1, port_no, username, passw, jump_host_channel_1)
ssh_2 = create_SSHClient(target_host_2, port_no, username, passw, jump_host_channel_2)
#delete old files in backup folder
command = "rm -rf /filepath/{*,.*}"
stdin, stdout, stderr = ssh_2.exec_command(command)
lines = stdout.readlines()
#print(lines)
#pause to ensure old directory is cleared
time.sleep(5)
#SCPCLient takes a paramiko transport as an argument, sets up file transfer connection
scp_1 = SCPClient(ssh_1.get_transport())
scp_2 = SCPClient(ssh_2.get_transport())
#get files from 1, store on local machine, put on 2
scp_1.get('/filepath/.', '/target_folder_local/', recursive=True)
scp_2.put('/target_folder_local/.', '/filepath/', recursive=True)
#remove temporary folder
shutil.rmtree(path)
#close connections
ssh_1.close()
ssh_2.close()
jump_host.close()
There's no API in SCPClient to skip hidden files or symbolic links.
For upload, it's easy, if you copy the SCPClient's code and modify it as you need. See the os.walk loop in _send_recursive function.
If you do not want to modify the SCPClient's code, you will have to iterate the files on your own, calling SCPClient.put for each. It will be somewhat less efficient, as it will start new SCP server for each file.
For download, you might be able to modify the SCPClient code to respond with non-zero code to C commands fed by the server for the files you do not want to download.
Check the _recv_file function. There where name is resolved, check for names or attributes of files you are not interested in downloading and do chan.send('\x01') and exit the function.
Though why do you want to use SCP? Use SFTP. It is much better suited for custom rules you need.
Paramiko does not have recursive SFTP transfer functionality (But pysftp does, see pysftp vs. Paramiko). But you won't be able to use it anyway, for the same reason you cannot use it with SCP. For your specific needs.
But check my answer to Python pysftp get_r from Linux works fine on Linux but not on Windows. It shows a simple recursive SFTP download code. Just modify it slightly to skip the files you do not want to download.
Something like
if (not S_ISLNK(mode)) and (not entry.filename.startswith(".")):
(see Checking if a file on SFTP server is a symbolic link, and deleting the symbolic link, using Python Paramiko/pysftp)
I'm trying to get latest new file in a directory of remote Linux server. The file in SFTP server is created every 4 hours and the file have specific name start with filegen_date_hour.json as per example below. In this case latest file 'filegen_20200101_0800.json' need to be transferred to my local directory.
filegen_20200101_0000.json
filegen_20200101_0400.json
filegen_20200101_0800.json
I use Python 3 code below, but got error
latestFile = max(listFile, key=os.path.getctime)
ValueError: max() arg is an empty sequence
SFTP code below
myHostname = "192.168.100.10"
myUsername = "user"
myPassword = "password"
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
with pysftp.Connection(host=myHostname, username=myUsername, password=myPassword, cnopts=cnopts) as sftp:
with sftp.cd('/home/operation/genfiles/'):
fileDir = '/home/operation/genfiles/filegen_*.json'
**#file have specific pattern with filegen_*.json**
listFile = glob.glob(fileDir)
latestFile = max(listFile, key=os.path.getctime)
sftp.get(latestFile)
Appreciate help on this matter. Thank you for your response and help.
First, you cannot use glob to list files on an SFTP server. The glob won't magically start querying SFTP server only because you have opened an SFTP connection before. It will still query local file system.
Use pysftp Connection.listdir. Though it does not support wildcards, so you will have to filter the files you want locally. Like here:
List files on SFTP server matching wildcard in Python using Paramiko
Only then you can try finding the latest file.
In general, you may use file modification time, as here:
How to download only the latest file from SFTP server with Paramiko?
The code is for Paramiko SFTPClient.listdir_attr, but it's the same with pysftp Connection.listdir_attr.
But in your case, I'm not sure if you can rely on the modification timestamp. It seems that you actually want to use the timestamp in the filename. With your file name format, you can simply pick the last file lexicographically.
import fnmatch
...
with sftp.cd('/home/operation/genfiles'):
files = []
for filename in sftp.listdir():
if fnmatch.fnmatch(filename, "filegen_*.json"):
files.append(filename)
latestFile = max(files)
Obligatory warning: Do not set cnopts.hostkeys = None, unless you do not care about security. For the correct solution see Verify host key with pysftp.
I have a SFTP server that contains a set of folders. I would like to download a file from one of these folders.
Can someone help me with the python code?
Where should I start?
Have you tried the example in pysftp's documentation?
import pysftp
with pysftp.Connection('hostname', username='me', password='secret') as sftp:
with sftp.cd('public'): # temporarily chdir to public
sftp.put('/my/local/filename') # upload file to public/ on remote
sftp.get('remote_file') # get a remote file
An addition from Michael Powers.
If your server is one account and your personal account is another one.
If your server account has root permission, then follow Michael's method.
If no, you may use io buffer to download this file
import io
server_sftp = pysftp.Connection(host=test_host, username=server_account,
password=server_pwd, cnopts=cn_opts)
your_user_sftp = pysftp.Connection(host=host_name, username = user_name,
password = user_pwd, cnopts=cn_opts)
try:
file_like=io.BytesIO()
server_account.getfo(server_file_location,file_like)
your_user_sftp.putfo(file_like, target_file_location)
finally:
flie_like.close()
server_sftp.close()
your_user_sftp.close()
I am using pysftp on Python and am trying to run a loop for a certain directory in the sftp server.
I don't know how to write the directory paths on sftp servers. I thought that connecting to the server and just writing the directory as below would work, but it doesn't. Please let me know how to write sftp paths so that python can read them.
sftp = pysftp.Connection('128.59.164.112', username = '', password = '');
source = sftp.u\'weatherForecast\'/dataRAW/2004/grib/tmax/
Try this:
import pysftp
sftp = pysftp.Connection('hostname', username='me', password='secret')
sftp.chdir('/home/user/development/stackoverflow')
ls = sftp.listdir()
for filename in ls:
print filename
You should read this: http://pysftp.readthedocs.org/en/latest/index.html
PS1: ; is optional in Python, but not Pythonic.
After enough trial and error I figured it out.
source = 'weatherForecast/dataRAW/2004/grib/tmax/'
destination= 'sftp.u\'weatherForecast\'/csv/2004/tmax'
This works.