Develop Reference

Uploading file using Paramiko in Python seemingly works, but the file cannot be found on the server

I am new to Python and sorry for my bad english.
I'm trying to save a file "toto.txt" from my HDD "d:" to my Synology NAS.
So I'll use paramiko for this, here is the code :
import paramiko
import os
ip_address = "my nas ip"
username = "my username"
password = "mypass"
utilisateur = os.getenv("USERNAME") // to get Windows username
remote_path = f"{utilisateur}/test.txt" // the file downloaded
local_path = "d:/toto.txt" //the file stored on my pc
ssh_client = paramiko.SSHClient()
ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh_client.connect(hostname=ip_address,username=username,password=password)
print("Connexion OK >", ip_address)
stdin, stdout, stderr = ssh_client.exec_command(f'mkdir {utilisateur}') //creating folder for the
user
sftp = ssh_client.open_sftp()
sftp.put(local_path,remote_path) // trying to send the file
sftp.close()
ssh_client.close()
i am not getting error but nothing is happening.
The folder is successful created but no file is sending in it.
Have someone an idea?
thanks a lot

If Paramiko does not throw any error, the upload was successful. The file just possibly ended in different location than you wanted/than you look to.
At least for a test, if not permanently, try an absolute absolute path. Make no guesses, use the exact path you see in your GUI SFTP client.
Another possibility is that the server automatically processes the uploaded file somehow and moves it away.

How To Read/Write Files From a SFTP Server in Python? [duplicate]

I'm working on a simple tool that transfers files to a hard-coded location with the password also hard-coded. I'm a python novice, but thanks to ftplib, it was easy:
import ftplib
info= ('someuser', 'password') #hard-coded
def putfile(file, site, dir, user=(), verbose=True):
"""
upload a file by ftp to a site/directory
login hard-coded, binary transfer
"""
if verbose: print 'Uploading', file
local = open(file, 'rb')
remote = ftplib.FTP(site)
remote.login(*user)
remote.cwd(dir)
remote.storbinary('STOR ' + file, local, 1024)
remote.quit()
local.close()
if verbose: print 'Upload done.'
if __name__ == '__main__':
site = 'somewhere.com' #hard-coded
dir = './uploads/' #hard-coded
import sys, getpass
putfile(sys.argv[1], site, dir, user=info)
The problem is that I can't find any library that supports sFTP. What's the normal way to do something like this securely?
Edit: Thanks to the answers here, I've gotten it working with Paramiko and this was the syntax.
import paramiko
host = "THEHOST.com" #hard-coded
port = 22
transport = paramiko.Transport((host, port))
password = "THEPASSWORD" #hard-coded
username = "THEUSERNAME" #hard-coded
transport.connect(username = username, password = password)
sftp = paramiko.SFTPClient.from_transport(transport)
import sys
path = './THETARGETDIRECTORY/' + sys.argv[1] #hard-coded
localpath = sys.argv[1]
sftp.put(localpath, path)
sftp.close()
transport.close()
print 'Upload done.'
Thanks again!

Paramiko supports SFTP. I've used it, and I've used Twisted. Both have their place, but you might find it easier to start with Paramiko.

You should check out pysftp https://pypi.python.org/pypi/pysftp it depends on paramiko, but wraps most common use cases to just a few lines of code.
import pysftp
import sys
path = './THETARGETDIRECTORY/' + sys.argv[1] #hard-coded
localpath = sys.argv[1]
host = "THEHOST.com" #hard-coded
password = "THEPASSWORD" #hard-coded
username = "THEUSERNAME" #hard-coded
with pysftp.Connection(host, username=username, password=password) as sftp:
sftp.put(localpath, path)
print 'Upload done.'

Here is a sample using pysftp and a private key.
import pysftp
def upload_file(file_path):
private_key = "~/.ssh/your-key.pem" # can use password keyword in Connection instead
srv = pysftp.Connection(host="your-host", username="user-name", private_key=private_key)
srv.chdir('/var/web/public_files/media/uploads') # change directory on remote server
srv.put(file_path) # To download a file, replace put with get
srv.close() # Close connection
pysftp is an easy to use sftp module that utilizes paramiko and pycrypto. It provides a simple interface to sftp.. Other things that you can do with pysftp which are quite useful:
data = srv.listdir() # Get the directory and file listing in a list
srv.get(file_path) # Download a file from remote server
srv.execute('pwd') # Execute a command on the server
More commands and about PySFTP here.

If you want easy and simple, you might also want to look at Fabric. It's an automated deployment tool like Ruby's Capistrano, but simpler and of course for Python. It's build on top of Paramiko.
You might not want to do 'automated deployment' but Fabric would suit your use case perfectly none the less. To show you how simple Fabric is: the fab file and command for your script would look like this (not tested, but 99% sure it will work):
fab_putfile.py:
from fabric.api import *
env.hosts = ['THEHOST.com']
env.user = 'THEUSER'
env.password = 'THEPASSWORD'
def put_file(file):
put(file, './THETARGETDIRECTORY/') # it's copied into the target directory
Then run the file with the fab command:
fab -f fab_putfile.py put_file:file=./path/to/my/file
And you're done! :)

fsspec is a great option for this, it offers a filesystem like implementation of sftp.
from fsspec.implementations.sftp import SFTPFileSystem
fs = SFTPFileSystem(host=host, username=username, password=password)
# list a directory
fs.ls("/")
# open a file
with fs.open(file_name) as file:
content = file.read()
Also worth noting that fsspec uses paramiko in the implementation.

With RSA Key then refer here
Snippet:
import pysftp
import paramiko
from base64 import decodebytes
keydata = b"""AAAAB3NzaC1yc2EAAAADAQABAAABAQDl"""
key = paramiko.RSAKey(data=decodebytes(keydata))
cnopts = pysftp.CnOpts()
cnopts.hostkeys.add(host, 'ssh-rsa', key)
with pysftp.Connection(host=host, username=username, password=password, cnopts=cnopts) as sftp:
with sftp.cd(directory):
sftp.put(file_to_sent_to_ftp)

Twisted can help you with what you are doing, check out their documentation, there are plenty of examples. Also it is a mature product with a big developer/user community behind it.

There are a bunch of answers that mention pysftp, so in the event that you want a context manager wrapper around pysftp, here is a solution that is even less code that ends up looking like the following when used
path = "sftp://user:p#ssw0rd#test.com/path/to/file.txt"
# Read a file
with open_sftp(path) as f:
s = f.read()
print s
# Write to a file
with open_sftp(path, mode='w') as f:
f.write("Some content.")
The (fuller) example: http://www.prschmid.com/2016/09/simple-opensftp-context-manager-for.html
This context manager happens to have auto-retry logic baked in in the event you can't connect the first time around (which surprisingly happens more often than you'd expect in a production environment...)
The context manager gist for open_sftp: https://gist.github.com/prschmid/80a19c22012e42d4d6e791c1e4eb8515

Paramiko is so slow. Use subprocess and shell, here is an example:
remote_file_name = "filename"
remotedir = "/remote/dir"
localpath = "/local/file/dir"
ftp_cmd_p = """
#!/bin/sh
lftp -u username,password sftp://ip:port <<EOF
cd {remotedir}
lcd {localpath}
get {filename}
EOF
"""
subprocess.call(ftp_cmd_p.format(remotedir=remotedir,
localpath=localpath,
filename=remote_file_name
),
shell=True, stdout=sys.stdout, stderr=sys.stderr)

PyFilesystem with its sshfs is one option. It uses Paramiko under the hood and provides a nicer paltform independent interface on top.
import fs
sf = fs.open_fs("sftp://[user[:password]#]host[:port]/[directory]")
sf.makedir('my_dir')
or
from fs.sshfs import SSHFS
sf = SSHFS(...

Here's a generic function that will download any given sftp url to a specified path
from urllib.parse import urlparse
import paramiko
url = 'sftp://username:password#hostname/filepath.txt'
def sftp_download(url, dest):
url = urlparse(url)
with paramiko.Transport((url.hostname, 22)) as transport:
transport.connect(None,url.username,url.password)
with paramiko.SFTPClient.from_transport(transport) as sftp:
sftp.get(url.path, dest)
Call it with
sftp_download(url, "/tmp/filepath.txt")

Upload new file to SFTP server using Paramiko without having to overwrite an existing file

I am trying to upload a file via SFTP to my server. But instead of just uploading it, i have to explicitly tell my skript what file to overwrite on the server. I don't know how to change that.
#!/usr/bin/python3
import paramiko
k = paramiko.RSAKey.from_private_key_file("/home/abdulkarim/.ssh/id_rsa")
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
print("connecting")
c.connect( hostname = "do-test", username = "abdulkarim", pkey = k )
print("connected")
sftp = c.open_sftp()
sftp.put('/home/abdulkarim/Skripte/data/test.txt', '/home/abdulkarim/test/test1.txt')
c.close()

In the below call, the second (remotepath) parameter refers to the path, where the file will be stored on the server. There is not requirement for the remote file to actually exist. It will be created.
sftp.put('/home/abdulkarim/Skripte/data/test.txt', '/home/abdulkarim/test/test1.txt')
Obligatory warning: Do not use AutoAddPolicy – You are losing a protection against MITM attacks by doing so. For a correct solution, see Paramiko "Unknown Server".

unable to download files from sftp server using python

i am writing a python script that helps me download files from sftp server to my local folder.
when i run the script it just downloads the blank document
i keep on trying but i am failing
i made a sftp server to test the code and the path specified in remote path is server root directory
i giving code below
import pysftp
myHostname = "192.168.56.1"
myUsername = "new45"
myPassword = "146515"
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
with pysftp.Connection(host=myHostname, username=myUsername, password=myPassword,cnopts=cnopts) as sftp:
# Define the file that you want to download from the remote directory
remoteFilePath = 'C:\\Users\\Simon\\Desktop\\ftp\\testfile.txt'
# Define the local path where the file will be saved
# or absolute "C:\Users\sdkca\Desktop\TUTORIAL.txt"
localFilePath = 'C:\\Users\\Simon\\Desktop\\ftp2\\textfile.txt'
sftp.get(remoteFilePath, localFilePath)
# connection closed automatically at the end of the with-block'''
please tell me whats the error why is blank file is being downloaded

Login to that server using any GUI SFTP client. And observe what path syntax your server is using. It's definitely not C:\Users\Simon\Desktop\ftp\testfile.txt. SFTP uses forward slashes. So it can be like /C:/Users/Simon/Desktop/ftp/testfile.txt. But even something completely different, particularly if the server or your account are chrooted.
You are probably trying to use the path syntax, which you would have used, were you directly on that target Windows system. But your SFTP server presents the files using a different path syntax. You need to adhere to that syntax.

Writing path for sftp server python

I am using pysftp on Python and am trying to run a loop for a certain directory in the sftp server.
I don't know how to write the directory paths on sftp servers. I thought that connecting to the server and just writing the directory as below would work, but it doesn't. Please let me know how to write sftp paths so that python can read them.
sftp = pysftp.Connection('128.59.164.112', username = '', password = '');
source = sftp.u\'weatherForecast\'/dataRAW/2004/grib/tmax/

Try this:
import pysftp
sftp = pysftp.Connection('hostname', username='me', password='secret')
sftp.chdir('/home/user/development/stackoverflow')
ls = sftp.listdir()
for filename in ls:
print filename
You should read this: http://pysftp.readthedocs.org/en/latest/index.html
PS1: ; is optional in Python, but not Pythonic.

After enough trial and error I figured it out.
source = 'weatherForecast/dataRAW/2004/grib/tmax/'
destination= 'sftp.u\'weatherForecast\'/csv/2004/tmax'
This works.