I'm trying to connect to SFTP from a notebook with databricks in python. My notebook looks like this:
import pysftp
import paramiko
# set parameters
host_name = 'xx.xxx.xxx.xxx'
username = 'FTP_USERNAME'
file_path_to_rsa_key = "/path/key_rsa"
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
# connect to SFTP
sftp = pysftp.Connection(host_name, username=username, private_key=file_path_to_rsa_key, cnopts=cnopts)
data = sftp.listdir()
sftp.close()
# Prints out the directories and files, line by line
for i in data:
print(i)
I have the following error:
Oops, unhandled type 3 ('unimplemented')
when running the following block:
try:
conn = pysftp.Connection(host_name, username=username, private_key=file_path_to_rsa_key, cnopts=cnopts)
print("connection established successfully")
except:
print('failed to establish connection to targeted server')
It print connection established successfully
What does it mean? What should I do? Is the issue with listdir()?
It seems that the issue is mostly because pysftp. I end up switching entirely to paramiko and everything is working fine. The SFTP server modified the login response and pysftp was too strict in the acceptable responses to take it without trouble. I noticed something similar when I tried to use a non-RSA private key that worked fine for ssh / sftp directly but not pysftp.
There is also some interesting information here pysftp vs. Paramiko
I am writing a program using pysftp, and it wants to verify the SSH host Key against C:\Users\JohnCalvin\.ssh\known_hosts.
Using PuTTY, the terminal program is saving it to the Registry [HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\SshHostKeys].
How do I reconcile the difference between pysftp and PuTTY?
My code is:
import pysftp as sftp
def push_file_to_server():
s = sftp.Connection(host='138.99.99.129', username='root', password='*********')
local_path = "testme.txt"
remote_path = "/home/testme.txt"
s.put(local_path, remote_path)
s.close()
push_file_to_server()
The error response I am receiving is:
E:\Program Files (x86)\Anaconda3\lib\site-packages\pysftp\__init__.py:61:
UserWarning: Failed to load HostKeys from C:\Users\JohnCalvin\.ssh\known_hosts. You will need to explicitly load HostKeys (cnopts.hostkeys.load(filename)) or disableHostKey checking (cnopts.hostkeys = None).
warnings.warn(wmsg, UserWarning)
Traceback (most recent call last):
File "E:\OneDrive\Python\GIT\DigitalCloud\pysftp_tutorial.py", line 14, in <module>
push_file_to_server()
File "E:\OneDrive\Python\GIT\DigitalCloud\pysftp_tutorial.py", line 7, in push_file_to_server
s = sftp.Connection(host='138.99.99.129', username='root', password='********')
File "E:\Program Files (x86)\Anaconda3\lib\site-packages\pysftp\__init__.py", line 132, in __init__
self._tconnect['hostkey'] = self._cnopts.get_hostkey(host)
File "E:\Program Files (x86)\Anaconda3\lib\site-packages\pysftp\__init__.py", line 71, in get_hostkey
raise SSHException("No hostkey for host %s found." % host) paramiko.ssh_exception.SSHException: No hostkey for host 138.99.99.129 found.
Exception ignored in: <bound method Connection.__del__ of <pysftp.Connection object at 0x00000222FF3A6BE0>>
Traceback (most recent call last):
File "E:\Program Files (x86)\Anaconda3\lib\site-packages\pysftp\__init__.py", line 1013, in __del__
self.close()
File "E:\Program Files (x86)\Anaconda3\lib\site-packages\pysftp\__init__.py", line 784, in close
if self._sftp_live:
AttributeError: 'Connection' object has no attribute '_sftp_live'
The pysftp has some bugs regarding host key handling, as described below. It also seems that the pysftp project was abandoned. Consider using Paramiko directly instead. The pysftp is just a wrapper on top of Paramiko and it does not add anything really significant. See pysftp vs. Paramiko.
For handling of host keys in Paramiko, see:
Paramiko "Unknown Server"
If you want to keep using pysftp, do not set cnopts.hostkeys = None (as the second most upvoted answer shows), unless you do not care about security. You lose a protection against Man-in-the-middle attacks by doing so.
Use CnOpts.hostkeys (returns HostKeys) to manage trusted host keys.
cnopts = pysftp.CnOpts(knownhosts='known_hosts')
with pysftp.Connection(host, username, password, cnopts=cnopts) as sftp:
where the known_hosts contains a server public key(s)] in a format like:
example.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQAB...
If you do not want to use an external file, you can also use
from base64 import decodebytes
# ...
keydata = b"""AAAAB3NzaC1yc2EAAAADAQAB..."""
key = paramiko.RSAKey(data=decodebytes(keydata))
cnopts = pysftp.CnOpts()
cnopts.hostkeys.add('example.com', 'ssh-rsa', key)
with pysftp.Connection(host, username, password, cnopts=cnopts) as sftp:
Though as of pysftp 0.2.9, this approach will issue a warning, what seems like a bug:
"Failed to load HostKeys" warning while connecting to SFTP server with pysftp
An easy way to retrieve the host key in the needed format is using OpenSSH ssh-keyscan:
$ ssh-keyscan example.com
# example.com SSH-2.0-OpenSSH_5.3
example.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQAB...
(due to a bug in pysftp, this does not work, if the server uses non-standard port – the entry starts with [example.com]:port + beware of redirecting ssh-keyscan to a file in PowerShell)
You can also make the application do the same automatically:
Use Paramiko AutoAddPolicy with pysftp
(It will automatically add host keys of new hosts to known_hosts, but for known host keys, it will not accept a changed key)
Though for an absolute security, you should not retrieve the host key remotely, as you cannot be sure, if you are not being attacked already.
See my article Where do I get SSH host key fingerprint to authorize the server?
It's for my WinSCP SFTP client, but most information there is valid in general.
If you need to verify the host key using its fingerprint only, see Python - pysftp / paramiko - Verify host key using its fingerprint.
One option is to disable the host key requirement:
import pysftp
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
with pysftp.Connection(host, username, password, cnopts=cnopts) as sftp:
sftp.put(local_path, remote_path)
You can find more info about that here:
https://stackoverflow.com/a/38355117/1060738
Important note:
By setting cnopts.hostkeys=None you'll lose the protection against Man-in-the-middle attacks by doing so. Use #martin-prikryl answer to avoid that.
Try to use the 0.2.8 version of pysftp library.
$ pip uninstall pysftp && pip install pysftp==0.2.8
And try with this:
try:
ftp = pysftp.Connection(host, username=user, password=password)
except:
print("Couldn't connect to ftp")
return False
Why this?
Basically is a bug with the 0.2.9 of pysftp
here all details
https://github.com/Yenthe666/auto_backup/issues/47
Cook book to use different ways of pysftp.CnOpts() and hostkeys options.
Source : https://pysftp.readthedocs.io/en/release_0.2.9/cookbook.html
Host Key checking is enabled by default. It will use ~/.ssh/known_hosts by default. If you wish to disable host key checking (NOT ADVISED) you will need to modify the default CnOpts and set the .hostkeys to None.
import pysftp
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
with pysftp.Connection('host', username='me', password='pass', cnopts=cnopts):
# do stuff here
To use a completely different known_hosts file, you can override CnOpts looking for ~/.ssh/known_hosts by specifying the file when instantiating.
import pysftp
cnopts = pysftp.CnOpts(knownhosts='path/to/your/knownhostsfile')
with pysftp.Connection('host', username='me', password='pass', cnopts=cnopts):
# do stuff here
If you wish to use ~/.ssh/known_hosts but add additional known host keys you can merge with update additional known_host format files by using .load method.
import pysftp
cnopts = pysftp.CnOpts()
cnopts.hostkeys.load('path/to/your/extra_knownhosts')
with pysftp.Connection('host', username='me', password='pass', cnopts=cnopts):
# do stuff here
If You try to connect by pysftp to "normal" FTP You have to set hostkey to None.
import pysftp
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
with pysftp.Connection(host='****',username='****',password='***',port=22,cnopts=cnopts) as sftp:
print('DO SOMETHING')
Connect to the server first with a Windows ssh client that uses the known_hosts file.
PuTTy stores the data in the windows registry,however OpenSSH uses the known_hosts file, and will add entries in there after you connect.
Default location for the file is %USERPROFILE%.ssh. I hope this helps
I've implemented auto_add_key in my pysftp github fork.
auto_add_key will add the key to known_hosts if auto_add_key=True
Once a key is present for a host in known_hosts this key will be checked.
Please reffer Martin Prikryl -> answer about security concerns.
Though for an absolute security, you should not retrieve the host key remotely, as you cannot be sure, if you are not being attacked already.
import pysftp as sftp
def push_file_to_server():
s = sftp.Connection(host='138.99.99.129', username='root', password='pass', auto_add_key=True)
local_path = "testme.txt"
remote_path = "/home/testme.txt"
s.put(local_path, remote_path)
s.close()
push_file_to_server()
Note: Why using context manager
import pysftp
with pysftp.Connection(host, username="whatever", password="whatever", auto_add_key=True) as sftp:
#do your stuff here
#connection closed
Hi We sort of had the same problem if I understand you well. So check what pysftp version you're using. If it's the latest one which is 0.2.9 downgrade to 0.2.8.
Check this out. https://github.com/Yenthe666/auto_backup/issues/47
FWIR, if authentication is only username & pw, add remote server ip address to known_hosts like ssh-keyscan -H 192.168.1.162 >> ~/.ssh/known_hosts for ref https://www.techrepublic.com/article/how-to-easily-add-an-ssh-fingerprint-to-your-knownhosts-file-in-linux/
I am using pysftp to upload a local file that is processed remotely, then returned to the SFTP, where I can download it. At the moment, I cannot get the code to recognize that the processed remote file has been uploaded. The code just waits forever, despite the remotely processed file successfully being uploaded.
While the code is running, if I refresh the connection on a service such as FileZilla, the code instantly recognizes the file is in and works perfectly. However, I need it to work without manually refreshing on FileZilla. I am not sure why the code cannot recognize itself that the file has been uploaded and is ready to be downloaded.
I've already tried disconnecting then re-connecting to the SFTP using a while statement, which was not successful.
import pysftp
import stat
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
srv = pysftp.Connection(host="host", username="username", password="passowrd", cnopts=cnopts)
print("Connection Made!")
# Put File In
srv.chdir(r'/path_to_remote_directory')
srv.put(r'C:\Path_to_local_file.csv')
print("File Uploaded")
while not srv.isfile(r'/Path_to_newly_uploaded_remote_file.csv'):
time.sleep(3)
print("Still Waiting")
if srv.isfile(r'/Path_to_newly_uploaded_remote_file.csv'):
print("File is Ready!")
srv.get('/Path_to_newly_uploaded_remote_file.csv', r'C:\Local_path_save_to/file.csv')
# Closes the connection
srv.close()
Just remove the /:
srv.isfile(r'/Path_to_newly_uploaded_remote_file.csv'):
->
srv.isfile(r'Path_to_newly_uploaded_remote_file.csv'):
NOTE:
Do not set cnopts.hostkeys = None,
unless you do not care about security. You lose a protection against
Man-in-the-middle attacks by doing so.
I've implemented auto_add_key in my pysftp github fork.
auto_add_key will add the key to known_hosts if auto_add_key=True
Once a key is present for a host in known_hosts this key will be checked.
Please refer to Martin Prikryl -> answer about security concerns.
Why using context manager is a good approach:
import pysftp
with pysftp.Connection(host, username="whatever", password="whatever", auto_add_key=True) as sftp:
#do your stuff here
#connection closed
EDIT:
Checked with my code and / was the problem ... please check spelling of your file and that you are in the correct working dir getcwd()_
import pysftp
import stat
import time
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None #do not do this !!!
srv = pysftp.Connection("host", username="user", password="pass", cnopts=cnopts)
print("Connection Made!")
# Put File In
srv.chdir(r'/upload')
srv.put(r'C:\Users\Fabian\Desktop\SFTP\testcsv.csv')
print("File Uploaded")
print(srv.getcwd()) #get the current folder ! <- you should looking here
while not srv.isfile(r'testcsv.csv'):
time.sleep(3)
print("Still Waiting")
if srv.isfile(r'testcsv.csv'):
print("File is Ready!")
srv.get('testcsv.csv', r'C:\Users\Fabian\Desktop\SFTP\testcsv_new.csv')
# Closes the connection
srv.close()
Found a working solution. I think it was caused by the old directory being cached. By pointing to the new directory every time for an update, it appears to work. Obvious solution now but took me a while. Posting in case anyone else runs into this.
while not srv.isfile(r'/Path_to_newly_uploaded_remote_file.csv'):
time.sleep(3)
srv.chdir(r'/Path_to_newly_uploaded_remote_file')
print("Still Waiting")
I want to send/download file from Droxbox via Python. I've tried pysftp and paramiko, but the connect() call will hang. Below is my code.
import paramiko
# create a client
ssh = paramiko.SSHClient()
# can alos choose .WarningPolicy(), .RejectPolicy()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname="dropbox.com", username="my username", password="my pw")
# The call will hang there
sftp code
import pysftp as sftp
cnopts = sftp.CnOpts()
cnopts.hostkeys = None
'''If I don't set up cnopts, I'll get error - No hostkey for host dropbox.com found.
But I got a warning if I set up the above cnopts -
CryptographyDeprecationWarning: Support for unsafe construction of
public numbers from encoded data will be removed in a future
version. Please use EllipticCurvePublicKey.from_encoded_point
self.ecdsa_curve.curve_class(), pointinfo
'''
s = sftp.Connection(host="dropbox.com",
username="xi#transcriptic.com",
password="Python#2018",
cnopts=cnopts)
Any suggestions would be greatly appreciated!
I don't believe that Dropbox supports SFTP, sadly.
My code:
import pysftp
s = pysftp.Connection(host='test.rebex.net', username='demo', password='password')
data = s.listdir()
s.close()
for i in data:
print i
I'm getting an error trying to connect to a SFTP server using pysftp.
This should be straight forward enough but I get the error below:
Traceback (most recent call last):
File "/Users/gavinhinfey/Documents/Python Files/sftp_test.py", line 3, in <module>
s = pysftp.Connection(host='test.rebex.net', username='demo', password='password')
File "build/bdist.macosx-10.6-intel/egg/pysftp.py", line 55, in __init__
File "build/bdist.macosx-10.5-intel/egg/paramiko/transport.py", line 303, in __init__
paramiko.SSHException: Unable to connect to test.rebex.net: [Errno 60] Operation timed out
Exception AttributeError: "'Connection' object has no attribute '_tranport_live'" in <bound method Connection.__del__ of <pysftp.Connection object at 0x101a5a810>> ignored
I've tried using different versions of python (mostly 2.7), I have all dependencies installed and I tried numerous sftp connections.
I'm using OS X 10.9.1.
updating the package didn't work for me, as it was already up-to-date (latest for python 2.7 at least)
Found a better aproach here.
1) You can manualy add the ssh key to the known_hosts file
ssh test.rebex.net
2) Or you can set a flag to ignore it
import pysftp
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None # disable host key checking.
with pysftp.Connection('host', username='me',private_key=private_key,
private_key_pass=private_key_password,
cnopts=cnopts) as sftp
# do stuff here
That initial error appears to be a problem connecting with the remote server (SSHException). The second (AttributeError), is from a bug in the code that occurs when the connection fails. It is fixed in the latest version of pysftp
https://pypi.python.org/pypi/pysftp
pip install -U pysftp
is your friend.
#Martin.Prikryl: setting hostkeys = None is very useful in the initial stage of coding with pysftp. Debugging a program that keeps failing for a known exception hides other problems that need attention--like making an actual connection. I can deal with the 'man in the middle' problem later once I know my code is actually working correctly.
#All: The current pysftp.CnOpts() object appears to have a bug:
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
The above code does not prevent host key checking.
python getfile_v3.py --help
Traceback (most recent call last):
File "getfile_v3.py", line 9, in
cnopts = pysftp.CnOpts()
File "c:\Program Files\Python\Python38\lib\site-packages\pysftp_init_.py", line 64, in init
raise HostKeysException('No Host Keys Found')
pysftp.exceptions.HostKeysException: No Host Keys Found
The second line doesn't get executed because the first does the host key check by default. If I set the key with:
cnopts = pysftp.CnOpts(hostkeys=None)
the same error results.
It appears that 'hostkeys' has been deprecated, and there is no way to disable the host key check.
Joe White
Workaround for SSHException: No hostkey for host test.rebex.net found
just to add it manually via ssh
ssh demo#test.rebex.net
it will ask for password, you need to enter "password"
it will show you the message
The authenticity of host 'test.rebex.net (ip-adress)' can't be established.
ECDSA key fingerprint is SHA256:OzvpQxxxV9F/ECMXbQ7B7zbKxxxxUno65c.
Are you sure you want to continue connecting (yes/no/[fingerprint])?
confirm action -> type "yes"
done, you will get a message, since now your sftp connection to 'test.rebex.net' should work
Warning: Permanently added 'test.rebex.net,ip' (ECDSA) to the list of known hosts.