I've been looking for SFTP python packages, ftpretty works fine for me:
https://pypi.org/project/ftpretty/
but I want to use a more secure protocol.
PySftp is obviously a bit outdated (Edit: it seems that pysftp is still frequently used, about the error please see below):
https://bitbucket.org/dundeemt/pysftp/src/master/
And throws me several errors on Win10, PyCharm, Python3.6:
C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\pysftp\__init__.py:61: UserWarning: Failed to load HostKeys from C:\Users\bobin\.ssh\known_hosts. You will need to explicitly load HostKeys (cnopts.hostkeys.load(filename)) or disableHostKey checking (cnopts.hostkeys = None).
warnings.warn(wmsg, UserWarning)
Traceback (most recent call last):
File "C:/Users/bobin/PycharmProjects/classtest/pysftptest.py", line 7, in <module>
with pysftp.Connection(host=myHostname, username=myUsername, password=myPassword) as sftp:
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\pysftp\__init__.py", line 132, in __init__
self._tconnect['hostkey'] = self._cnopts.get_hostkey(host)
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\pysftp\__init__.py", line 71, in get_hostkey
raise SSHException("No hostkey for host %s found." % host)
paramiko.ssh_exception.SSHException: No hostkey for host s233.goserver.host found.
Exception ignored in: <bound method Connection.__del__ of <pysftp.Connection object at 0x00000235B0695048>>
Traceback (most recent call last):
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\pysftp\__init__.py", line 1013, in __del__
self.close()
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\pysftp\__init__.py", line 784, in close
if self._sftp_live:
AttributeError: 'Connection' object has no attribute '_sftp_live'
Process finished with exit code 1
This thread seemed relevant to me but it's already 12years old:
SFTP in Python? (platform independent)
And the paramiko package is also throwing me errors:
Traceback (most recent call last):
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\paramiko\sftp_client.py", line 130, in __init__
server_version = self._send_version()
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\paramiko\sftp.py", line 134, in _send_version
t, data = self._read_packet()
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\paramiko\sftp.py", line 201, in _read_packet
x = self._read_all(4)
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\paramiko\sftp.py", line 188, in _read_all
raise EOFError()
EOFError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/bobin/PycharmProjects/classtest/paramikotest.py", line 12, in <module>
sftp = paramiko.SFTPClient.from_transport(transport)
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\paramiko\sftp_client.py", line 170, in from_transport
return cls(chan)
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\paramiko\sftp_client.py", line 132, in __init__
raise SSHException("EOF during negotiation")
paramiko.ssh_exception.SSHException: EOF during negotiation
Process finished with exit code 1
I have come so far to understand that I probably need a keyfile that I can find out by first connecting to my webspace using e.g. filezilla:
How To Extract SFTP SSH Key From Key Cache in FileZilla FTP Client
My question is: how do I establish an SFTP connection to my host, which is webgo: https://www.webgo.de/hilfe/content/76/52/de/was-ist-sftp.html
EDIT: providing no host_key as follows:
import pysftp
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
myHostname = "host"
myUsername = "user"
myPassword = "pass"
with pysftp.Connection(host=myHostname, username=myUsername, password=myPassword, cnopts=cnopts, port=22) as sftp:
print("Connection succesfully stablished ... ")
sftp.put('C:\TEMP\Capture.PNG', preserve_mtime=True)
still throws me an error for providing no host_keys:
C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\pysftp\__init__.py:61: UserWarning: Failed to load HostKeys from C:\Users\bobin\.ssh\known_hosts. You will need to explicitly load HostKeys (cnopts.hostkeys.load(filename)) or disableHostKey checking (cnopts.hostkeys = None).
warnings.warn(wmsg, UserWarning)
EDIT2: tried prettyftp but my provider is refusing the connection:
Traceback (most recent call last):
File "C:/Users/bobin/PycharmProjects/classtest/testftp.py", line 15, in <module>
f.put('C:\TEMP\Capture.PNG', 'Capture230.PNG')
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\ftpretty.py", line 119, in put
self.conn.storbinary('STOR %s' % remote_file, local_file)
File "C:\Users\bobin\AppData\Local\Programs\Python\Python36\lib\ftplib.py", line 513, in storbinary
return self.voidresp()
File "C:\Users\bobin\AppData\Local\Programs\Python\Python36\lib\ftplib.py", line 249, in voidresp
resp = self.getresp()
File "C:\Users\bobin\AppData\Local\Programs\Python\Python36\lib\ftplib.py", line 242, in getresp
raise error_temp(resp)
ftplib.error_temp: 425 Unable to build data connection: Operation not permitted
Used following code snippet, setting secure=False worked again:
from ftpretty import ftpretty
# Minimal
f = ftpretty('host','user','pass', port=21, secure=True)
f.put('C:\TEMP\Capture.PNG', 'Capture230.PNG')
f.close()
For the first error, it seems like a bug in pysftp.
You can have a look at the Connection class here on line 76, and the attribute _sftp_live is defined on line 134, so this is definitely an error occurring at runtime without being validated correctly. I have also been able to find this related error, which likely explains the cause of this issue; the solution is also mentioned in the error if you want to explicitly fix it.
I would still consider using ftpretty. It does use TLS for security and a pretty safe wrapper, you can simply enable it by setting the secure parameter to True (secure=True) - which by default is set as False.
Related
I'm trying to store a python dict in memory through redis, I was following the pypi doc, when I try to instance RedisCluster i got this error:
from redis.cluster import RedisCluster as Redis # this line works
rc = Redis(host='localhost', port=6379) # here is the problem
Traceback (most recent call last):
File "/home/developer/.pyenv/versions/redisTesting/lib/python3.9/site-packages/redis/cluster.py", line 1306, in initialize
raise RedisClusterException(
redis.exceptions.RedisClusterException: Cluster mode is not enabled on this node
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/developer/.pyenv/versions/3.9.5/lib/python3.9/code.py", line 90, in runcode
exec(code, self.locals)
File "<input>", line 1, in <module>
File "/home/developer/.pyenv/versions/redisTesting/lib/python3.9/site-packages/redis/cluster.py", line 507, in __init__
self.nodes_manager = NodesManager(
File "/home/developer/.pyenv/versions/redisTesting/lib/python3.9/site-packages/redis/cluster.py", line 1128, in __init__
self.initialize()
File "/home/developer/.pyenv/versions/redisTesting/lib/python3.9/site-packages/redis/cluster.py", line 1334, in initialize
raise RedisClusterException(
redis.exceptions.RedisClusterException: ERROR sending "cluster slots" command to redis server 127.0.0.1:6379. error: Cluster mode is not enabled on this node
I know that the problem is Cluster mode is not enabled on this node but I didn't find a way to solve this error, how can enable the cluster mode on the node?
Also I find a way to store this dict in memory with
import redis
r = redis.Redis()
r.hmset({
"color": "green",
"price": 99.99,
"style": "baseball",
"quantity": 200,
"npurchased": 0,
})
but this got me a deprecate warning <input>:1: DeprecationWarning: Redis.hmset() is deprecated. Use Redis.hset() instead. and when I tryed to use r.hset() the terminal got me redis.exceptions.DataError: Invalid input of type: 'dict'. Convert to a bytes, string, int or float first.
Change redis.cluster to redis and connection would succeed.
#from redis.cluster import RedisCluster as Redis # this line works (but assumes you are connecting to Redis Cluster)
from redis import Redis
rc = Redis(host='localhost', port=6379)
Background:
spark standalone cluster mode on k8s
spark 2.2.1
hadoop 2.7.6
run code in python, not in pyspark
client mode, not cluster mode
The pyspark code in python, not in pyspark env.
Every code can work and get it down. But 'sometimes', when the code finish and exit, below error will show up even time.sleep(10) after spark.stop().
{{py4j.java_gateway:1038}} INFO - Error while receiving.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/py4j-0.10.4-py2.7.egg/py4j/java_gateway.py", line 1035, in send_command
raise Py4JNetworkError("Answer from Java side is empty")
Py4JNetworkError: Answer from Java side is empty
[2018-11-22 09:06:40,293] {{root:899}} ERROR - Exception while sending command.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/py4j-0.10.4-py2.7.egg/py4j/java_gateway.py", line 883, in send_command
response = connection.send_command(command)
File "/usr/lib/python2.7/site-packages/py4j-0.10.4-py2.7.egg/py4j/java_gateway.py", line 1040, in send_command
"Error while receiving", e, proto.ERROR_ON_RECEIVE)
Py4JNetworkError: Error while receiving
[2018-11-22 09:06:40,293] {{py4j.java_gateway:443}} DEBUG - Exception while shutting down a socket
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/py4j-0.10.4-py2.7.egg/py4j/java_gateway.py", line 441, in quiet_shutdown
socket_instance.shutdown(socket.SHUT_RDWR)
File "/usr/lib64/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
File "/usr/lib64/python2.7/socket.py", line 170, in _dummy
raise error(EBADF, 'Bad file descriptor')
error: [Errno 9] Bad file descriptor
I guess the reason is parent process python try to get log message from terminated child process 'jvm'. But the wired thing is the error not always raise...
Any suggestion?
This root-cause is 'py4j' log-level.
I set python log-level to DEBUG, this let the 'py4j' client & 'java' raise connection error when close pyspark.
So setting python log-level to INFO or more higher level will resolve this problem.
ref: Gateway raises an exception when shut down
ref: Tune down the logging level for callback server messages
ref: PySpark Internals
I am using the Flask-WeRobot plugin and I am seeing an error being thrown here:
class FileStorage(SessionStorage):
"""
FileStorage 会把你的 Session 数据以 dbm 形式储存在文件中。
:param filename: 文件名， 默认为 ``werobot_session``
"""
def __init__(self, filename='werobot_session'):
print filename
try:
self.db = dbm.open(filename, "c")
except Exception, e:
print e
raise e
This is what is being outputted:
Traceback (most recent call last):
File "application.py", line 18, in <module>
another_robot = WeRoBot(token='abcdefg', enable_session=True)
File "/Users/vng/Dropbox/Code/Paw/venv/lib/python2.7/site-packages/flask_werobot.py", line 42, in __init__
super(WeRoBot, self).__init__(*args, **kwargs)
File "/Users/vng/Dropbox/Code/Paw/venv/lib/python2.7/site-packages/werobot/robot.py", line 47, in __init__
filename=os.path.abspath("werobot_session")
File "/Users/vng/Dropbox/Code/Paw/venv/lib/python2.7/site-packages/werobot/session/filestorage.py", line 20, in __init__
self.db = dbm.open(filename, "c")
File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/anydbm.py", line 85, in open
return mod.open(file, flag, mode)
gdbm.error: (35, 'Resource temporarily unavailable')
At first I thought it would have something to do with permissions, but changing the file, werobot_session didn't work.
I am on OSX running python 2.7. This code works on my ubuntu production server.
Any ideas what's causing this?
We find this bug too and we are working on it now.
It seems that it's a bug of gdbm. And everything works fine on linux.
You can check this issue for further information.
https://github.com/whtsky/WeRoBot/issues/102
I'm trying to use paramiko for SSH, but got an error:
>>> import paramiko
>>> ssh = paramiko.SSHClient()
>>> ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
>>> ssh.connect('54.***.***.110', key_filename='D:\Keys\MyOWN\priv.ppk')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build\bdist.win32\egg\paramiko\client.py", line 366, in connect
File "build\bdist.win32\egg\paramiko\client.py", line 515, in _auth
File "build\bdist.win32\egg\paramiko\agent.py", line 343, in __init__
File "build\bdist.win32\egg\paramiko\agent.py", line 66, in _connect
File "build\bdist.win32\egg\paramiko\agent.py", line 83, in _send_message
File "build\bdist.win32\egg\paramiko\win_pageant.py", line 123, in send
File "build\bdist.win32\egg\paramiko\win_pageant.py", line 89, in _query_pageant
File "build\bdist.win32\egg\paramiko\_winapi.py", line 273, in get_security_attributes_for_user
File "build\bdist.win32\egg\paramiko\_winapi.py", line 222, in descriptor
NameError: global name 'descriptor' is not defined
Regarding this issue - it was solved, but - I still have this error (latest paramiko version, downloaded from it's Github).
May be - there is some other libs, to wok via SSH with RSA-key authorization?
Or - any way to solve this NameError...
Seems like the issue is not really solved (I too downloaded latest zip: it can also be seen on [GitHub]: paramiko/paramiko - (v1.15.2) paramiko/paramiko/_winapi.py), so you'll have to fix it yourself in your paramiko installation files (fixed in v1.15.3):
Edit your ${PYTHON_DIR}\build\bdist.win32\egg\paramiko\_winapi.py (${PYTHON_DIR} is just a placeholder for your Python installation directory),
and at lines 222 and 223 simply replace descriptor by value:
self._descriptor = descriptor
self.lpSecurityDescriptor = ctypes.addressof(descriptor)
should become:
self._descriptor = value
self.lpSecurityDescriptor = ctypes.addressof(value)
I used to get this type of errors. I restarted the machine and it got solved!
But I think there is a bug in the paramiko library.
Changing descriptor by value as explained by CristiFati works fine.
I met the issue too, please try to set the allow_agent=False, and it should be resolved.
import paramiko
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('54.***.***.110', key_filename='D:\Keys\MyOWN\priv.ppk', allow_agent=False)
I'm trying to write a quick python script to grab some logs with sftp. My first inclination was to use Pysftp, since it seemed like it made it very simple. It worked great, until it got to a larger file. I got an error while getting any file over about 13 MB. I then decided to try writing what I needed directly in Paramiko, rather than relying on the extra layer of Pysftp. After figuring out how to do that, I ended up getting the exact same error. Here's the Paramiko code, as well as the trace from the error I get. Does anyone have any idea why this would have an issue pulling any largish files? Thanks.
# Create tranport and connect
transport = paramiko.Transport((host, 22))
transport.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(transport)
# List of the log files in c:
files = sftp.listdir('c:/logs')
# Now pull them, logging as you go
for f in files:
if f[0].lower() == 't' or f[:3].lower() == 'std':
logger.info('Pulling {0}'.format(f))
sftp.get('c:/logs/{0}'.format(f), output_dir +'/{0}'.format(f))
# Close the connection
sftp.close()
transport.close()
And here's the error:
No handlers could be found for logger "paramiko.transport"
Traceback (most recent call last):
File "pull_logs.py", line 420, in <module> main()
File "pull_logs.py", line 410, in main
pull_logs(username, host, password, location)
File "pull_logs.py", line 142, in pull_logs
sftp.get('c:/logs/{0}'.format(f), output_dir +'/{0}'.format(f))
File "/Users/me/my_site/site_packages/paramiko/sftp_client.py", line 676, in get
size = self.getfo(remotepath, fl, callback)
File "/Users/me/my_site/site_packages/paramiko/sftp_client.py", line 645, in getfo
data = fr.read(32768)
File "/Users/me/my_site/site_packages/paramiko/file.py", line 153, in read
new_data = self._read(read_size)
File "/Users/me/my_site/site_packages/paramiko/sftp_file.py", line 152, in _read
data = self._read_prefetch(size)
File "/Users/me/my_site/site_packages/paramiko/sftp_file.py", line 132, in _read_prefetch
self.sftp._read_response()
File "/Users/me/my_site/site_packages/paramiko/sftp_client.py", line 721, in _read_response
raise SSHException('Server connection dropped: %s' % (str(e),))
paramiko.SSHException: Server connection dropped: