Develop Reference

Why do I get error "Expected a message object, but got kind: " when using Datastore API?

I'm using this Datastore API documentation and here is my code in Python:
class DatastoreConf:
def __init__(self):
self.datastore_client = datastore.Client()
def insert_entity(self):
complete_key = self.datastore_client.key("Task", "sampleTask")
task = datastore.Entity(key=complete_key)
task.update(
{
"category": "Personal",
"done": False,
"priority": 4,
"description": "Learn Cloud Datastore",
}
)
self.datastore_client.put(task)
d = DatastoreConf()
d.insert_entity()
I've already set GOOGLE_APPLICATION_CREDENTIALS variable to my service account key and I have sufficient roles, but I'm getting the error
TypeError: Expected a message object, but got kind: "Task" name: "sampleTask".
The traceback that I got is:
Traceback (most recent call last):
File "C:/Users/user/project/datastore_configuration.py", line 24, in <module>
d.get_entity()
File "C:/Users/user/project/datastore_configuration.py", line 21, in get_entity
self.datastore_client.put(task)
File "C:\Users\user\project\venv\lib\site-packages\google\cloud\datastore\client.py", line 575, in put
self.put_multi(entities=[entity], retry=retry, timeout=timeout)
File "C:\Users\user\projectvenv\lib\site-packages\google\cloud\datastore\client.py", line 612, in put_multi
current.put(entity)
File "C:\Users\user\project\venv\lib\site-packages\google\cloud\datastore\batch.py", line 227, in put
_assign_entity_to_pb(entity_pb, entity)
File "C:\Users\user\project\venv\lib\site-packages\google\cloud\datastore\batch.py", line 373, in _assign_entity_to_pb
bare_entity_pb = helpers.entity_to_protobuf(entity)
File "C:\Users\user\project\venv\lib\site-packages\google\cloud\datastore\helpers.py", line 207, in entity_to_protobuf
key_pb = entity.key.to_protobuf()
File "C:\Users\user\project\venv\lib\site-packages\google\cloud\datastore\key.py", line 298, in to_protobuf
key.path.append(element)
Do you have any idea what causes this error?

I resolved the problem. The error was due to version difference. Datastore worked with library protobuf 3.20.1 but didn't with 4.21.0.
I think it's beacause Firestore is now developed much more and the versions in Datastore are no longer supported.

Most recent SFTP python package and best practices

I've been looking for SFTP python packages, ftpretty works fine for me:
https://pypi.org/project/ftpretty/
but I want to use a more secure protocol.
PySftp is obviously a bit outdated (Edit: it seems that pysftp is still frequently used, about the error please see below):
https://bitbucket.org/dundeemt/pysftp/src/master/
And throws me several errors on Win10, PyCharm, Python3.6:
C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\pysftp\__init__.py:61: UserWarning: Failed to load HostKeys from C:\Users\bobin\.ssh\known_hosts. You will need to explicitly load HostKeys (cnopts.hostkeys.load(filename)) or disableHostKey checking (cnopts.hostkeys = None).
warnings.warn(wmsg, UserWarning)
Traceback (most recent call last):
File "C:/Users/bobin/PycharmProjects/classtest/pysftptest.py", line 7, in <module>
with pysftp.Connection(host=myHostname, username=myUsername, password=myPassword) as sftp:
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\pysftp\__init__.py", line 132, in __init__
self._tconnect['hostkey'] = self._cnopts.get_hostkey(host)
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\pysftp\__init__.py", line 71, in get_hostkey
raise SSHException("No hostkey for host %s found." % host)
paramiko.ssh_exception.SSHException: No hostkey for host s233.goserver.host found.
Exception ignored in: <bound method Connection.__del__ of <pysftp.Connection object at 0x00000235B0695048>>
Traceback (most recent call last):
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\pysftp\__init__.py", line 1013, in __del__
self.close()
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\pysftp\__init__.py", line 784, in close
if self._sftp_live:
AttributeError: 'Connection' object has no attribute '_sftp_live'
Process finished with exit code 1
This thread seemed relevant to me but it's already 12years old:
SFTP in Python? (platform independent)
And the paramiko package is also throwing me errors:
Traceback (most recent call last):
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\paramiko\sftp_client.py", line 130, in __init__
server_version = self._send_version()
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\paramiko\sftp.py", line 134, in _send_version
t, data = self._read_packet()
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\paramiko\sftp.py", line 201, in _read_packet
x = self._read_all(4)
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\paramiko\sftp.py", line 188, in _read_all
raise EOFError()
EOFError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/bobin/PycharmProjects/classtest/paramikotest.py", line 12, in <module>
sftp = paramiko.SFTPClient.from_transport(transport)
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\paramiko\sftp_client.py", line 170, in from_transport
return cls(chan)
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\paramiko\sftp_client.py", line 132, in __init__
raise SSHException("EOF during negotiation")
paramiko.ssh_exception.SSHException: EOF during negotiation
Process finished with exit code 1
I have come so far to understand that I probably need a keyfile that I can find out by first connecting to my webspace using e.g. filezilla:
How To Extract SFTP SSH Key From Key Cache in FileZilla FTP Client
My question is: how do I establish an SFTP connection to my host, which is webgo: https://www.webgo.de/hilfe/content/76/52/de/was-ist-sftp.html
EDIT: providing no host_key as follows:
import pysftp
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
myHostname = "host"
myUsername = "user"
myPassword = "pass"
with pysftp.Connection(host=myHostname, username=myUsername, password=myPassword, cnopts=cnopts, port=22) as sftp:
print("Connection succesfully stablished ... ")
sftp.put('C:\TEMP\Capture.PNG', preserve_mtime=True)
still throws me an error for providing no host_keys:
C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\pysftp\__init__.py:61: UserWarning: Failed to load HostKeys from C:\Users\bobin\.ssh\known_hosts. You will need to explicitly load HostKeys (cnopts.hostkeys.load(filename)) or disableHostKey checking (cnopts.hostkeys = None).
warnings.warn(wmsg, UserWarning)
EDIT2: tried prettyftp but my provider is refusing the connection:
Traceback (most recent call last):
File "C:/Users/bobin/PycharmProjects/classtest/testftp.py", line 15, in <module>
f.put('C:\TEMP\Capture.PNG', 'Capture230.PNG')
File "C:\Users\bobin\PycharmProjects\classtest\venv\lib\site-packages\ftpretty.py", line 119, in put
self.conn.storbinary('STOR %s' % remote_file, local_file)
File "C:\Users\bobin\AppData\Local\Programs\Python\Python36\lib\ftplib.py", line 513, in storbinary
return self.voidresp()
File "C:\Users\bobin\AppData\Local\Programs\Python\Python36\lib\ftplib.py", line 249, in voidresp
resp = self.getresp()
File "C:\Users\bobin\AppData\Local\Programs\Python\Python36\lib\ftplib.py", line 242, in getresp
raise error_temp(resp)
ftplib.error_temp: 425 Unable to build data connection: Operation not permitted
Used following code snippet, setting secure=False worked again:
from ftpretty import ftpretty
# Minimal
f = ftpretty('host','user','pass', port=21, secure=True)
f.put('C:\TEMP\Capture.PNG', 'Capture230.PNG')
f.close()

For the first error, it seems like a bug in pysftp.
You can have a look at the Connection class here on line 76, and the attribute _sftp_live is defined on line 134, so this is definitely an error occurring at runtime without being validated correctly. I have also been able to find this related error, which likely explains the cause of this issue; the solution is also mentioned in the error if you want to explicitly fix it.
I would still consider using ftpretty. It does use TLS for security and a pretty safe wrapper, you can simply enable it by setting the secure parameter to True (secure=True) - which by default is set as False.

How to fix - TypeError: can't set attributes of built-in/extension type 'set'

I'm relatively new to python and I'm trying to make a simple GUI chat with python. It is programmed to ask for a nickname when a client joins the server. All works fine until the part where I enter the nickname. When I enter the nickname I get these errors from the server and client respectively, I'll provide the tracebacks as well.
ConnectionResetError: [Errno 104] Connection reset by peer
(From server)
traceback:
Traceback (most recent call last):
File "server.py", line 51, in <module>
receive()
File "server.py", line 44, in receive
broadcast(f"{nickname} entered to the chat!\n".encode('utf-8'))
File "server.py", line 17, in broadcast
client.send(message)
TypeError: can't set attributes of built-in/extension type 'set'
(from client)
traceback:
Traceback (most recent call last):
File "C:/Users/ISINDU WICKRAMASEKAR/PycharmProjects/guichat/client.py", line 94, in <module>
client = Client(HOST, PORT)
File "C:/Users/ISINDU WICKRAMASEKAR/PycharmProjects/guichat/client.py", line 22, in __init__
set.gui_done = False
The code for the server and client are also linked.
server -> https://pastebin.com/0W7Cw9Cu
client-> https://pastebin.com/FES2UNc1
What I have tried:
I tried googling for answers and I can't say I didn't get any, but I didn't understand how to implement those solutions for my issue. These are the links I referred to
Python handling socket.error: [Errno 104] Connection reset by peer
python can't set attributes of built-in/extension type 'object'

I think the error message points us to the problem pretty precisely.
Traceback (most recent call last):
File "C:/Users/ISINDU WICKRAMASEKAR/PycharmProjects/guichat/client.py", line 94, in <module>
client = Client(HOST, PORT)
File "C:/Users/ISINDU WICKRAMASEKAR/PycharmProjects/guichat/client.py", line 22, in __init__
set.gui_done = False
TypeError: can't set attributes of built-in/extension type 'set' (from client)
On line 22 of client.py, you try to assign set.gui_done = False; you probably meant self.gui_done.

Your problem is this line
set.gui_done = False
set is a protected keyword in python. It's reserved for the set data type. With this line youre telling the interpreter to change the gui_done attribute of the in-built set type to false which is not allowed. That's why you're getting a TypeError. If it wasn't an in-built type you'd still get an AttributeError because set doesn't have a gui_done attribute. You probably meant to use self instead of set

How to execute get command in redis cluster mode using python?

I want to perform the following scenario on redis using python.
Using command line:
1) redis-cli -c
2) redis_prompt >> get some_string
I tried using redis and rediscluster modules, but with no luck. Below are the codes that i tried:
1)
r = redis.Redis(host='123.123.123.123', port=6379, db=0)
r.get('some srting')
Got the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\redis\client.py", line 880, in get
return self.execute_command('GET', name)
File "C:\Python27\lib\site-packages\redis\client.py", line 573, in execute_command
return self.parse_response(connection, command_name, **options)
File "C:\Python27\lib\site-packages\redis\client.py", line 585, in parse_response
response = connection.read_response()
File "C:\Python27\lib\site-packages\redis\connection.py", line 582, in read_response
raise response
redis.exceptions.ResponseError: MOVED 9442 172.16.176.36:6380
2)
from rediscluster import StrictRedisCluster
startup_nodes = [{"host": "123.123.123.123", "port": "7000"}]
rc_readonly = StrictRedisCluster(startup_nodes=startup_nodes, decode_responses=True, readonly_mode=True)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\rediscluster\client.py", line 181, in __init__
**kwargs
File "C:\Python27\lib\site-packages\rediscluster\connection.py", line 353, in __init__
**connection_kwargs)
File "C:\Python27\lib\site-packages\rediscluster\connection.py", line 141, in __init__
self.nodes.initialize()
File "C:\Python27\lib\site-packages\rediscluster\nodemanager.py", line 240, in initialize
raise RedisClusterException("Redis Cluster cannot be connected. Please provide at least one reachable node.")
rediscluster.exceptions.RedisClusterException: Redis Cluster cannot be connected. Please provide at least one reachable node.
Any help would be great.

Ok, so this error means one of the following:
In these lines of code (1):
r = redis.Redis(host='123.123.123.123', port=6379, db=0)
r.get('some srting')
you try to fetch the key "some sting" from host 123.123.123.123 and
port 6379. The error redis.exceptions.ResponseError: MOVED 9442
172.16.176.36:6380 means that this key lies in 172.16.176.36:6380. So try to connect with ./redis-cli -c -p 6380 in this node and then
execute CLUSTER GETKEYSINSLOT 6380 3 to see if this key is actually
here.
From (2) this error: rediscluster.exceptions.RedisClusterException: Redis Cluster cannot be connected. Please provide at least one reachable node means that something is wrong with the cluster. Maybe, you have not made correct the initialization.
First in /home/{{user}}/redis-yourversion/utils/create-cluster you
will find create-cluster.sh. Set up port and host and nodes and then
execute ./create-cluster start ./create-cluster-create. But first you
have to start redis server in cluster mode. Go to redis.conf and in
the configuration parameters you will see cluster-enabled no. Set up
to yes. After this you must run ./redis-server ../redis.conf (give
the path for redis.conf). If cluster enabled is correct you will see Running in cluster mode
Important thing to remember:
The parameter -c in terminal means that redis nodes can execute the redirections and find the correct node that the key belongs without MOVED errors. If you have a driver suppose in python to manipulate the nodes and fetch keys or whatever you must manage these redirections by yourself.

how to upload files on a windows sharing network using python

I am using a mac machine. I have access to a windows share network. I am able to connect to it from my mac using 'smb://location'.
Using python, I want to be able to download/upload files on the windows share network using my access credentials. How can I do it.
The shared location looks like smb://v1/abc. I am able to connect to this. Now I want to create a file in a folder within this connection.
I want to create a file inside the folder d2. I tried using pysmb module. I tried the following:
>>> fp = open('local_file.txt', 'w')
>>> conn = SMBConnection('user', 'password', 'test_v1','v1', use_ntlm_v2 = True)
>>> conn.retrieveFile('share', 'abc/d2/check.csv', fp)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/8417/repos/workspace/venv/lib/python2.7/site-packages/smb/SMBConnection.py", line 267, in retrieveFile
return self.retrieveFileFromOffset(service_name, path, file_obj, 0L, -1L, timeout)
File "/Users/8417/repos/workspace/venv/lib/python2.7/site-packages/smb/SMBConnection.py", line 283, in retrieveFileFromOffset
raise NotConnectedError('Not connected to server')
smb.base.NotConnectedError: Not connected to server
Also tried:
>>> conn = SMBConnection('user', 'password', 'test_v1','v1/abc', use_ntlm_v2 = True)
>>> conn.retrieveFile('share', 'd2/check.csv', fp)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/8417/repos/workspace/venv/lib/python2.7/site-packages/smb/SMBConnection.py", line 267, in retrieveFile
return self.retrieveFileFromOffset(service_name, path, file_obj, 0L, -1L, timeout)
File "/Users/8417/repos/workspace/venv/lib/python2.7/site-packages/smb/SMBConnection.py", line 283, in retrieveFileFromOffset
raise NotConnectedError('Not connected to server')
smb.base.NotConnectedError: Not connected to server

The SMBConnection constructor doesn't establish the connection, you need to call the connect method before retrieving the file.