Develop Reference

How to return value from C program to Python script

When I try to return value from c code to python code i got error.
Traceback (most recent call last):
File "python.py", line 54, in <module>
print("\n\n\n\RESULT: ", str(result, "utf-8"))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 245: invalid start byte
in c function i try to return json string which got hex data - which I could parse in python and than make another calculation.
Example of returned string is "{"data":"0x123132"}"
In python i use
import ctypes
my_functions = ctypes.cdll.LoadLibrary("./my_functions.so")
my_functions.getJson.argtypes = (ctypes.c_char_p,)
my_functions.EthereumProcessor.restype = ctypes.c_char_p
result=my_functions.getJson()
print("\n\n\n\RESULT: ", str(result, "utf-8"))

Porting Python2 to Python3 - Reading Bytes from a socket and using unpack() correctly

There is code here that I am trying to convert from Python2 to Python3. In this section of code, data is received from a socket. data is declared to be an empty string and then concatenated. This is an important Python2 to 3 distinction. In Python3, the 'received' variable is of type Bytes and thus needs to be converted to string first via the use of str(). However, str() needs an encoding parameter. What would the default one be for python 2? I've tried several different encodings (latin-1 and such) but they seem to not match up with a magic value that is defined here after being unpacked here
"\xffSMB seems to decode correctly while "\xfeSMB does not.
I have adjusted the non_polling_read function and the NetBIOSSessionPacket class as follows. Full modified source code available on GitHub.
def non_polling_read(self, read_length, timeout):
data = b''
bytes_left = read_length
while bytes_left > 0:
try:
ready, _, _ = select.select([self._sock.fileno()], [], [], timeout)
if not ready:
raise NetBIOSTimeout
received = self._sock.recv(bytes_left)
if len(received) == 0:
raise NetBIOSError('Error while reading from remote', ERRCLASS_OS, None)
data = data + received
bytes_left = read_length - len(data)
except select.error as ex:
if ex[0] != errno.EINTR and ex[0] != errno.EAGAIN:
raise NetBIOSError('Error occurs while reading from remote', ERRCLASS_OS, ex[0])
return data
class NetBIOSSessionPacket:
def __init__(self, data=0):
self.type = 0x0
self.flags = 0x0
self.length = 0x0
if data == 0:
self._trailer = ''
else:
try:
self.type = data[0]
if self.type == NETBIOS_SESSION_MESSAGE:
self.length = data[1] << 16 | (unpack('!H', data[2:4])[0])
else:
self.flags = data[1]
self.length = unpack('!H', data[2:4])[0]
self._trailer = data[4:]
except Exception as e:
import traceback
traceback.print_exc()
raise NetBIOSError('Wrong packet format ')
When I start the server and issue 'smbclient -L 127.0.0.1 -d 4' from the commandline, the server first creates a libs.nmb.NetBIOSTCPSession which appears to be working well. Once it tries to unwrap the libs.nmb.NetBIOSSessionPacket, it throws an exception.
Traceback (most recent call last):
File "/root/PycharmProjects/HoneySMB/libs/smbserver.py", line 3975, in processRequest
packet = smb.NewSMBPacket(data=data)
File "/root/PycharmProjects/HoneySMB/libs/smb.py", line 690, in __init__
Structure.__init__(self, **kargs)
File "/root/PycharmProjects/HoneySMB/libs/structure.py", line 77, in __init__
self.fromString(data)
File "/root/PycharmProjects/HoneySMB/libs/structure.py", line 144, in fromString
self[field[0]] = self.unpack(field[1], data[:size], dataClassOrCode = dataClassOrCode, field = field[0])
File "/root/PycharmProjects/HoneySMB/libs/structure.py", line 288, in unpack
raise Exception("Unpacked data doesn't match constant value %r should be %r" % (data, answer))
Exception: ("Unpacked data doesn't match constant value b'\\xffSMB' should be 'ÿSMB'", 'When unpacking field \'Signature | "ÿSMB | b\'\\xffSMBr\\x00\\x00\\x00\\x00\\x18C\\xc8\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\xfe\\xff\\x00\\x00\\x00\\x00\\x00\\xb1\\x00\\x02PC NETWORK PROGRAM 1.0\\x00\\x02MICROSOFT NETWORKS 1.03\\x00\\x02MICROSOFT NETWORKS 3.0\\x00\\x02LANMAN1.0\\x00\\x02LM1.2X002\\x00\\x02DOS LANMAN2.1\\x00\\x02LANMAN2.1\\x00\\x02Samba\\x00\\x02NT LANMAN 1.0\\x00\\x02NT LM 0.12\\x00\\x02SMB 2.002\\x00\\x02SMB 2.???\\x00\'[:4]\'')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/PycharmProjects/HoneySMB/libs/smbserver.py", line 3597, in handle
resp = self.__SMB.processRequest(self.__connId, p.get_trailer())
File "/root/PycharmProjects/HoneySMB/libs/smbserver.py", line 3979, in processRequest
packet = smb2.SMB2Packet(data=data)
File "/root/PycharmProjects/HoneySMB/libs/smb3structs.py", line 435, in __init__
Structure.__init__(self,data)
File "/root/PycharmProjects/HoneySMB/libs/structure.py", line 77, in __init__
self.fromString(data)
File "/root/PycharmProjects/HoneySMB/libs/structure.py", line 144, in fromString
self[field[0]] = self.unpack(field[1], data[:size], dataClassOrCode = dataClassOrCode, field = field[0])
File "/root/PycharmProjects/HoneySMB/libs/structure.py", line 288, in unpack
raise Exception("Unpacked data doesn't match constant value %r should be %r" % (data, answer))
Exception: ("Unpacked data doesn't match constant value b'\\xffSMB' should be 'þSMB'", 'When unpacking field \'ProtocolID | "þSMB | b\'\\xffSMBr\\x00\\x00\\x00\\x00\\x18C\\xc8\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\xfe\\xff\\x00\\x00\\x00\\x00\\x00\\xb1\\x00\\x02PC NETWORK PROGRAM 1.0\\x00\\x02MICROSOFT NETWORKS 1.03\\x00\\x02MICROSOFT NETWORKS 3.0\\x00\\x02LANMAN1.0\\x00\\x02LM1.2X002\\x00\\x02DOS LANMAN2.1\\x00\\x02LANMAN2.1\\x00\\x02Samba\\x00\\x02NT LANMAN 1.0\\x00\\x02NT LM 0.12\\x00\\x02SMB 2.002\\x00\\x02SMB 2.???\\x00\'[:4]\'')
Now it's obvious why this throws an exception. After all, 0xFF does NOT equal ÿ or þ. The question is why is 0xFF the value it tries to write in there in the first place when it should be two different values?
The original python 2 code seems to be quite close to C (using unpack and importing cstring). Is there an obvious benefit to this here or could this be done more simply?
My actual question is:
In the original code, there is no reference to any encoding anywhere. So how is this divined then? Where does it translate 0xFF to ÿ?

Retrieving error 'utf-8' codec can't decode byte 0xb2 in position 17: invalid start byte

client side
I have following code sending array from client to server. I am retreiving UTF-8 decode error. I don't prefer to load data pickle or zlib compress. I am pyzmq socket to send data and receive data.
def send_array(socket, A, flags=0, copy=True, track=False):
"""send a numpy array with metadata"""
md = dict(
dtype = str(A.dtype),
shape = A.shape,
)
socket.send_json(md, flags|zmq.SNDMORE)
return socket.send(A, flags, copy=copy, track=track)
def forward_data(frame, count):
source = cv2.imdecode(np.fromstring(base64.b64decode(frame), dtype=np.uint8), 1)
image = img_to_array(source)
image = image.reshape((1, image.shape[0], image.shape[1], image.shape[2]))
preds = m1.predict(preprocess_input(image))
send_array(dest_socket,preds)
server side
def recv_array(socket, flags=0, copy=True, track=False):
"""recv a numpy array"""
md = socket.recv_json(flags=flags)
msg = socket.recv(flags=flags, copy=copy, track=track)
buf = np.buffer(msg)
A = numpy.frombuffer(buf, dtype=md['dtype'])
return A.reshape(md['shape'])
def handle_image_processing(frame, count):
frame_object=recv_array(socket)
tmp = np.zeros(frame_object.shape)
for i in range( 0, 1 ):
tmp[i,:] = tmp[i, :]
predictions_result = m2.predict( tmp )
label_vgg16 = decode_predictions( predictions_result )
print(label_vgg16)
Error retrieving
Traceback (most recent call last):
handle_image_processing(frame,count)
File "C:/Pause-Resume-Approach-Updated/server-b.py", line 82, in handle_image_processing
frame_object=recv_array(socket)
File "C:/Pause-Resume-Approach-Updated/server-b.py", line 68, in recv_array
md = socket.recv_json(flags=flags)
File "C:\Users\learn\miniconda3\envs\tensorflow\lib\site-packages\zmq\sugar\socket.py", line 690, in recv_json
return self._deserialize(msg, lambda buf: jsonapi.loads(buf, **kwargs))
File "C:\Users\learn\miniconda3\envs\tensorflow\lib\site-packages\zmq\sugar\socket.py", line 515, in _deserialize
return load(recvd)
File "C:\Users\learn\miniconda3\envs\tensorflow\lib\site-packages\zmq\sugar\socket.py", line 690, in <lambda>
return self._deserialize(msg, lambda buf: jsonapi.loads(buf, **kwargs))
File "C:\Users\learn\miniconda3\envs\tensorflow\lib\site-packages\zmq\utils\jsonapi.py", line 54, in loads
s = s.decode('utf8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb2 in position 17: invalid start byte
Thanks

While your code uses (properly) the zmq.SNDMORE flags on constructing the multipart-message payload before sending, the receiving part ignores any & all zmq.RCVMORE-driven testing, if any next part of the delivered message ought be consumed.
Something like this ought work :
while True:
...
if socket.getsockopt( zmq.RCVMORE ):
continue
else:
break
I have never tested a mixed methods python-code on multi-frame messages. The native ZeroMQ API is clear in this and one can read with zmq.RCVMORE-testing until the last part of the multi-frame message, while python ported besides the .recv() a set of derived method, .recv_json() being used here. If that port keeps the native API, the repaired code shall handle the flags as expected, if not, expect problems from some pyzmq side effects, not present in the original API.
If not happy with the pyzmq builtin method .send_obj(), may use a python struct module to very efficiently pack bytefields mapped data-elements ( .shape, .dtype ) and the <array>.data()-method.
Code shall also enjoy to become more robust against not receiving a multi-part, but a plain or even an empty at all message envelope, be it due to one's own human error or due to an adversary intrusion.

UnicodeDecodeError: 'utf-8' codec can't decode byte (python)

I'm playing around, trying to pull a bitcoin block template using the getblocktemplate method. I got a response but when I try to decode it from bytes I'm getting an error. Check it out:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
HOST = "184.70.15.166" # choose a node from blockchain.info
PORT = 8333
s.connect((HOST, PORT))
s.send(msg)
s.recv(1024)
x = random.randrange(320)
print(x)
data = json.dumps({'version': '2.0', 'id': 'x', 'method': 'getblocktemplate'}).encode('utf-8').strip()
s.send(data)
resp = s.recv(2048).decode('utf8')
print (resp)
# prints id number -
315
# prints response (in bytes) -
b'\xf9\xbe\xb4\xd9verack\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00]\xf6\xe0\xe2\xf9\xbe\xb4\xd9alert\x00\x00\x00\x00\x00\x00\x00\xa8\x00\x00\x00\x1b\xf9\xaa\xea`\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\x7f\x00\x00\x00\x00\xff\xff\xff\x7f\xfe\xff\xff\x7f\x01\xff\xff\xff\x7f\x00\x00\x00\x00\xff\xff\xff\x7f\x00\xff\xff\xff\x7f\x00/URGENT: Alert key compromised, upgrade required\x00F0D\x02 e?\xeb\xd6A\x0fG\x0fk\xae\x11\xca\xd1\x9cHA;\xec\xb1\xac,\x17\xf9\x08\xfd\x0f\xd5;\xdc:\xbdR\x02 m\x0e\x9c\x96\xfe\x88\xd4\xa0\xf0\x1e\xd9\xde\xda\xe2\xb6\xf9\xe0\r\xa9L\xad\x0f\xec\xaa\xe6n\xcfh\x9b\xf7\x1bP\xf9\xbe\xb4\xd9ping\x00\x00\x00\x00\x00\x00\x00\x00\x08\x00\x00\x00-8\xc9\x03\xd7\xbc\x0f\xc9\xc2\x1d36'
# try to decode from bytes and get this error -
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 699, in runfile
execfile(filename, namespace)
File "/usr/lib/python3/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 88, in execfile
exec(compile(open(filename, 'rb').read(), filename, 'exec'), namespace)
File "/home/myfolder/Desktop/bitcoin/bitcoin 2017/connection.py", line 66, in <module>
print (resp.decode('utf8'))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf9 in position 0: invalid start byte
What do you think? I've been looking all over the web and can't find a fix that works.

An immediate solution is using binascii like this way:
For example, your data is:
a = b'\xf9\xbe\xb4\xd9verack\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00]\xf6\xe0\xe2\xf9\xbe\xb4\xd9alert\x00\x00\x00\x00\x00\x00\x00\xa8\x00\x00\x00\x1b\xf9\xaa\xea`\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\x7f\x00\x00\x00\x00\xff\xff\xff\x7f\xfe\xff\xff\x7f\x01\xff\xff\xff\x7f\x00\x00\x00\x00\xff\xff\xff\x7f\x00\xff\xff\xff\x7f\x00/URGENT: Alert key compromised, upgrade required\x00F0D\x02 e?\xeb\xd6A\x0fG\x0fk\xae\x11\xca\xd1\x9cHA;\xec\xb1\xac,\x17\xf9\x08\xfd\x0f\xd5;\xdc:\xbdR\x02 m\x0e\x9c\x96\xfe\x88\xd4\xa0\xf0\x1e\xd9\xde\xda\xe2\xb6\xf9\xe0\r\xa9L\xad\x0f\xec\xaa\xe6n\xcfh\x9b\xf7\x1bP\xf9\xbe\xb4\xd9ping\x00\x00\x00\x00\x00\x00\x00\x00\x08\x00\x00\x00-8\xc9\x03\xd7\xbc\x0f\xc9\xc2\x1d36'
You can do like this:
import binascii
# binascii.b2a_uu takes at most 45 bytes at once
# this why i'm splitting the data into chunks of 45 bytes at most
b = [a[k:k+45] for k in range(0, len(a), 45)]
# You can also use:
# binascii.b2a_uu(k).decode('UTF8')
final = "".join(binascii.b2a_uu(k).decode() for k in b)
print(final)
Output:
M^;ZTV79E<F%C:P !=]N#B^;ZTV6%L97)T *# ;
M^:KJ8 $ /___W\ ____?_[__W\!____?P #___]_
M /___W\ +U521T5.5#H#06QE<G0#:V5Y(&-O;7!R;VUI<V5D+"!U<&=R861E
M(')E<75I<F5D $8P1 (#93_KUD$/1P]KKA'*T9Q(03OLL:PL%_D(_0_5.]PZ
MO5("(&T.G);^B-2#\![9WMKBMOG##:E,K0_LJN9NSVB;]QM0^;ZTV7!I;F<
7 # M.,D#U[P/R<(=,S8

I found a python rpc library for bitcoin called
python-bitcoinrpc
It solved my problem:
rpc_connection = AuthServiceProxy('http://%s:%s#127.0.0.1:8332'%('rpc_username','rpc_password'))
block_template = rpc_connection.getblocktemplate()
print(block_template)
Friendly API and works fantastically.

Python: error loading JSON object

I'm trying to load the following JSON string in python:
{
"Motivo_da_Venda_Perdida":"",
"Data_Visita":"2015-03-17 08:09:55",
"Cliente":{
"Distribuidor1_Modelo":"",
"RG":"",
"Distribuidor1_Marca":"Selecione",
"PlataformaMilho1_Quantidade":"",
"Telefone_Fazenda":"",
"Pulverizador1_Quantidade":"",
"Endereco_Fazenda":"",
"Nome_Fazenda":"",
"Area_Total_Fazenda":"",
"PlataformaMilho1_Marca":"Selecione",
"Trator1_Modelo":"",
"Tipo_Cultura3":"Selecione",
"Tipo_Cultura4":"Selecione",
"Cultura2_Hectares":"",
"Colheitadeira1_Quantidade":"",
"Tipo_Cultura1":"Soja",
"Tipo_Cultura2":"Selecione",
"Plantadeira1_Marca":"Stara",
"Autopropelido1_Modelo":"",
"Email_Fazenda":"",
"Autopropelido1_Marca":"Stara",
"Distribuidor1_Quantidade":"",
"PlataformaMilho1_Modelo":"",
"Trator1_Marca":"Jonh deere",
"Email":"",
"CPF":"46621644000",
"Endereco_Rua":"PAQUINHAS, S/N",
"Caixa_Postal_Fazenda":"",
"Cidade_Fazenda":"",
"Plantadeira1_Quantidade":"",
"Colheitadeira1_Marca":"New holland",
"Data_Nascimento":"2015-02-20",
"Cultura4_Hectares":"",
"Nome_Cliente":"MILTON CASTIONE",
"Cep_Fazenda":"",
"Telefone":"5491290687",
"Cultura3_Hectares":"",
"Trator1_Quantidade":"",
"Cultura1_Hectares":"",
"Autopropelido1_Quantidade":"",
"Pulverizador1_Modelo":"",
"Caixa_Postal":"",
"Estado":"RS",
"Endereco_Numero":"",
"Cidade":"COLORADO",
"Colheitadeira1_Modelo":"",
"Pulverizador1_Marca":"Selecione",
"CEP":"99460000",
"Inscricao_Estadual":"0",
"Plantadeira1_Modelo":"",
"Estado_Fazenda":"RS",
"Bairro":""
},
"Quilometragem":"00",
"Modelo_Pretendido":"Selecione",
"Quantidade_Prevista_Aquisicao":"",
"Id_Revenda":"1",
"Contato":"05491290687",
"Pendencia_Para_Proxima_Visita":"",
"Data_Proxima_Visita":"2015-04-17 08:09:55",
"Valor_de_Venda":"",
"Maquina_Usada":"0",
"Id_Vendedor":"2",
"Propensao_Compra":"Propensao_Compra_Frio",
"Comentarios":"despertar compra",
"Sistema_Compra":"Sistema_Compra_Finame",
"Outro_Produto":"",
"Data_Prevista_Aquisicao":"2015-04-17 08:09:55",
"Objetivo_Visita":"Despertar_Interesse",
"Tipo_Contato":"Telefonico"}
however I get the following error when I try to load it
File "python_file.py", line 107, in busca_proxima_mensagem
Visita = json.loads(corpo)
File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 369, in decode
raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 1 column 2 - line 6 column 84 (char 1 - 1020)
but this JSON seems to be valid according to this site: http://jsonformatter.curiousconcept.com/ What am I doing wrong? Why can't I load this string as a JSON object?
I'm trying to load the string from AWS SQS like this:
import json
...
result = fila.get_messages(1, 30, 'SentTimestamp')
for message in result:
corpo = message.get_body()
Visita = json.loads(corpo)
OK, so I figured out what is causing me problems: There is a slash as a value of a key
"Endereco_Rua":"PAQUINHAS, S/N",
However I'm telling python to filter that out (code below), but it's not working. How can I remove that? Can do it on the origin that created the data, as I don't have access to the interface the user uses to fill in.
result = fila.get_messages(1, 30, 'SentTimestamp')
for message in result:
corpo = message.get_body()
corpo = corpo.replace("/", "") #Filtering slashes
Visita = json.loads(corpo)

Found a solution! Beside the slash caracter, sometimes this error also happened with no visible cause. Ended up solving this by adding the following lines in my python code:
1) At the start of my code, along with other python imports
from boto.sqs.message import RawMessage
2) Changing my SQS queue to use/fetch raw data:
fila = sqs_conn.get_queue(constantes.fila_SQS)
fila.set_message_class(RawMessage)
Hope this helps anyone who is having the same issue.