Develop Reference

Some conversion issues between the byte and strings

Here is the I am trying:
import struct
#binary_data = open("your_binary_file.bin","rb").read()
#your binary data would show up as a big string like this one when you .read()
binary_data = '\x44\x69\x62\x65\x6e\x7a\x6f\x79\x6c\x70\x65\x72\x6f\x78\x69\x64\x20\x31\
x32\x30\x20\x43\x20\x30\x33\x2e\x30\x35\x2e\x31\x39\x39\x34\x20\x31\x34\x3a\x32\
x34\x3a\x33\x30'
def search(text):
#convert the text to binary first
s = ""
for c in text:
s+=struct.pack("b", ord(c))
results = binary_data.find(s)
if results == -1:
print ("no results found")
else:
print ("the string [%s] is found at position %s in the binary data"%(text, results))
search("Dibenzoylperoxid")
search("03.05.1994")
And this is the error I am getting:
Traceback (most recent call last):
File "dec_new.py", line 22, in <module>
search("Dibenzoylperoxid")
File "dec_new.py", line 14, in search
s+=struct.pack("b", ord(c))
TypeError: Can't convert 'bytes' object to str implicitly
Kindly, let me know what I can do to make it functional properly.
I am using Python 3.5.0.

s = ""
for c in text:
s+=struct.pack("b", ord(c))
This won't work because s is a string, and struct.pack returns a bytes, and you can't add a string and a bytes.
One possible solution is to make s a bytes.
s = b""
... But it seems like a lot of work to convert a string to a bytes this way. Why not just use encode()?
def search(text):
#convert the text to binary first
s = text.encode()
results = binary_data.find(s)
#etc
Also, "your binary data would show up as a big string like this one when you .read()" is not, strictly speaking, true. The binary data won't show up as a big string, because it is a bytes, not a string. If you want to create a bytes literal that resembles what might be returned by open("your_binary_file.bin","rb").read(), use the bytes literal syntax binary_data = b'\x44\x69<...etc...>\x33\x30'

How do you use a variable argument on base64.b64encode but when i am not using the prompt window?

This question is similar to this one here but if I put this into this code like so:
import base64
theone = input('Enter your plaintext: ')
encoded = str(base64.b64encode(theone))
encoded = base64.b64encode(encoded.encode('ascii'))
encoded = encoded[2:]
o = len(encoded)
o = o-1
encoded = encoded[:o]
print(encoded)
it raises this problem:
line 58, in b64encode
encoded = binascii.b2a_base64(s, newline=False)
TypeError: a bytes-like object is required, not 'str'
And then if I remove this line of code:
encoded = base64.b64encode(encoded.encode('ascii'))
then it raises the same error. I'm not sure what to do from here and I would be grateful for any help.

You seem to be having problems with bytes and strings. The value returned by input is a string (str), but base64.b64encode expects bytes (bytes).
If you print a bytes instance you see something like
b'spam'
To remove the leading 'b' you need to decode back to a str.
To make your code work, pass bytes to base64.b64encode, and decode the result to print it.
>>> theone = input('Enter your plaintext: ')
Enter your plaintext: Hello World!
>>> encoded = base64.b64encode(theone.encode())
>>> encoded
b'SGVsbG8gV29ybGQh'
>>> print(encoded.decode())
SGVsbG8gV29ybGQh

I can't figure out how to fix TypeError: can't concat str to bytes mean

I wrote a function to see encrypt a given text file. The below code is a small portion of the function.
#pad it before encrypting it
elif len(chunk) % 16 != 0:
chunk += ' ' * (16 - len(chunk) % 16)
#write encrypted data into output file
out_file.write(encryptor.encrypt(chunk))
Whenever I try to use the function I get an error that points to the last line saying
"TyprError: can't concat str to bytes". I'm not sure what I need to do in order to fix this error. I've tried a few things and they end leading me into more similar errors. Any guidance would be greatly appreciated.
The encryptor is below.
encryptor = PKCS1_OAEP.new(pub_key)

Your encryption method encryptor.encrypt() very likely accepts bytes as argument, not str. It also returns bytes very likely. So I suggest you to use the encode/decode methods as follows (example of utf-8 coding):
out_file.write(encryptor.encrypt(chunk.encode('utf-8')).decode('utf-8'))

You are trying to mix and match incompatible data types. Here is an example that will throw the same error:
str1 = bytes(123)
str2 = str(123)
str1 + str2
Go through your example and find where you are trying to concat a bytes value to a str value and make them match types.

Convertion between ISO-8859-2 and UTF-8 in Python

I'm wondering how can I convert ISO-8859-2 (latin-2) characters (I mean integer or hex values that represents ISO-8859-2 encoded characters) to UTF-8 characters.
What I need to do with my project in python:
Receive hex values from serial port, which are characters encoded in ISO-8859-2.
Decode them, this is - get "standard" python unicode strings from them.
Prepare and write xml file.
Using Python 3.4.3
txt_str = "ąęłóźć"
txt_str.decode('ISO-8859-2')
Traceback (most recent call last): File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute 'decode'
The main problem is still to prepare valid input for the "decode" method (it works in python 2.7.10, and thats the one I'm using in this project). How to prepare valid string from decimal value, which are Latin-2 code numbers?
Note that it would be uber complicated to receive utf-8 characters from serial port, thanks to devices I'm using and communication protocol limitations.
Sample data, on request:
68632057
62206A75
7A647261
B364206F
20616775
777A616E
616A2061
6A65696B
617A20B6
697A7970
6A65B361
70697020
77F36469
62202C79
6E647572
75206A65
7963696C
72656D75
6A616E20
73726F67
206A657A
65647572
77207972
73772065
00000069
This is some sample data. ISO-8859-2 pushed into uint32, 4 chars per int.
bit of code that manages unboxing:
l = l[7:].replace(",", "").replace(".", "").replace("\n","").replace("\r","") # crop string from uart, only data left
vl = [l[0:2], l[2:4], l[4:6], l[6:8]] # list of bytes
vl = vl[::-1] # reverse them - now in actual order
To get integer value out of hex string I can simply use:
int_vals = [int(hs, 16) for hs in vl]

Your example doesn't work because you've tried to use a str to hold bytes. In Python 3 you must use byte strings.
In reality, if you're using PySerial then you'll be reading byte strings anyway, which you can convert as required:
with serial.Serial('/dev/ttyS1', 19200, timeout=1) as ser:
s = ser.read(10)
# Py3: s == bytes
# Py2.x: s == str
my_unicode_string = s.decode('iso-8859-2')
If your iso-8895-2 data is actually then encoded to ASCII hex representation of the bytes, then you have to apply an extra layer of encoding:
with serial.Serial('/dev/ttyS1', 19200, timeout=1) as ser:
hex_repr = ser.read(10)
# Py3: hex_repr == bytes
# Py2.x: hex_repr == str
# Decodes hex representation to bytes
# Eg. b"A3" = b'\xa3'
hex_decoded = codecs.decode(hex_repr, "hex")
my_unicode_string = hex_decoded.decode('iso-8859-2')
Now you can pass my_unicode_string to your favourite XML library.

Interesting sample data. Ideally your sample data should be a direct print of the raw data received from PySerial. If you actually are receiving the raw bytes as 8-digit hexadecimal values, then:
#!python3
from binascii import unhexlify
data = b''.join(unhexlify(x)[::-1] for x in b'''\
68632057
62206A75
7A647261
B364206F
20616775
777A616E
616A2061
6A65696B
617A20B6
697A7970
6A65B361
70697020
77F36469
62202C79
6E647572
75206A65
7963696C
72656D75
6A616E20
73726F67
206A657A
65647572
77207972
73772065
00000069'''.splitlines())
print(data.decode('iso-8859-2'))
Output:
W chuj bardzo długa nazwa jakiejś zapyziałej pipidówy, brudnej ulicyumer najgorszej rudery we wsi
Google Translate of Polish to English:
The dick very long name some zapyziałej Small Town , dirty ulicyumer worst hovel in the village

This topic is closed. Working code, that handles what need to be done:
x=177
x.to_bytes(1, byteorder='big').decode("ISO-8859-2")

How do I 'declare' an empty bytes variable?

How do I initialize ('declare') an empty bytes variable in Python 3?
I am trying to receive chunks of bytes, and later change that to a
utf-8 string.
However, I'm not sure how to initialize the initial variable that will
hold the entire series of bytes. This variable is called msg.
I can't initialize it as None, because you can't add a bytes and a
NoneType. I can't initialize it as a unicode string, because then
I will be trying to add bytes to a string.
Also, as the receiving program evolves it might get me in to a mess
with series of bytes that contain only parts of characters.
I can't do without a msg initialization, because then msg would be
referenced before assignment.
The following is the code in question
def handleClient(conn, addr):
print('Connection from:', addr)
msg = ?
while 1:
chunk = conn.recv(1024)
if not chunk:
break
msg = msg + chunk
msg = str(msg, 'UTF-8')
conn.close()
print('Received:', unpack(msg))

Just use an empty byte string, b''.
However, concatenating to a string repeatedly involves copying the string many times. A bytearray, which is mutable, will likely be faster:
msg = bytearray() # New empty byte array
# Append data to the array
msg.extend(b"blah")
msg.extend(b"foo")
To decode the byte array to a string, use msg.decode(encoding='utf-8').

bytes() works for me;
>>> bytes() == b''
True

Use msg = bytes('', encoding = 'your encoding here').
Encase you want to go with the default encoding, simply use msg = b'', but this will garbage the whole buffer if its not in the same encoding

As per documentation:
Blockquote
socket.recv(bufsize[, flags])
Receive data from the socket. The return value is a string representing the data received.
Blockquote
So, I think msg="" should work just fine:
>>> msg = ""
>>> msg
''
>>> len(msg)
0
>>>

To allocate bytes of some arbitrary length do
bytes(bytearray(100))