Develop Reference

Finding a Python 2.7 -> 3.8 struct.Pack("H") into Strings for joining into Lists?

This is.. a long one. So I apologize for any inconsistency regarding code and problems. I'll be sure to try and add as much of the source code as I can to make sure the issue is as clear as possible.
This project at work is an attempt at converting Python 2 to 3, and thus far has been mildly straightforward. My coworker and I have reached a point though where no amount of googling or searching has given a straight answer, so here we are.
Alright, Starting offwith...
Python 2 code:
listBytes[102:104]=struct.pack('H',rotation_deg*100) # rotational position in degrees
listBytes[202:204]=struct.pack('H',rotation_deg*100) # rotational position in degrees
listBytes[302:304]=struct.pack('H',rotation_deg*100) # rotational position in degrees
# this continues on for a while in the same fashion
Where rotation_deg is a float between 0.00 and 359.99 (but for testing is almost always changing between 150-250)
For the purpose of testing, we're going to make rotation_deg be 150.00 all the time.
a = listBytes[102:104]=struct.pack('H',150.00*100)
print a
print type(a)
The print out of the following is:
�:
<type 'str'>
From what I understand, in the Python2 version of struct.pack, it is packing the floats as shorts, which then are "added" to the list as a short. Python 2 sees it as a string, and adds no encoding to the string (will get to that later for python 3). All simple and good, then a few more bits and bobs of dropping more stuff into the list and we get to:
return ''.join(listBytes)
Which, is being sent back to a simple variable:
bytes=self.UpdatePacket(bytes,statusIndex,rotation_deg,StatusIdList,StatusValueList, Stat,offsetUTC)
To then be sent along as a string through
sock.sendto(bytes , (host, port) )
This all comes together to look like this:
A string with a bunch of bytes (I think)
This is the working version, in which we are sending the bytes along the socket, data is being retrieved, and everyone is happy. If I missed anything, please let me know, otherwise, lets move to...
Python 3
This is where the Fun Begins
There are a few changes that are required between Python 2 and 3 right off the bat.
struct.pack('H',rotation_deg*100) requires an INT type to be packed, meaning all instances of packing had to be given int(rotatin_deg*100) as to not error the program.
sock.sendto(bytes, (host, port)) did not work anymore as the socket needed a bytes object to send something. No more strings that look like bytes, they had to be properly encoded to send properly. So now this becomes sock.sendto(bytes.encode(), (host, port)) to properly encode the "bytes" string.
As more of a background, the length of listBytes should always be 1206. Anymore and our socket won't work properly, and the issue is that no matter what we try with this python 3 code, the .join seems to be sending a LOT more than just byte objects, often quintupling the length of listBytes and breaking the socket.sendto .
listBytes[102:104] = struct.pack('H', int(rotation_deg * 100)) # rotational position in degrees
listBytes[202:204] = struct.pack('H', int(rotation_deg * 100)) # rotational position in degrees
listBytes[302:304] = struct.pack('H', int(rotation_deg * 100)) # rotational position in degrees
# continues on in this fashion again
return ''.join(str(listBytes))
returns to:
bytes = self.UpdatePacket(bytes, statusIndex, rotation_deg, StatusIdList, StatusValueList, Stat, offsetUTC)
sock.sendto(bytes.encode(), (host, port))
Here's where things start getting weird
a = struct.pack('H', int(150.00 * 100))
returns:
b'\x98:', with it's type being <class 'bytes'>, which is fine and the value we want, except we specifically need to store this variable into the list as maybe a string... to encode it later to send as a byte object for the socket.
You're starting to see the problem, yes?
The thing is, we've tried just about every technique to convert the two bytes that struct.pack returns into a string of some kind, and we've been able to convert it over, but then we run into the issue of the .join being evil.
Remember when I was talking about listBytes had to remain a size of 1206 or else it would break? For some reason, if we .join literally anything other than the two bytes as a string, we think python is trying to add a bunch of other stuff that we don't need.
So for now, we're focusing on trying to match the python 2 equivalent to python 3.
Here's what we've tried
binascii.hexlify(struct.pack('H', int(150.00 * 100))).decode() returns '983a'
str(struct.pack('H', int(150.00 * 100.00)).decode()) returns an error, 'utf-8' codec can't decode byte 0x98 in position 0: invalid start byte
str(struct.pack('H', int(150.00 * 100.00)).decode("utf-16")) returns '㪘'. Can't even begin to understand that.
return b''.join(listBytes) returns an error because there are int's at the start of the list.
return ''.join(str(listBytes)).encode('utf-8') still is adding a bunch of nonsense.
Now we get to the .join, and the first loop around it seems.. fine? It has 1206 as listBytes length before .joining, but on the second loop around, it creates a massive influx of junk, making the list 5503 in length. Third go around it becomes 27487, and finally on the last go around, it becomes to large for the socket to handle and I get slapped with [WinError 10040] A message sent on a datagram socket was larger than the internal message buffer or some other network limit, or the buffer used to receive a datagram into was smaller than the datagram itself
Phew, if you made it this far, thank you. Any help at all would be extremely appreciated. If you have questions or I'm missing something, let me know.
Thanks!

You’re just trying too hard. It may be helpful to think of bytes and unicode rather than the ambiguous str type (which is the former in Python 2 and the latter in Python 3). struct always produces bytes, and socket needs to send bytes, regardless of the language version. Put differently, everything outside your process is bytes, although some represent characters in some encoding chosen by the rest of the system. Characters are a data type used to manipulate text within your program, just like floats are a data type used to manipulate “real numbers”. You have no characters here, so you definitely don’t need encode.
So just accumulate your bytes objects and then join them with b''.join (if you can’t just directly use the buffer into which your slice assignments seem to be writing) to keep them as bytes.

pyserial Python2/3 string headache

I'm using pyserial to communicate with some sensors which use the Modbus protocol. In Python 2.7, this works perfectly:
import serial
c = serial.Serial('port/address') # connect to sensor
msg = "\xFE\x44\x00\x08\x02\x9F\x25" # 7 hex bytes(?)
c.write(msg) # send signal
r = c.read(7) # read 7 hex bytes (?).
In Python 3, this does not work. I know it's something to do with differences in how Python 2/3 handle binary vs. unicode strings. I've found numerous other threads suggesting the solution should be to simply prepend a b on my message (msg=b""\xFE\x44\x00\x08\x02\x9F\x25") to specify it as a binary string but this does not work for my case.
Any insights? What should I be sending in Python 3 so the sensor recieves the same signal? I'm at my wit's end...
I should add that I'm totally new to serial connections (well... 1 week old), and (despite reading around quite a bit) I struggle with understanding different character/string formats... Hence question marks in comments above. Please pitch answers appropriately :).
Thanks in advance!

write expects argument to be str, not bytes, so passing b"\xFE\x44\x00\x08\x02\x9F\x25" directly to it won't work. You need to convert bytes to str first: c.write(b"\xFE\x44\x00\x08\x02\x9F\x25".decode()) should work.

Solution
It turned out that specifying the input as a byte string (msg=b""\xFE\x44\x00\x08\x02\x9F\x25") did work. Initial error was from a typo in the msg string...
Secondary errors arose from how the outputs were handled - in Python 2 ord() had to be applied to the indexed output to return integers, in Python 3 integers can be extracted directly from the output by indexing (i.e. no ord() necessary).
Hope this might help someone in the future...

Converting an IP Address string to exactly 4 bytes in Python

So this is very simple, but I'm having trouble getting this to work. I want to, for example, if the incoming IP address string is '168.108.114.22', convert this to a bytes object like:
\xA8\x6C\x72\x16
Basically each part of the IP address is converted to it's hexadecimal equivalent.
I've tried so many ways but couldn't get what I want. String manipulation, using socket.inet_aton, packing, etc. I want to be able to send these bytes over a socket and then receive and parse them at the other end, but I am having trouble just getting my bytes object created and looking like that.

Python's inet_aton function should do what you need, it does return a string containing exactly 4 bytes:
import socket
print socket.inet_aton('168.108.114.22')
print socket.inet_aton('65.66.67.68')
These would display:
¨lr
ABCD
And to convert the four characters back again using inet_ntoa:
print socket.inet_ntoa('\xA8\x6C\x72\x16')
print socket.inet_ntoa('ABCD')
Giving:
65.66.67.68

this
ip='168.108.114.22'
b_out = bytes(map(int,ip.split('.')))
print(b_out)
on python 3 produces
b'\xa8lr\x16'
which should be what you are looking for, if I understand correctly.
Note: there are more specific and optimized utility functions to manipulate IP addresses

zeromq: unicode not allowed, use send_unicode

I am getting error on zeromq python while sending strings through ROUTER SOCKET. String type messages are receveid successfully but some times, a unicode message throws exception "Type Error: unicode not allowed. use send_unicode". Although I have been trying to use msg.encode('utf-8'). BUt I cant figure out a way to get over with it.
I am on python 2.7.3. I am not using pyzmq (import zmq only). Looking forward to your suggesitons :) Thanks
if backend in sockets:
request=backend.recv_multipart()
#print ("Backend Thread is ready")
worker_id,client_id = request[:2]
if client_id != b"READY" and len(request) > 3:
#print (len(request))
empty2,reply = request[2:]
router_socket.send_multipart([client_id, reply.encode('utf-8')])

The problem was resolved only thing was that I needed to convert the unicode strings back to ascii by using string.encode('ascii')

I got the same error. My erroneous code was:
socket.send("Server message to client3")
You must convert the message to bytes to solve it. To do so, just add b like this:
socket.send(b"Server message to client3")
Is it better to convert strings to byte, then bytes to strings when data sent through network, and why?

So, because PyZMQ is actually a good library they have some docs.
https://pyzmq.readthedocs.io/en/latest/unicode.html
What it says is that the str object changed it's nature over the course of history of Python evolution.
In Python 3 str is a collection of characters and in Python 2 it is a simple wrapper (with some sugar) for char* that we know from C :).
Docs explain why the people behind pyZMQ chose to make the differences explicit - performance is the answer.
To send strings in Python3 you should use the right method, which is send_string, probably the other way around for Python2 (to send unicode you should use send_unicode).
It is however recommended to stick to bytes and explicitly provide correct encoding and decoding where needed.
Also you are using pyzmq... the module name "zmq" comes from pyzmq library/package.
To confront this statement use: pip list | grep zmq (or pip list | select-string zmq for Windows).

sending raw bits to the terminal in python

As I understand it, files like /dev/urandom provide just a constant stream of bits. The terminal emulator then tries to interpret them as strings, which results in a mess of unrecognised characters.
How would I go about doing the same thing in python, send a string of ones and zeros to the terminal as "raw bits"?
edit
I may have to clarify:
Say for example the string I want to "print" is 1011100. On an ascii system, the output should be "\". If I cat /dev/urandom, it provides a constant stream of bits. Which get printed like this: "���c�g/�t]+__��-�;". That's what I want.

Stephano: the key is the incomplete answer by "#you" above - the chr function :
import random, sys
for i in xrange(500):
sys.stdout.write(chr(random.randrange(256)))

Use the chr function. I takes an input between 0 and 255 and returns a string containing the character corresponding to that value.

And from another question on StackOverflow you can get a _bin function.
def _bin(x, width):
return ''.join(str((x>>i)&1) for i in xrange(width-1,-1,-1))
Then simply put call _bin(ord(x), 8) where x is a character (string of length one)

import sys, random
while True:
sys.stdout.write(chr(random.getrandbits(8)))
sys.stdout.flush()