Develop Reference

pyserial Python2/3 string headache

I'm using pyserial to communicate with some sensors which use the Modbus protocol. In Python 2.7, this works perfectly:
import serial
c = serial.Serial('port/address') # connect to sensor
msg = "\xFE\x44\x00\x08\x02\x9F\x25" # 7 hex bytes(?)
c.write(msg) # send signal
r = c.read(7) # read 7 hex bytes (?).
In Python 3, this does not work. I know it's something to do with differences in how Python 2/3 handle binary vs. unicode strings. I've found numerous other threads suggesting the solution should be to simply prepend a b on my message (msg=b""\xFE\x44\x00\x08\x02\x9F\x25") to specify it as a binary string but this does not work for my case.
Any insights? What should I be sending in Python 3 so the sensor recieves the same signal? I'm at my wit's end...
I should add that I'm totally new to serial connections (well... 1 week old), and (despite reading around quite a bit) I struggle with understanding different character/string formats... Hence question marks in comments above. Please pitch answers appropriately :).
Thanks in advance!

write expects argument to be str, not bytes, so passing b"\xFE\x44\x00\x08\x02\x9F\x25" directly to it won't work. You need to convert bytes to str first: c.write(b"\xFE\x44\x00\x08\x02\x9F\x25".decode()) should work.

Solution
It turned out that specifying the input as a byte string (msg=b""\xFE\x44\x00\x08\x02\x9F\x25") did work. Initial error was from a typo in the msg string...
Secondary errors arose from how the outputs were handled - in Python 2 ord() had to be applied to the indexed output to return integers, in Python 3 integers can be extracted directly from the output by indexing (i.e. no ord() necessary).
Hope this might help someone in the future...

hex header of file, magic numbers, python

I have a program in Python which analyses file headers and decides which file type it is. (https://github.com/LeoGSA/Browser-Cache-Grabber)
The problem is the following:
I read first 24 bytes of a file:
with open (from_folder+"/"+i, "rb") as myfile:
header=str(myfile.read(24))
then I look for pattern in it:
if y[1] in header:
shutil.move (from_folder+"/"+i,to_folder+y[2]+i+y[3])
where y = ['/video', r'\x47\x40\x00', '/video/', '.ts']
y[1] is the pattern and = r'\x47\x40\x00'
the file has it inside, as you can see from the picture below.
the program does NOT find this pattern (r'\x47\x40\x00') in the file header.
so, I tried to print header:
You see? Python sees it as 'G#' instead of '\x47\x40'
and if i search for 'G#'+r'\x00' in header - everything is ok. It finds it.
Question: What am I doing wrong? I want to look for r'\x47\x40\x00' and find it. Not for some strange 'G#'+r'\x00'.
OR
why python sees first two numbers as 'G#' and not as '\x47\x40', though the rest of header it sees in HEX? Is there a way to fix it?

with open (from_folder+"/"+i, "rb") as myfile:
header=myfile.read(24)
header = str(binascii.hexlify(header))[2:-1]
the result I get is:
And I can work with it
4740001b0000b00d0001c100000001efff3690e23dffffff
P.S. But anyway, if anybody will explain what was the problem with 2 first bytes - I would be grateful.

In Python 3 you'll get bytes from a binary read, rather than a string.
No need to convert it to a string by str.
Print will try to convert bytes to something human readable.
If you don't want that, convert your bytes to e.g. hex representations of the integer values of the bytes by:
aBytes = b'\x00\x47\x40\x00\x13\x00\x00\xb0'
print (aBytes)
print (''.join ([hex (aByte) for aByte in aBytes]))
Output as redirected from the console:
b'\x00G#\x00\x13\x00\x00\xb0'
0x00x470x400x00x130x00x00xb0
You can't search in aBytes directly with the in operator, since aBytes isn't a string but an array of bytes.
If you want to apply a string search on '\x00\x47\x40', use:
aBytes = b'\x00\x47\x40\x00\x13\x00\x00\xb0'
print (aBytes)
print (r'\x'.join ([''] + ['%0.2x'%aByte for aByte in aBytes]))
Which will give you:
b'\x00G#\x00\x13\x00\x00\xb0'
\x00\x47\x40\x00\x13\x00\x00\xb0
So there's a number of separate issues at play here:
print tries to print something human readable, which succeeds only for the first two chars.
You can't directly search for bytearrays in bytearrays with in, so convert them to a string containing fixed length hex representations as substrings, as shown.

Hexadecimal Memory Address to Assembly

I am following a buffer overflow tutorial. I have set up my NOP block, I also set up my shell code, now I need to append the return address to the end of my string. I know my return address is :
0xbfffef40
however I need to write it in the form:
xd0\xce\xff\xff (that's just an example address to show what format I need)
I'm not sure how to carry out the conversion between the two.

You can use struct.pack like this:
import struct
struct.pack('<L', 0xbfffef40)
Check the documentation of struct.pack if you want to change the endianness.

C++ wxsocket TCP server send unsigned char array but python client get 4 more bytes in

Hi I sent an wxImage using C++ TCP socket to python TCP clientlike this:
C++ TCP Server is like this:
//this part is the TCP server m_sock send the unsigned char* ImageData
std::stringstream imageStream1;
imageStream1 << ImageData;
m_sock->Write(imageStream1.str().c_str(), imageStream1.str().length());
//Then I send a simple string "hello"
std::stringstream dataStream2;
dataStream2 << "hello";
m_sock->Write(dataStream2.str().c_str(), dataStream2.str().length());
dataStream2.clear();
So I receive the two message in python
// This is used to receive the image data with the exact same bytes of the ImageData
packet = socket.recv(ImageDataSize)
myWxImage.SetData(packet)
// This is used to receive the second "hello" message
packet = socket.recv(1000)
I can received the image successfully. But when I print the message, it shows "****hello" but not "hello". There is an additional 4 bytes string in front of it. the four "*" is something python can not print out. what is it? Can I get rid of it?

std::stringstream imageStream1;
imageStream1 << ImageData;
m_sock->Write(imageStream1.str().c_str(), imageStream1.str().length());
looks like a bad idea. I don't know the type of ImageData, but converting it to a std::stringstream is definitely not the best way to go about this. If ImageData just contains a reference to a raw image buffer, use that. Images are not strings, and obviously, things will go wrong if you get the c_str() from something that might contain \x00 in its data (because that's what normally terminates C strings).
Then:
//Then I send a simple string "hello"
std::stringstream dataStream2;
dataStream2 << "hello";
m_sock->Write(dataStream2.str().c_str(), dataStream2.str().length());
dataStream2.clear();
Um, do you understand what you're writing here? You take a perfectly valid C string , "hello", you push it onto a stringstream, just to get a C string out again?
To be honest: I think you're copy&pasting code from an example without understanding it. You should go back and understand every single line before usage.

How to remove the "****"
In general to just remove characters from a python string you can do this:
cleaned_data = str(packet).replace("*", "")
Please remember that when you receive data in packet that you are receiving bytes and not a string, so if you try to print directly python has do an implicit conversion for you. In this case, I think it would be better to do an explicit conversion and remove the beginning.
However, it doesn't solve the problem of why you are getting the characters in the first place. These 4 "*" could potentially stem from an encoding issue.
Trouble Shooting
It would be worth it to connect the python program to debugger (maybe PyCharm) so that you could see the actual values that are before "Hello", this would give you an idea of where to look so that you do not have to deal with conversions from bytes to unicode or whatever locale your console is in.
If you could get that and post that info it might help others to help you.

How do you convert string to bytes

I am using python v2.7.3 and am trying to get a conversion to work but am having some issues.
This is code that works the way I would like it to:
testString = "\x00\x13\xA2\x00\x40\xAA\x15\x47"
print 'Test String:',testString
This produces the following result
TestString: ¢#ªG
Now I load the same string as above along with some other data:
\x00\x13\xA2\x00\x40\xAA\x15\x47123456
into a SQLite3 database and then pull it from the database as such:
cur.execute('select datafield from databasetable')
rows = cur.fetchall()
if len(rows) == 0:
print 'Sorry Found Nothing!'
else:
print row[0][:32]
This however produces the following result:
\x00\x13\xA2\x00\x40\xAA\x15\x47
I can not figure out how to convert the database stored string to the bytes string, if that is what it is, as the first snippet of code does. I actually need it to load into a variable in that format so I can pass it to a function for further processing.
The following I have tried:
print "My Addy:",bytes(row[0][:32])
print '{0}'.format(row[0][:32])
...
They all produce the same results...
Please
First, Can anyone tell me what format the first results are in? I think its bytes format but am not sure.
Second, How can I convert the database stored text into
Any help and I would be eternally grateful.
Thanks in advance,
Ed

The problem is that you're not storing the value in the database properly. You want to store a sequence of bytes, but you're storing an escaped version of those bytes instead.
When entering string literals into a programming language, you can use escape codes in your source code to access non-printing characters. That's what you've done in your first example:
testString = "\x00\x13\xA2\x00\x40\xAA\x15\x47"
print 'Test String:',testString
But this is processing done by the Python interpreter as it's reading through your program and executing it.
Change the database column to a binary blob instead of a string, then go back to the code you're using to store the bytes in SQLite3, and have it store the actual bytes ('ABC', 3 bytes) instead of an escaped string ('\x41\x42\x43', 12 characters).
If you truly need to store the escaped string in SQLite3 and convert it at run-time, you might be able to use ast.literal_eval() to evaluate it as a string literal.