I'm trying to convert 64-bit symbolic string to 32-bit. My advisor said, what that 64-bit symbolic string is a Hex string and I should only cut all the bits after 8th left bit. I decided to do it in Python by mask:
that_string & 0xFFFFFFFF
But it's impossible to do without converting string to int():
int('0x'+that_string, 16) & 0xFFFFFFFF
But then 'that_string' becomes a truly Integer and I can't convert her back to string. It's not possible to make chr(int('0x'+that_string, 16) & 0xFFFFFFFF), it causes in problem:
ValueError: chr() arg not in range(256)
Also it's not possible to do decode() because of another Error:
bash ~$: (that_string).decode('hex')
... File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/encodings/hex_codec.py", line 42, in hex_decode
output = binascii.a2b_hex(input)
TypeError: Odd-length string
I tried to search for another opportunity, but didn't get any suitable solution.
And, yes, I tried base64 - library, but my adviser said, what that's wrong idea and what that's not a solution.
I'm much embarrassed. Could You help me, please? May be, there are other ways exist, aren't they?
Take the rightmost 8 digits with a slice:
str[-8:]
where str is a string containing the hexadecimal representation of your input data.
If str has fewer than 8 digits, then they will all be returned by this expression, which is what you want.
For example:
>>> str = '0123456789abcdef'
>>> str[-8:]
'89abcdef'
>>> str = 'abcd'
>>> str[-8:]
'abcd'
Related
I am using python 3.8.5, and trying to convert from an integer in the range (0,65535) to a pair of bytes. I am currently using the following code:
from struct import pack
input_integer = 2111
bytes_val = voltage.to_bytes(2,'little')
output_data = struct.pack('bb',bytes_val[1],bytes_val[0])
print(output_data)
This produces the following output:
b'\x08?'
This \x08 is 8 in hex, the most significant byte, and ? is 63 in ascii. So together, the numbers add up to 2111 (8*256+63=2111). What I can't figure out is why the least significant byte is coming out in ascii instead of hex? It's very strange to me that it's in a different format than the MSB right next to it. I want it in hex for the output data, and am trying to figure out how to achieve that.
I have also tried modifying the format string in the last line to the following:
output_data = struct.pack('cc',bytes_val[1],bytes_val[0])
which produces the following error:
struct.error: char format requires a bytes object of length 1
I checked the types at each step, and it looks like bytes_val is a bytearray of length 2, but when I take one of the individual elements, say bytes_val[1], it is an integer rather than a byte array.
Any ideas?
All your observations can be verified from the docs for the bytes class:
While bytes literals and representations are based on ASCII text, bytes objects actually behave like immutable sequences of integers
In Python strings any letters and punctuation are represented by themselves in ASCII, while any control codes by their hexadecimal value (0-31, 127). You can see this by printing ''.join(map(chr, range(128))). Bytes literals follow the same convention, except that individual byte elements are integer, i.e., output_data[0].
If you want to represent everything as hex
>>> output_data.hex()
'083f'
>>> bytes.fromhex('083f') # to recover
b'\x08?'
As of version 3.8 bytes.hex() now supports optional sep and bytes_per_sep parameters to insert separators between bytes in the hex output.
>>> b'abcdef'.hex(' ', 2)
'6162 6364 6566'
I am new to Python & I am trying to learn how to XOR hex encoded ciphertexts against one another & then derive the ASCII value of this.
I have tried some of the functions as outlined in previous posts on this subject - such as bytearray.fromhex, binascii.unhexlify, decode("hex") and they have all generated different errors (obviously due to my lack of understanding). Some of these errors were due to my python version (python 3).
Let me give a simple example, say I have a hex encoded string ciphertext_1 ("4A17") and a hex endoded string ciphertext_2. I want to XOR these two strings and derive their ASCII value. The closest that I have come to a solution is with the following code:
result=hex(int(ciphertext_1, 16) ^ int(ciphertext_2, 16))
print(result)
This prints me a result of: 0xd07
(This is a hex string is my understanding??)
I then try to convert this to its ASCII value. At the moment, I am trying:
binascii.unhexliy(result)
However this gives me an error: "binascii.Error: Odd-length string"
I have tried the different functions as outlined above, as well as trying to solve this specific error (strip function gives another error) - however I have been unsuccessful. I realise my knowledge and understanding of the subject are lacking, so i am hoping someone might be able to advise me?
Full example:
#!/usr/bin/env python
import binascii
ciphertext_1="4A17"
ciphertext_2="4710"
result=hex(int(ciphertext_1, 16) ^ int(ciphertext_2, 16))
print(result)
print(binascii.unhexliy(result))
from binascii import unhexlify
ciphertext_1 = "4A17"
ciphertext_2 = "4710"
xored = (int(ciphertext_1, 16) ^ int(ciphertext_2, 16))
# We format this integer: hex, no leading 0x, uppercase
string = format(xored, 'X')
# We pad it with an initial 0 if the length of the string is odd
if len(string) % 2:
string = '0' + string
# unexlify returns a bytes object, we decode it to obtain a string
print(unhexlify(string).decode())
#
# Not much appears, just a CR followed by a BELL
Or, if you prefer the repr of the string:
print(repr(unhexlify(string).decode()))
# '\r\x07'
When doing byte-wise operations like XOR, it's often easier to work with bytes objects (since the individual bytes are treated as integers). From this question, then, we get:
ciphertext_1 = bytes.fromhex("4A17")
ciphertext_2 = bytes.fromhex("4710")
XORing the bytes can then be accomplished as in this question, with a comprehension. Then you can convert that to a string:
result = [c1 ^ c2 for (c1, c2) in zip(ciphertext_1, ciphertext_2)]
result = ''.join(chr(c) for c in result)
I would probably take a slightly different angle and create a bytes object instead of a list, which can be decoded into your string:
result = bytes(b1 ^ b2 for (b1, b2) in zip(ciphertext_1, ciphertext_2)).decode()
So, I've been banging my head against the wall for too long on what seems like it should be an easy data conversion. I am writing in python and passing to another module a hex value that is converted with a wrapper to c type uint_64t. the problem is I am getting this hex value via the python library argparse. When it takes in this value, for example lets use the value 0x3f, it saves it as a string. If I try to cast this as an int it throws the error:
"ValueError: invalid literal for int() with base 10: '0x3f'"
If I create a variable hex = 0x3f however, when I print it out, it gives me the appropriate integer value. (which is great since I'm creating a uint) I am just confused how to make the conversion from string to int if a cast doesn't work. I have seen plenty of examples on turning this string into a hex value by letter (in other words take each individual character of the ascii string '0x3f' and give it a hex value) but I haven't been able to find an example of the situation I am looking for. Apologies if I'm bringing up something that has been answered time and again.
Try specifying that the int is in base 16:
>>> int("0x3f", 16)
63
You could also use ast.literal_eval, which should be able to read any string that could be used as an integer literal. You don't even have to specify a base for this one, as long as the 0x prefix is present.
>>> import ast
>>> #hexadecimal
>>> ast.literal_eval("0x3f")
63
>>> #binary
>>> ast.literal_eval("0b01010")
10
>>> #octal
>>> ast.literal_eval("0712")
458
>>> #decimal
>>> ast.literal_eval("42")
42
int takes an optional second argument, which is the numerical base to use for conversion. The default is 10 (decimal conversion) but you can change it to 16 (for hex) or 0 (for automatic conversion based on prefix):
>>> int('0b1010', 0)
10
>>> int('0x3f', 0)
63
>>> int('0o777', 0)
511
>>> int('1234', 0)
1234
I feel like a complete tool for posting this, it is so basic and I cant believe I have wasted the last two days on this problem. I've tried all the solutions I can find on this (seriously, I will show you my internet history) but to no avail. Here is the problem:
I am parsing a serial string in from a uC. It is 52 bytes long and contains a lot of different variables of data. The data in encoded in packed binary coded decimal.
Ex: .....blah.....0x01 0x5E .....blah
015E hex gives 350 decimal. This is the value I want. I am reading in the serial string just fine, I used binascii.hexifiy to print the bytes to ensure it is corrent. I use
data = ser.read()
and placed the data in an array if an newline is not received. I have tried making the array a bytearray, list, anything that I could find, but none work.
I want to send the required two byte section to a defined method.
def makeValue(highbyte, lowbyte)
When I try to use unpack, join, pack, bit manipulation, string concentation, I keep getting the same problem.
Because 0x01 and 0x5E are not valid int numbers (start of heading and ^ in ASCII), it wont work. It wont even let me join the numbers first because it's not a valid int.
using hex(): hex argument can't be converted to hex.
Joining the strings: invalid literal for int() with base 16: '\x01^'
using int: invalid literal for int() with base 10: '\x01^'
Packing a struct: struct.error: cannot convert argument to integer
Seriously, am I missing something really basic here? All the examples I can find make use of all the functions above perfectly but they specificy the hex numbers '0x1234', or the numbers they are converting are actual ASCII numbers. Please help.
EDIT
I got it, ch3ka set me on the right track, thanks a million!
I don't know why it wouldn't work before but I hex'ed both values
one = binascii.hexlify(line[7])
two = binascii.hexlify(line[8])
makeValue(one, two)`
and then used the char makeValues ch3ka defined:
def makeValue(highbyte, lowbyte)
print int(highbyte, 16)*256 + int(lowbyte, 16)
Thanks again!!!
you are interpreting the values as chars. Feeding chars to int() won't work, you have to feed the values as strings, like so: int("0x5E", 16). What you are attempting is in fact int(chr(int("0x5E", 16)),16), which is int("^",16) and will of course not work.
Do you expect these results?
makevalue('0x01', '0x5E') -> 350 0x15e 0b101011110
makevalue('0xFF', '0x00') -> 65280 0xff00 0b1111111100000000
makevalue('0x01', '0xFF') -> 511 0x1ff 0b111111111
makevalue('0xFF', '0xFF') -> 65535 0xffff 0b1111111111111111
If so, you can use this:
def makeValue(highbyte, lowbyte):
return int(highbyte, 16)*256 + int(lowbyte, 16)
or the IMO more ugly and errorprone:
def makeValue(highbyte, lowbyte):
return int(highbyte+lowbyte[2:], 16) # strips leading "0x" from lowbyte be4 concat
I have a long Hex string that represents a series of values of different types. I need to convert this Hex String into bytes or bytearray so that I can extract each value from the raw data. How can I do this?
For example, the string "ab" should convert to the bytes b"\xab" or equivalent byte array. Longer example:
>>> # what to use in place of `convert` here?
>>> convert("8e71c61de6a2321336184f813379ec6bf4a3fb79e63cd12b")
b'\x8eq\xc6\x1d\xe6\xa22\x136\x18O\x813y\xeck\xf4\xa3\xfby\xe6<\xd1+'
Suppose your hex string is something like
>>> hex_string = "deadbeef"
Convert it to a bytearray (Python 3 and 2.7):
>>> bytearray.fromhex(hex_string)
bytearray(b'\xde\xad\xbe\xef')
Convert it to a bytes object (Python 3):
>>> bytes.fromhex(hex_string)
b'\xde\xad\xbe\xef'
Note that bytes is an immutable version of bytearray.
Convert it to a string (Python ≤ 2.7):
>>> hex_data = hex_string.decode("hex")
>>> hex_data
"\xde\xad\xbe\xef"
There is a built-in function in bytearray that does what you intend.
bytearray.fromhex("de ad be ef 00")
It returns a bytearray and it reads hex strings with or without space separator.
provided I understood correctly, you should look for binascii.unhexlify
import binascii
a='45222e'
s=binascii.unhexlify(a)
b=[ord(x) for x in s]
Assuming you have a byte string like so
"\x12\x45\x00\xAB"
and you know the amount of bytes and their type you can also use this approach
import struct
bytes = '\x12\x45\x00\xAB'
val = struct.unpack('<BBH', bytes)
#val = (18, 69, 43776)
As I specified little endian (using the '<' char) at the start of the format string the function returned the decimal equivalent.
0x12 = 18
0x45 = 69
0xAB00 = 43776
B is equal to one byte (8 bit) unsigned
H is equal to two bytes (16 bit) unsigned
More available characters and byte sizes can be found here
The advantages are..
You can specify more than one byte and the endian of the values
Disadvantages..
You really need to know the type and length of data your dealing with
You can use the Codecs module in the Python Standard Library, i.e.
import codecs
codecs.decode(hexstring, 'hex_codec')
You should be able to build a string holding the binary data using something like:
data = "fef0babe"
bits = ""
for x in xrange(0, len(data), 2)
bits += chr(int(data[x:x+2], 16))
This is probably not the fastest way (many string appends), but quite simple using only core Python.
A good one liner is:
byte_list = map(ord, hex_string)
This will iterate over each char in the string and run it through the ord() function. Only tested on python 2.6, not too sure about 3.0+.
-Josh