I ask a Measurement Device to give me some Data. At first it tells me how many bytes of data are in the storage. It is always 14. Then it gives me the data which i have to encode into hex. It is Python 2.7 can´t use newer versions. Line 6 to 10 tells the Device to give me the measured data.
Line 12 to 14 is the encoding to Hex. In other Programs it works. but when i print result(Line 14) then i get a Hex number with 13 Bytes PLUS 1 which can not be correct because it has an L et the end. I guess it is some LONG or whatever. and i dont need the last Byte. but i do think it changes the Data too, which is picked out from Line 15 and up. at first in Hex. Then it is converted into Int.
Is it possible that the L has an effect on the Data or not?
How can i fix it?
1 ap.write(b"ML\0")
rmemb = ap.read(2)
print(rmemb)
rmemb = int(rmemb)+1
5 rmem = rmemb #must be and is 14 Bytes
addmem = ("MR:%s\0" % rmem)
# addmem = ("MR:14\0")
ap.write(addmem.encode())
10 time.sleep(1)
test = ap.read(rmem)
result = hex(int(test.encode('hex'), 16))
print(result)
15 ftflash = result[12:20]
ftbg = result[20:28]
print(ftflash)
print(ftbg)
ftflash = int(ftflash, 16)
20 # print(ftflash)
ftbg = int(ftbg, 16)
# print(ftbg)
OUTPUT:
14
0x11bd5084c0b000001ce00000093L
b000001c
e0000009
Python 2 has two built-in integer types, int and long. hex returns a string representing a Python hexadecimal literal, and in Python 2, that means that longs get an L at the end, to signify that it's a long.
I am given this number 427021005928, which i am supposed to change into a base64 encoded string and then decode the base64 string to get a plain text.
This decimal value 427021005928 when converted to binary gives 110001101101100011011110111010001101000 which corresponds to 'Y2xvdGg=', which is what i want. Got the conversion from (https://cryptii.com/pipes/binary-to-base64)
And then finally i decode 'Y2xvdGg=' to get the text cloth.
My problem is i do not have any idea how to use Python to get from either the decimal or binary value to get 'Y2xvdGg='
Some help would be appreciated!
NOTE: I only have this value 427021005928 at the start. I need to get the base64 and plaintext answers.
One elegant way would be using [Python 3]: struct - Interpret bytes as packed binary data, but given the fact that Python numbers are not fixed size, some additional computation would be required (for example, the number is 5 bytes long).
Apparently, the online converter, applied the base64 encoding on the number's memory representation, which can be obtained via [Python 3]: int.to_bytes(length, byteorder, *, signed=False)(endianness is important, and in this case it's big):
For the backwards process, reversed steps are required. There are 2 alternatives:
Things being done manually (this could also be applied to the "forward" process)
Using int.from_bytes
>>> import base64
>>>
>>> number = 427021005928
>>>
>>> number_bytes = number.to_bytes((number.bit_length() + 7) // 8, byteorder="big") # Here's where the magic happens
>>> number_bytes, number_bytes.decode()
(b'cloth', 'cloth')
>>>
>>> encoded = base64.b64encode(number_bytes)
>>> encoded, encoded.decode() # Don't let yourself tricked by the variable and method names resemblance
(b'Y2xvdGg=', 'Y2xvdGg=')
>>>
>>> # Now, getting the number back
...
>>> decoded = base64.b64decode(encoded)
>>> decoded
b'cloth'
>>>
>>> final_number0 = sum((item * 256 ** idx for idx, item in enumerate(reversed(decoded))))
>>> final_number0
427021005928
>>> number == final_number0
True
>>>
>>> # OR using from_bytes
...
>>> final_number1 = int.from_bytes(decoded, byteorder="big")
>>> final_number1
427021005928
>>> final_number1 == number
True
For more details on bitwise operations, check [SO]: Output of crc32b in PHP is not equal to Python (#CristiFati's answer).
Try this (https://docs.python.org/3/library/stdtypes.html#int.to_bytes)
>>> import base64
>>> x=427021005928
>>> y=x.to_bytes(5,byteorder='big').decode('utf-8')
>>> base64.b64encode(y.encode()).decode()
'Y2xvdGg='
>>> y
'cloth'
try
number = 427021005928
encode = base64.b64encode(bytes(number))
decode = base64.b64decode(encodeNumber)
The function below converts an unsigned 64 bit integer into base64 representation, and back again. This is particularly helpful for encoding database keys.
We first encode the integer into a byte array using little endian, and automatically remove any extra leading zeros. Then convert to base64, removing the unnecessary = sign. Note the flag url_safe which makes the solution non-base64 compliant, but works better with URLs.
def int_to_chars(number, url_safe = True):
'''
Convert an integer to base64. Used to turn IDs into short URL slugs.
:param number:
:param url_safe: base64 may contain "/" and "+", which do not play well
with URLS. Set to True to convert "/" to "-" and "+" to
"_". This no longer conforms to base64, but looks better
in URLS.
:return:
'''
if number < 0:
raise Exception("Cannot convert negative IDs.")
# Encode the long, long as little endian.
packed = struct.pack("<Q", number)
# Remove leading zeros
while len(packed) > 1 and packed[-1] == b'\x00':
packed = packed[:-1]
encoded = base64.b64encode(packed).split(b"=")[0]
if url_safe:
encoded = encoded.replace(b"/", b"-").replace(b"+", b".")
return encoded
def chars_to_int(chars):
'''Reverse of the above function. Will work regardless of whether
url_safe was set to True or False.'''
# Make sure the data is in binary type.
if isinstance(chars, six.string_types):
chars = chars.encode('utf8')
# Do the reverse of the url_safe conversion above.
chars = chars.replace(b"-", b"/").replace(b".", b"+")
# First decode the base64, adding the required "=" padding.
b64_pad_len = 4 - len(chars) % 4
decoded = base64.b64decode(chars + b"="*b64_pad_len)
# Now decode little endian with "0" padding, which are leading zeros.
int64_pad_len = 8 - len(decoded)
return struct.unpack("<Q", decoded + b'\x00' * int64_pad_len)[0]
You can do following conversions by using python
First of all import base64 by using following syntax
>>> import base64
For converting text to base64 do following
encoding
>>> base64.b64encode("cloth".encode()).decode()
'Y2xvdGg='
decoding
>>> base64.b64decode("Y2xvdGg=".encode()).decode()
'cloth'
I am trying to send a value over serial port. I am unsure how to convert it into the correct format. I tried bin() for binary and format() but it didn't work.
result = 2
ser.open()
ser.write(b'1')
time.sleep(3)
ser.write(result) # wrong format
ser.write("{0:b}".format(result)) # TypeError: unicode strings
# are not supported, please
# encode to bytes: '10'
The first write operation sends '1' as binary string. Now I want the variable to be send as binary string as well.
write() requires a bytes object.
>>> help(serial.Serial.write)
Help on function write in module serial.serialwin32:
write(self, data)
Output the given byte string over the serial port.
To convert an integer to bytes, call int.to_bytes().
>>> result = 2
>>> b = result.to_bytes(4, 'little')
>>> b
b'\x02\x00\x00\x00'
>>> # to convert back to an integer
>>> int.from_bytes(b, 'little')
2
Like this :
import binascii
def write(num):
pack = binascii.unlexlify("%04X"%num)
ser.write(pack)
Important point: Which number system used on device (8,16,32,64 bit) ?
8 Bit = 1 Byte(0-255)
16Bit = 2 Byte(0-65535)
32Bit = 4 Byte(like upside)(0-4294967295)
All ranges are UNSIGNED(look up), but float
got extra definition !
You can't type binary 1 value with keyboard :
binascii.unlexlify("%01X"%1) so equal to \x01 (of course you can use struct package)
write() method accepts a string parameter. You can convert result to string with str() builtin function like this.
result = str(result)
Python 2:
result = 2
ser.open()
ser.write(b'1')
time.sleep(3)
ser.write(str(result))
Python 3:
You have to encode string in bytes.
result = 2
ser.open()
ser.write(b'1')
time.sleep(3)
ser.write(str(result).encode('utf-8'))
I'm working on a project where the output size is very important. As my outputs are numbers between 0 and 100, I'm trying to write them as bytes (or unsigned chars).
However, I'm getting errors when trying to read them.
Here is a simple example:
test_filename='test.b'
g=(3*ones(shape=[1000])).astype('c')
g.tofile(test_filename)
with open(test_filename, "rb") as f:
bytes = f.read(1)
num = int(bytes.encode('hex'), 1)
print num
Here is the error I get, somehow the bytes.encode thingy excepts a binary string or something of that sort (not sure of course):
ValueError Traceback (most recent call last)
<ipython-input-43-310a447041fe> in <module>()
----> 1 num = int(bytes.encode('hex'), 1)
2 print num
ValueError: int() base must be >= 2 and <= 36
I should state that I would later need to read the output files in C++.
Thanks in advance,
Gil
There is some iffiness to this based on the version of python you are using.
If python2, which I assume you are using because of the print statement, the main problem you have is that you are getting a string from the read, so if the value is say 50 you would get an ascii value of 2 if you print it. You need to tell python that those bits should be in an int type not a str type and a simple cast does not do that.
I personally would use the struct package and do the following:
with open(test_filename, "rb") as f:
bytes = f.read(1)
num = struct.unpack("B", bytes)[0]
print num
Another option would be to encode the string to hex and read it in as a hex string (which looks like is what you are trying):
num = int(bytes.encode("hex_codec"), 16))
print num
One final option would be to put the string in a bytearray and pull the first byte:
num = bytearray(bytes)[0]
print num
If you are actually using python 3 this is simpler because you will get back a bytes object (if so dont name a variable bytes, very confusing). With a bytes object you can just pull the first element out which will be pulled out as an int:
num = bytes[0]
print num
I want the shortest possible way of representing an integer in a URL. For example, 11234 can be shortened to '2be2' using hexadecimal. Since base64 uses is a 64 character encoding, it should be possible to represent an integer in base64 using even less characters than hexadecimal. The problem is I can't figure out the cleanest way to convert an integer to base64 (and back again) using Python.
The base64 module has methods for dealing with bytestrings - so maybe one solution would be to convert an integer to its binary representation as a Python string... but I'm not sure how to do that either.
This answer is similar in spirit to Douglas Leeder's, with the following changes:
It doesn't use actual Base64, so there's no padding characters
Instead of converting the number first to a byte-string (base 256), it converts it directly to base 64, which has the advantage of letting you represent negative numbers using a sign character.
import string
ALPHABET = string.ascii_uppercase + string.ascii_lowercase + \
string.digits + '-_'
ALPHABET_REVERSE = dict((c, i) for (i, c) in enumerate(ALPHABET))
BASE = len(ALPHABET)
SIGN_CHARACTER = '$'
def num_encode(n):
if n < 0:
return SIGN_CHARACTER + num_encode(-n)
s = []
while True:
n, r = divmod(n, BASE)
s.append(ALPHABET[r])
if n == 0: break
return ''.join(reversed(s))
def num_decode(s):
if s[0] == SIGN_CHARACTER:
return -num_decode(s[1:])
n = 0
for c in s:
n = n * BASE + ALPHABET_REVERSE[c]
return n
>>> num_encode(0)
'A'
>>> num_encode(64)
'BA'
>>> num_encode(-(64**5-1))
'$_____'
A few side notes:
You could (marginally) increase the human-readibility of the base-64 numbers by putting string.digits first in the alphabet (and making the sign character '-'); I chose the order that I did based on Python's urlsafe_b64encode.
If you're encoding a lot of negative numbers, you could increase the efficiency by using a sign bit or one's/two's complement instead of a sign character.
You should be able to easily adapt this code to different bases by changing the alphabet, either to restrict it to only alphanumeric characters or to add additional "URL-safe" characters.
I would recommend against using a representation other than base 10 in URIs in most cases—it adds complexity and makes debugging harder without significant savings compared to the overhead of HTTP—unless you're going for something TinyURL-esque.
All the answers given regarding Base64 are very reasonable solutions. But they're technically incorrect. To convert an integer to the shortest URL safe string possible, what you want is base 66 (there are 66 URL safe characters).
That code looks something like this:
from io import StringIO
import urllib
BASE66_ALPHABET = u"0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz-_.~"
BASE = len(BASE66_ALPHABET)
def hexahexacontadecimal_encode_int(n):
if n == 0:
return BASE66_ALPHABET[0].encode('ascii')
r = StringIO()
while n:
n, t = divmod(n, BASE)
r.write(BASE66_ALPHABET[t])
return r.getvalue().encode('ascii')[::-1]
Here's a complete implementation of a scheme like this, ready to go as a pip installable package:
https://github.com/aljungberg/hhc
You probably do not want real base64 encoding for this - it will add padding etc, potentially even resulting in larger strings than hex would for small numbers. If there's no need to interoperate with anything else, just use your own encoding. Eg. here's a function that will encode to any base (note the digits are actually stored least-significant first to avoid extra reverse() calls:
def make_encoder(baseString):
size = len(baseString)
d = dict((ch, i) for (i, ch) in enumerate(baseString)) # Map from char -> value
if len(d) != size:
raise Exception("Duplicate characters in encoding string")
def encode(x):
if x==0: return baseString[0] # Only needed if don't want '' for 0
l=[]
while x>0:
l.append(baseString[x % size])
x //= size
return ''.join(l)
def decode(s):
return sum(d[ch] * size**i for (i,ch) in enumerate(s))
return encode, decode
# Base 64 version:
encode,decode = make_encoder("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/")
assert decode(encode(435346456456)) == 435346456456
This has the advantage that you can use whatever base you want, just by adding appropriate
characters to the encoder's base string.
Note that the gains for larger bases are not going to be that big however. base 64 will only reduce the size to 2/3rds of base 16 (6 bits/char instead of 4). Each doubling only adds one more bit per character. Unless you've a real need to compact things, just using hex will probably be the simplest and fastest option.
To encode n:
data = ''
while n > 0:
data = chr(n & 255) + data
n = n >> 8
encoded = base64.urlsafe_b64encode(data).rstrip('=')
To decode s:
data = base64.urlsafe_b64decode(s + '===')
decoded = 0
while len(data) > 0:
decoded = (decoded << 8) | ord(data[0])
data = data[1:]
In the same spirit as other for some “optimal” encoding, you can use 73 characters according to RFC 1738 (actually 74 if you count “+” as usable):
alphabet = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz_`\"!$'()*,-."
encoded = ''
while n > 0:
n, r = divmod(n, len(alphabet))
encoded = alphabet[r] + encoded
and the decoding:
decoded = 0
while len(s) > 0:
decoded = decoded * len(alphabet) + alphabet.find(s[0])
s = s[1:]
The easy bit is converting the byte string to web-safe base64:
import base64
output = base64.urlsafe_b64encode(s)
The tricky bit is the first step - convert the integer to a byte string.
If your integers are small you're better off hex encoding them - see saua
Otherwise (hacky recursive version):
def convertIntToByteString(i):
if i == 0:
return ""
else:
return convertIntToByteString(i >> 8) + chr(i & 255)
You don't want base64 encoding, you want to represent a base 10 numeral in numeral base X.
If you want your base 10 numeral represented in the 26 letters available you could use: http://en.wikipedia.org/wiki/Hexavigesimal.
(You can extend that example for a much larger base by using all the legal url characters)
You should atleast be able to get base 38 (26 letters, 10 numbers, +, _)
Base64 takes 4 bytes/characters to encode 3 bytes and can only encode multiples of 3 bytes (and adds padding otherwise).
So representing 4 bytes (your average int) in Base64 would take 8 bytes. Encoding the same 4 bytes in hex would also take 8 bytes. So you wouldn't gain anything for a single int.
a little hacky, but it works:
def b64num(num_to_encode):
h = hex(num_to_encode)[2:] # hex(n) returns 0xhh, strip off the 0x
h = len(h) & 1 and '0'+h or h # if odd number of digits, prepend '0' which hex codec requires
return h.decode('hex').encode('base64')
you could replace the call to .encode('base64') with something in the base64 module, such as urlsafe_b64encode()
If you are looking for a way to shorten the integer representation using base64, I think you need to look elsewhere. When you encode something with base64 it doesn't get shorter, in fact it gets longer.
E.g. 11234 encoded with base64 would yield MTEyMzQ=
When using base64 you have overlooked the fact that you are not converting just the digits (0-9) to a 64 character encoding. You are converting 3 bytes into 4 bytes so you are guaranteed your base64 encoded string would be 33.33% longer.
I maintain a little library named zbase62: http://pypi.python.org/pypi/zbase62
With it you can convert from a Python 2 str object to a base-62 encoded string and vice versa:
Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> d = os.urandom(32)
>>> d
'C$\x8f\xf9\x92NV\x97\x13H\xc7F\x0c\x0f\x8d9}\xf5.u\xeeOr\xc2V\x92f\x1b=:\xc3\xbc'
>>> from zbase62 import zbase62
>>> encoded = zbase62.b2a(d)
>>> encoded
'Fv8kTvGhIrJvqQ2oTojUGlaVIxFE1b6BCLpH8JfYNRs'
>>> zbase62.a2b(encoded)
'C$\x8f\xf9\x92NV\x97\x13H\xc7F\x0c\x0f\x8d9}\xf5.u\xeeOr\xc2V\x92f\x1b=:\xc3\xbc'
However, you still need to convert from integer to str. This comes built-in to Python 3:
Python 3.2 (r32:88445, Mar 25 2011, 19:56:22)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> d = os.urandom(32)
>>> d
b'\xe4\x0b\x94|\xb6o\x08\xe9oR\x1f\xaa\xa8\xe8qS3\x86\x82\t\x15\xf2"\x1dL%?\xda\xcc3\xe3\xba'
>>> int.from_bytes(d, 'big')
103147789615402524662804907510279354159900773934860106838120923694590497907642
>>> x= _
>>> x.to_bytes(32, 'big')
b'\xe4\x0b\x94|\xb6o\x08\xe9oR\x1f\xaa\xa8\xe8qS3\x86\x82\t\x15\xf2"\x1dL%?\xda\xcc3\xe3\xba'
To convert from int to bytes and vice versa in Python 2, there is not a convenient, standard way as far as I know. I guess maybe I should copy some implementation, such as this one: https://github.com/warner/foolscap/blob/46e3a041167950fa93e48f65dcf106a576ed110e/foolscap/banana.py#L41 into zbase62 for your convenience.
I needed a signed integer, so I ended up going with:
import struct, base64
def b64encode_integer(i):
return base64.urlsafe_b64encode(struct.pack('i', i)).rstrip('=\n')
Example:
>>> b64encode_integer(1)
'AQAAAA'
>>> b64encode_integer(-1)
'_____w'
>>> b64encode_integer(256)
'AAEAAA'
I'm working on making a pip package for this.
I recommend you use my bases.py https://github.com/kamijoutouma/bases.py which was inspired by bases.js
from bases import Bases
bases = Bases()
bases.toBase16(200) // => 'c8'
bases.toBase(200, 16) // => 'c8'
bases.toBase62(99999) // => 'q0T'
bases.toBase(200, 62) // => 'q0T'
bases.toAlphabet(300, 'aAbBcC') // => 'Abba'
bases.fromBase16('c8') // => 200
bases.fromBase('c8', 16) // => 200
bases.fromBase62('q0T') // => 99999
bases.fromBase('q0T', 62) // => 99999
bases.fromAlphabet('Abba', 'aAbBcC') // => 300
refer to https://github.com/kamijoutouma/bases.py#known-basesalphabets
for what bases are usable
For your case
I recommend you use either base 32, 58 or 64
Base-64 warning: besides there being several different standards, padding isn't currently added and line lengths aren't tracked. Not recommended for use with APIs that expect formal base-64 strings!
Same goes for base 66 which is currently not supported by both bases.js and bases.py but it might in the future
I'd go the 'encode integer as binary string, then base64 encode that' method you suggest, and I'd do it using struct:
>>> import struct, base64
>>> base64.b64encode(struct.pack('l', 47))
'LwAAAA=='
>>> struct.unpack('l', base64.b64decode(_))
(47,)
Edit again:
To strip out the extra 0s on numbers that are too small to need full 32-bit precision, try this:
def pad(str, l=4):
while len(str) < l:
str = '\x00' + str
return str
>>> base64.b64encode(struct.pack('!l', 47).replace('\x00', ''))
'Lw=='
>>> struct.unpack('!l', pad(base64.b64decode('Lw==')))
(47,)
Pure python, no dependancies, no encoding of byte strings etc. , just turning a base 10 int into base 64 int with the correct RFC 4648 characters:
def tetrasexagesimal(number):
out=""
while number>=0:
if number == 0:
out = 'A' + out
break
digit = number % 64
out = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"[digit] + out
number /= 64 # //= 64 for py3 (thank spanishgum!)
if number == 0:
break
return out
tetrasexagesimal(1)
As it was mentioned here in comments you can encode a data using 73 characters that are not escaped in URL.
I found two places were this Base73 URL encoding is used:
https://git.nolog.cz/NoLog.cz/f.bain/src/branch/master/static/script.js JS based URL shortener
https://gist.github.com/LoneFry/3792021 in PHP
But in fact you may use more characters like /, [, ], :, ; and some others. Those characters are escaped only when you doing encodeURIComponent i.e. you need to pass data via get parameter.
So in fact you can use up to 82 characters. The full alphabet is !$&'()*+,-./0123456789:;=#ABCDEFGHIJKLMNOPQRSTUVWXYZ[]_abcdefghijklmnopqrstuvwxyz~. I sorted all the symbols by their code so when Base82URL numbers are sorted as plain strings they are keep the same order.
I tested in Chrome and Firefox and they are works fine but may be confusing for regular users. But I used such ids for an internal API calls where nobody sees them.
Unsigned integer 32 bit may have a maximum value of 2^32=4294967296
And after encoding to the Base82 it will take 6 chars: $0~]mx.
I don't have a code in Python but here is a JS code that generates a random id (int32 unsigned) and encodes it into the Base82URL:
/**
* Convert uint32 number to Base82 url safe
* #param {int} number
* #returns {string}
*/
function toBase82Url(number) {
// all chars that are not escaped in url
let keys = "!$&'()*+,-./0123456789:;=#ABCDEFGHIJKLMNOPQRSTUVWXYZ[]_abcdefghijklmnopqrstuvwxyz~"
let radix = keys.length
let encoded = []
do {
let index = number% radix
encoded.unshift(keys.charAt(index))
number = Math.trunc(number / radix)
} while (number !== 0)
return encoded .join("")
}
function generateToken() {
let buf = new Uint32Array(1);
window.crypto.getRandomValues(buf)
var randomInt = buf[0]
return toBase82Url(randomInt)
}