A coordinate x,y is encoded in a integer using this function
# Convert X Y position to MAP file index
def index_from_xy(x, y):
return (y - 16) << 16 | (x & 65535)
It seems that x has been converted into an unsigned short.
x and y are within the range [-32767 32767]
What is the function that will convert the index into a x, y tuple ?
All python integers are long under the hood (unless you're playing with bigger numbers).
To extract x and y, just reverse the steps of the above function.
def int_to_signed_short(value):
return -(value & 0x8000) | (value & 0x7fff)
def xy_from_index(index):
x, y = index & 65535, (index >> 16) + 16
return map(int_to_signed_short, [x, y])
In more detail, your function takes the two numbers and shifts them in binary so they don't overlap with each other.
x & 65535 only keeps the 16 rightmost bits of x, since 65535 is 16 1s in binary. Read more about bitwise AND.
(y - 16) << 16 shifts the number y - 16 16 bits to the left. So if your number was XXXXX in binary, it will become XXXXX0000000000000000. Read more about bitwise shift.
A | B will OR the bits of both numbers. Since A has 16 0s to the right and B is at most 16 bits long, A and B will not interfere with each other's bits. Read more about bitwise OR
Once you understand that, it should become clear how my function is the inverse of this.
Example
>>> index = index_from_xy(1234, 5678)
>>> index
371066066
>>> xy_from_index(index)
(1234, 5678)
Related
In Python the << operator does the following:
Returns x with the bits shifted to the left by y places (and new bits on the right-hand-side are zeros). This is the same as multiplying x by 2**y.
I want to have another version where it fills 1 on the new bits. Is there a built in function for this, if not how do I write it?
Is x = (x << y) | ((1 << y) - 1) what you're looking for?
First, we shift x left y bits:
x = 21
y = 2
bin(x) == 10101 (in binary)
x << y == 1010100
Then, the number ((1 << y) - 1) gives us a number with only the y lowest bits set to 1,
e.g.
1 << 2 == 100
(1 << 2) - 1 == 011
Finally, we or them to get 1010100 | 11 == 1010111.
There's no such function built in, or, for that matter, available in any extension library I know of.
It's pretty easy to do yourself, though. Say the int is n and you want to shift it left by s bits.
First flip all the bits of n (change all 0 bits to 1, and all 1 bits to 0). Then shift left by s. That adds s 0 bits on the right. Then flip the bits again. The new trailing 0 bits are changed to 1 bits, and the original bits of n are restored.
>>> n = 5
>>> bin(n)
'0b101'
>>> ~(~n << 6)
383
>>> bin(_)
'0b101111111'
I'm a little confused by the ~ operator. Code goes below:
a = 1
~a #-2
b = 15
~b #-16
How does ~ do work?
I thought, ~a would be something like:
0001 = a
1110 = ~a
why not?
You are exactly right. It's an artifact of two's complement integer representation.
In 16 bits, 1 is represented as 0000 0000 0000 0001. Inverted, you get 1111 1111 1111 1110, which is -2. Similarly, 15 is 0000 0000 0000 1111. Inverted, you get 1111 1111 1111 0000, which is -16.
In general, ~n = -n - 1
The '~' operator is defined as:
"The bit-wise inversion of x is defined as -(x+1). It only applies to integral numbers."Python Doc - 5.5
The important part of this sentence is that this is related to 'integral numbers' (also called integers). Your example represents a 4 bit number.
'0001' = 1
The integer range of a 4 bit number is '-8..0..7'. On the other hand you could use 'unsigned integers', that do not include negative number and the range for your 4 bit number would be '0..15'.
Since Python operates on integers the behavior you described is expected. Integers are represented using two's complement. In case of a 4 bit number this looks like the following.
7 = '0111'
0 = '0000'
-1 = '1111'
-8 = '1000'
Python uses 32bit for integer representation in case you have a 32-bit OS. You can check the largest integer with:
sys.maxint # (2^31)-1 for my system
In case you would like an unsigned integer returned for you 4 bit number you have to mask.
'0001' = a # unsigned '1' / integer '1'
'1110' = ~a # unsigned '14' / integer -2
(~a & 0xF) # returns 14
If you want to get an unsigned 8 bit number range (0..255) instead just use:
(~a & 0xFF) # returns 254
It looks like I found simpler solution that does what is desired:
uint8: x ^ 0xFF
uint16: x ^ 0xFFFF
uint32: x ^ 0xFFFFFFFF
uint64: x ^ 0xFFFFFFFFFFFFFFFF
You could also use unsigned ints (for example from the numpy package) to achieve the expected behaviour.
>>> import numpy as np
>>> bin( ~ np.uint8(1))
'0b11111110'
The problem is that the number represented by the result of applying ~ is not well defined as it depends on the number of bits used to represent the original value. For instance:
5 = 101
~5 = 010 = 2
5 = 0101
~5 = 1010 = 10
5 = 00101
~5 = 11010 = 26
However, the two's complement of ~5 is the same in all cases:
two_complement(~101) = 2^3 - 2 = 6
two_complement(~0101) = 2^4 - 10 = 6
two_complement(~00101) = 2^5 - 26 = 6
And given that the two's complement is used to represent negative values, it makes sense to consider ~5 as the negative value, -6, of its complement.
So, more formally, to arrive at this result we have:
flipped zeros and ones (that's equivalent to taking the ones' complement)
taken two's complement
applied negative sign
and if x is a n-digit number:
~x = - two_complement(one_complement(x)) = - two_complement(2^n - 1 - x) = - (2^n - (2^n - 1 - x)) = - (x + 1)
I am writing code that will have negative and positive numbers all 16 bits long with the MSB being the sign aka two's complement. This means the smallest number I can have is -32768 which is 1000 0000 0000 0000 in two's complement form. The largest number I can have is 32767 which is 0111 1111 1111 1111.
The issue I am having is python is representing the negative numbers with the same binary notation as positive numbers just putting a minus sign out the front i.e. -16384 is displayed as -0100 0000 0000 0000 what I want to be displayed for a number like -16384 is 1100 0000 0000 0000.
I am not quite sure how this can be coded. This is the code i have. Essentially if the number is between 180 and 359 its going to be negative. I need to display this as a twos compliment value. I dont have any code on how to display it because i really have no idea how to do it.
def calculatebearingActive(i):
numTracks = trackQty_Active
bearing = (((i)*360.0)/numTracks)
if 0< bearing <=179:
FC = (bearing/360.0)
FC_scaled = FC/(2**(-16))
return int(FC_scaled)
elif 180<= bearing <=359:
FC = -1*(360-bearing)/(360.0)
FC_scaled = FC/(2**(-16))
return int(FC_scaled)
elif bearing ==360:
FC = 0
return FC
If you're doing something like
format(num, '016b')
to convert your numbers to a two's complement string representation, you'll want to actually take the two's complement of a negative number before stringifying it:
format(num if num >= 0 else (1 << 16) + num, '016b')
or take it mod 65536:
format(num % (1 << 16), '016b')
The two's complement of a value is the one's complement plus one.
You can write your own conversion function based on that:
def to_binary(value):
result = ''
if value < 0:
result = '-'
value = ~value + 1
result += bin(value)
return result
The result looks like this:
>>> to_binary(10)
'0b1010'
>>> to_binary(-10)
'-0b1010'
Edit: To display the bits without the minus in front you can use this function:
def to_twoscomplement(bits, value):
if value < 0:
value = ( 1<<bits ) + value
formatstring = '{:0%ib}' % bits
return formatstring.format(value)
>>> to_twoscomplement(16, 3)
'0000000000000011'
>>> to_twoscomplement(16, -3)
'1111111111111101'
If you actually want to store the numbers using 16 bits, you can use struct.
import struct
>>> struct.pack('h', 32767)
'\xff\x7f'
>>> struct.pack('h', -32767)
'\x01\x80'
You can unpack using unpack
>>> a = struct.pack('h', 32767)
>>> struct.unpack('H', a)
32767
Python can hold unlimited integer values, the bit representation will adopt to hold any number you put. So such technical details as two complement does not make sense in this context. In C 'b1111111111111111' means -1 for int16 and 65535 for uint16 or int32. In python it is always is 65535 as the int will adopt to hold such values.
I think this is why they opted to add - in front of negative numbers regardless of string representation (binary, oct, hex, decimal ...).
If you wish to replicate the C behaviour and get the negative represented in two complement form you have the following options:
1 int > uint > bin using numpy
The most straight forward way is to dump the value as signed limited int and read it as unsigned.
If you have access to numpy the code is pretty easy to read:
>>> bin(np.int16(-30).astype(np.uint16))
'0b1111111111100010'
>>> bin(np.int16(-1).astype(np.uint16))
'0b1111111111111111'
>>> bin(np.int16(-2).astype(np.uint16))
'0b1111111111111110'
>>> bin(np.int16(-16).astype(np.uint16))
'0b1111111111110000'
2 int > uint > bin using struct
You can do the similar think with struct but it is slightly harder to understand
>>> bin(struct.unpack('>H', struct.pack('>h', 30))[0])
'0b1111111111100010'
>>> bin(struct.unpack('>H', struct.pack('>h', -1))[0])
'0b1111111111111111'
>>> bin(struct.unpack('>H', struct.pack('>h', -2))[0])
'0b1111111111111110'
>>> bin(struct.unpack('>H', struct.pack('>h', -16))[0])
'0b1111111111110000'
Note: h is signed and H is unsigned int 16, and '>' stands for bigendian it comes handy if you want to read bytes directly without converting them back to int
3. int using struct, then read byte by byte and convert to bin
>>> ''.join(f'{byte:08b}' for byte in struct.pack('>h', -1<<15))
'1000000000000000'
>>> ''.join(f'{byte:08b}' for byte in struct.pack('>h', -1))
'1111111111111111'
>>> ''.join(f'{byte:08b}' for byte in struct.pack('>h', -2))
'1111111111111110'
>>> ''.join(f'{byte:08b}' for byte in struct.pack('>h', -16))
'1111111111110000'
Note this has some quirks, as you need to remember to enforce the byte binary representation to be 8 digits long with '08'.
3 Directly from math,
Lastly you can go check what wikipedia says about 2 complement representation and get it implemented directly from the math formula Two complement > Subtraction from 2N
>>> bin(2**16 -16)
'0b1111111111110000'
>>> bin(2**16 -3)
'0b1111111111111101'
This looks super simple but it is hard to understand if you are not versed in the way 2 complement representation works.
Since you haven't given any code examples, I can't be sure what's going on. Based on the numbers in your example, I don't think you're using bin(yourint) because you're output doesn't contain 0b. Maybe you're already slicing that off in your examples.
If you are storing your binary data as strings, you could do something like:
def handle_negatives(binary_string):
If binary_string < 0:
binary_string = '1' + str(binary_string)[1:]
Return binary_string
Here are some caveats in printing complements(1s and 2s) in binary form, in python:
UNSIGNED RANGE: 0 to (2^k)-1 for k bit number
ex: 0 to (2^32)-1 numbers
ex: 0 to 7 for 3 bit unsigned numbers (count = 8)
SIGNED RANGE: -2^(k-1) to +2^(k-1)-1 for 1+k bit number (k-1 is for dividing current range k into two equal half)
ex: -2^31 to +(2^31)-1 numbers
ex -8 to +7 for 1+3 bit signed numbers (count = 8)
bin(int)->str converts an integer to binary string
CAVEAT: 1. Since in python there is no limit to length of integer
for ~x or !x (1s_complement/not/negate) we can't determine how many bits after MSB needs to be 1
so python just prints out unsigned value of target negative number in binary format with a
'-' sign in the beginning
ex: for x = b'01010'(10) we get ~x = -0b'1011' (-11)
but we should be getting -16+5 = -11
(-b'10000'+b'00101') = -b'10101' (-11 signed) or (21 unsigned)
to get real binary value after negation(or 1s complement) one could simulate it
NOTE: 2^0 is always 1, so (2**0 == 1) in python
NOTE: (1 << k) is always 2^k (left shift is 2 raised to the power k)
ex: bin((1 << k)-1 - x) which is ((2^k)-1)-x (1s complement)
ex: bin((1 << k)-1 - x) + 1 which is (2^k)-x (2s complement)
2. Same goes for reverse parsing of signed binary string to int:
ex: int("-0b'0101'", 2) gives -5 but instead it actually is -11 assuming -0b represents all bits
from MSB till current to be like 1111......0101 which is actually -16+5 = -11
BUT due to PYTHON's limitation of representing signed binary values we need to adhere to
current way of parsing considering unsigned binary strings with sign in front for -ve numbers
# NOTE: how the significant prefix zeros doesn't matter in both +ve and -ve cases
# Byte type inputs
x = b'+1010' # valid +ve number byte string
x = b'1010' # valid +ve number byte string
x = b'-1010' # valid -ve number byte string
x = b'+01010' # valid +ve number byte string
x = b'01010' # valid +ve number byte string
x = b'-01010' # valid -ve number byte string
int(b'101') # interprets as base 10 for each digit
int(b'101', 2) # interprets as base 2 for each digit
int(b'101', 8) # interprets as base 8 for each digit
int(b'101', 10) # interprets as base 10 for each digit
int(b'101', 16) # interprets as base 16 for each digit
# String type inputs
x = '+1010' # valid +ve number string
x = '1010' # valid +ve number string
x = '-1010' # valid -ve number string
x = '+01010' # valid +ve number string
x = '01010' # valid +ve number string
x = '-01010' # valid -ve number string
int('101') # interprets as base 10 for each digit
int('101', 2) # interprets as base 2 for each digit
int('101', 8) # interprets as base 8 for each digit
int('101', 10) # interprets as base 10 for each digit
int('101', 16) # interprets as base 16 for each digit
# print(bin(int(x, 2)), int(x,2), ~int(x, 2), bin(~int(x,2)), "-"+bin((1<<k)-1 - int(x,2)))
k = 5 # no of bits
assert 2**0 == 1 # (2^0 is always 1)
_2k = (1 << k) # (2^k == left shift (2^0 or 1) by k times == multiply 2 by k times)
x = '01010' # valid +ve number string
x = int(x,2)
print("input:", x) # supposed to be 1s complement of binStr but due to python's limitation,
# we consider it -(unsigned binStr)
_1s = '-'+bin((_2k-1)-x)
print("1s complement(negate/not): ", _1s, ~x)
_2s = '-'+bin(_2k-x)
print("2s complement(1s +1): ", _2s, ~x+1)
output:
k = 5 (5 bit representation)
input: 10
1s complement(negate/not): -0b10101 -11
2s complement(1s +1): -0b10110 -10
k=32 (32 bit representation)
input: 10
1s complement(negate/not): -0b11111111111111111111111111110101 -11
2s complement(1s +1): -0b11111111111111111111111111110110 -10
Let's say I have this number i = -6884376.
How do I refer to it as to an unsigned variable?
Something like (unsigned long)i in C.
Assuming:
You have 2's-complement representations in mind; and,
By (unsigned long) you mean unsigned 32-bit integer,
then you just need to add 2**32 (or 1 << 32) to the negative value.
For example, apply this to -1:
>>> -1
-1
>>> _ + 2**32
4294967295L
>>> bin(_)
'0b11111111111111111111111111111111'
Assumption #1 means you want -1 to be viewed as a solid string of 1 bits, and assumption #2 means you want 32 of them.
Nobody but you can say what your hidden assumptions are, though. If, for example, you have 1's-complement representations in mind, then you need to apply the ~ prefix operator instead. Python integers work hard to give the illusion of using an infinitely wide 2's complement representation (like regular 2's complement, but with an infinite number of "sign bits").
And to duplicate what the platform C compiler does, you can use the ctypes module:
>>> import ctypes
>>> ctypes.c_ulong(-1) # stuff Python's -1 into a C unsigned long
c_ulong(4294967295L)
>>> _.value
4294967295L
C's unsigned long happens to be 4 bytes on the box that ran this sample.
To get the value equivalent to your C cast, just bitwise and with the appropriate mask. e.g. if unsigned long is 32 bit:
>>> i = -6884376
>>> i & 0xffffffff
4288082920
or if it is 64 bit:
>>> i & 0xffffffffffffffff
18446744073702667240
Do be aware though that although that gives you the value you would have in C, it is still a signed value, so any subsequent calculations may give a negative result and you'll have to continue to apply the mask to simulate a 32 or 64 bit calculation.
This works because although Python looks like it stores all numbers as sign and magnitude, the bitwise operations are defined as working on two's complement values. C stores integers in twos complement but with a fixed number of bits. Python bitwise operators act on twos complement values but as though they had an infinite number of bits: for positive numbers they extend leftwards to infinity with zeros, but negative numbers extend left with ones. The & operator will change that leftward string of ones into zeros and leave you with just the bits that would have fit into the C value.
Displaying the values in hex may make this clearer (and I rewrote to string of f's as an expression to show we are interested in either 32 or 64 bits):
>>> hex(i)
'-0x690c18'
>>> hex (i & ((1 << 32) - 1))
'0xff96f3e8'
>>> hex (i & ((1 << 64) - 1)
'0xffffffffff96f3e8L'
For a 32 bit value in C, positive numbers go up to 2147483647 (0x7fffffff), and negative numbers have the top bit set going from -1 (0xffffffff) down to -2147483648 (0x80000000). For values that fit entirely in the mask, we can reverse the process in Python by using a smaller mask to remove the sign bit and then subtracting the sign bit:
>>> u = i & ((1 << 32) - 1)
>>> (u & ((1 << 31) - 1)) - (u & (1 << 31))
-6884376
Or for the 64 bit version:
>>> u = 18446744073702667240
>>> (u & ((1 << 63) - 1)) - (u & (1 << 63))
-6884376
This inverse process will leave the value unchanged if the sign bit is 0, but obviously it isn't a true inverse because if you started with a value that wouldn't fit within the mask size then those bits are gone.
Python doesn't have builtin unsigned types. You can use mathematical operations to compute a new int representing the value you would get in C, but there is no "unsigned value" of a Python int. The Python int is an abstraction of an integer value, not a direct access to a fixed-byte-size integer.
Since version 3.2 :
def unsignedToSigned(n, byte_count):
return int.from_bytes(n.to_bytes(byte_count, 'little', signed=False), 'little', signed=True)
def signedToUnsigned(n, byte_count):
return int.from_bytes(n.to_bytes(byte_count, 'little', signed=True), 'little', signed=False)
output :
In [3]: unsignedToSigned(5, 1)
Out[3]: 5
In [4]: signedToUnsigned(5, 1)
Out[4]: 5
In [5]: unsignedToSigned(0xFF, 1)
Out[5]: -1
In [6]: signedToUnsigned(0xFF, 1)
---------------------------------------------------------------------------
OverflowError Traceback (most recent call last)
Input In [6], in <cell line: 1>()
----> 1 signedToUnsigned(0xFF, 1)
Input In [1], in signedToUnsigned(n, byte_count)
4 def signedToUnsigned(n, byte_count):
----> 5 return int.from_bytes(n.to_bytes(byte_count, 'little', signed=True), 'little', signed=False)
OverflowError: int too big to convert
In [7]: signedToUnsigned(-1, 1)
Out[7]: 255
Explanations : to/from_bytes convert to/from bytes, in 2's complement considering the number as one of size byte_count * 8 bits. In C/C++, chances are you should pass 4 or 8 as byte_count for respectively a 32 or 64 bit number (the int type).
I first pack the input number in the format it is supposed to be from (using the signed argument to control signed/unsigned), then unpack to the format we would like it to have been from. And you get the result.
Note the Exception when trying to use fewer bytes than required to represent the number (In [6]). 0xFF is 255 which can't be represented using a C's char type (-128 ≤ n ≤ 127). This is preferable to any other behavior.
You could use the struct Python built-in library:
Encode:
import struct
i = -6884376
print('{0:b}'.format(i))
packed = struct.pack('>l', i) # Packing a long number.
unpacked = struct.unpack('>L', packed)[0] # Unpacking a packed long number to unsigned long
print(unpacked)
print('{0:b}'.format(unpacked))
Out:
-11010010000110000011000
4288082920
11111111100101101111001111101000
Decode:
dec_pack = struct.pack('>L', unpacked) # Packing an unsigned long number.
dec_unpack = struct.unpack('>l', dec_pack)[0] # Unpacking a packed unsigned long number to long (revert action).
print(dec_unpack)
Out:
-6884376
[NOTE]:
> is BigEndian operation.
l is long.
L is unsigned long.
In amd64 architecture int and long are 32bit, So you could use i and I instead of l and L respectively.
[UPDATE]
According to the #hl037_ comment, this approach works on int32 not int64 or int128 as I used long operation into struct.pack(). Nevertheless, in the case of int64, the written code would be changed simply using long long operand (q) in struct as follows:
Encode:
i = 9223372036854775807 # the largest int64 number
packed = struct.pack('>q', i) # Packing an int64 number
unpacked = struct.unpack('>Q', packed)[0] # Unpacking signed to unsigned
print(unpacked)
print('{0:b}'.format(unpacked))
Out:
9223372036854775807
111111111111111111111111111111111111111111111111111111111111111
Next, follow the same way for the decoding stage. As well as this, keep in mind q is long long integer — 8byte and Q is unsigned long long
But in the case of int128, the situation is slightly different as there is no 16-byte operand for struct.pack(). Therefore, you should split your number into two int64.
Here's how it should be:
i = 10000000000000000000000000000000000000 # an int128 number
print(len('{0:b}'.format(i)))
max_int64 = 0xFFFFFFFFFFFFFFFF
packed = struct.pack('>qq', (i >> 64) & max_int64, i & max_int64)
a, b = struct.unpack('>QQ', packed)
unpacked = (a << 64) | b
print(unpacked)
print('{0:b}'.format(unpacked))
Out:
123
10000000000000000000000000000000000000
111100001011110111000010000110101011101101001000110110110010000000011110100001101101010000000000000000000000000000000000000
just use abs for converting unsigned to signed in python
a=-12
b=abs(a)
print(b)
Output:
12
I'm searching for a simple algorithm that 'combines' two 2bytes integers into one unique 4bytes integer.
The two 2bytes integers are both whole positive numbers in the range 0..65535.
I want to create one 4bytes integer that is the exact combination of both, in a way that will make it easy to:
(1) given the two 2bytes integers --> calculate the value of that 4bytes integer.
(2) given the 4bytes integer --> parse the contents of the two 2bytes integers.
Any idea how to achieve this in python?
How about:
def combine(n, m):
return (n << 16) | m
def extract(c):
return (c >> 16), c & 0xffff
This solution places one of the 2-byte integers in the upper half of a 32-bit word, and the other into the lower half. In order to extract the values, simply take the upper half of the word (c >> 16), and the lower half (c & 0xffff).
>>> i1, i2 = 345, 12
>>> i1 * 0x10000 + i2
22609932
>>> divmod(22609932, 0x10000)
(345, 12)