Develop Reference

Why is 2 * x * x faster than 2 * ( x * x ) in Python 3.x, for integers?

The following Python 3.x integer multiplication takes on average between 1.66s and 1.77s:
import time
start_time = time.time()
num = 0
for x in range(0, 10000000):
# num += 2 * (x * x)
num += 2 * x * x
print("--- %s seconds ---" % (time.time() - start_time))
if I replace 2 * x * x with 2 *(x * x), it takes between 2.04 and 2.25. How come?
On the other hand it is the opposite in Java: 2 * (x * x) is faster in Java. Java test link: Why is 2 * (i * i) faster than 2 * i * i in Java?
I ran each version of the program 10 times, here are the results.
2 * x * x | 2 * (x * x)
---------------------------------------
1.7717654705047607 | 2.0789272785186768
1.735931396484375 | 2.1166207790374756
1.7093875408172607 | 2.024367570877075
1.7004504203796387 | 2.047525405883789
1.6676218509674072 | 2.254328966140747
1.699510097503662 | 2.0949244499206543
1.6889283657073975 | 2.0841963291168213
1.7243537902832031 | 2.1290600299835205
1.712965488433838 | 2.1942825317382812
1.7622807025909424 | 2.1200053691864014

First of all, note that we don't see the same thing in Python 2.x:
>>> timeit("for i in range(1000): 2*i*i")
51.00784397125244
>>> timeit("for i in range(1000): 2*(i*i)")
50.48330092430115
So this leads us to believe that this is due to how integers changed in Python 3: specifically, Python 3 uses long (arbitrarily large integers) everywhere.
For small enough integers (including the ones we're considering here), CPython actually just uses the O(MN) grade-school digit by digit multiplication algorithm (for larger integers it switches to the Karatsuba algorithm). You can see this yourself in the source.
The number of digits in x*x is roughly twice that of 2*x or x (since log(x2) = 2 log(x)). Note that a "digit" in this context is not a base-10 digit, but a 30-bit value (which are treated as single digits in CPython's implementation). Hence, 2 is a single-digit value, and x and 2*x are single-digit values for all iterations of the loop, but x*x is two-digit for x >= 2**15. Hence, for x >= 2**15, 2*x*x only requires single-by-single-digit multiplications whereas 2*(x*x) requires a single-by-single and a single-by-double-digit multiplication (since x*x has 2 30-bit digits).
Here's a direct way to see this (Python 3):
>>> timeit("a*b", "a,b = 2, 123456**2", number=100000000)
5.796971936999967
>>> timeit("a*b", "a,b = 2*123456, 123456", number=100000000)
4.3559221399999615
Again, compare this to Python 2, which doesn't use arbitrary-length integers everywhere:
>>> timeit("a*b", "a,b = 2, 123456**2", number=100000000)
3.0912468433380127
>>> timeit("a*b", "a,b = 2*123456, 123456", number=100000000)
3.1120400428771973
(One interesting note: If you look at the source, you'll see that the algorithm actually has a special case for squaring numbers (which we're doing here), but even still this is not enough to overcome the fact that 2*(x*x) just requires processing more digits.)

Python intern representation of integers is special, it uses slots of 30 bits :
In [6]: sys.getsizeof(2**30-1)
Out[6]: 28 # one slot + heading
In [7]: sys.getsizeof(2**30)
Out[7]: 32 # two slots
So everything happens as if Python counts in base B = 2**30 = 1 073 741 824 ~1 billion.
For a human who want to calculate 2*4*4, two ways :
(2*4)*4 = 8*4 =32 = 30 + 2 is immediate if you knows your add tables.
2*(4*4) = 2*16 = 2*10 + 2*6 = (2*10+10) + 2 = 30 + 2 since we have to put the operation down.
Python have the same problem. If x is a number such than 2x < B < x² , let x² = aB+b , with a,b <B. x² is stored in 2 slots, which I note (a|b). Computations leads to (without managing carries here):
(x*x)*2 => (a|b)*2 => (2*a|2*b)
(2*x)*x => (2x)*x =>(2a|2b)
in the first case the 2* operation is done two times, against only one in the first case. That explains the difference.

If your benchmark is right (didn't check), it may come from the fact that Python integers may be two different things : native integers when they are small (with a quick computation), and big integers when they increase in size (slower computation). The first syntax keeps the size smaller after the first operation while the second syntax may lead to two operations involving big integers.

From what I can tell, it comes down to a little bit more memory access in the version using 2 * (x * x). I printed the disassembled bytecode and it seems to prove that:
Relevant part of 2 * x * x:
7 28 LOAD_FAST 1 (num)
30 LOAD_CONST 3 (2)
32 LOAD_FAST 2 (x)
34 BINARY_MULTIPLY
36 LOAD_FAST 2 (x)
38 BINARY_MULTIPLY
40 INPLACE_ADD
42 STORE_FAST 1 (num)
44 JUMP_ABSOLUTE 24
Relevant part of 2 * (x * x):
7 28 LOAD_FAST 1 (num)
30 LOAD_CONST 3 (2)
32 LOAD_FAST 2 (x)
34 LOAD_FAST 2 (x)
36 BINARY_MULTIPLY <=== 1st multiply x*x in a temp value
38 BINARY_MULTIPLY <=== then multiply result with 2
40 INPLACE_ADD
42 STORE_FAST 1 (num)
44 JUMP_ABSOLUTE 24

Precise math.log(x, 2) in Python

In python, I need to get the rounded down logarithm of positive integers for base 2, including big numbers.
However, since floating point math is used, I might get bad results, for example:
>>> import math
>>> int(math.log(281474976710655, 2))
48
However:
>>> 2 ** 48
281474976710656
So the correct result, rounded down, should be 47.
How can I get the correct value?

In Python 3, integers have a .bit_length method, so you should use that to get your rounded down base 2 logarithm.
Here's a short demo:
m = 2 ** 1000
for n in (281474976710655, m-1, m, m+1):
a = n.bit_length() - 1
b = 2 ** a
print(a, b <= n < 2 * b)
output
47 True
999 True
1000 True
1000 True

In python 3 ints even have an efficient .bit_length() method!
>>> (281474976710655).bit_length()
48
>>> (281474976710656).bit_length()
49
In python 2, instead of using floating point math, count the number of bits:
def log2(n):
assert n >= 1
return len(bin(n)) - 3 # bin() returns a string starting with '0b'
(Edited following this comment)

Compressing a list of integers in Python

I have a list of positive (random) integers with the following properties:
Number of elements: 78495
Maximum value of element: 999982
Length of list when converted to a string: 517115 (string looks like "6,79384,238956,...")
Size of list in text file on disk: 520 kb
I am trying to use this list as a precomputed list for an online judge problem because it takes a long time to actually generate this list. However, it is too large to be accepted if I paste it directly into the source code, which has a cap of 50 kb.
I looked into zlib as a way to compress the string but it only seemed to cut the size in half.
Is there a way to really shrink this down so I can unpack it / use it in the source code?

Given your definition ...
it is a list of smallest-k values for which 10^k = 1 mod p for primes p > 5
... am I wrong to believe that your values are of the form (p - 1) / x where x is an integer significantly smaller than p?
For instance, for p < 50, we have:
p = 7 : 10^6 = 1 (mod 7) => k = 6 = (p - 1) / 1 => x = 1
p = 11 : 10^2 = 1 (mod 11) => k = 2 = (p - 1) / 5 => x = 5
p = 13 : 10^6 = 1 (mod 13) => k = 6 = (p - 1) / 2 => x = 2
p = 17 : 10^16 = 1 (mod 17) => k = 16 = (p - 1) / 1 => x = 1
p = 19 : 10^18 = 1 (mod 19) => k = 18 = (p - 1) / 1 => x = 1
p = 23 : 10^22 = 1 (mod 23) => k = 22 = (p - 1) / 1 => x = 1
p = 29 : 10^28 = 1 (mod 29) => k = 28 = (p - 1) / 1 => x = 1
p = 31 : 10^15 = 1 (mod 31) => k = 15 = (p - 1) / 2 => x = 2
p = 37 : 10^3 = 1 (mod 37) => k = 3 = (p - 1) / 12 => x = 12
p = 41 : 10^5 = 1 (mod 41) => k = 5 = (p - 1) / 8 => x = 8
p = 43 : 10^21 = 1 (mod 43) => k = 21 = (p - 1) / 2 => x = 2
p = 47 : 10^46 = 1 (mod 47) => k = 46 = (p - 1) / 1 => x = 1
The list of x values should compress much better than the list of k values. (For instance, I'd be willing to bet that the most frequent value of x will be '1'.)
And because it's rather easy and fast to compute primes up to 1 million (which I think is your upper bound), you may be able to quickly rebuild the list of k values based on the compressed list of x values and the real-time computed list of primes.
You probably should have explained from the beginning what exactly you were trying to compress to get more accurate answers.

In short, no.
log(2, 999982) ~= 20
So the largest number would take 20 bits to store. Let's say that on average, each number takes 10 bits to store (evenly distributed between 0 and the max).
~80,000 numbers * 10 bits per number = 800,000 bits = 100,000 bytes
So these numbers, stored as efficiently as possible, would take ~100KB of space.
Compression will only work if there's some non-randomness to the numbers. If they're truly random, as you say, then a general compression algorithm won't be able to make this any smaller, so 100KB is about the best you can hope to do.
EDIT
Note that things are even worse, in that you want to paste these into source code, so you can't just use arbitrary binary data. You'll need something text-friendly, like base64 encoding, which will add another ~33% of overhead. Also, you can't really store numbers based on the average number of bits required, because you'd need some way to know how many bits were used by each individual number. There are possible encoding schemes, but all will carry some additional overhead.
SECOND EDIT
Based on the comments above, the data is not actually random as originally stated. A general compression algorithm therefore might work, and if not, there are presumably other solutions (e.g. just shipping the code that generated the numbers in the first place, which is likely smaller than 50KB).

The best text compression available offers a (roughly) 12-17% compression ratio (62.4-90 kB) so you're not going to meet your threshold. Your data are random, as well, which generally makes compression algorithms perform worse.
Look at an alternative approach, such as making your RNG process faster, or if you don't need a full list (just some integers), create a separate "producer" thread to generate random integers (involving whatever actual math you are using) and a "consumer" thread that does work on those integers as they come in. That way, your program could perhaps still do work, even if it would take a long time to generate a full list.

Here I've tested easily availiable algorithms in python on two strings: one is generated randomly with non uniform distribution, another one has some structure. It seems, lzma does better
# check the compression ratio
import lzma
import zlib
import gzip
import bz2
import zipfile
import tarfile
compressors = ['lzma','zlib','gzip','bz2']
a = np.exp(np.random.rand(1024))
b = np.arange(1024)
b[32] = -10
b[96] = 20000
a = bytes(a)
b = bytes(b)
for i in range(len(compressors)):
print("{} compression ratio: ".format(compressors[i]))
a_lzma = eval(compressors[i]).compress(a)
b_lzma = eval(compressors[i]).compress(b)
print(float(len(a_lzma))/len(a),float(len(b_lzma))/len(b))
print("\n")
The output:
lzma compression ratio:
0.93115234375 0.08984375
zlib compression ratio:
0.95068359375 0.1944580078125
gzip compression ratio:
0.9521484375 0.196533203125
bz2 compression ratio:
0.9925537109375 0.1268310546875

modulus of negative numbers in Python [duplicate]

This question already has answers here:
How does the modulo (%) operator work on negative numbers in Python?
(12 answers)
Closed last month.
23 % -5 = -2
23 % 5 = 3
Can someone explain to me how I can understand this because I have an exam tomorrow. I want to say its because -5 * -5 =25 then 25 -2 = 23 which is how they get the 23. Is this correct?

In Python, the sign of the remainder is the same as the sign of the denominator (which differs from languages like C, where it is the same as the sign of the numerator).
Mathematically, you are always guaranteed that if a, b = divmod(n, d), then a*d + b == n.
Note that 23//5 == 4 and 23//-5 == -5 (Python always does floor division). Thus, we have 4*5 + 3 == 23 and -5*-5 - 2 == 23, as you have said.

Lets write it out as N=kM+R.
We have 23 = -5*(-5) - 2, and 23 = 4*5 + 3.

The simplest way of looking at problem for your purposes is to consider the definition that:
a mod n = R where the remainder R must satisfy 0<= R
So for mod -5 arithmetic, 0<= R < -4 i.e. R can be one of 0, -1, -2, -3, -4
that is you effectively subtract (or add) n from a until you bring R into the range above:
So
23 % 5 is (23-4*5) = 23-20 = 3
but
23 % -5 is (23+5*(-5)) = 23-25 = -2

Well, 23 % 5 = 3 since 4*5 = 20 and when you divide 23 by 20 you obtain a remainder of 3. You can think of it as the closet you can go without going over.
As for 23 % -5, well that answer differs from one programming language to another.
For Python it's -2 because it will always return the value of the divisor and it's because 5*5 = 25 and when you divide 23 by 25 in Python you obtain a remainder of -2 (since it must be negative because the divisor was negative) so we have 25 - 2 = 23.
It's worth noting that the formal mathematical definition states that b is a positive integer.

% in Python uses "Modulo operation" ; it's different from taking the reminder of a division operation such that.
a - int(a/n) * n
although it is sometimes equivalent in some computer languages.
The math expression can be found explict here: http://en.wikipedia.org/wiki/Modulo_operation
So obviously, in Python "%" operation uses the following expression:
mod(a, n) = a - n * floor(a / n)
Therefore,
23%-5 = mod(23,-5) = 23 - (-5) * floor(23/-5) = 23 - (-5) * -5 = -2
and
23%5 = mod(23, 5) = 23 - 5 * floor(23/5) = 23 - 5 * 4 = 3
In addition, you my find it's interesting that
-23%5 = mod(-23,5) = (-23) - 5 * floor(-23/5) = -23 - 5 * (-5) = 2
since floor() action will take the integer value toward negative infinity.

Python quotient vs remainder

The python 2.6 docs state that x % y is defined as the remainder of x / y (http://docs.python.org/library/stdtypes.html#numeric-types-int-float-long-complex). I am not clear on what is really occurring though, as:
for i in range(2, 11):
print 1.0 % i
prints "1.0" ten times, rather than "0.5, 0.333333, 0.25" etc. as I expected (1/2 = 0.5, etc).

Modulo is performed in the integer context, not fractional (remainders are integers). Therefore:
1 % 1 = 0 (1 times 1 plus 0)
1 % 2 = 1 (2 times 0 plus 1)
1 % 3 = 1 (3 times 0 plus 1)
6 % 3 = 0 (3 times 2 plus 0)
7 % 3 = 1 (3 times 2 plus 1)
8 % 3 = 2 (3 times 2 plus 2)
etc
How do I get the actual remainder of x / y?
By that I presume you mean doing a regular floating point division?
for i in range(2, 11):
print 1.0 / i

I think you can get the result you want by doing something like this:
for i in range(2, 11):
print 1.0*(1 % i) / i
This computes the (integer) remainder as explained by others. Then you divide by the denominator again, to produce the fractional part of the quotient.
Note that I multiply the result of the modulo operation by 1.0 to ensure that a floating point division operation is done (rather than integer division, which will result in 0).

You've confused division and modulus.
"0.5, 0.333333, 0.25" etc. as I expected (1/2 = 0.5, etc)."
That's the result of division.
Not modulus.
Modulus (%) is the remainder left over after integer division.
Your sample values are simple division, which is the / operator. Not the % operator.

Wouldn't dividing 1 by an number larger than it result in 0 with remainder 1?
The number theorists in the crowd may correct me, but I think modulus/remainder is defined only on integers.

We can have 2 types of division, that we can define through the return types:
Float: a/b. For example: 3/2=1.5
def division(a,b):
return a/b
Int: a//b and a%b. For example: 3//2=1 and 3%2=1
def quotient(a,b):
return a//b
def remainder(a,b):
return a%b