Develop Reference

Why random.randint() is much slower than random.getrandbits()

A made a test which compares random.randint() and random.getrandbits() in Python. The result shows that getrandbits is way faster than randint.
from random import randint, getrandbits, seed
from datetime import datetime
def ts():
return datetime.now().timestamp()
def diff(st, en):
print(f'{round((en - st) * 1000, 3)} ms')
seed(111)
N = 6
rn = []
st = ts()
for _ in range(10**N):
rn.append(randint(0, 1023))
en = ts()
diff(st,en)
rn = []
st = ts()
for _ in range(10**N):
rn.append(getrandbits(10))
en = ts()
diff(st,en)
The numbers range is exactly the same, because 10 random bits is range from 0 (in case of 0000000000) to 1023 (in case of 1111111111).
The output of the code:
590.509 ms
91.01 ms
As you can see, getrandbits under the same conditions is almost 6.5 times faster. But why? Can somebody explain that?

randint uses randrange which uses _randbelow which is _randbelow_with_get_randbits (if it is present) which uses getrandbits. However, randrange includes overhead on top of the underlying call to getrandbits because it must check for the start and step arguments. Even though there are fast paths for these checks, they are still there. There are definitely other things contributing to the 6.5 slowdown when using randint, but this answer atleast shows you that randint will always be slower than getrandbits.

getrandbits() is a very short function going to native code almost immediately:
def getrandbits(self, k):
"""getrandbits(k) -> x. Generates an int with k random bits."""
if k < 0:
raise ValueError('number of bits must be non-negative')
numbytes = (k + 7) // 8 # bits / 8 and rounded up
x = int.from_bytes(_urandom(numbytes), 'big')
return x >> (numbytes * 8 - k) # trim excess bits
_urandom is os.urandom, which you can't even find in os.py in the same folder, as it's native code. See details here: Where can I find the source code of os.urandom()?
randint() is a return self.randrange(a, b+1) call, where randrange() is a long and versatile function, almost 100 lines of Python code, its "fast track" ends in a return istart + self._randbelow(width) after circa 50 lines of checking and initializing.
_randbelow() is actually a choice, depending on the existence of getrandbits() which exists in your case, so probably this one is running:
def _randbelow_with_getrandbits(self, n):
"Return a random int in the range [0,n). Returns 0 if n==0."
if not n:
return 0
getrandbits = self.getrandbits
k = n.bit_length() # don't use (n-1) here because n can be 1
r = getrandbits(k) # 0 <= r < 2**k
while r >= n:
r = getrandbits(k)
return r
so randint() ends in the same getrandbits() after doing a lot of other things, and that's pretty much how it's slower than it.

The underlying reason is the function that getrandbits calls is written in C, look at the source code at GitHub Python repository
You will find that getrandbits under the hood calls urandom from os module, and if you look at the implementation of this function, you'll find that it is indeed written in C.
os_urandom_impl(PyObject *module, Py_ssize_t size)
/*[clinic end generated code: output=42c5cca9d18068e9 input=4067cdb1b6776c29]*/
{
PyObject *bytes;
int result;
if (size < 0)
return PyErr_Format(PyExc_ValueError,
"negative argument not allowed");
bytes = PyBytes_FromStringAndSize(NULL, size);
if (bytes == NULL)
return NULL;
result = _PyOS_URandom(PyBytes_AS_STRING(bytes), PyBytes_GET_SIZE(bytes));
if (result == -1) {
Py_DECREF(bytes);
return NULL;
}
return bytes;
}
Talking about randint, if you look at the source code for this function, it calls randrange, which further calls _randbelow_with_get_randbits.
def randrange(self, start, stop=None, step=_ONE):
"""Choose a random item from range(start, stop[, step]).
This fixes the problem with randint() which includes the
endpoint; in Python this is usually not what you want.
"""
# This code is a bit messy to make it fast for the
# common case while still doing adequate error checking.
try:
istart = _index(start)
except TypeError:
istart = int(start)
if istart != start:
_warn('randrange() will raise TypeError in the future',
DeprecationWarning, 2)
raise ValueError("non-integer arg 1 for randrange()")
_warn('non-integer arguments to randrange() have been deprecated '
'since Python 3.10 and will be removed in a subsequent '
'version',
DeprecationWarning, 2)
if stop is None:
# We don't check for "step != 1" because it hasn't been
# type checked and converted to an integer yet.
if step is not _ONE:
raise TypeError('Missing a non-None stop argument')
if istart > 0:
return self._randbelow(istart)
raise ValueError("empty range for randrange()")
# stop argument supplied.
try:
istop = _index(stop)
except TypeError:
istop = int(stop)
if istop != stop:
_warn('randrange() will raise TypeError in the future',
DeprecationWarning, 2)
raise ValueError("non-integer stop for randrange()")
_warn('non-integer arguments to randrange() have been deprecated '
'since Python 3.10 and will be removed in a subsequent '
'version',
DeprecationWarning, 2)
width = istop - istart
try:
istep = _index(step)
except TypeError:
istep = int(step)
if istep != step:
_warn('randrange() will raise TypeError in the future',
DeprecationWarning, 2)
raise ValueError("non-integer step for randrange()")
_warn('non-integer arguments to randrange() have been deprecated '
'since Python 3.10 and will be removed in a subsequent '
'version',
DeprecationWarning, 2)
# Fast path.
if istep == 1:
if width > 0:
return istart + self._randbelow(width)
raise ValueError("empty range for randrange() (%d, %d, %d)" % (istart, istop, width))
# Non-unit step argument supplied.
if istep > 0:
n = (width + istep - 1) // istep
elif istep < 0:
n = (width + istep + 1) // istep
else:
raise ValueError("zero step for randrange()")
if n <= 0:
raise ValueError("empty range for randrange()")
return istart + istep * self._randbelow(n)
def randint(self, a, b):
"""Return random integer in range [a, b], including both end points.
"""
return self.randrange(a, b+1)
As you can see, there are so many things going on with Try/Except block in randrange function and as they are written in Python, due to all these overheads and control flow, it's slow.

This is the code that gets executed when you call randint:
#!/usr/bin/python3
import random
random.randint(0, 1023)
$ python3 -m trace -t s.py
zen importlib._bootstrap>(194): s.py(4): random.randint(0, 1023)
--- modulename: random, funcname: randint
random.py(339): return self.randrange(a, b+1)
--- modulename: random, funcname: randrange
random.py(301): istart = int(start)
random.py(302): if istart != start:
random.py(304): if stop is None:
random.py(310): istop = int(stop)
random.py(311): if istop != stop:
random.py(313): width = istop - istart
random.py(314): if step == 1 and width > 0:
random.py(315): return istart + self._randbelow(width)
--- modulename: random, funcname: _randbelow_with_getrandbits
random.py(241): if not n:
random.py(243): getrandbits = self.getrandbits
random.py(244): k = n.bit_length() # don't use (n-1) here because n can be 1
random.py(245): r = getrandbits(k) # 0 <= r < 2**k
random.py(246): while r >= n:
random.py(248): return r
So the answer is that there's some extra boilerplate that ends up calling getrandbits, which you call directly otherwise. Note also that calling _randbelow_with_getrandbits with a power of two exhibits the worst case behavior: It will sample getrandbits(11) until it gets a number with 10 bits, so it will discard half of the values (this is not the main reason why it is slower, but adding the rejection sampling logic to your calls to getrandbits make them ~twice as slow).
Some of the answers say that getrandbits end up calling os.urandom. That is not true.
The regular getrandbits function is implemented directly in C in _random_Random_getrandbits_impl. The one that calls os.urandom is from SystemRandom.
You can check it by using the trace module:
#!/usr/bin/python3
import random
r = random.SystemRandom()
r.getrandbits(5)
random.getrandbits(5)
$ python3 -m trace -t s.py
...
<frozen importlib._bootstrap>(186): <frozen importlib._bootstrap>(187): <frozen importlib._bootstrap>(191): <frozen importlib._bootstrap>(192): <frozen importlib._bootstrap>(194): s.py(4): r = random.SystemRandom()
--- modulename: random, funcname: __init__
random.py(123): self.seed(x)
--- modulename: random, funcname: seed
random.py(800): return None
random.py(124): self.gauss_next = None
s.py(6): r.getrandbits(5)
--- modulename: random, funcname: getrandbits
random.py(786): if k < 0:
random.py(788): numbytes = (k + 7) // 8 # bits / 8 and rounded up
random.py(789): x = int.from_bytes(_urandom(numbytes), 'big')
random.py(790): return x >> (numbytes * 8 - k) # trim excess bits
s.py(8): random.getrandbits(5)

Is it possible to update funcion variables every time it's been called?

I'm trying to solve an ODE system with solve_ivp and i want to change the local variables of the function every time it's been called by the solver.
In particular I wand to update the lagrange multipliers (lambdas_i) so that the next time step of solve_ivp, uses the values of the previous one.
(The ''reconstruct'' function is from a python module that uses a method to reconstruct a size distribution from given moments)
Is there a way to do this? I'll post the code below:
import time
import numpy as np
import scipy.integrate as integrate
from numpy import sqrt, sin, cos, pi
import math
import pylab as pp
from pymaxent import reconstruct
start = time.time()
'''Initialize variables'''
t=np.linspace(0, 60,31)
tspan=(0,60)
initial_m=[]
for i in range(4):
def distr(L,i=i):
return (L**i)*0.0399*np.exp(-((L-50)**2)/200)
m, err=integrate.quad(distr, 0, np.inf)
print('m(',i,')=',m)
initial_m.append(m)
''' Solving ode system using Maximum Entropy, G(L)=1+0.002*L'''
def moments(t,y):
m0 = y[0]
m1 = y[1]
m2 = y[2]
m3 = y[3]
Lmean=m1
σ=(m2-Lmean**2)**(1/2)
Lmin=Lmean-3*σ
Lmax=Lmean+4*σ
bnds=[Lmin,Lmax]
L=np.linspace(Lmin,Lmax)
sol, lambdas_i= reconstruct(mu=y ,bnds=bnds)
print(lambdas_i)
dm0dt=0
def moment1(L):
return(sol(L)+0.002*sol(L)*L)
dm1dt, err1=integrate.quad(moment1,Lmin,Lmax)
def moment2(L):
return(2*L*(sol(L)+0.002*sol(L)*L))
dm2dt, err2=integrate.quad(moment2,Lmin,Lmax)
def moment3(L):
return(3*L**2*(sol(L)+0.002*sol(L)*L))
dm3dt, err3=integrate.quad(moment3,Lmin,Lmax)
return(dm0dt,dm1dt,dm2dt,dm3dt)
'''Χρήση της BDF, step by step'''
r=integrate.solve_ivp(moments,tspan,initial_m,method='BDF',jac=None,t_eval=t,rtol=10**(-3))
end = time.time()
print('Total time =',{end-start})

Here is one way to achieve what you want. I won't be using your actual code, but using a simpler example problem, and you can use the same strategy to solve yours.
def seq():
l = [1, 0]
while True:
p = l[1]
n = l[0] + p
l = [p, n]
print(n)
input()
Above is an example function that would print the next (or first) term of the fibonacci sequence. (In which each term is the sum of the two previous terms.) In this case, the input is solely used to pause between each iteration (as it is infinite). Now to transform this into a generator, to allow more flexibility, you could rewrite the function as:
def seq():
l = [1, 0]
while True:
p = l[1]
n = l[0] + p
l = [p, n]
yield n
Now, if you wanted to get the same result, you could:
for item in seq():
print(item)
input()
However, this is mostly useless. The point of the generator comes in when you want to gather the next number of the sequence, but at any point of the code. Not necessarily inside the loop that repeats itself until you're done. You could achieve this:
gen = seq()
next(gen) # returns 1
next(gen) # returns 1
next(gen) # returns 2
next(gen) # returns 3
next(gen) # returns 5
And so on...
Another way to solve this problem would be to use a global variable instead of a local variable.
l = [1, 0]
def seq():
p = l[1]
n = l[0] + p
l[0] = p
l[1] = n
return n
The variable l is defined outside the function, not inside, so it will not be discarded when the function exits. (To run the code the same way as before:)
while True:
print(seq())
input()
Both of these should be implementable in your code.

Get the size of a conditional subset of a list

Assume that you have a list with an arbitrary amounts of items, and you wish to get the number of items that match a specific conditions. I though of two ways to do this in a sensible manner but I am not sure which one is best (more pythonic) - or if there is perhaps a better option (without sacrificing too much readability).
import numpy.random as nprnd
import timeit
my = nprnd.randint(1000, size=1000000)
def with_len(my_list):
much = len([t for t in my_list if t >= 500])
def with_sum(my_list):
many = sum(1 for t in my_list if t >= 500)
t1 = timeit.Timer('with_len(my)', 'from __main__ import with_len, my')
t2 = timeit.Timer('with_sum(my)', 'from __main__ import with_sum, my')
print("with len:",t1.timeit(1000)/1000)
print("with sum:",t2.timeit(1000)/1000)
Performance is almost identical between these two cases. However, which of these is more pythonic? Or is there a better alternative?
For those who are curious, I tested the proposed solutions (from comments and answers) and these are the results:
import numpy as np
import timeit
import functools
my = np.random.randint(1000, size=100000)
def with_len(my_list):
return len([t for t in my_list if t >= 500])
def with_sum(my_list):
return sum(1 for t in my_list if t >= 500)
def with_sum_alt(my_list):
return sum(t >= 500 for t in my_list)
def with_lambda(my_list):
return functools.reduce(lambda a, b: a + (1 if b >= 500 else 0), my_list, 0)
def with_np(my_list):
return len(np.where(my_list>=500)[0])
t1 = timeit.Timer('with_len(my)', 'from __main__ import with_len, my')
t2 = timeit.Timer('with_sum(my)', 'from __main__ import with_sum, my')
t3 = timeit.Timer('with_sum_alt(my)', 'from __main__ import with_sum_alt, my')
t4 = timeit.Timer('with_lambda(my)', 'from __main__ import with_lambda, my')
t5 = timeit.Timer('with_np(my)', 'from __main__ import with_np, my')
print("with len:", t1.timeit(1000)/1000)
print("with sum:", t2.timeit(1000)/1000)
print("with sum_alt:", t3.timeit(1000)/1000)
print("with lambda:", t4.timeit(1000)/1000)
print("with np:", t5.timeit(1000)/1000)
Python 2.7
('with len:', 0.02201753337348283)
('with sum:', 0.022727363518455238)
('with sum_alt:', 0.2370256687439941) # <-- very slow!
('with lambda:', 0.026367264818657078)
('with np:', 0.0005811764306089913) # <-- very fast!
Python 3.6
with len: 0.017649643657480736
with sum: 0.0182978007766851
with sum_alt: 0.19659815740239048
with lambda: 0.02691670741400111
with np: 0.000534095418615152

The 2nd one, with_sum is more pythonic in the sense that it uses much less memory as it doesn't build the whole list because the generator expression is fed to sum().

I'm with #Chris_Rands. But as far as performance is concerned, there is a faster way using numpy:
import numpy as np
def with_np(my_list):
return len(np.where(my_list>=500)[0])

Explanation for tail(?) recursion when calling the method twice

Let's have the following code:
import time
def rec1(len):
if( len < 2): return 1;
return 2*rec1(len-1);
def rec2(len):
if( len < 2): return 1;
return rec2(len-1) + rec2(len-1);
def callAndReport(len, method):
time1 = time.time()
answer = method(len)
time2 = time.time()
print("{0} with {1}:{2} in {3:.0f} ms".format(len,method.__name__,answer, (time2-time1)*1000))
if __name__ == '__main__':
callAndReport(20,rec1)
callAndReport(20,rec2)
print('')
callAndReport(23,rec1)
callAndReport(23,rec2)
This code produces the following output:
20 with rec1:524288 in 0 ms
20 with rec2:524288 in 642 ms
23 with rec1:4194304 in 0 ms
23 with rec2:4194304 in 4613 ms
Could someone explain the execution time difference? I have few ideas but I would like to be sure.
For completeness, the original problem I had was the method below (which can be easily express as a for loop, but that's not the point here):
def find11s_rec(len):
if len<2: return 0
if len== 2: return 1;
return find11s_rec(len-2)+find11s_rec(len-1)+2**(len-2)
.

That's because while rec1 only uses rec1 once, rec2 uses rec2 twice. And then those inner rec2 calls will each call rec2 twice. The number of function calls will rise exponentially. While rec1 might use x calls, rec2 would use 2^x calls. In computer science terms, rec1 is O(x) while rec2 is O(2^x). In more complicated cases, dangerous recursion might inobvious; so use a debugger to find out what is actually being done.

The complexity of rec1 is O(n) and the complexity of rec2 is O(2^n). That's a big performance difference.
rec2(n) = rec2(n-1) + rec2(n-1)
= (rec2(n-2) + rec2(n-2)) + (rec2(n-2) + rec2(n-2)) = 4 * rec2(n-2)
...
rec2(n) = (2^n)*rec2(1)
= O(2^n)

Help me finish this Python 3.x self-challenge

This is not homework.
I saw this article praising Linq library and how great it is for doing combinatorics stuff, and I thought to myself: Python can do it in a more readable fashion.
After half hour of dabbing with Python I failed. Please finish where I left off. Also, do it in the most Pythonic and efficient way possible please.
from itertools import permutations
from operator import mul
from functools import reduce
glob_lst = []
def divisible(n): return (sum(j*10^i for i,j in enumerate(reversed(glob_lst))) % n == 0)
oneToNine = list(range(1, 10))
twoToNine = oneToNine[1:]
for perm in permutations(oneToNine, 9):
for n in twoToNine:
glob_lst = perm[1:n]
#print(glob_lst)
if not divisible(n):
continue
else:
# Is invoked if the loop succeeds
# So, we found the number
print(perm)
Thanks!

Here's a short solution, using itertools.permutations:
from itertools import permutations
def is_solution(seq):
return all(int(seq[:i]) % i == 0 for i in range(2, 9))
for p in permutations('123456789'):
seq = ''.join(p)
if is_solution(seq):
print(seq)
I've deliberately omitted the divisibility checks by 1 and by 9, since they'll always be satisfied.

Here's my solution. I like all things bottom-up ;-). On my machine it runs about 580 times faster (3.1 msecs vs. 1.8 secs) than Marks:
def generate(digits, remaining=set('123456789').difference):
return (n + m
for n in generate(digits - 1)
for m in remaining(n)
if int(n + m) % digits == 0) if digits > 0 else ['']
for each in generate(9):
print(int(each))
EDIT: Also, this works, and twice as fast (1.6 msecs):
from functools import reduce
def generate():
def digits(x):
while x:
x, y = divmod(x, 10)
yield y
remaining = set(range(1, 10)).difference
def gen(numbers, decimal_place):
for n in numbers:
for m in remaining(digits(n)):
number = 10 * n + m
if number % decimal_place == 0:
yield number
return reduce(gen, range(2, 10), remaining())
for each in generate():
print(int(each))

Here's my solution (not as elegant as Mark's, but it still works):
from itertools import permutations
for perm in permutations('123456789'):
isgood = 1
for i in xrange(9):
if(int(''.join(perm[:9-i])) % (9-i)):
isgood = 0
break
if isgood:
print ''.join(perm)

this is my solution, it is very similar to Marks, but it runs about twice as fast
from itertools import permutations
def is_solution(seq):
if seq[-1]=='9':
for i in range(8,1,-1):
n = -(9-i)
if eval(seq[:n]+'%'+str(i))==0:
continue
else:return False
return True
else:return False
for p in permutations('123456789'):
seq = ''.join(p)
if is_solution(seq):
print(seq)