I'm in the process of working on programming project that involves some pretty extensive Monte Carlo simulation in Python, and as such the generation of a tremendous number of random numbers. Very nearly all of them, if not all of them, will be able to be generated by Python's built in random module.
I'm something of a coding newbie, and unfamiliar with efficient and inefficient ways to do things. Is it faster to generate say, all the random numbers as a list, and then iterate through that list, or generate a new random number each time a function is called, which will be in a very large loop?
Or some other, undoubtedly more clever method?
Generate a random number each time. Since the inner workings of the loop only care about a single random number, generate and use it inside the loop.
Example:
# do this:
import random
for x in xrange(SOMEVERYLARGENUMBER):
n = random.randint(1,1000) # whatever your range of random numbers is
# Do stuff with n
# don't do this:
import random
# This list comprehension generates random numbers in a list
numbers = [random.randint(1,1000) for x in xrange(SOMEVERYLARGENUMBER)]
for n in numbers:
# Do stuff with n
Obviously, in practical terms it really doesn't matter, unless you're dealing with billions and billions of iterations, but why bother generating all those numbers if you're only going to be using one at a time?
import random
for x in (random.randint(0,80) for x in xrange(1000*1000)):
print x
The code between parentheses will only generate one item at a time, so it's memory safe.
Python builtin random module, e.g. random.random(), random.randint(), (some distributions also available, you probably want gaussian) does about 300K samples/s.
Since you are doing numerical computation, you probably use numpy anyway, that offers better performance if you cook random number one array at a time instead of one number at a time and wider choice of distributions. 60K/s * 1024 (array length), that's ~60M samples/s.
You can also read /dev/urandom on Linux and OSX. my hw/sw (osx laptop) manages ~10MB/s.
Surely there must be faster ways to generate random numbers en masse, e.g.:
from Crypto.Cipher import AES
from Crypto.Util import Counter
import secrets
aes = AES.new(secrets.token_bytes(16), AES.MODE_CTR, secrets.token_bytes(16), counter=Counter.new(128))
data = "0" * 2 ** 20
with open("filler.bin", "wb") as f:
while True:
f.write(aes.encrypt(data))
This generates 200MB/s on a single core of i5-4670K
Common ciphers like aes and blowfish manage 112MB/s and 70MB/s on my stack. Furthermore modern processors make aes even faster up to some 700MB/s see this link to test runs on few hardware combinations. (edit: link broken). You could use weaker ECB mode, provided you feed distinct inputs into it, and achieve up to 3GB/s.
Stream cipher are better suited for the task, e.g. RC4 tops out at 300MB/s on my hardware, you may get best results from most popular ciphers as more effort was spent optimising those both and software.
Code to generate 10M random numbers efficiently and faster:
import random
l=10000000
listrandom=[]
for i in range (l):
value=random.randint(0,l)
listrandom.append(value)
print listrandom
Time taken included the I/O time lagged in printing on screen:
real 0m27.116s
user 0m24.391s
sys 0m0.819s
Using Numpy -
import numpy as np
np.random.randint(low="put the range like 1 to 100, so put '1' in
low",high="100",size="1000000")
Related
So I am trying to find a way to get a massively improbable condition based on random generations. To better explain, here's an example:
from random import *
import ctypes
random1 = [randint(0, 2 ** 32 - 1) for j in range(10000000)]
while True:
random2 = [randint(0, 2 ** 32 - 1) for i in range(10000000)]
if set(random2) == set(random1):
MessageBox = ctypes.windll.user32.MessageBoxW
MessageBox(None, 'Match', 'Output', 0)
break
Because of the limitations and functionality of mersene twister, and the uniformity of its distribution of numbers, it's quite likely we will generate 10 milion numbers in both lists, where when order doesn't matter and duplicates are removed, they will match quite often.
This is not that rare, but the following code is slightly better:
from random import *
import ctypes
while True:
if random() == 0.0:
MessageBox = ctypes.windll.user32.MessageBoxW
MessageBox(None, 'Match', 'Output', 0)
break
This is a lot less likely to happen, but with massive single core performance, quite common today, it's still easily likely to get a match in 1~ day. The probability being 1/2^56, and with the limitations of mersenne twister in mind, it's not that unlikely to happen.
Is there a good way to write a condition utilizing randomness in python that will truly be extremely unlikely to happen?.. That would say, take a year to crack, or more.
Alternatively I thought to turn to hash matching... creating a random SHA256 hash, then generating random big data, and hashing it through sha256 to try and match the hash. But I don't know a way to observe probability in that case.
You may be interested in the geometric distribution, which counts the number of failures before the first success (some works say it counts that number plus the first success instead). As an example, the probability of getting no failures in a row is 1/2, one failure in a row is 1/4, two in a row is 1/8, three in a row is 1/16, and so on. If we take a zero-bit to mean failure and a one-bit to mean success, that means that with more zero-bits, it becomes less probable for that many zero-bits to be generated at random. As an example of an "improbable event", you can treat 30 or more zero-bits in a row as improbable.
Mersenne Twister and pseudorandom number generators (PRNGs) in general have cycles. The size of this cycle affects how many zero-bits the PRNG can generate in a row. For example, Mersenne Twister has a cycle of 2^19937 - 1 numbers, so that in theory, it can cycle through all states except for the all-zeros state. Thus, it can generate no more than 19937 * 2 zero-bits in a row. (This is if we treat Mersenne Twister as outputting individual bits, rather than 32 bits, at a time.)
This is in contrast to nondeterministic random number generators (RNGs), which don't have cycles but still generate random-behaving numbers. If the numbers it generates are independent, uniform, and random bits, then there is no telling how many zero-bits the RNG can randomly generate at most. One example of an RNG that uses nondeterminism is found in Python's secrets module, specifically secrets.randbelow(). (In practice, this module is likely to use a PRNG, but may gather "entropy" from nondeterministic sources from time to time, so that in practice the module's RNG is nondeterministic.)
I'm using Python for a project, and for part of that project, I need to get a scrambled permutation of a library of strings that I feed into the method. I want to get a replicable result out, so I am currently using the permutation method of the itertools library and then looping through it until I reach a certain permutation. However, looping through a bunch of times is obviously not the best for time, especially with scrambles with more elements (20+). Is there anyone who has worked with itertools before who can tell me how to find the scramble faster, or is there someone who can give me a better method for scrambling?
Code Below VVV
import itertools
from itertools import permutations
def genkey(library, toget):
scrambled = permutations(library)
iternum = 0
for fin in scrambled:
if iternum == toget:
return(list(fin))
else:
iternum += 1
print('Permutation out of range')
return
My main aim is just to find a way that I can get a replicable scramble that is seemingly random but can be replicated based on an int that I fed in. I'm open to any suggestions.
It sounds like you can use the random module:
import random
random.seed(1) # Choose another number here if you want a different outcome
scrambled = random.sample(library, k=len(library))
Everytime this is run with the same parameter to seed() you should get the same value for scrambled.
this is my first post here, so i'm sorry if i didn't follow the rules
i recently learned python, i know the basics and i like writing famous sets and plot them, i've wrote codes for the hofstadter sequence, a logistic sequence and succeeded in both
now i've tried writing mandelbrot's sequence without any complex parameters, but actually doing it "by hand"
for exemple if Z(n) is my complexe(x+iy) variable and C(n) my complexe number (c+ik)
i write the sequence as {x(n)=x(n-1)^2-y(n-1)^2+c ; y(n)=2.x(n-1).y(n-1)+c}
from math import *
import matplotlib.pyplot as plt
def mandel(p,u):
c=5
k=5
for i in range(p):
c=5
k=k-10/p
for n in range(p):
c=c-10/p
x=0
y=0
for m in range (u):
x=x*x-y*y + c
y=2*x*y + k
if sqrt(x*x+y*y)>2:
break
if sqrt(x*x+y*y)<2:
X=X+[c]
Y=Y+[k]
print (round((i/p)*100),"%")
return (plt.plot(X,Y,'.')),(plt.show())
p is the width and number of complexe parameters i want, u is the number of iterations
this is what i get as a result :
i think it's just a bit close to what i want.
now for my questions, how can i make the function faster? and how can i make it better ?
thanks a lot !
A good place to start would be to profile your code.
https://docs.python.org/2/library/profile.html
Using the cProfile module or the command line profiler, you can find the inefficient parts of your code and try to optimize them. If I had to guess without personally profiling it, your array appending is probably inefficient.
You can either use a numpy array that is premade at an appropriate size, or in pure python you can make an array with a given size (like 50) and work through that entire array. When it fills up, append that array to your main array. This reduces the number of times the array has to be rebuilt. The same could be done with a numpy array.
Quick things you could do though
if sqrt(x*x+y*y)>2:
should become this
if x*x+y*y>4:
Remove calls to sqrt if you can, its faster to just exponentiate the other side by 2. Multiplication is cheaper than finding roots.
Another thing you could do is this.
print (round((i/p)*100),"%")
should become this
# print (round((i/p)*100),"%")
You want faster code?...remove things not related to actually plotting it.
Also, you break a for loop after a comparison then make the same comparison...Do what you want to after the comparison and then break it...No need to compute that twice.
I need to create a large bytearry of a specific size but the size is not known prior to run time. The bytes need to be fairly random. The bytearray size may be as small as a few KBs but as large as a several MB. I do not want to iterate byte-by-byte. This is too slow -- I need performance similar to numpy.random. However, I do not have the numpy module available for this project. Is there something part of a standard python install that will do this? Or do i need to compile my own using C?
for those asking for timings:
>>> timeit.timeit('[random.randint(0,128) for i in xrange(1,100000)]',setup='import random', number=100)
35.73110193696641
>>> timeit.timeit('numpy.random.random_integers(0,128,100000)',setup='import numpy', number=100)
0.5785652013481126
>>>
The os module provides urandom, even on Windows:
bytearray(os.urandom(1000000))
This seems to perform as quickly as you need, in fact, I get better timings than your numpy (though our machines could be wildly different):
timeit.timeit(lambda:bytearray(os.urandom(1000000)), number=10)
0.0554857286941
There are several possibilities, some faster than os.urandom. Also consider whether the data has to be generated deterministically from a random seed. This is invaluable for unit tests where failures have to be reproducible.
short and pithy:
lambda n:bytearray(map(random.getrandbits,(8,)*n))
I've use the above for unit tests and it was fast enough but can it be done faster?
using itertools:
lambda n:bytearray(itertools.imap(random.getrandbits,itertools.repeat(8,n))))
itertools and struct producing 8 bytes per iteration
lambda n:(b''.join(map(struct.Struct("!Q").pack,itertools.imap(
random.getrandbits,itertools.repeat(64,(n+7)//8)))))[:n]
Anything based on b''.join will fill 3-7x the memory consumed by the final bytearray with temporary objects since it queues up all the sub-strings before joining them together and python objects have lots of storage overhead.
Producing large chunks with a specialized function gives better performance and avoids filling memory.
import random,itertools,struct,operator
def randbytes(n,_struct8k=struct.Struct("!1000Q").pack_into):
if n<8000:
longs=(n+7)//8
return struct.pack("!%iQ"%longs,*map(
random.getrandbits,itertools.repeat(64,longs)))[:n]
data=bytearray(n);
for offset in xrange(0,n-7999,8000):
_struct8k(data,offset,
*map(random.getrandbits,itertools.repeat(64,1000)))
offset+=8000
data[offset:]=randbytes(n-offset)
return data
Performance
.84 MB/s :original solution with randint:
4.8 MB/s :bytearray(getrandbits(8) for _ in xrange(n)): (solution by other poster)
6.4MB/s :bytearray(map(getrandbits,(8,)*n))
7.2 MB/s :itertools and getrandbits
10 MB/s :os.urandom
23 MB/s :itertools and struct
35 MB/s :optimised function (holds for len = 100MB ... 1KB)
Note:all tests used 10KB as the string size. Results were consistent up till intermediate results filled memory.
Note:os.urandom is meant to provide secure random seeds. Applications expand that seed with their own fast PRNG. Here's an example, using AES in counter mode as a PRNG:
import os
seed=os.urandom(32)
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
from cryptography.hazmat.backends import default_backend
backend = default_backend()
cipher = Cipher(algorithms.AES(seed), modes.CTR(b'\0'*16), backend=backend)
encryptor = cipher.encryptor()
nulls=b'\0'*(10**5) #100k
from timeit import timeit
t=timeit(lambda:encryptor.update(nulls),number=10**5) #1GB, (100K*10k)
print("%.1f MB/s"%(1000/t))
This produces pseudorandom data at 180 MB/s. (no hardware AES acceleration, single core) That's only ~5x the speed of the pure python code above.
Addendum
There's a pure python crypto library waiting to be written. Putting the above techniques together with hashlib and stream cipher techniques looks promising. Here's a teaser, a fast string xor (42MB/s).
def xor(a,b):
s="!%iQ%iB"%divmod(len(a),8)
return struct.pack(s,*itertools.imap(operator.xor,
struct.unpack(s,a),
struct.unpack(s,b)))
What's wrong with just including numpy? Anyhow, this creates a random N-bit integer:
import random
N = 100000
bits = random.getrandbits(N)
So if you needed to see if the value of the j-th bit is set or not, you can do bits & (2**j)==(2**j)
EDIT: He asked for byte array not bit array. Ned's answer is better: your_byte_array= bytearray((random.getrandbits(8) for i in xrange(N))
import random
def randbytes(n):
for _ in xrange(n):
yield random.getrandbits(8)
my_random_bytes = bytearray(randbytes(1000000))
There's probably something in itertools that could help here, there always is...
My timings indicate that this goes about five times faster than [random.randint(0,128) for i in xrange(1,100000)]
I need to run a MonteCarlo simulations in parallel on different machines. The code is in c++, but the program is set up and launched with a python script that set a lot of things, in particular the random seed. The function setseed thake a 4 bytes unsigned integer
Using a simple
import time
setseed(int(time.time()))
is not very good because I submit the jobs to a queue on a cluster, they remain pending for some minutes then they starts, but the start time is impredicible, it can be that two jobs start at the same time (seconds), so I switch to:
setseet(int(time.time()*100))
but I'm not happy. What is the best solution? Maybe I can combine information from: time, machine id, process id. Or maybe the best solution is to read from /dev/random (linux machines)?
How to read 4 bytes from /dev/random?
f = open("/dev/random","rb")
f.read(4)
give me a string, I want an integer!
Reading from /dev/random is a good idea. Just convert the 4 byte string into an Integer:
f = open("/dev/random","rb")
rnd_str = f.read(4)
Either using struct:
import struct
rand_int = struct.unpack('I', rnd_string)[0]
Update Uppercase I is needed.
Or multiply and add:
rand_int = 0
for c in rnd_str:
rand_int <<= 8
rand_int += ord(c)
You could simply copy over the four bytes into an integer, that should be the least of your worries.
But parallel pseudo-random number generation is a rather complex topic and very often not done well. Usually you generate seeds on one machine and distribute them to the others.
Take a look at SPRNG, which handles exactly your problem.
If this is Linux or a similar OS, you want /dev/urandom -- it always produces data immediately.
/dev/random may stall waiting for the system to gather randomness. It does produce cryptographic-grade random numbers, but that is overkill for your problem.
You can use a random number as the seed, which has the advantage of being operating-system agnostic (no /dev/random needed), with no conversion from string to int:
Why not simply use
random.randrange(-2**31, 2**31)
as the seed of each process? Slightly different starting times give wildly different seeds, this way…
You could also alternatively use the random.jumpahead method, if you know roughly how many random numbers each process is going to use (the documentation of random.WichmannHill.jumpahead is useful).