I need to generate a controlled sequence of pseudo-random numbers, given an initial parameter. For that I'm using the standard python random generator, seeded by this parameter. I'd like to make sure that I will generate the same sequence across systems (Operating system, but also Python version).
In summary: Does python ensure the reproducibility / portability of it's pseudo-random number generator across implementation and versions?
No, it doesn't. There's no such promise in the random module's documentation.
What the docs do contain is this remark:
Changed in version 2.3: MersenneTwister replaced Wichmann-Hill as the default generator
So a different RNG was used prior to Python 2.3.
So far, I've been using numpy.random.RandomState for reproducible pseudo-randomness, though it too does not make the formal promise you're after.
If you want full reproducibility, you might want to include a copy of random's source in your program, or hack together a "P²RNG" (pseudo-pseudo-RNG) from hashlib.
Not necessarily.
As described in the documentation, the random module has used the Mersenne twister to generate random numbers since version 2.3, but used Wichmann-Hill before that.
(If a seed is not provided, the method of obtaining the seed also does depend on the operating system, the Python version, and factors such as the system time).
#reubano - 3.2 changed the integer functions in random, to produce more evenly distributed (which inevitably means different) output.
That change was discussed in Issue9025, where the team discuss whether they have an obligation to stick to the previous output, even when it was defective. They conclude that they do not. The docs for the module guarantee consistency for random.random() - one might assume that the functions which call it (like random.randrange()) are implicitly covered under that guarantee, but that doesn't seem to be the case.
Just as a heads up: in addition to the 2.3 change, python 3 gives numbers from python 2.x from randrange and probably other functions, even if the numbers from random.random are similar.
I just found out that there is also a difference between python3.7 and python3.8.
The following code behaves the same
from random import Random
seed = 317
rand = Random(seed)
rand.getrandbits(64)
but if you use from _random import Random instead, it behaves differently.
Related
I'm looking for more information about random. Primarily what was the seed and how many steps have happened in that pseudo random sequence & Was that seed a datetime, i.e. did random use the system provided random or default to time?
I looked at dir(random) and didn't see anything super promising...
The documentation states:
Almost all module functions depend on the basic function random(),
which generates a random float uniformly in the semi-open range [0.0,
1.0). Python uses the Mersenne Twister as the core generator. It produces 53-bit precision floats and has a period of 2**19937-1. The
underlying implementation in C is both fast and threadsafe. The
Mersenne Twister is one of the most extensively tested random number
generators in existence. However, being completely deterministic, it
is not suitable for all purposes, and is completely unsuitable for
cryptographic purposes.
You can study the source code here, in particular the default implementation "seeds from current time or from an operating system specific randomness source if available"
From the docs:
random.seed(a=None, version=2) Initialize the random number generator.
If a is omitted or None, the current system time is used. If
randomness sources are provided by the operating system, they are used
instead of the system time (see the os.urandom() function for details
on availability).
But...if it's truly random...(and I thought I read it uses Mersenne, so it's VERY random)...what's the point in seeding it? Either way the outcome is unpredictable...right?
The default is probably best if you want different random numbers with each run. If for some reason you need repeatable random numbers, in testing for instance, use a seed.
The module actually seeds the generator (with OS-provided random data from urandom if possible, otherwise with the current date and time) when you import the module, so there's no need to manually call seed().
(This is mentioned in the Python 2.7 documentation but, for some reason, not the 3.x documentation. However, I confirmed in the 3.x source that it's still done.)
If the automatic seeding weren't done, you'd get the same sequence of numbers every time you started your program, same as if you manually use the same seed every time.
But...if it's truly random
No, it's pseudo random. If it uses Mersenne Twister, that too is a PRNG.
It's basically an algorithm that generates the exact same sequence of pseudo random numbers out of a given seed. Generating truly random numbers requires special hardware support, it's not something you can do by a pure algorithm.
You might not need to seed it since it seeds itself on first use, unless you have some other or better means of providing a seed than what is time based.
If you use the random numbers for things that are not security related, a time based seed is normally fine. If you use if for security/cryptography, note what the docs say: "and is completely unsuitable for cryptographic purposes"
If you want to reproduce your results, you seed the generator with a known value so you get the same sequence every time.
A Mersenne twister, the random number generator, used by Python is seeded by the operating system served random numbers by default on those platforms where it is possible (Unixen, Windows); however on other platforms the seed defaults to the system time which means very repeatable values if the system time has a bad precision. On such systems seeding with known better random values is thus beneficial. Note that on Python 3 specifically, if version 2 is used, you can pass in any str, bytes, or bytearray to seed the generator; thus taking use of the Mersenne twister's large state better.
Another reason to use a seed value is indeed to guarantee that you get the same sequence of random numbers again and again - by reusing the known seed. Quoting the docs:
Sometimes it is useful to be able to reproduce the sequences given by
a pseudo random number generator. By re-using a seed value, the same
sequence should be reproducible from run to run as long as multiple
threads are not running.
Most of the random module’s algorithms and seeding functions are
subject to change across Python versions, but two aspects are
guaranteed not to change:
If a new seeding method is added, then a backward compatible seeder will be offered.
The generator’s random() method will continue to produce the same sequence when the compatible seeder is given the same seed.
For this however, you mostly want to use the random.Random instances instead of using module global methods (the multiple threads issue, etc).
Finally also note that the random numbers produced by Mersenne twister are unsuitable for cryptographical use; whereas they appear very random, it is possible to fully recover the internal state of the random generator by observing only some hundreds of values from the generator. For cryptographical algorithms, you want to use the SystemRandom class.
In most cases I would say there is no need to care about. But if someone is really willing to do something wired and (s)he could roughly figure out your system time when your code was running, they might be able to brute force replay your random numbers and see which series fits. But I would say this is quite unlikely in most cases.
I am trying to use a different core random generator in python. On reading random module docs I found that I have to override existing random,seed,getstate and setstate method. But I can't find random function in random .py module. I am using python 3.4.
Can you elaborate slightly? I don't have the reputation to comment yet, but when you say 'different core random generator' are you referring to changing the underlying structure? Python uses a Mersenne Twister as the core generator, which is robust and efficient.
In terms of just generating a random number, you can really just use
import random
my_random_number_between_0_and_1 = random.random()
my_random_number_between_0_and_1 # print out your new random number
You can also use random.seed() and if you do not pass in a seed, the current system time is used, which is pseudorandom as well.
I hope some of this helps! Please let me know if not. Apologies for not understanding entirely.
Here is what the docs say:
Class Random can also be subclassed if you want to use a different basic generator of your own devising: in that case, override the random(), seed(), getstate(), and setstate() methods. Optionally, a new generator can supply a getrandbits() method — this allows randrange() to produce selections over an arbitrarily large range.
This class is, as noted in the comments, implemented in C. GVR wrote a translation in py 3.2 that is not maintained but that you can use for reference regarding syntax, arguments, etc. An exemplary random function can be found on class SystemRandom.
From the docs:
random.seed(a=None, version=2) Initialize the random number generator.
If a is omitted or None, the current system time is used. If
randomness sources are provided by the operating system, they are used
instead of the system time (see the os.urandom() function for details
on availability).
But...if it's truly random...(and I thought I read it uses Mersenne, so it's VERY random)...what's the point in seeding it? Either way the outcome is unpredictable...right?
The default is probably best if you want different random numbers with each run. If for some reason you need repeatable random numbers, in testing for instance, use a seed.
The module actually seeds the generator (with OS-provided random data from urandom if possible, otherwise with the current date and time) when you import the module, so there's no need to manually call seed().
(This is mentioned in the Python 2.7 documentation but, for some reason, not the 3.x documentation. However, I confirmed in the 3.x source that it's still done.)
If the automatic seeding weren't done, you'd get the same sequence of numbers every time you started your program, same as if you manually use the same seed every time.
But...if it's truly random
No, it's pseudo random. If it uses Mersenne Twister, that too is a PRNG.
It's basically an algorithm that generates the exact same sequence of pseudo random numbers out of a given seed. Generating truly random numbers requires special hardware support, it's not something you can do by a pure algorithm.
You might not need to seed it since it seeds itself on first use, unless you have some other or better means of providing a seed than what is time based.
If you use the random numbers for things that are not security related, a time based seed is normally fine. If you use if for security/cryptography, note what the docs say: "and is completely unsuitable for cryptographic purposes"
If you want to reproduce your results, you seed the generator with a known value so you get the same sequence every time.
A Mersenne twister, the random number generator, used by Python is seeded by the operating system served random numbers by default on those platforms where it is possible (Unixen, Windows); however on other platforms the seed defaults to the system time which means very repeatable values if the system time has a bad precision. On such systems seeding with known better random values is thus beneficial. Note that on Python 3 specifically, if version 2 is used, you can pass in any str, bytes, or bytearray to seed the generator; thus taking use of the Mersenne twister's large state better.
Another reason to use a seed value is indeed to guarantee that you get the same sequence of random numbers again and again - by reusing the known seed. Quoting the docs:
Sometimes it is useful to be able to reproduce the sequences given by
a pseudo random number generator. By re-using a seed value, the same
sequence should be reproducible from run to run as long as multiple
threads are not running.
Most of the random module’s algorithms and seeding functions are
subject to change across Python versions, but two aspects are
guaranteed not to change:
If a new seeding method is added, then a backward compatible seeder will be offered.
The generator’s random() method will continue to produce the same sequence when the compatible seeder is given the same seed.
For this however, you mostly want to use the random.Random instances instead of using module global methods (the multiple threads issue, etc).
Finally also note that the random numbers produced by Mersenne twister are unsuitable for cryptographical use; whereas they appear very random, it is possible to fully recover the internal state of the random generator by observing only some hundreds of values from the generator. For cryptographical algorithms, you want to use the SystemRandom class.
In most cases I would say there is no need to care about. But if someone is really willing to do something wired and (s)he could roughly figure out your system time when your code was running, they might be able to brute force replay your random numbers and see which series fits. But I would say this is quite unlikely in most cases.
Simple enough question:
I'm using python random module to generate random integers. I want to know what is the suggested value to use with the random.seed() function? Currently I am letting this default to the current time, but this is not ideal. It seems like a string literal constant (similar to a password) would also not be ideal/strong
Suggestions?
Thanks,
-aj
UPDATE:
The reason I am generating random integers is for generation of test data. The numbers do not need to be reproducable.
According to the documentation for random.seed:
If x is omitted or None, current system time is used; current system time is also used to initialize the generator when the module is first imported. If randomness sources are provided by the operating system, they are used instead of the system time (see the os.urandom() function for details on availability).
If you don't pass something to seed, it will try to use operating-system provided randomness sources instead of the time, which is always a better bet. This saves you a bit of work, and is about as good as it's going to get. Regarding availability, the docs for os.urandom tell us:
On a UNIX-like system this will query /dev/urandom, and on Windows it will use CryptGenRandom.
Cross-platform random seeds are the big win here; you can safely omit a seed and trust that it will be random enough on almost every platform you'll use Python on. Even if Python falls back to the time, there's probably only a millisecond window (or less) to guess the seed. I don't think you'll run into any trouble using the current time anyway -- even then, it's only a fallback.
For most cases using current time is good enough. Occasionally you need to use a fixed number to generate pseudo random numbers for comparison purposes.
Setting the seed is for repeatability, not security. If anything, you make the system less secure by having a fixed seed than one that is constantly changing.
Perhaps it is not a problem in your case, but ont problem with using the system time as the seed is that someone who knows roughly when your system was started may be able to guess your seed (by trial) after seeing a few numbers from the sequence.
eg, don't use system time as the seed for your online poker game
If you are using random for generating test data I would like to suggest that reproducibility can be important.
Just think to an use case: for data set X you get some weird behaviour (eg crash). Turns out that data set X shows some feature that was not so apparent from the other data sets Y and Z and uncovers a bug which had escapend your test suites. Now knowing the seed is useful so that you can precisely reproduce the bug and you can fix it.