Pythonでのランダムデータの生成（ガイド）

>>> # Don't call `random.seed()` yet
>>> import random
>>> random.random()
0.35553263284394376
>>> random.random()
0.6101992345575074

>>> random.seed(444)
>>> random.random()
0.3088946587429545
>>> random.random()
0.01323751590501987

>>> random.seed(444)  # Re-seed
>>> random.random()
0.3088946587429545
>>> random.random()
0.01323751590501987

>>> random.randint(0, 10)
7
>>> random.randint(500, 50000)
18601

>>> random.randrange(1, 10)
5

>>> random.uniform(20, 30)
27.42639687016509
>>> random.uniform(30, 40)
36.33865802745107

>>> items = ['one', 'two', 'three', 'four', 'five']
>>> random.choice(items)
'four'

>>> random.choices(items, k=2)
['three', 'three']
>>> random.choices(items, k=3)
['three', 'five', 'four']

>>> random.sample(items, 4)
['one', 'five', 'four', 'three']

>>> random.shuffle(items)
>>> items
['four', 'three', 'two', 'one', 'five']

import string

def unique_strings(k: int, ntokens: int,
               pool: str=string.ascii_letters) -> set:
    """Generate a set of unique string tokens.

    k: Length of each token
    ntokens: Number of tokens
    pool: Iterable of characters to choose from

    For a highly optimized version:
    https://stackoverflow.com/a/48421303/7954504
    """

    seen = set()

    # An optimization for tightly-bound loops:
    # Bind these methods outside of a loop
    join = ''.join
    add = seen.add

    while len(seen) < ntokens:
        token = join(random.choices(pool, k=k))
        add(token)
    return seen

>>> unique_strings(k=4, ntokens=5)
{'AsMk', 'Cvmi', 'GIxv', 'HGsZ', 'eurU'}

>>> unique_strings(5, 4, string.printable)
{"'O*1!", '9Ien%', 'W=m7<', 'mUD|z'}

>>> [random.random() for _ in range(5)]
[0.021655420657909374,
 0.4031628347066195,
 0.6609991871223335,
 0.5854998250783767,
 0.42886606317322706]

>>> import numpy as np
>>> np.random.seed(444)
>>> np.set_printoptions(precision=2)  # Output decimal fmt.

>>> # Return samples from the standard normal distribution
>>> np.random.randn(5)
array([ 0.36,  0.38,  1.38,  1.18, -0.94])

>>> np.random.randn(3, 4)
array([[p` is the probability of choosing each element
>>> np.random.choice([0, 1], p=[0.6, 0.4], size=(5, 4))
array([[In the syntax for `+randn(d0, d1, ..., dn)+`, the parameters `+d0, d1, ..., dn+` are optional and indicate the shape of the final object. Here, `+np.random.randn(3, 4)+` creates a 2d array with 3 rows and 4 columns. The data will be https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables[i.i.d.], meaning that each data point is drawn independent of the others.

Another common operation is to create a sequence of random Boolean values, `+True+` or `+False+`. One way to do this would be with `+np.random.choice([True, False])+`. However, it’s actually about 4x faster to choose from `+(0, 1)+` and then https://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.view.html[view-cast] these integers to their corresponding Boolean values:

[.repl-toggle]#>>>#

[source,python,repl]

What about generating correlated data? Let’s say you want to simulate two correlated time series. One way of going about this is with NumPy’s https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.multivariate_normal.html#numpy.random.multivariate_normal[`+multivariate_normal()+`] function, which takes a covariance matrix into account. In other words, to draw from a single normally distributed random variable, you need to specify its mean and variance (or standard deviation).

To sample from the https://en.wikipedia.org/wiki/Multivariate_normal[multivariate normal] distribution, you specify the means and covariance matrix, and you end up with multiple, correlated series of data that are each approximately normally distributed.

However, rather than covariance, https://en.wikipedia.org/wiki/correlation[correlation] is a measure that is more familiar and intuitive to most. It’s the covariance normalized by the product of standard deviations, and so you can also define covariance in terms of correlation and standard deviation:

https://files.realpython.com/media/scalar_equation.2ef9746c8834.jpg[image:https://files.realpython.com/media/scalar_equation.2ef9746c8834.jpg[Covariance in Scalar Form,width=620,height=88]]

So, could you draw random samples from a multivariate normal distribution by specifying a correlation matrix and standard deviations? Yes, but you’ll need to get the above https://blogs.sas.com/content/iml/2010/12/10/converting-between-correlation-and-covariance-matrices.html[into matrix form] first. Here, *_S_ *is a vector of the standard deviations,* _P_ *is their correlation matrix, and* _C_* is the resulting (square) covariance matrix:

https://files.realpython.com/media/matrix_equation.d70f9fd73960.jpg[image:https://files.realpython.com/media/matrix_equation.d70f9fd73960.jpg[Covariance in Matrix Form,width=610,height=78]]

This can be expressed in NumPy as follows:

[source,python]

Now, you can generate two time series that are correlated but still random:

[.repl-toggle]#>>>#

[source,python,repl]

You can think of `+data+` as 500 pairs of inversely correlated data points. Here’s a sanity check that you can back into the original inputs, which approximate `+corr+`, `+stdev+`, and `+mean+` from above:

[.repl-toggle]#>>>#

[source,python,repl]

Before we move on to CSPRNGs, it might be helpful to summarize some `+random+` functions and their `+numpy.random+` counterparts:

[cols=",,",options="header",]
|===
|Python `+random+` Module |NumPy Counterpart |Use
|`+random()+` |https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.rand.html#numpy.random.rand[`+rand()+`] |Random float in [0.0, 1.0)
|`+randint(a, b)+` |https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.random_integers.html#numpy.random.random_integers[`+random_integers()+`] |Random integer in [a, b]
|`+randrange(a, b[, step])+` |https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.randint.html#numpy.random.randint[`+randint()+`] |Random integer in [a, b)
|`+uniform(a, b)+` |https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.uniform.html#numpy.random.uniform[`+uniform()+`] |Random float in [a, b]
|`+choice(seq)+` |https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.choice.html#numpy.random.choice[`+choice()+`] |Random element from `+seq+`
|`+choices(seq, k=1)+` |https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.choice.html#numpy.random.choice[`+choice()+`] |Random `+k+` elements from `+seq+` with replacement
|`+sample(population, k)+` |https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.choice.html#numpy.random.choice[`+choice()+`] with `+replace=False+` |Random `+k+` elements from `+seq+` without replacement
|`+shuffle(x[, random])+` |https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.shuffle.html#numpy.random.shuffle[`+shuffle()+`] |Shuffle the sequence `+x+` in place
|`+normalvariate(mu, sigma)+` or `+gauss(mu, sigma)+` |https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.normal.html#numpy.random.normal[`+normal()+`] |Sample from a normal distribution with mean `+mu+` and standard deviation `+sigma+`
|===

*Note*: NumPy is specialized for building and manipulating large, multidimensional arrays. If you just need a single value, `+random+` will suffice and will probably be faster as well. For small sequences, `+random+` may even be faster too, because NumPy does come with some overhead.

Now that you’ve covered two fundamental options for PRNGs, let’s move onto a few more secure adaptations.

=== CSPRNGs in Python

[[osurandom-about-as-random-as-it-gets]]
==== `+os.urandom()+`: About as Random as It Gets

Python’s https://docs.python.org/library/os.html#os.urandom[`+os.urandom()+`] function is used by both https://github.com/python/cpython/blob/b225cb770fb17596298f5a05c41a7c90c470c4f8/Lib/secrets.py#L47[`+secrets+`] and https://github.com/python/cpython/blob/b225cb770fb17596298f5a05c41a7c90c470c4f8/Lib/uuid.py#L621[`+uuid+`] (both of which you’ll see here in a moment). Without getting into too much detail, `+os.urandom()+` generates operating-system-dependent random bytes that can safely be called cryptographically secure:

* On Unix operating systems, it reads random bytes from the special file `+/dev/urandom+`, which in turn “allow access to environmental noise collected from device drivers and other sources.” (Thank you, https://en.wikipedia.org/wiki//dev/random[Wikipedia].) This is garbled information that is particular to your hardware and system state at an instance in time but at the same time sufficiently random.
 *On Windows, the C++ function https://msdn.microsoft.com/en-us/library/windows/desktop/aa379942(v=vs.85).aspx[`+CryptGenRandom()+`] is used. This function is still technically pseudorandom, but it works by generating a seed value from variables such as the process ID, memory status, and so on.

With `+os.urandom()+`, there is no concept of manually seeding. While still technically pseudorandom, this function better aligns with how we think of randomness. The only argument is the number of https://docs.python.org/library/stdtypes.html#bytes[bytes] to return:

[.repl-toggle]#>>>#

[source,python,repl]

Before we go any further, this might be a good time to delve into a mini-lesson on https://docs.python.org/howto/unicode.html[character encoding]. Many people, including myself, have some type of allergic reaction when they see `+bytes+` objects and a long line of `+\x+` characters. However, it’s useful to know how sequences such as `+x+` above eventually get turned into strings or numbers.

`+os.urandom()+` returns a sequence of single bytes:

[.repl-toggle]#>>>#

[source,python,repl]

But how does this eventually get turned into a Python `+str+` or sequence of numbers?

First, recall one of the fundamental concepts of computing, which is that a byte is made up of 8 bits. You can think of a bit as a single digit that is either 0 or 1. A byte effectively chooses between 0 and 1 eight times, so both `+01101100+` and `+11110000+` could represent bytes. Try this, which makes use of Python https://realpython.com/python-f-strings/[f-strings] introduced in Python 3.6, in your interpreter:

[.repl-toggle]#>>>#

[source,python,repl]

This is equivalent to `+[bin(i) for i in range(256)]+`, with some special formatting. https://docs.python.org/3/library/functions.html#bin[`+bin()+`] converts an integer to its binary representation as a string.

Where does that leave us? Using `+range(256)+` above is not a random choice. (No pun intended.) Given that we are allowed 8 bits, each with 2 choices, there are `+2* * 8 == 256+` possible bytes “combinations.”

This means that each byte maps to an integer between 0 and 255. In other words, we would need more than 8 bits to express the integer 256. You can verify this by checking that `+len(f'{256:0>8b}')+` is now 9, not 8.

Okay, now let’s get back to the `+bytes+` data type that you saw above, by constructing a sequence of the bytes that correspond to integers 0 through 255:

[.repl-toggle]#>>>#

[source,python,repl]

If you call `+list(bites)+`, you’ll get back to a Python list that runs from 0 to 255. But if you just print `+bites+`, you get an ugly looking sequence littered with backslashes:

[.repl-toggle]#>>>#

[source,python,repl]

These backslashes are escape sequences, and `+\xhh+` https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals[represents] the character with hex value `+hh+`. Some of the elements of `+bites+` are displayed literally (printable characters such as letters, numbers, and punctuation). Most are expressed with escapes. `+\x08+` represents a keyboard’s backspace, while `+\x13+` is a https://en.wikipedia.org/wiki/Carriage_return[carriage return] (part of a new line, on Windows systems).

If you need a refresher on hexadecimal, Charles Petzold’s https://realpython.com/asins/0735611319/[_Code: The Hidden Language_] is a great place for that. Hex is a base-16 numbering system that, instead of using 0 through 9, uses 0 through 9 and _a_ through _f_ as its basic digits.

Finally, let’s get back to where you started, with the sequence of random bytes `+x+`. Hopefully this makes a little more sense now. Calling `+.hex()+` on a `+bytes+` object gives a `+str+` of hexadecimal numbers, with each corresponding to a decimal number from 0 through 255:

[.repl-toggle]#>>>#

[source,python,repl]

One last question: how is `+b.hex()+` 12 characters long above, even though `+x+` is only 6 bytes? This is because two hexadecimal digits correspond precisely to a single byte. The `+str+` version of `+bytes+` will always be twice as long as far as our eyes are concerned.

Even if the byte (such as `+\x01+`) does not need a full 8 bits to be represented, `+b.hex()+` will always use two hex digits per byte, so the number 1 will be represented as `+01+` rather than just `+1+`. Mathematically, though, both of these are the same size.

*Technical Detail*: What you’ve mainly dissected here is how a `+bytes+` object becomes a Python `+str+`. One other technicality is how `+bytes+` produced by `+os.urandom()+` get converted to a `+float+` in the interval [0.0, 1.0), as in the https://github.com/python/cpython/blob/c6040638aa1537709add895d24cdbbb9ee310fde/Lib/random.py#L676[cryptographically secure version] of `+random.random()+`. If you’re interested in exploring this further, https://github.com/realpython/materials/blob/master/random-data/bytes_to_int.py[this code snippet] demonstrates how `+int.from_bytes()+` makes the initial conversion to an integer, using a base-256 numbering system.

With that under your belt, let’s touch on a recently introduced module, `+secrets+`, which makes generating secure tokens much more user-friendly.

==== Python’s Best Kept `+secrets+`

Introduced in Python 3.6 by https://www.python.org/dev/peps/pep-0506/[one of the more colorful PEPs] out there, the `+secrets+` module is intended to be the de facto Python module for generating cryptographically secure random bytes and strings.

You can check out the https://github.com/python/cpython/blob/3.6/Lib/secrets.py[source code] for the module, which is short and sweet at about 25 lines of code. `+secrets+` is basically a wrapper around `+os.urandom()+`. It exports just a handful of functions for generating random numbers, bytes, and strings. Most of these examples should be fairly self-explanatory:

[.repl-toggle]#>>>#

[source,python,repl]

Now, how about a concrete example? You’ve probably used URL shortener services like https://tinyurl.com[tinyurl.com] or https://bit.ly[bit.ly] that turn an unwieldy URL into something like https://bit.ly/2IcCp9u. Most shorteners don’t do any complicated hashing from input to output; they just generate a random string, make sure that string has not already been generated previously, and then tie that back to the input URL.

Let’s say that after taking a look at the https://www.iana.org/domains/root/db[Root Zone Database], you’ve registered the site *short.ly*. Here’s a function to get you started with your service:

[source,python]

Is this a full-fledged real illustration? No. I would wager that bit.ly does things in a slightly more advanced way than storing its gold mine in a global Python dictionary that is not persistent between sessions. However, it’s roughly accurate conceptually:

[.repl-toggle]#>>>#

[source,python,repl]

*Hold On: *One thing you may notice is that both of these results are of length 7 when you requested 5 bytes. _Wait, I thought that you said the result would be twice as long?_ Well, not exactly, in this case. There is one more thing going on here: `+token_urlsafe()+` uses base64 encoding, where each character is 6 bits of data. (It’s 0 through 63, and corresponding characters. The characters are A-Z, a-z, 0-9, and +/.)

If you originally specify a certain number of bytes `+nbytes+`, the resulting length from `+secrets.token_urlsafe(nbytes)+` will be `+math.ceil(nbytes* 8/6)+`, which you can https://github.com/realpython/materials/blob/master/random-data/urlsafe.py[prove] and investigate further if you’re curious.

The bottom line here is that, while `+secrets+` is really just a wrapper around existing Python functions, it can be your go-to when security is your foremost concern.

=== One Last Candidate: `+uuid+`

One last option for generating a random token is the `+uuid4()+` function from Python’s https://docs.python.org/library/uuid.html[`+uuid+`] module. A https://tools.ietf.org/html/rfc4122.html[UUID] is a Universally Unique IDentifier, a 128-bit sequence (`+str+` of length 32) designed to “guarantee uniqueness across space and time.” `+uuid4()+` is one of the module’s most useful functions, and this function https://github.com/python/cpython/blob/78392885c9b08021c89649728053d31503d8a509/Lib/uuid.py#L623[also uses `+os.urandom()+`]:

[.repl-toggle]#>>>#

[source,python,repl]

The nice thing is that all of `+uuid+`’s functions produce an instance of the `+UUID+` class, which encapsulates the ID and has properties like `+.int+`, `+.bytes+`, and `+.hex+`:

[.repl-toggle]#>>>#

[source,python,repl]

You may also have seen some other variations: `+uuid1()+`, `+uuid3()+`, and `+uuid5()+`. The key difference between these and `+uuid4()+` is that those three functions all take some form of input and therefore don’t meet the definition of “random” to the extent that a Version 4 UUID does:

* `+uuid1()+` uses your machine’s host ID and current time by default. Because of the reliance on current time down to nanosecond resolution, this version is where UUID derives the claim “guaranteed uniqueness across time.”
 *`+uuid3()+` and `+uuid5()+` both take a namespace identifier and a name. The former uses an https://en.wikipedia.org/wiki/MD5[MD5] hash and the latter uses SHA-1.

`+uuid4()+`, conversely, is entirely pseudorandom (or random). It consists of getting 16 bytes via `+os.urandom()+`, converting this to a https://en.wikipedia.org/wiki/Endianness[big-endian] integer, and doing a number of bitwise operations to comply with the https://tools.ietf.org/html/rfc4122.html#section-4.1.1[formal specification].

Hopefully, by now you have a good idea of the distinction between different “types” of random data and how to create them. However, one other issue that might come to mind is that of collisions.

In this case, a collision would simply refer to generating two matching UUIDs. What is the chance of that? Well, it is technically not zero, but perhaps it is close enough: there are `+2* * 128+` or 340 https://en.wikipedia.org/wiki/Names_of_large_numbers[undecillion] possible `+uuid4+` values. So, I’ll leave it up to you to judge whether this is enough of a guarantee to sleep well.

One common use of `+uuid+` is in Django, which has a https://docs.djangoproject.com/en/2.0/ref/models/fields/#uuidfield[`+UUIDField+`] that is often used as a primary key in a model’s underlying relational database.

=== Why Not Just “Default to” `+SystemRandom+`?

In addition to the secure modules discussed here such as `+secrets+`, Python’s `+random+` module actually has a little-used class called https://github.com/python/cpython/blob/b225cb770fb17596298f5a05c41a7c90c470c4f8/Lib/random.py#L666[`+SystemRandom+`] that uses `+os.urandom()+`. (`+SystemRandom+`, in turn, is also used by `+secrets+`. It’s all a bit of a web that traces back to `+urandom()+`.)

At this point, you might be asking yourself why you wouldn’t just “default to” this version? Why not “always be safe” rather than defaulting to the deterministic `+random+` functions that aren’t cryptographically secure ?

I’ve already mentioned one reason: sometimes you want your data to be deterministic and reproducible for others to follow along with.

But the second reason is that CSPRNGs, at least in Python, tend to be meaningfully slower than PRNGs. Let’s test that with a script, https://github.com/realpython/materials/blob/master/random-data/timed.py[`+timed.py+`], that compares the PRNG and CSPRNG versions of `+randint()+` using Python’s `+timeit.repeat()+`:

[source,python]

Now to execute this from the shell:

[source,sh]

A 5x timing difference is certainly a valid consideration in addition to cryptographic security when choosing between the two.

=== Odds and Ends: Hashing

One concept that hasn’t received much attention in this tutorial is that of https://en.wikipedia.org/wiki/Cryptographic_hash_function[hashing], which can be done with Python’s https://docs.python.org/3/library/hashlib.html[`+hashlib+`] module.

A hash is designed to be a one-way mapping from an input value to a fixed-size string that is virtually impossible to reverse engineer. As such, while the result of a hash function may “look like” random data, it doesn’t really qualify under the definition here.

=== Recap

You’ve covered a lot of ground in this tutorial. To recap, here is a high-level comparison of the options available to you for engineering randomness in Python:

[cols=",,",options="header",]
|===
|Package/Module |Description |Cryptographically Secure
|https://docs.python.org/library/random.html[`+random+`] |Fasty & easy random data using Mersenne Twister |No
|https://docs.scipy.org/doc/numpy/reference/routines.random.html[`+numpy.random+`] |Like `+random+` but for (possibly multidimensional) arrays |No
|https://docs.python.org/library/os.html[`+os+`] |Contains `+urandom()+`, the base of other functions covered here |Yes
|https://docs.python.org/library/secrets.html[`+secrets+`] |Designed to be Python’s de facto module for generating secure random numbers, bytes, and strings |Yes
|https://docs.python.org/library/uuid.html[`+uuid+`] |Home to a handful of functions for building 128-bit identifiers |Yes, `+uuid4()+`
|===

Feel free to leave some totally random comments below, and thanks for reading.

=== Additional Links

* https://www.random.org/[Random.org] offers “true random numbers to anyone on the Internet” derived from atmospheric noise.
* The https://docs.python.org/3.6/library/random.html#examples-and-recipes[Recipes] section from the `+random+` module has some additional tricks.
* The seminal paper on the http://www.math.sci.hiroshima-u.ac.jp/%7Em-mat/MT/ARTICLES/mt.pdf[Mersienne Twister] appeared in 1997, if you’re into that kind of thing.
* The https://docs.python.org/3.6/library/itertools.html#itertools-recipes[Itertools Recipes] define functions for choosing randomly from a combinatoric set, such as from combinations or permutations.
* http://scikit-learn.org/stable/datasets/index.html#sample-generators[Scikit-Learn] includes various random sample generators that can be used to build artificial datasets of controlled size and complexity.
* Eli Bendersky digs into `+random.randint()+` in his article https://eli.thegreenplace.net/2018/slow-and-fast-methods-for-generating-random-integers-in-python/#id2[Slow and Fast Methods for Generating Random Integers in Python].
* Peter Norvig’s a http://nbviewer.jupyter.org/url/norvig.com/ipython/Probability.ipynb[Concrete Introduction to Probability using Python] is a comprehensive resource as well.
* The Pandas library includes a https://github.com/pandas-dev/pandas/blob/f9cc39fb1391cb05f55232367f6547ff9ea615b8/pandas/util/testing.py#L2513[context manager] that can be used to set a temporary random state.
* From Stack Overflow:
** https://stackoverflow.com/q/50559078/7954504[Generating Random Dates In a Given Range]
** https://stackoverflow.com/q/48421142/7954504[Fastest Way to Generate a Random-like Unique String with Random Length]
** https://stackoverflow.com/q/21187131/7954504[How to Use `+random.shuffle()+` on a Generator]
** https://stackoverflow.com/q/31389481/7954504[Replace Random Elements in a NumPy Array]
** https://stackoverflow.com/q/14720799/7954504[Getting Numbers from/dev/random in Python]