Related
I have created a sequence alignment tool to compare two strands of DNA (X and Y) to find the best alignment of substrings from X and Y. The algorithm is summarized here (https://en.wikipedia.org/wiki/Smith–Waterman_algorithm). I have been able to generate a lists of lists, filling them all with zeros, to represent my matrix. I created a scoring algorithm to return a numerical score for each kind of alignment between bases (eg. plus 4 for a match). Then I created an alignment algorithm that should put a score in each coordinate of my "matrix". However, when I go to print the matrix, it only returns the original with all zeros (rather than actual scores).
I know there are other methods of implementing this method (with numpy for example), so could you please tell me why this specific code (below) does not work? Is there a way to modify it, so that it does work?
code:
def zeros(X: int, Y: int):
lenX = len(X) + 1
lenY = len(Y) + 1
matrix = []
for i in range(lenX):
matrix.append([0] * lenY)
def score(X, Y):
if X[n] == Y[m]: return 4
if X[n] == '-' or Y[m] == '-': return -4
else: return -2
def SmithWaterman(X, Y, score):
for n in range(1, len(X) + 1):
for m in range(1, len(Y) + 1):
align = matrix[n-1, m-1] + (score(X[n-1], Y[m-1]))
indelX = matrix[n-1, m] + (score(X[n-1], Y[m]))
indelY = matrix[n, m-1] + (score(X[n], Y[m-1]))
matrix[n, m] = max(align, indelX, indelY, 0)
print(matrix)
zeros("ACGT", "ACGT")
output:
[[0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]]
The reason it's just printing out the zeroed out matrix is that the SmithWaterman function is never called, so the matrix is never updated.
You would need to do something like
# ...
SmithWaterman(X, Y, score)
print(matrix)
# ...
However, If you do this, you will find that this code is actually quite broken in many other ways. I've gone through and annotated some of the syntax errors and other issues with the code:
def zeros(X: int, Y: int):
# ^ ^ incorrect type annotations. should be str
lenX = len(X) + 1
lenY = len(Y) + 1
matrix = []
for i in range(lenX):
matrix.append([0] * lenY)
# A more "pythonic" way of expressing the above would be:
# matrix = [[0] * len(Y) + 1 for _ in range(len(x) + 1)]
def score(X, Y):
# ^ ^ shadowing variables from outer scope. this is not a bug per se but it's considered bad practice
if X[n] == Y[m]: return 4
# ^ ^ variables not defined in scope
if X[n] == '-' or Y[m] == '-': return -4
# ^ ^ variables not defined in scope
else: return -2
def SmithWaterman(X, Y, score): # this function is never called
# ^ unnecessary function passed as parameter. function is defined in scope
for n in range(1, len(X) + 1):
for m in range(1, len(Y) + 1):
align = matrix[n-1, m-1] + (score(X[n-1], Y[m-1]))
# ^ invalid list lookup. should be: matrix[n-1][m-1]
indelX = matrix[n-1, m] + (score(X[n-1], Y[m]))
# ^ out of bounds error when m == len(Y)
indelY = matrix[n, m-1] + (score(X[n], Y[m-1]))
# ^ out of bounds error when n == len(X)
matrix[n, m] = max(align, indelX, indelY, 0)
# this should be nested in the inner for-loop. m, n, indelX, and indelY are not defined in scope here
print(matrix)
zeros("ACGT", "ACGT")
Say, we have f(t) = v * t + A * sin(w * t). I call such functions "saw-like":
I want to solve saw(t) = C, that is, find a root of saw(t) - C (still "saw-like").
I tried writing down a ternary search for function abs(saw(t) - C) to find its minima. If we are lucky (or crafty), it would be the root. Unfortunately, my code does not always work: sometimes we get stuck in those places:
My code (python3):
def calculate(fun):
eps = 0.000000001
eps_l = 0.1
x = terns(fun, 0, 100000000000000)
t = terns(fun, 0, x)
cnt = 0
while fun(x) > eps:
t = x
x = terns(fun, 0, t)
if abs(t - x) < eps_l:
cnt += 1
# A sorry attempt pass some wrong value as a right one.
# Gets us out of an infinite loop at least.
if cnt == 10:
break
return t
def terns(f, l, r):
eps = 0.00000000001
while r - l > eps:
x_1 = l + (r - l) / 3
x_2 = r - (r - l) / 3
if f(x_1) < f(x_2):
r = x_2
else:
l = x_1
return (l + r) / 2
So, how is it done? Is using ternary search the right way?
My other idea was somehow sending the equation over to the net, passing it to Wolfram Alpha and fetching the answers. Yet, I don't how it's done, as I am not quite fluent at python.
How could this be done?
I got one puzzle and I want to solve it using Python.
Puzzle:
A merchant has a 40 kg weight which he used in his shop. Once, it fell
from his hands and was broken into 4 pieces. But surprisingly, now he
can weigh any weight between 1 kg to 40 kg with the combination of
these 4 pieces.
So question is, what are weights of those 4 pieces?
Now I wanted to solve this in Python.
The only constraint i got from the puzzle is that sum of 4 pieces is 40. With that I could filter all the set of 4 values whose sum is 40.
import itertools as it
weight = 40
full = range(1,41)
comb = [x for x in it.combinations(full,4) if sum(x)==40]
length of comb = 297
Now I need to check each set of values in comb and try all the combination of operations.
Eg if (a,b,c,d) is the first set of values in comb, I need to check a,b,c,d,a+b,a-b, .................a+b+c-d,a-b+c+d........ and so on.
I tried a lot, but i am stuck at this stage, ie how to check all these combination of calculations to each set of 4 values.
Question :
1) I think i need to get a list all possible combination of [a,b,c,d] and [+,-].
2) does anyone have a better idea and tell me how to go forward from here?
Also, I want to do it completely without help of any external libraries, need to use only standard libraries of python.
EDIT : Sorry for the late info. Its answer is (1,3,9,27), which I found a few years back. I have checked and verified the answer.
EDIT : At present, fraxel's answer works perfect with time = 0.16 ms. A better and faster approach is always welcome.
Regards
ARK
Earlier walk-through anwswer:
We know a*A + b*B + c*C + d*D = x for all x between 0 and 40, and a, b, c, d are confined to -1, 0, 1. Clearly A + B + C + D = 40. The next case is x = 39, so clearly the smallest move is to remove an element (it is the only possible move that could result in successfully balancing against 39):
A + B + C = 39, so D = 1, by neccessity.
next:
A + B + C - D = 38
next:
A + B + D = 37, so C = 3
then:
A + B = 36
then:
A + B - D = 35
A + B - C + D = 34
A + B - C = 33
A + B - C - D = 32
A + C + D = 31, so A = 9
Therefore B = 27
So the weights are 1, 3, 9, 27
Really this can be deduced immediately from the fact that they must all be multiples of 3.
Interesting Update:
So here is some python code to find a minimum set of weights for any dropped weight that will span the space:
def find_weights(W):
weights = []
i = 0
while sum(weights) < W:
weights.append(3 ** i)
i += 1
weights.pop()
weights.append(W - sum(weights))
return weights
print find_weights(40)
#output:
[1, 3, 9, 27]
To further illustrate this explaination, one can consider the problem as the minimum number of weights to span the number space [0, 40]. It is evident that the number of things you can do with each weight is trinary /ternary (add weight, remove weight, put weight on other side). So if we write our (unknown) weights (A, B, C, D) in descending order, our moves can be summarised as:
ABCD: Ternary:
40: ++++ 0000
39: +++0 0001
38: +++- 0002
37: ++0+ 0010
36: ++00 0011
35: ++0- 0012
34: ++-+ 0020
33: ++-0 0021
32: ++-- 0022
31: +0++ 0100
etc.
I have put ternary counting from 0 to 9 alongside, to illustrate that we are effectively in a trinary number system (base 3). Our solution can always be written as:
3**0 + 3**1 +3**2 +...+ 3**N >= Weight
For the minimum N that this holds true. The minimum solution will ALWAYS be of this form.
Furthermore, we can easily solve the problem for large weights and find the minimum number of pieces to span the space:
A man drops a known weight W, it breaks into pieces. His new weights allow him to weigh any weight up to W. How many weights are there, and what are they?
#what if the dropped weight was a million Kg:
print find_weights(1000000)
#output:
[1, 3, 9, 27, 81, 243, 729, 2187, 6561, 19683, 59049, 177147, 531441, 202839]
Try using permutations for a large weight and unknown number of pieces!!
Here is a brute-force itertools solution:
import itertools as it
def merchant_puzzle(weight, pieces):
full = range(1, weight+1)
all_nums = set(full)
comb = [x for x in it.combinations(full, pieces) if sum(x)==weight]
funcs = (lambda x: 0, lambda x: x, lambda x: -x)
for c in comb:
sums = set()
for fmap in it.product(funcs, repeat=pieces):
s = sum(f(x) for x, f in zip(c, fmap))
if s > 0:
sums.add(s)
if sums == all_nums:
return c
>>> merchant_puzzle(40, 4)
(1, 3, 9, 27)
For an explanation of how it works, check out the answer Avaris gave, this is an implementation of the same algorithm.
You are close, very close :).
Since this is a puzzle you want to solve, I'll just give pointers. For this part:
Eg if (a,b,c,d) is the first set of values in comb, i need to check
a,b,c,d,a+b,a-b, .................a+b+c-d,a-b+c+d........ and so on.
Consider this: Each weight can be put to one scale, the other or neither. So for the case of a, this can be represented as [a, -a, 0]. Same with the other three. Now you need all possible pairings with these 3 possibilities for each weight (hint: itertools.product). Then, a possible measuring of a pairing (lets say: (a, -b, c, 0)) is merely the sum of these (a-b+c+0).
All that is left is just checking if you could 'measure' all the required weights. set might come handy here.
PS: As it was stated in the comments, for the general case, it might not be necessary that these divided weights should be distinct (for this problem it is). You might reconsider itertools.combinations.
I brute forced the hell out of the second part.
Do not click this if you don't want to see the answer. Obviously, if I was better at permutations, this would have required a lot less cut/paste search/replace:
http://pastebin.com/4y2bHCVr
I don't know Python syntax, but maybe you can decode this Scala code; start with the 2nd for-loop:
def setTo40 (a: Int, b: Int, c: Int, d: Int) = {
val vec = for (
fa <- List (0, 1, -1);
fb <- List (0, 1, -1);
fc <- List (0, 1, -1);
fd <- List (0, 1, -1);
prod = fa * a + fb * b + fc * c + fd * d;
if (prod > 0)
) yield (prod)
vec.toSet
}
for (a <- (1 to 9);
b <- (a to 14);
c <- (b to 20);
d = 40-(a+b+c)
if (d > 0)) {
if (setTo40 (a, b, c, d).size > 39)
println (a + " " + b + " " + c + " " + d)
}
With weights [2, 5, 15, 18] you can also measure all objects between 1 and 40kg, although some of them will need to be measured indirectly. For example, to measure an object weighting 39kg, you would first compare it with 40kg and the balance would pend to the 40kg side (because 39 < 40), but then if you remove the 2kg weight it would pend to the other side (because 39 > 38) and thus you can conclude the object weights 39kg.
More interestingly, with weights [2, 5, 15, 45] you can measure all objects up to 67kg.
If anyone doesn't want to import a library to import combos/perms, this will generate all possible 4-move strategies...
# generates permutations of repeated values
def permutationsWithRepeats(n, v):
perms = []
value = [0] * n
N = n - 1
i = n - 1
while i > -1:
perms.append(list(value))
if value[N] < v:
value[N] += 1
else:
while (i > -1) and (value[i] == v):
value[i] = 0
i -= 1
if i > -1:
value[i] += 1
i = N
return perms
# generates the all possible permutations of 4 ternary moves
def strategy():
move = ['-', '0', '+']
perms = permutationsWithRepeats(4, 2)
for i in range(len(perms)):
s = ''
for j in range(4):
s += move[perms[i][j]]
print s
# execute
strategy()
My solution as follows:
#!/usr/bin/env python3
weight = 40
parts = 4
part=[0] * parts
def test_solution(p, weight,show_result=False):
cv=[0,0,0,0]
for check_weight in range(1,weight+1):
sum_ok = False
for parts_used in range(2 ** parts):
for options in range(2 ** parts):
for pos in range(parts):
pos_neg = int('{0:0{1}b}'.format(options,parts)[pos]) * 2 - 1
use = int('{0:0{1}b}'.format(parts_used,parts)[pos])
cv[pos] = p[pos] * pos_neg * use
if sum(cv) == check_weight:
if show_result:
print("{} = sum of:{}".format(check_weight, cv))
sum_ok = True
break
if sum_ok:
continue
else:
return False
return True
for part[0] in range(1,weight-parts):
for part[1] in range(part[0]+1, weight - part[0]):
for part[2] in range( part[1] + 1 , weight - sum(part[0:2])):
part[3] = weight - sum(part[0:3])
if test_solution(part,weight):
print(part)
test_solution(part,weight,True)
exit()
It gives you all the solutions for the given weights
More dynamic than my previous answer, so it also works with other numbers. But breaking up into 5 peaces takes some time:
#!/usr/bin/env python3
weight = 121
nr_of_parts = 5
# weight = 40
# nr_of_parts = 4
weight = 13
nr_of_parts = 3
part=[0] * nr_of_parts
def test_solution(p, weight,show_result=False):
cv=[0] * nr_of_parts
for check_weight in range(1,weight+1):
sum_ok = False
for nr_of_parts_used in range(2 ** nr_of_parts):
for options in range(2 ** nr_of_parts):
for pos in range(nr_of_parts):
pos_neg = int('{0:0{1}b}'.format(options,nr_of_parts)[pos]) * 2 - 1
use = int('{0:0{1}b}'.format(nr_of_parts_used,nr_of_parts)[pos])
cv[pos] = p[pos] * pos_neg * use
if sum(cv) == check_weight:
if show_result:
print("{} = sum of:{}".format(check_weight, cv))
sum_ok = True
break
if sum_ok:
continue
else:
return False
return True
def set_parts(part,position, nr_of_parts, weight):
if position == 0:
part[position] = 1
part, valid = set_parts(part,position+1,nr_of_parts,weight)
return part, valid
if position == nr_of_parts - 1:
part[position] = weight - sum(part)
if part[position -1] >= part[position]:
return part, False
return part, True
part[position]=max(part[position-1]+1,part[position])
part, valid = set_parts(part, position + 1, nr_of_parts, weight)
if not valid:
part[position]=max(part[position-1]+1,part[position]+1)
part=part[0:position+1] + [0] * (nr_of_parts - position - 1)
part, valid = set_parts(part, position + 1, nr_of_parts, weight)
return part, valid
while True:
part, valid = set_parts(part, 0, nr_of_parts, weight)
if not valid:
print(part)
print ('No solution posible')
exit()
if test_solution(part,weight):
print(part,' ')
test_solution(part,weight,True)
exit()
else:
print(part,' ', end='\r')
Does some standard Python module contain a function to compute modular multiplicative inverse of a number, i.e. a number y = invmod(x, p) such that x*y == 1 (mod p)? Google doesn't seem to give any good hints on this.
Of course, one can come up with home-brewed 10-liner of extended Euclidean algorithm, but why reinvent the wheel.
For example, Java's BigInteger has modInverse method. Doesn't Python have something similar?
Python 3.8+
y = pow(x, -1, p)
Python 3.7 and earlier
Maybe someone will find this useful (from wikibooks):
def egcd(a, b):
if a == 0:
return (b, 0, 1)
else:
g, y, x = egcd(b % a, a)
return (g, x - (b // a) * y, y)
def modinv(a, m):
g, x, y = egcd(a, m)
if g != 1:
raise Exception('modular inverse does not exist')
else:
return x % m
If your modulus is prime (you call it p) then you may simply compute:
y = x**(p-2) mod p # Pseudocode
Or in Python proper:
y = pow(x, p-2, p)
Here is someone who has implemented some number theory capabilities in Python: http://www.math.umbc.edu/~campbell/Computers/Python/numbthy.html
Here is an example done at the prompt:
m = 1000000007
x = 1234567
y = pow(x,m-2,m)
y
989145189L
x*y
1221166008548163L
x*y % m
1L
You might also want to look at the gmpy module. It is an interface between Python and the GMP multiple-precision library. gmpy provides an invert function that does exactly what you need:
>>> import gmpy
>>> gmpy.invert(1234567, 1000000007)
mpz(989145189)
Updated answer
As noted by #hyh , the gmpy.invert() returns 0 if the inverse does not exist. That matches the behavior of GMP's mpz_invert() function. gmpy.divm(a, b, m) provides a general solution to a=bx (mod m).
>>> gmpy.divm(1, 1234567, 1000000007)
mpz(989145189)
>>> gmpy.divm(1, 0, 5)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ZeroDivisionError: not invertible
>>> gmpy.divm(1, 4, 8)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ZeroDivisionError: not invertible
>>> gmpy.divm(1, 4, 9)
mpz(7)
divm() will return a solution when gcd(b,m) == 1 and raises an exception when the multiplicative inverse does not exist.
Disclaimer: I'm the current maintainer of the gmpy library.
Updated answer 2
gmpy2 now properly raises an exception when the inverse does not exists:
>>> import gmpy2
>>> gmpy2.invert(0,5)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ZeroDivisionError: invert() no inverse exists
As of 3.8 pythons pow() function can take a modulus and a negative integer. See here. Their case for how to use it is
>>> pow(38, -1, 97)
23
>>> 23 * 38 % 97 == 1
True
Here is a one-liner for CodeFights; it is one of the shortest solutions:
MMI = lambda A, n,s=1,t=0,N=0: (n < 2 and t%N or MMI(n, A%n, t, s-A//n*t, N or n),-1)[n<1]
It will return -1 if A has no multiplicative inverse in n.
Usage:
MMI(23, 99) # returns 56
MMI(18, 24) # return -1
The solution uses the Extended Euclidean Algorithm.
Sympy, a python module for symbolic mathematics, has a built-in modular inverse function if you don't want to implement your own (or if you're using Sympy already):
from sympy import mod_inverse
mod_inverse(11, 35) # returns 16
mod_inverse(15, 35) # raises ValueError: 'inverse of 15 (mod 35) does not exist'
This doesn't seem to be documented on the Sympy website, but here's the docstring: Sympy mod_inverse docstring on Github
Here is a concise 1-liner that does it, without using any external libraries.
# Given 0<a<b, returns the unique c such that 0<c<b and a*c == gcd(a,b) (mod b).
# In particular, if a,b are relatively prime, returns the inverse of a modulo b.
def invmod(a,b): return 0 if a==0 else 1 if b%a==0 else b - invmod(b%a,a)*b//a
Note that this is really just egcd, streamlined to return only the single coefficient of interest.
I try different solutions from this thread and in the end I use this one:
def egcd(a, b):
lastremainder, remainder = abs(a), abs(b)
x, lastx, y, lasty = 0, 1, 1, 0
while remainder:
lastremainder, (quotient, remainder) = remainder, divmod(lastremainder, remainder)
x, lastx = lastx - quotient*x, x
y, lasty = lasty - quotient*y, y
return lastremainder, lastx * (-1 if a < 0 else 1), lasty * (-1 if b < 0 else 1)
def modinv(a, m):
g, x, y = self.egcd(a, m)
if g != 1:
raise ValueError('modinv for {} does not exist'.format(a))
return x % m
Modular_inverse in Python
Here is my code, it might be sloppy but it seems to work for me anyway.
# a is the number you want the inverse for
# b is the modulus
def mod_inverse(a, b):
r = -1
B = b
A = a
eq_set = []
full_set = []
mod_set = []
#euclid's algorithm
while r!=1 and r!=0:
r = b%a
q = b//a
eq_set = [r, b, a, q*-1]
b = a
a = r
full_set.append(eq_set)
for i in range(0, 4):
mod_set.append(full_set[-1][i])
mod_set.insert(2, 1)
counter = 0
#extended euclid's algorithm
for i in range(1, len(full_set)):
if counter%2 == 0:
mod_set[2] = full_set[-1*(i+1)][3]*mod_set[4]+mod_set[2]
mod_set[3] = full_set[-1*(i+1)][1]
elif counter%2 != 0:
mod_set[4] = full_set[-1*(i+1)][3]*mod_set[2]+mod_set[4]
mod_set[1] = full_set[-1*(i+1)][1]
counter += 1
if mod_set[3] == B:
return mod_set[2]%B
return mod_set[4]%B
The code above will not run in python3 and is less efficient compared to the GCD variants. However, this code is very transparent. It triggered me to create a more compact version:
def imod(a, n):
c = 1
while (c % a > 0):
c += n
return c // a
from the cpython implementation source code:
def invmod(a, n):
b, c = 1, 0
while n:
q, r = divmod(a, n)
a, b, c, n = n, c, b - q*c, r
# at this point a is the gcd of the original inputs
if a == 1:
return b
raise ValueError("Not invertible")
according to the comment above this code, it can return small negative values, so you could potentially check if negative and add n when negative before returning b.
To figure out the modular multiplicative inverse I recommend using the Extended Euclidean Algorithm like this:
def multiplicative_inverse(a, b):
origA = a
X = 0
prevX = 1
Y = 1
prevY = 0
while b != 0:
temp = b
quotient = a/b
b = a%b
a = temp
temp = X
a = prevX - quotient * X
prevX = temp
temp = Y
Y = prevY - quotient * Y
prevY = temp
return origA + prevY
Well, here's a function in C which you can easily convert to python. In the below c function extended euclidian algorithm is used to calculate inverse mod.
int imod(int a,int n){
int c,i=1;
while(1){
c = n * i + 1;
if(c%a==0){
c = c/a;
break;
}
i++;
}
return c;}
Translates to Python Function
def imod(a,n):
i=1
while True:
c = n * i + 1;
if(c%a==0):
c = c/a
break;
i = i+1
return c
Reference to the above C function is taken from the following link C program to find Modular Multiplicative Inverse of two Relatively Prime Numbers
I am looking for a hash functions family generator that could generate a family of hash functions given a set of parameters. I haven't found any such generator so far.
Is there a way to do that with the hashlib package ?
For example I'd like to do something like :
h1 = hash_function(1)
h2 = hash_function(2)
...
and h1 and h2 would be different hash functions.
For those of you who might know about it, I am trying to implement a min-hashing algorithm on a very large dataset.
Basically, I have a very large set of features (100 millions to 1 billion) for a given document, and I need to create 1000 to 10000 different random permutations for this set of features.
I do NOT want to build the random permutations explicitly so the technique I would like to use in the following :
generate a hash function h and consider that for two indices r and s
r appears before s in the permutation if h(r) < h(s) and do that for 100 to 1000 different hash functions.
Are there any known libraries that I might have missed ? Or any standard way of generating families of hash functions with python that you might be aware of ?
I'd just do something like (if you don't need thread-safety -- not hard to alter if you DO need thread safety -- and assuming a 32-bit Python version):
import random
_memomask = {}
def hash_function(n):
mask = _memomask.get(n)
if mask is None:
random.seed(n)
mask = _memomask[n] = random.getrandbits(32)
def myhash(x):
return hash(x) ^ mask
return myhash
As mentioned above, you can use universal hashing for minhash.
For example:
import random
def minhash():
d1 = set(random.randint(0, 2000) for _ in range(1000))
d2 = set(random.randint(0, 2000) for _ in range(1000))
jacc_sim = len(d1.intersection(d2)) / len(d1.union(d2))
print("jaccard similarity: {}".format(jacc_sim))
N_HASHES = 200
hash_funcs = []
for i in range(N_HASHES):
hash_funcs.append(universal_hashing())
m1 = [min([h(e) for e in d1]) for h in hash_funcs]
m2 = [min([h(e) for e in d2]) for h in hash_funcs]
minhash_sim = sum(int(m1[i] == m2[i]) for i in range(N_HASHES)) / N_HASHES
print("min-hash similarity: {}".format(minhash_sim))
def universal_hashing():
def rand_prime():
while True:
p = random.randrange(2 ** 32, 2 ** 34, 2)
if all(p % n != 0 for n in range(3, int((p ** 0.5) + 1), 2)):
return p
m = 2 ** 32 - 1
p = rand_prime()
a = random.randint(0, p)
if a % 2 == 0:
a += 1
b = random.randint(0, p)
def h(x):
return ((a * x + b) % p) % m
return h
Reference
#alex's answer is great and concise, but the hash functions it generates are not "very different from each other".
Let's look at the Pearson correlation between 10000 samples of 10000 hashes that put the results in 100 bins
%%time # 1min 14s
n=10000
hashes = [hash_function(i) for i in range(n)]
median_pvalue(hashes, n=n)
# 1.1614081043690444e-06
I.e. the median p_value is 1e-06 which is far from random. Here's an example if it were truly random :
%%time # 4min 15s
hashes = [lambda _ : random.randint(0,100) for _ in range(n)]
median_pvalue(hashes, n=n)
# 0.4979718236429698
Using Carter and Wegman method you could get:
%%time # 1min 43s
hashes = HashFamily(100).draw_hashes(n)
median_pvalue(hashes, n=n)
# 0.841929288037321
Code to reproduce :
from scipy.stats.stats import pearsonr
import numpy as np
import random
_memomask = {}
def hash_function(n):
mask = _memomask.get(n)
if mask is None:
random.seed(n)
mask = _memomask[n] = random.getrandbits(32)
def myhash(x):
return hash(x) ^ mask
return myhash
class HashFamily():
r"""Universal hash family as proposed by Carter and Wegman.
.. math::
\begin{array}{ll}
h_{{a,b}}(x)=((ax+b)~{\bmod ~}p)~{\bmod ~}m \ \mid p > m\\
\end{array}
Args:
bins (int): Number of bins to hash to. Better if a prime number.
moduler (int,optional): Temporary hashing. Has to be a prime number.
"""
def __init__(self, bins, moduler=None):
if moduler and moduler <= bins:
raise ValueError("p (moduler) should be >> m (buckets)")
self.bins = bins
self.moduler = moduler if moduler else self._next_prime(np.random.randint(self.bins + 1, 2**32))
# do not allow same a and b, as it could mean shifted hashes
self.sampled_a = set()
self.sampled_b = set()
def _is_prime(self, x):
"""Naive is prime test."""
for i in range(2, int(np.sqrt(x))):
if x % i == 0:
return False
return True
def _next_prime(self, n):
"""Naively gets the next prime larger than n."""
while not self._is_prime(n):
n += 1
return n
def draw_hash(self, a=None, b=None):
"""Draws a single hash function from the family."""
if a is None:
while a is None or a in self.sampled_a:
a = np.random.randint(1, self.moduler - 1)
assert len(self.sampled_a) < self.moduler - 2, "please give a bigger moduler"
self.sampled_a.add(a)
if b is None:
while b is None or b in self.sampled_b:
b = np.random.randint(0, self.moduler - 1)
assert len(self.sampled_b) < self.moduler - 1, "please give a bigger moduler"
self.sampled_b.add(b)
return lambda x: ((a * x + b) % self.moduler) % self.bins
def draw_hashes(self, n, **kwargs):
"""Draws n hash function from the family."""
return [self.draw_hash() for i in range(n)]
def median_pvalue(hashes, buckets=100, n=1000):
p_values = []
for j in range(n-1):
a = [hashes[j](i) % buckets for i in range(n)]
b = [hashes[j+1](i) % buckets for i in range(n)]
p_values.append(pearsonr(a,b)[1])
return np.median(p_values)
Note that my implementation is of Carter and Wegman is very naive (e.g. generation of prime numbers). It could be made shorter and quicker.
You should consider using universal hashing. My answer and code can be found here: https://stackoverflow.com/a/25104050/207661
The universal hash family is a set of hash functions H of size m, such that any two (district) inputs collide with probability at most 1/m when the hash function h is drawn randomly from set H.
Based on the formulation in Wikipedia, use can use the following code:
import random
def is_prime(n):
if n==2 or n==3: return True
if n%2==0 or n<2: return False
for i in range(3, int(n**0.5)+1, 2):
if n%i==0:
return False
return True
# universal hash functions
class UniversalHashFamily:
def __init__(self, number_of_hash_functions, number_of_buckets, min_value_for_prime_number=2, bucket_value_offset=0):
self.number_of_buckets = number_of_buckets
self.bucket_value_offset = bucket_value_offset
primes = []
number_to_check = min_value_for_prime_number
while len(primes) < number_of_hash_functions:
if is_prime(number_to_check):
primes.append(number_to_check)
number_to_check += random.randint(1, 1000)
self.hash_function_attrs = []
for i in range(number_of_hash_functions):
p = primes[i]
a = random.randint(1, p)
b = random.randint(0, p)
self.hash_function_attrs.append((a, b, p))
def __call__(self, function_index, input_integer):
a, b, p = self.hash_function_attrs[function_index]
return (((a*input_integer + b)%p)%self.number_of_buckets) + self.bucket_value_offset
Example usage:
We can create a hash family consists of 20 hash functions, each one map the input to 100 buckets.
hash_family = UniversalHashFamily(20, 100)
And get the hashed values like:
input_integer = 1234567890 # sample input
hash_family(0, input_integer) # the output of the first hash function, i.e. h0(input_integer)
hash_family(1, input_integer) # the output of the second hash function, i.e. h1(input_integer)
# ...
hash_family(19, input_integer) # the output of the last hash function, i.e. h19(input_integer)
If you are interested in the universal hash family for string inputs, you can use the following code. But please note that this code may not be the optimized solution for string hashing.
class UniversalStringHashFamily:
def __init__(self, number_of_hash_functions, number_of_buckets, min_value_for_prime_number=2, bucket_value_offset=0):
self.number_of_buckets = number_of_buckets
self.bucket_value_offset = bucket_value_offset
primes = []
number_to_check = max(min_value_for_prime_number, number_of_buckets)
while len(primes) < number_of_hash_functions:
if is_prime(number_to_check):
primes.append(number_to_check)
number_to_check += random.randint(1, 1000)
self.hash_function_attrs = []
for i in range(number_of_hash_functions):
p = primes[i]
a = random.randint(1, p)
a2 = random.randint(1, p)
b = random.randint(0, p)
self.hash_function_attrs.append((a, b, p, a2))
def hash_int(self, int_to_hash, a, b, p):
return (((a*int_to_hash + b)%p)%self.number_of_buckets) + self.bucket_value_offset
def hash_str(self, str_to_hash, a, b, p, a2):
str_to_hash = "1" + str_to_hash # this will ensure that universality is not affected, see wikipedia for more detail
l = len(str_to_hash)-1
int_to_hash = 0
for i in range(l+1):
int_to_hash += ord(str_to_hash[i]) * (a2 ** (l-i))
int_to_hash = int_to_hash % p
return self.hash_int(int_to_hash, a, b, p)
def __call__(self, function_index, str_to_hash):
a, b, p, a2 = self.hash_function_attrs[function_index]
return self.hash_str(str_to_hash, a, b, p, a2)