Develop Reference

Numpy mean of flattened large array slower than mean of mean of all axes

Running Numpy version 1.19.2, I get better performance cumulating the mean of every individual axis of an array than by calculating the mean over an already flattened array.
shape = (10000,32,32,3)
mat = np.random.random(shape)
# Call this Method A.
%%timeit
mat_means = mat.mean(axis=0).mean(axis=0).mean(axis=0)
14.6 ms ± 167 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
mat_reshaped = mat.reshape(-1,3)
# Call this Method B
%%timeit
mat_means = mat_reshaped.mean(axis=0)
135 ms ± 227 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
This is odd, since doing the mean multiple times has the same bad access pattern (perhaps even worse) than the one on the reshaped array. We also do more operations this way. As a sanity check, I converted the array to FORTRAN order:
mat_reshaped_fortran = mat.reshape(-1,3, order='F')
%%timeit
mat_means = mat_reshaped_fortran.mean(axis=0)
12.2 ms ± 85.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
This yields the performance improvement I expected.
For Method A, prun gives:
36 function calls in 0.019 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
3 0.018 0.006 0.018 0.006 {method 'reduce' of 'numpy.ufunc' objects}
1 0.000 0.000 0.019 0.019 {built-in method builtins.exec}
3 0.000 0.000 0.019 0.006 _methods.py:143(_mean)
3 0.000 0.000 0.000 0.000 _methods.py:59(_count_reduce_items)
1 0.000 0.000 0.019 0.019 <string>:1(<module>)
3 0.000 0.000 0.019 0.006 {method 'mean' of 'numpy.ndarray' objects}
3 0.000 0.000 0.000 0.000 _asarray.py:86(asanyarray)
3 0.000 0.000 0.000 0.000 {built-in method numpy.array}
3 0.000 0.000 0.000 0.000 {built-in method numpy.core._multiarray_umath.normalize_axis_index}
6 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance}
6 0.000 0.000 0.000 0.000 {built-in method builtins.issubclass}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
While for Method B:
14 function calls in 0.166 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.166 0.166 0.166 0.166 {method 'reduce' of 'numpy.ufunc' objects}
1 0.000 0.000 0.166 0.166 {built-in method builtins.exec}
1 0.000 0.000 0.166 0.166 _methods.py:143(_mean)
1 0.000 0.000 0.000 0.000 _methods.py:59(_count_reduce_items)
1 0.000 0.000 0.166 0.166 <string>:1(<module>)
1 0.000 0.000 0.166 0.166 {method 'mean' of 'numpy.ndarray' objects}
1 0.000 0.000 0.000 0.000 _asarray.py:86(asanyarray)
1 0.000 0.000 0.000 0.000 {built-in method numpy.array}
1 0.000 0.000 0.000 0.000 {built-in method numpy.core._multiarray_umath.normalize_axis_index}
2 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance}
2 0.000 0.000 0.000 0.000 {built-in method builtins.issubclass}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
Note: np.setbufsize(1e7) doesn't seem to have any effect.
What is the reason for this performance difference?

Let's call your original matrix mat. mat.shape = (10000,32,32,3). Visually, this is like having a "stack" of 10,000 * 32x32x3 * rectangular prisms (I think of them as LEGOs) of floats.
Now lets think about what you did in terms of floating point operations (flops):
In Method A, you do mat.mean(axis=0).mean(axis=0).mean(axis=0). Let's break this down:
You take the mean of each position (i,j,k) across all 10,000 LEGOs. This gives you back a single LEGO of size 32x32x3 which now contains the first set of means. This means you have performed 10,000 additions and 1 division per mean, of which there are 32323 = 3072. In total, you've done 30,723,072 flops.
You then take the mean again, this time of each position (j,k), where i is now the number of the layer (vertical position) you are currently on. This gives you a piece of paper with 32x3 means written on it. You have performed 32 additions and 1 divisions per mean, of which there are 32*3 = 96. In total, you've done 3,168 flops.
Finally, you take the mean of each column k, where j is now the row you are currently on. This gives you a stub with 3 means written on it. You have performed 32 additions and 1 division per mean, of which there are 3. In total, you've done 99 flops.
The grand total of all this is 30,723,072 + 3,168 + 99 = 30,726,339 flops.
In Method B, you do mat_reshaped = mat.reshape(-1,3); mat_means = mat_reshaped.mean(axis=0). Let's break this down:
You reshaped everything, so mat is a long roll of paper of size 10,240,000x3. You take the mean of each column k, where j is now the row you are currently on. This gives you a stub with 3 means written on it. You have performed 10,240,000 additions and 1 division per mean, of which there are 3. In total, you've done 30,720,003 flops.
So now you're saying to yourself "What! All of that work, only to show that the slower method actually does ~less~ work?! " Here's the problem: Although Method B has less work to do, it does not have a lot less work to do, meaning just from a flop standpoint, we would expect things to be similar in terms of runtime.
You also have to consider the size of your reshaped array in Method B: a matrix with 10,240,000 rows is HUGE!!! It's really hard/inefficient for the computer to access all of that, and more memory accesses means longer runtimes. The fact is that in its original 10,000x32x32x3 shape, the matrix was already partitioned into convenient slices that the computer could access more efficiently: this is actually a common technique when handling giant matrices Jaime's response to a similar question or even this article: both talk about how breaking up a big matrix into smaller slices helps your program be more memory efficient, therefore making it run faster.

How to improve performance of bi-level image type in MagickWand Python?

I am working on a python script which converts DXF files from AutoCAD into high resolution bi-level TIFF files using the MagickWand bindings library. The script works exactly as expected, but I am looking for ways to improve the performance.
Here is the script:
import dxfgrabber
import cv2
import numpy as np
import sys
from wand.image import Image
RESOLUTION_DPI = 1000
def drawPolyLines(pl):
if(pl.is_closed):
points = []
for pt in pl.points:
points += [[xToPix(pt[0]),yToPix(pt[1])]]
points = np.array(points,np.int32)
cv2.fillPoly(canvas,[points],0,cv2.LINE_8)
dxf = dxfgrabber.readfile("812b.dxf")
shapes = dxf.entities.get_entities()
in_limMin = dxf.header['$LIMMIN']
in_limMax = dxf.header['$LIMMAX']
pix_limMin = tuple([int(z * RESOLUTION_DPI) for z in in_limMin])
pix_limMax = tuple([int(z * RESOLUTION_DPI) for z in in_limMax])
#Translate x,y values to pixels
def xToPix(coord):
return int((coord*RESOLUTION_DPI))
def yToPix(coord, ymax=pix_limMax[1]):
return int(ymax - (coord*RESOLUTION_DPI))
canvas = np.zeros((pix_limMax[1],pix_limMax[0]), np.uint8)
canvas.fill(1)
for shape in shapes:
if shape.dxftype == 'POLYLINE':
drawPolyLines(shape)
with Image.from_array(canvas) as img:
img.resolution = RESOLUTION_DPI
img.compression='rle'
img.type='bilevel'
img.depth = 1
img.save(filename='result.tif')
It takes about 30 seconds to convert one file and I am looking to improve this. A cProfile sorted by cumulative time shows the following:
$ python -m cProfile -s 'cumtime' test.py
324966 function calls (322147 primitive calls) in 37.550 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
488/1 0.001 0.000 37.557 37.557 {built-in method builtins.exec}
1 0.000 0.000 37.557 37.557 test.py:1(<module>)
3 0.000 0.000 31.180 10.393 image.py:1013(wrapped)
1 30.500 30.500 30.500 30.500 image.py:2451(type)
1 4.329 4.329 4.329 4.329 image.py:9251(save)
1 1.485 1.485 1.487 1.487 image.py:8754(from_array)
1 0.681 0.681 0.681 0.681 image.py:1629(depth)
16 0.001 0.000 0.446 0.028 __init__.py:1(<module>)
208/5 0.002 0.000 0.229 0.046 <frozen importlib._bootstrap>:986(_find_and_load)
207/5 0.001 0.000 0.228 0.046 <frozen importlib._bootstrap>:956(_find_and_load_unlocked)
197/6 0.001 0.000 0.225 0.038 <frozen importlib._bootstrap>:650(_load_unlocked)
160/6 0.000 0.000 0.225 0.037 <frozen importlib._bootstrap_external>:777(exec_module)
290/6 0.000 0.000 0.221 0.037 <frozen importlib._bootstrap>:211(_call_with_frames_removed)
330/3 0.001 0.000 0.179 0.060 {built-in method builtins.__import__}
198/23 0.000 0.000 0.173 0.008 <frozen importlib._bootstrap>:1017(_handle_fromlist)
197/58 0.000 0.000 0.161 0.003 <frozen importlib._bootstrap>:549(module_from_spec)
22/3 0.000 0.000 0.159 0.053 <frozen importlib._bootstrap_external>:1099(create_module)
22/3 0.029 0.001 0.159 0.053 {built-in method _imp.create_dynamic}
1 0.000 0.000 0.158 0.158 __init__.py:41(readfile)
1 0.000 0.000 0.158 0.158 __init__.py:52(readfile_as_asc)
1 0.000 0.000 0.116 0.116 __init__.py:61(_read_encoded_file)
1 0.110 0.110 0.110 0.110 {method 'fill' of 'numpy.ndarray' objects}
As you can see, almost the entire run time is spent in the resulting call of setting the image type to bilevel. I've dug into the MagickWand source a bit and found a dead-end when the SetImageType() method is called from within MagickImage.MagickSetImageType(). I'm wondering if there is a thresholding operation which is converting every pixel to a single bit value. This is probably time consuming, and pointless in this case since the values of the matrix are only 0 or 1. Unfortunately, the opencv draw methods require 8bit integer type and I do not know if there is another data type I can feed into MagickWand that will improve performance. Looking for any suggestions to speed up this operation. Thanks.

strange result from timeit

I tried to repeat the functionality of IPython %time, but for some strange reason, results of testing of some function are horrific.
IPython:
In [11]: from random import shuffle
....: import numpy as np
....: def numpy_seq_el_rank(seq, el):
....: return sum(seq < el)
....:
....: seq = np.array(xrange(10000))
....: shuffle(seq)
....:
In [12]: %timeit numpy_seq_el_rank(seq, 10000//2)
10000 loops, best of 3: 46.1 µs per loop
Python:
from timeit import timeit, repeat
def my_timeit(code, setup, rep, loops):
result = repeat(code, setup=setup, repeat=rep, number=loops)
return '%d loops, best of %d: %0.9f sec per loop'%(loops, rep, min(result))
np_setup = '''
from random import shuffle
import numpy as np
def numpy_seq_el_rank(seq, el):
return sum(seq < el)
seq = np.array(xrange(10000))
shuffle(seq)
'''
np_code = 'numpy_seq_el_rank(seq, 10000//2)'
print 'Numpy seq_el_rank:\n\t%s'%my_timeit(code=np_code, setup=np_setup, rep=3, loops=100)
And its output:
Numpy seq_el_rank:
100 loops, best of 3: 1.655324947 sec per loop
As you can see, in python i made 100 loops instead 10000 (and get 35000 times slower result) as in ipython, because it takes really long time. Can anybody explain why result in python is so slow?
UPD:
Here is cProfile.run('my_timeit(code=np_code, setup=np_setup, rep=3, loops=10000)') output:
30650 function calls in 4.987 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 4.987 4.987 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 <timeit-src>:2(<module>)
3 0.001 0.000 4.985 1.662 <timeit-src>:2(inner)
300 0.006 0.000 4.961 0.017 <timeit-src>:7(numpy_seq_el_rank)
1 0.000 0.000 4.987 4.987 Lab10.py:47(my_timeit)
3 0.019 0.006 0.021 0.007 random.py:277(shuffle)
1 0.000 0.000 0.002 0.002 timeit.py:121(__init__)
3 0.000 0.000 4.985 1.662 timeit.py:185(timeit)
1 0.000 0.000 4.985 4.985 timeit.py:208(repeat)
1 0.000 0.000 4.987 4.987 timeit.py:239(repeat)
2 0.000 0.000 0.000 0.000 timeit.py:90(reindent)
3 0.002 0.001 0.002 0.001 {compile}
3 0.000 0.000 0.000 0.000 {gc.disable}
3 0.000 0.000 0.000 0.000 {gc.enable}
3 0.000 0.000 0.000 0.000 {gc.isenabled}
1 0.000 0.000 0.000 0.000 {globals}
3 0.000 0.000 0.000 0.000 {isinstance}
3 0.000 0.000 0.000 0.000 {len}
3 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
29997 0.001 0.000 0.001 0.000 {method 'random' of '_random.Random' objects}
2 0.000 0.000 0.000 0.000 {method 'replace' of 'str' objects}
1 0.000 0.000 0.000 0.000 {min}
3 0.003 0.001 0.003 0.001 {numpy.core.multiarray.array}
1 0.000 0.000 0.000 0.000 {range}
300 4.955 0.017 4.955 0.017 {sum}
6 0.000 0.000 0.000 0.000 {time.clock}

Well, one issue is that you're misreading the results. ipython is telling you how long it took each of the 10,000 iterations for the set of 10,000 iterations with the lowest total time. The timeit.repeat module is reporting how long the whole round of 100 iterations took (again, for the shortest of three). So the real discrepancy is 46.1 µs per loop (ipython) vs. 16.5 ms per loop (python), still a factor of ~350x difference, but not 35,000x.
You didn't show profiling results for ipython. Is it possible that in your ipython session, you did either from numpy import sum or from numpy import *? If so, you'd have been timing the numpy.sum (which is optimized for numpy arrays and would run several orders of magnitude faster), while your python code (which isolated the globals in a way that ipython does not) ran the normal sum (that has to convert all the values to Python ints and sum them).
If you check your profiling output, virtually all of your work is being done in sum; if that part of your code was sped up by several orders of magnitude, the total time would reduce similarly. That would explain the "real" discrepancy; in the test case linked above, it was a 40x difference, and that was for a smaller array (the smaller the array, the less numpy can "show off") with more complex values (vs. summing 0s and 1s here I believe).
The remainder (if any) is probably an issue of how the code is being evaled slightly differently, or possibly weirdness with the random shuffle (for consistent tests, you'd want to seed random with a consistent seed to make the "randomness" repeatable) but I doubt that's a difference of more than a few percent.

There could be any number of reasons this code is running slower in one implementation of python than another. One may be optimized differently than another, one may pre-compile certain parts while the other is fully interpreted. The only way to figure out why is to profile your code.
https://docs.python.org/2/library/profile.html
import cProfile
cProfile.run('repeat(code, setup=setup, repeat=rep, number=loops)')
Will give a result similar to
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 <stdin>:1(testing)
1 0.000 0.000 0.000 0.000 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
1 0.000 0.000 0.000 0.000 {method 'upper' of 'str' objects}
Which shows you when function calls were made, how many times they were made and how long they took.

Python Boto Dynamodb very slow performance for small record set retrieval on range keys

I am testing dynamodb via boto and have found it to be surprisingly slow in retrieving data sets based on hashkey, rangekey condition queries. I have seen some discussion about the oddity that causes ssl (is_secure) to perform about 6x faster then non-ssl and I can confirm that finding. But even using ssl I am seeing 1-2 seconds to retrieve 300 records using a hashkey/range key condition on a fairly small data set (less then 1K records).
Running profilehooks profiler I see a lot of extraneous time spent in ssl.py to the order of 20617 ncalls to retrieve the 300 records. It seems like even at 10 calls per record it's still 6x more then I would expect. This is on a medium instance-- though the same results occur on a micro instance. 500 Reads/sec 1000 writes/sec provisioning with no throttles logged.
I have looked at doing a batch request but the inability to use range key conditions eliminates that option for me.
Any ideas on where I'm loosing time would be greatly appreciated!!
144244 function calls in 2.083 CPU seconds
Ordered by: cumulative time, internal time, call count
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.001 0.001 2.083 2.083 eventstream.py:427(session_range)
107 0.006 0.000 2.081 0.019 dynamoDB.py:36(rangeQ)
408 0.003 0.000 2.073 0.005 layer2.py:493(query)
107 0.001 0.000 2.046 0.019 layer1.py:435(query)
107 0.002 0.000 2.040 0.019 layer1.py:119(make_request)
107 0.006 0.000 1.988 0.019 connection.py:699(_mexe)
107 0.001 0.000 1.916 0.018 httplib.py:956(getresponse)
107 0.002 0.000 1.913 0.018 httplib.py:384(begin)
662 0.049 0.000 1.888 0.003 socket.py:403(readline)
20617 0.040 0.000 1.824 0.000 ssl.py:209(recv)
20617 0.036 0.000 1.785 0.000 ssl.py:130(read)
20617 1.748 0.000 1.748 0.000 {built-in method read}
107 0.002 0.000 1.738 0.016 httplib.py:347(_read_status)
107 0.001 0.000 0.170 0.002 mimetools.py:24(__init__)
107 0.000 0.000 0.165 0.002 rfc822.py:88(__init__)
107 0.007 0.000 0.165 0.002 httplib.py:230(readheaders)
107 0.001 0.000 0.031 0.000 __init__.py:332(loads)
107 0.001 0.000 0.028 0.000 decoder.py:397(decode)
107 0.008 0.000 0.026 0.000 decoder.py:408(raw_decode)
107 0.001 0.000 0.026 0.000 httplib.py:910(request)
107 0.003 0.000 0.026 0.000 httplib.py:922(_send_request)
107 0.001 0.000 0.025 0.000 connection.py:350(authorize)
107 0.004 0.000 0.024 0.000 auth.py:239(add_auth)
3719 0.011 0.000 0.019 0.000 layer2.py:31(item_object_hook)
301 0.010 0.000 0.018 0.000 item.py:38(__init__)
22330 0.015 0.000 0.015 0.000 {method 'append' of 'list' objects}
107 0.001 0.000 0.012 0.000 httplib.py:513(read)
214 0.001 0.000 0.011 0.000 httplib.py:735(send)
856 0.002 0.000 0.010 0.000 __init__.py:1034(debug)
214 0.001 0.000 0.009 0.000 ssl.py:194(sendall)
107 0.000 0.000 0.008 0.000 httplib.py:900(endheaders)
107 0.001 0.000 0.008 0.000 httplib.py:772(_send_output)
107 0.001 0.000 0.008 0.000 auth.py:223(string_to_sign)
856 0.002 0.000 0.008 0.000 __init__.py:1244(isEnabledFor)
137 0.001 0.000 0.008 0.000 httplib.py:603(_safe_read)
214 0.001 0.000 0.007 0.000 ssl.py:166(send)
214 0.007 0.000 0.007 0.000 {built-in method write}
3311 0.006 0.000 0.006 0.000 item.py:186(__setitem__)
107 0.001 0.000 0.006 0.000 auth.py:95(sign_string)
137 0.001 0.000 0.006 0.000 socket.py:333(read)

This isn't a complete answer but I thought it was worth posting it at this time.
I've heard reports like this from a couple of people over the last few weeks. I was able to reproduce the anomaly of HTTPS being considerably faster than HTTP but wasn't able to track it down. It seemed like that problem was unique to Python/boto but it turns out the same issue was found on C#/.Net and investigating that it was found that the underlying problem was the use of the Nagle's algorithm in the Python and .Net libraries. In .Net, it's easy to turn this off but it's not as easy in Python, unfortunately.
To test this, I wrote a simple script that performed 1000 GetItem requests in a loop. The item that was being fetch was very small, well under 1K. Running this on Python 2.6.7 on an m1.medium instance in the us-east-1 region produced these results:
>>> http_data = speed_test(False, 1000)
dynamoDB_speed_test - RUNTIME = 53.120193
Throttling exceptions: 0
>>> https_data = speed_test(True, 1000)
dynamoDB_speed_test - RUNTIME = 8.167652
Throttling exceptions: 0
Note that there is sufficient provisioned capacity in the table to avoid any throttling from the service and the unexpected gap between HTTP and HTTPS is clear.
I next ran the same test in Python 2.7.2:
>>> http_data = speed_test(False, 1000)
dynamoDB_speed_test - RUNTIME = 5.668544
Throttling exceptions: 0
>>> https_data = speed_test(True, 1000)
dynamoDB_speed_test - RUNTIME = 7.425210
Throttling exceptions: 0
So, 2.7 seems to have fixed this issue. I then applied a simple patch to httplib.py in 2.6.7. The patch simply sets the TCP_NO_DELAY property of the socket associated with the HTTPConnection object, like this:
self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
I then re-ran the test on 2.6.7:
>>> http_data = speed_test(False, 1000)
dynamoDB_speed_test - RUNTIME = 5.914109
Throttling exceptions: 0
>>> https_data = speed_test(True, 1000)
dynamoDB_speed_test - RUNTIME = 5.137570
Throttling exceptions: 0
Even better although still an expectedly faster time with HTTPS than HTTP. It's hard to know whether that difference is significant or not.
So, I'm looking for ways to programmatically configure the socket for HTTPConnection objects to have TCP_NO_DELAY configured correctly. It's not easy to get at that in the httplib.py. My best advice for the moment is to use Python 2.7, if possible.

Single pass algorithm for finding the topX percent of items

I'm looking for a single-pass algorithm for finding the topX percent of floats in a stream where I do not know the total number ahead of time ... but its on the order of 5-30 million floats. It needs to be single-pass since the data is generated on the fly and recreate the exact stream a second time.
The algorithm I have so far is to keep a sorted list of the topX items that I've seen so far. As the stream continues I enlarge the list as needed. Then I use bisect_left to find the insertion point if needed.
Below is the algorithm I have so far:
from bisect import bisect_left
from random import uniform
from itertools import islice
def data_gen(num):
for _ in xrange(num):
yield uniform(0,1)
def get_top_X_percent(iterable, percent = 0.01, min_guess = 1000):
top_nums = sorted(list(islice(iterable, int(percent*min_guess)))) #get an initial guess
for ind, val in enumerate(iterable, len(top_nums)):
if int(percent*ind) > len(top_nums):
top_nums.insert(0,None)
newind = bisect_left(top_nums, val)
if newind > 0:
top_nums.insert(newind, val)
top_nums.pop(0)
return top_nums
if __name__ == '__main__':
num = 1000000
all_data = sorted(data_gen(num))
result = get_top_X_percent(all_data)
assert result[0] == all_data[-int(num*0.01)], 'Too far off, lowest num:%f' % result[0]
print result[0]
In the real case the data does not come from any standard distribution (otherwise I could use some statistics knowledge).
Any suggestions would be appreciated.

I'm not sure there's any way to actually do that reliably, as the range denoted by the "top X percent" can grow unpredictably as you see more elements. Consider the following input:
101 102 103 104 105 106 107 108 109 110 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0...
If you wanted the top 25% of elements, you'd end up picking 101 and 102 out of the first ten elements, but after seeing enough zeroes after there you'd eventually have to end up selecting all of the first ten. This same pattern can be expanded to any sufficiently large stream -- it's always possible to end up getting misled by appearances and discarding elements that you actually should have kept. As such, unless you know the exact length of the stream ahead of time, I don't think this is possible (short of keeping every element in memory until you hit the end of the stream).

You must store the entire stream in memory.
Proof: You have a sequence of numbers, n1,…,nk. The value of k is unknown. How do you know when ni can be forgotten? When you have seen x*k/100 numbers greater than ni. However, since k is unknown, you can never do this.
So the only "one-pass" algorithm must store the entire sequence in memory.

As the other answers have discussed, you can't really do a whole lot better than just storing the entire stream in memory. Consider doing it that way, especially since 5-30 million floats is probably only 40-240 MB of memory which is manageable.
Given that you store the entire stream, the algorithmically fastest way to get the topX percent is by first finding the cutoff element (the smallest element that is in the topX percent) using a linear-time selection algorithm:
http://en.wikipedia.org/wiki/Selection_algorithm
Then, make another pass through the stream and filter out all the elements smaller than the cutoff element.
This method is linear time and linear space, which is the best you can hope for.

Below is the output from running this with cProfile. It seems that your code is doing fine, as most calls are 0.000 (percall). It seems its slow simply because you have many items to process. If you wanted to optimize, you'd have to try and pop less items, as that is what has it has been called 999999, which seems to be unnecessary.
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 __future__.py:48(<module>)
1 0.000 0.000 0.000 0.000 __future__.py:74(_Feature)
7 0.000 0.000 0.000 0.000 __future__.py:75(__init__)
1 0.001 0.001 0.001 0.001 bisect.py:1(<module>)
1 0.001 0.001 0.001 0.001 hashlib.py:55(<module>)
6 0.000 0.000 0.000 0.000 hashlib.py:91(__get_openssl_constructor)
1 0.000 0.000 0.000 0.000 os.py:743(urandom)
1 0.000 0.000 0.000 0.000 random.py:100(seed)
1000000 0.731 0.000 0.876 0.000 random.py:355(uniform)
1 0.003 0.003 0.004 0.004 random.py:40(<module>)
1 0.000 0.000 0.000 0.000 random.py:647(WichmannHill)
1 0.000 0.000 0.000 0.000 random.py:72(Random)
1 0.000 0.000 0.000 0.000 random.py:797(SystemRandom)
1 0.000 0.000 0.000 0.000 random.py:91(__init__)
1 2.498 2.498 13.313 13.313 test.py:12(get_top_X_percent)
1 0.006 0.006 16.330 16.330 test.py:3(<module>)
1000001 0.545 0.000 1.422 0.000 test.py:8(data_gen)
1000000 1.744 0.000 1.744 0.000 {_bisect.bisect_left}
1 0.000 0.000 0.000 0.000 {_hashlib.openssl_md5}
1 0.000 0.000 0.000 0.000 {_hashlib.openssl_sha1}
1 0.000 0.000 0.000 0.000 {_hashlib.openssl_sha224}
1 0.000 0.000 0.000 0.000 {_hashlib.openssl_sha256}
1 0.000 0.000 0.000 0.000 {_hashlib.openssl_sha384}
1 0.000 0.000 0.000 0.000 {_hashlib.openssl_sha512}
1 0.000 0.000 0.000 0.000 {binascii.hexlify}
1 0.000 0.000 0.000 0.000 {function seed at 0x100684a28}
6 0.000 0.000 0.000 0.000 {getattr}
6 0.000 0.000 0.000 0.000 {globals}
1000004 0.125 0.000 0.125 0.000 {len}
1 0.000 0.000 0.000 0.000 {math.exp}
2 0.000 0.000 0.000 0.000 {math.log}
1 0.000 0.000 0.000 0.000 {math.sqrt}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
1009989 0.469 0.000 0.469 0.000 {method 'insert' of 'list' objects}
999999 8.477 0.000 8.477 0.000 {method 'pop' of 'list' objects}
1000000 0.146 0.000 0.146 0.000 {method 'random' of '_random.Random' objects}
1 0.000 0.000 0.000 0.000 {posix.close}
1 0.000 0.000 0.000 0.000 {posix.open}
1 0.000 0.000 0.000 0.000 {posix.read}
2 1.585 0.792 3.006 1.503 {sorted}
BTW, you can use cProfile with:
python -m cProfile test.py