I am simulating flipping 999 coins 1000 times, and draw a distribution of sample mean, which may take a long time (about 21 seconds). Is there a better way to do this? a faster way to run for loop, for instance. will vectorizing be useful?
import datetime
import numpy as np
sample_mean_dis = []
start_time = datetime.datetime.now()
# to draw a distribution of sample mean
for i in range(1000):
if not (i%100):
print('iterate: ', i)
sums_1000coins = []
# simulate 1k repetition of experiment_1
# and consider this opertation as a sample
# and compute the sample mean
for i in range(1000):
# this is simulating experiment_1 which flip 999 coins
# and sum heads
coins = np.random.randint(2, size=999)
sums_1000coins.append(np.sum(1 == coins))
sample_mean_dis.append(np.mean(sums_1000coins))
end_time = datetime.datetime.now()
elapsedTime = end_time - start_time
print("Elapsed time: %d seconds" % (elapsedTime.total_seconds()))
To flip 999 coins and see which come up heads, read 999 bits of random data (a bit can either be 0 or 1 with probability 50/50, just like a coin) and then count how many bits are set to 1.
import random
bin(random.getrandbits(999)).count("1")
the above will probably return a number close to 499.5
To flip 999 coins 1000 times do the above in a for loop:
num_heads = [bin(random.getrandbits(999)).count("1") for _ in range(1000)]
num_heads will be a list of 1000 integers normally distributed around 499.5 (999/2).
Related
import numpy as np
from time import time
import matplotlib.pyplot as plt
np.random.seed(27)
mysetup = "from math import sqrt"
begin=time()
i=int(input("Number of rows in first matrix"))
k=int(input("Number of column in first and rows in second matrix"))
j=int(input("Number of columns in second matrix"))
A = np.random.randint(1,10,size = (i,k))
B = np.random.randint(1,10,size = (k,j))
def multiply_matrix(A,B):
global C
if A.shape[1]==B.shape[0]:
C=np.zeros((A.shape[0],B.shape[1]),dtype=int)
for row in range(i):
for col in range(j):
for elt in range(0,len(B)):
C[row,col] += A[row,elt]*B[elt,col]
return C
else:
return "Cannot multiply A and B"
print(f"Matrix A:\n {A}\n")
print(f"Matrix B:\n {B}\n")
D=print(multiply_matrix(A, B))
end=time()
t=print(end-begin)
x=[0,100,10]
y=[100,100,1000]
plt.plot(x,y)
plt.xlabel('Time taken for the program to run')
plt.ylabel('Order of the matrix multiplication')
plt.show()
In the program, I have generated random elements for the matrices to be multiplied.Basically I am trying to compute the time it takes to multiply two matrices.The i,j and k will be considered as the order used for the matrix.As we cannot multiply matrices where number of columns of the first is not equal to the number of the rows in the second, I have already given them the variable 'k'.
Initially I considered to increment the order of the matrix using for loop but wasn't able to do so. I want the graph to display the time it took to multiply the matrices on the x axis and the order of the resultant matrix on the y axis.
There is a problem in the logic I applied but I am not able to find out how to do this problem as I am a beginner in programming
I was expecting to get the result as Y axis having a scale ranging from 0 to 100 with a difference of 10 and x axis with a scale of 100 to 1000 with a difference of 100.
The thousandth entity on the x axis will correspond to the time it took to compute the multiplication of two matrices with numbers of rows and columns as 1000.
Suppose the time it took to compute this was 200seconds. So the graph should be showing the point(1000,200).
Some problematic points I'd like to address -
You're starting the timer before the user chooses an input - which can differ, we want to be as precise as possible, thus we need to only calculate how much time it takes for the multiply_matrix function to run.
Because you're taking an input - it means that each run you will get one result, and one result is only a single point - not a full graph, so we need to get rid of the user input and generate our own.
Moreover to point #2 - we are not interested in giving "one shot" for each matrix order - that means that when we want to test how much time it takes to multiply two matrices of order 300 (for example) - we need to do it N times and take the average in order to be more precise, not to mention we are generating random numbers, and it is possible that some random generated matrices will be easier to compute than other... although taking the average over N tests is not 100% accurate - it does help.
You don't need to set C as a global variable as it can be a local variable of the function multiply_matrix that we anyways return. Also this is not the usage of globals as even with the global C - it will be undefined in the module level.
This is not a must, but it can improve a little bit your program - use time.perf_counter() as it uses the clock with the highest (available) resolution to measure a short duration, and it avoids precision loss by the float type.
You need to change the axes because we want to see how the time is affected by the order of the matrices, not the opposite! (so our X axis is now the order and the Y is the average time it took to multiply them)
Those fixes translate to this code:
Calculating how much it takes for multiply_matrix only.
begin = time.perf_counter()
C = multiply_matrix(A, B)
end = time.perf_counter()
2+3. Generating our own data, looping from order 1 to order maximum_order, taking 50 tests for each order:
maximum_order = 50
tests_number_for_each_order = 50
def generate_matrices_to_graph():
matrix_orders = [] # our X
multiply_average_time = [] # our Y
for order in range(1, maximum_order):
print(order)
times_for_each_order = []
for _ in range(tests_amount_for_each_order):
# generating random square matrices of size order.
A = np.random.randint(1, 10, size=(order, order))
B = np.random.randint(1, 10, size=(order, order))
# getting the time it took to compute
begin = time.perf_counter()
multiply_matrix(A, B)
end = time.perf_counter()
# adding it to the times list
times_for_each_order.append(end - begin)
# adding the data about the order and the average time it took to compute
matrix_orders.append(order)
multiply_average_time.append(sum(times_for_each_order) / tests_amount_for_each_order) # average
return matrix_orders, multiply_average_time
Minor changes to multiply_matrix as we don't need i, j, k from the user:
def multiply_matrix(A, B):
matrix_order = A.shape[1]
C = np.zeros((matrix_order, matrix_order), dtype=int)
for row in range(matrix_order):
for col in range(matrix_order):
for elt in range(0, len(B)):
C[row, col] += A[row, elt] * B[elt, col]
return C
and finally call generate_matrices_to_graph
# calling the generate_data_and_compute function
plt.plot(*generate_matrices_to_graph())
plt.xlabel('Matrix order')
plt.ylabel('Time [in seconds]')
plt.show()
Some outputs:
We can see that when our tests_number_for_each_order is small, the graph loses precision and crisp.
Going from order 1-40 with 1 test for each order:
Going from order 1-40 with 30 tests for each order:
Going from order 1-40 with 80 tests for each order:
I love this kind of questions:
import numpy as np
from time import time
import matplotlib.pyplot as plt
np.random.seed(27)
dim = []
times = []
for i in range(1,10001,10):
A = np.random.randint(1,10,size=(1,i))
B = np.random.randint(1,10,size=(i,1))
begin = time()
C = A*B
times.append(time()-begin)
dim.append(i)
plt.plot(times,dim)
This is a simplified test in which I tested 1 dimension matrices, (1,1)(1,1), (1,10)(10,1), (1,20)(20,1) and so on...
But you can make a double iteration to change also the "outer" dimension of the matrices and see how this affect the computational time
I have a dataset carrying N arrays, each array may have a fixed or variable length. I am going to extract a window from the dataset and counts the occurrence of a given number. To make it simple, I assume each array contains 90 numbers. The model will pick a random position on each array and extract M consecutive numbers started from that position as a window strip, and thus total N strip will be obtained. I assume each strip has the same size so all N strips composing a view (window). I have a C++ code to perform the job and work very well, for a dataset of 5x90, repeating the window sampling 1000000 times, it takes roughly 0.002 seconds to finish every 10,000 iterations. I am trying to port the C++ to Python such that I could use the powerful data frame (Pandas) and other tools for related analysis. After some research, it seems that the Numpy array is a good representation of the data array but since the size of each array varied, I need to warp all arrays into a list. The code I have shown below (majority of irrelated code are removed)
import numpy as np
import random
import time
class dataSource:
_data = None
_NCOLS = 0
_NROWS = 0
# NCOLS, NROWS is the dimension of a window (view) of the data
# _data is a list of NCOLS Numpy array, each array carry 90 numbers
def __init__(self, NCOLS, NROWS):
# assuming each array has 90 numbers for this exmaple, it could be varied in actual case
src = [[1]*90]*NCOLS # just assume all numbers in the array to be 1 for this example
self._data = [np.array(cc) for cc in src]
self._NCOLS = NCOLS
self._NROWS = NROWS
def extractView(self, view):
pos = [random.randint(0, len(cc)-1-self._NROWS) for cc in self._data]
for col in range(len(self._data)):
view[:, col] = self._data[col][pos[col]:pos[col]+self._NROWS]
class dataView:
_NCOLS = 0
_NROWS = 0
_view = None
def __init__(self, NCOLS, NROWS):
self._NCOLS = NCOLS
self._NROWS = NROWS
self._view = np.zeros([self._NROWS, self._NCOLS], order='F')
def __setitem__(self, key, value):
self._view[key] = value
def count(self, elem):
return np.count_nonzero(self._view==elem)
if __name__ == '__main__':
ds = dataSource(5, 3)
dv = dataView(5, 3)
batchCount, totalTime, startTime = 1, 0, time.process_time()
for n in range(10000000):
if ((n+1)%10000==0):
dt = time.process_time() - startTime
totalTime += dt
print(totalTime / batchCount, ' ', dt)
batchCount += 1
startTime = time.process_time()
ds.extractView(dv)
cnt = dv.count(1) # this could then be used for other analysis, dv may be used by other functions
Running this code in Python 3.8.8 showing that the code takes 0.23 seconds to finish every 10,000 iterations. The Python code is about 100 times slower than C++. In an actual case, the data source could have up to 50 arrays and each may have 50 to 500 numbers, so the run times in Python could be even longer. To apply the C++ code in the real data, it takes about 5 hours to finish all iterations, based on the above estimation, it takes 500 hours to do the same job in Python. I understand that there may be some limit in Python that makes it hard to boost the speed up, but if anything that I could do to optimize my code so to reduce the time (even 10%) will help a lot.
I am working on a project which is aiming to show difference between good form and bad form of an exercise. To do this we collected the acceleration data with wrist based accelerometer. The image above shows 2 set of a fitness execise (bench press). Each set has 10 repetitions. And the image below shows 10 repetitions of 1 set.I have a raw data set which consist of 10 set of an execises. What I want to do is splitting the raw data to 10 parts which will contain the part between 2 black line in the image above so I can analyze the data easily. My supervisor gave me a starting point which is choosing cutpoint in the each set. He said take a cutpoint, find the first interruption time start cutting at 3 sec before that time and count to 10 and finish cutting.
This an idea that I don't know how to apply. At least, if you can tell how to cut a dataframe according to cutpoint I would be greatful.
Well, I found another way to detect periodic parts of my accelerometer data. So, Here is my code:
import numpy as np
from peakdetect import peakdetect
import datetime as dt
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
from matplotlib import style
from pandas import DataFrame as df
style.use('ggplot')
def get_periodic(path):
periodics = []
data_frame = df.from_csv(path)
data_frame.columns = ['z', 'y', 'x']
if path.__contains__('1'):
if path.__contains__('bench'):
bench_press_1_week = data_frame.between_time('11:24', '11:52')
peak_indexes = get_peaks(bench_press_1_week.y, lookahead=3000)
for i in range(0, len(peak_indexes)):
time_indexes = bench_press_1_week.index.tolist()
start_time = time_indexes[0]
periodic_start = start_time.to_datetime() + dt.timedelta(0, peak_indexes[i] / 100)
periodic_end = periodic_start + dt.timedelta(0, 60)
periodic = bench_press_1_week.between_time(periodic_start.time(), periodic_end.time())
periodics.append(periodic)
return periodics
def get_peaks(data, lookahead):
peak_indexes = []
correlation = np.correlate(data, data, mode='full')
realcorr = correlation[correlation.size / 2:]
maxpeaks, minpeaks = peakdetect(realcorr, lookahead=lookahead)
for i in range(0, len(maxpeaks)):
peak_indexes.append(maxpeaks[i][0])
return peak_indexes
def show_segment_plot(data, periodic_area, exercise_name):
plt.figure(8)
gs = gridspec.GridSpec(7, 2)
ax = plt.subplot(gs[:2, :])
plt.title(exercise_name)
ax.plot(data)
k = 0
for i in range(2, 7):
for j in range(0, 2):
ax = plt.subplot(gs[i, j])
title = "{} {}".format(k + 1, ".Set")
plt.title(title)
ax.plot(periodic_area[k])
k = k + 1
plt.show()
Firstly, this question gave me another perspective for my problem. The image below shows the raw accelerometer data of bench press with 10 sets. Here it has 3 axis(x,y,z) and it's major axis is y(Blue on the image).
I used autocorrelation function for detecting the periodic parts, In the image above every peak represents 1 set of execises. With this peak detection algorithm I found each peak's x-axis value,
In[196]: maxpeaks
Out[196]:
[[16204, 32910.14013671875],
[32281, 28726.95849609375],
[48515, 24583.898681640625],
[64436, 22088.130859375],
[80335, 19582.248291015625],
[96699, 16436.567626953125],
[113081, 12100.027587890625],
[129027, 8098.98486328125],
[145184, 5387.788818359375]]
Basically, each x-value represent samples. My sampling frequency was 100Hz so 16204/100 = 162,04 seconds. To find the time of periodic part I added 162,04 sec to started time. Each bench press took aproximatelly 1 min and in this example, exercise's starting time was 11:24, for first periodic part's start time is 11:26 and ending time is 1 min after. There is some lag but yes best solution that I found is this.
I want to repeat a task several times during an interval of time and I want to do that in an uniform way so if I want to do the task 4 times in 1 second it should be executed at t = 0, 0.25, 0.5 and 0.75.
So now I am doing:
import math
import socket
s = socket.socket(...) #not important
time_step = 1./num_times_executed
for _ in num_times_executed:
now = time.time()
s.sendto(...) #action i do
time.sleep(max(0,time_step-(time.time()-now)))
However there is a lot of overhead, the bigger the loop is the more drift I get. For example with num_times_executed = 800, it takes 1.1 seconds so ~ 10% wrong...
Is there a way to do that with a good precision ?
time_step = 1./num_times_executed
start = time.time()
for i in num_times_executed:
s.sendto(...) #action i do
next_send_time = start + (i+1) * time_step
time.sleep(max(0,next_send_time - time.time()))
Now you're not going to get any drift because the time steps are firmly set values off of a start time. Previously the little calculations happening before setting now = time.time() would cause a tiny drift, but now so long as your time_step is long enough to execute the s.sendto(...) command, you shouldn't have any drift.
I am trying to optimize a function that returns a subcovariance matrix from the full covariance matrix given the desired memmbers. The full covariance matrix could contains 500+ items and I could be looking at a variable number of members each time but likely 20 or less per call. This gets called 10,000+ times.
My code works but I was wondering how to optimize it.
def subcovar(covar,elements):
numelements = elements.shape[0]
subcovar = np.zeros((numelements,numelements))
for i in range(0,numelements):
for j in range(0,numelements):
subcovar[i,j]= covar[elements[i],elements[j]]
return subcovar
Thanks
Paul
Why not use matrix slicing? [offical python docs]
Here's a SO thread on building a sub-matrix by extracting arbitrary (ie non-sequential) rows and columns
Slicing of a numpy 2d array, or how do I extract an mxm submatrix from an nxn array (n>m)
Based on Mike's response I went back, looked at slicing and found the following:
Old way time = 0.016201255250881966 sec, choosing a 20x20 from a 500x500
New way time = 0.0016199544633708396 sec, choosing a 20x20 from a 500x500
10x faster
Old way time = 0.09903732167528723 sec, choosing a 50x50 from a 500x500
New way time = 0.00229222701258387 sec, choosing a 50x50 from a 500x500
43x faster
Old way time = 2.669313751708479 sec, choosing a 250x250 from a 500x500
New way time = 0.003080821529599664 sec, choosing a 250x250 from a 500x500
866x faster
oldway
def subcovarold(covar,elements):
start2 = time.clock()
covar = np.arange(250000).reshape((500, 500))
elements = np.arange(0,500,25) # this is the elements to choose
numelements = elements.shape[0]
subcovar= np.zeros((numelements,numelements))
for i in range(0,numelements):
for j in range(0,numelements):
subcovar[i,j]= covar[elements[i],elements[j]]
end2 = time.clock()
print ("time ", end2 - start2)
return subcovar
newway
def subcovarnew(covar,elements):
start2 = time.clock()
covar = np.arange(250000).reshape((500, 500))
elements = np.arange(0,500,2)
msize = elements.shape[0]
ii = elements.reshape(msize,1)
jj = elements.reshape(1,msize)
subcovar = covar[ii,jj]
end2 = time.clock()
print ("time ", end2 - start2)
return subcovar
Thanks
Paul