Performing Arithmetic operations on nested dataframe containing a list - python

I have a dataframe called dailyHistogram defined as follows:
dailyHistogram = pd.DataFrame({'NumVisits':[[0 for x in range(1440)]
for y in range (180)],
'DoW': [0]*ReportingDateRange.size
}
,columns=['NumVisits','DoF'],
index=ReportingDateRange)
Where NumVisits is a two-dimensional array (1440 by 180) and holds a histogram of some activity in 180 days. DoW is simply a column which holds the day of the week.
The index in this dataframe is the dates on which the activities occurred.
My problem is in performing any operations on dailyHistogram["NumVisits"].
Here's what dailyHistogram["NumVisits"] looks like:
> dailyHistogram["NumVisits"]
> Out[193]:
> 2016-01-01 [5, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
>
> 2016-01-02 [2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
>
> 2016-01-03 [6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
>
> 2016-01-04 [8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
>
> 2016-06-26 [3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
>
> 2016-06-27 [4, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
>
> 2016-06-28 [7, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
>
> 2016-06-29 [7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
>
> 2016-06-30 [4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
>
> Freq: D, Name: NumVisits, dtype: object
I'd like to sum up all vectors in "NumVisits" for a particular day of the week, but no arithmetic operations seem to be possible on dailyHistogram["NumVisits"]


That is because NumVisits is a list, and to perform arithmetic on the contents of a list, you need to explicitly apply your functions. For example:
df['NumVisits'].apply(sum)
For element by element sum in each row:
import numpy as np
df['NumVisits'].apply(np.cumsum)
For sum across all rows, for each element in row:
np.array(dailyHistogram['NumVisits'].tolist()).sum(axis=0)

Related

am trying to convert this for loop into list comprehension?

L=[]
for i in range(11):
L.append(1)
for z in range(i):
L.append(0)
L.append(1) #this is to add a 1 at the end of list
print(L)

This should give the desired result - for i in the desired range, generate the tuple (1,0,0,0...) and then flatten the list of tuples. Finally, append the trailing 1.
>>> [x for i in range(11) for x in (1,)+(0,)*i] + [1]
[1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]

How to compare the values of same index in multiple list and print the start time and end time if condition not satisfy?

Here, i have 6 lists, all of them has same length of data. one is time which contains time from one start point to one end point and another five list contains signals.
time = [11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67]
A = [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0]
B = [0, 0, 0, 0, 0, 0, 0 ,0 ,0 ,0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0, 0, 0, 0, 0, 2, 2, 2, 2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2]
C = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]
D = [0, 0, 0, 0, 0, 0, 0 ,0 ,0 ,0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 2, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2]
E = [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]
Here first i want to compare list A and B. if in list A 0 comes and in the same index 2 comes in B and if it is True then in second condition check in the same index in other three list there C should be 0, D should be 0 and E should be 1. if this condition satisfy then it is passed but in case in some point it comes different value then i need the start time and end time.
or j in range(len(time)):
lis = []
lis2 = []
for i in range(len(A)):
if(A[i] == 0 and B[i] == 2):
if C == 0 and D == 0 and E == 1:
lis.append(time[i])
else:
lis2.append(time[i])
print lis
print lis2
Using this code i've got the time where it is not satisfying but this isn't what i want.
i want the start time and end time like this
OUTPUT - [33,42] or [33,34,35,36,37,38,39,40,41,42]
Because in this time period 1st condition is True and from where it fails 2nd condition from there it should print the time till 1st condition True like i've given in output, then no need to check further.
Thank You In Advance.

I think this is what you want.
or j in range(len(time)):
lis = []
lis2 = []
bool = false
for i in range(len(A)):
if bool:
break
if(A[i] == 0 and B[i] == 2):
if C == 0 and D == 0 and E == 1:
lis.append(time[i])
else:
bool = true
lis2.append(time[i])
print lis
print lis2

Using numpy, you can do the following:
import numpy as np
A = np.array(A)
B = np.array(B)
C = np.array(C)
D = np.array(D)
E = np.array(E)
time = np.array(time)
print time[(A == 0)*(B == 2)*(C == 0)*(D == 0)*(E == 1)]
By the way, your example is wrong. The correct result is [32, 34, 35, 36, 37, 39, 40, 48, 49, 50, 51, 52], thus there are two periods with the correct pattern (from 31 to 40 and from 48 to 52).

Indexing 2 dimensional array in python

I've been trying to change a single item in a 2-dimensional array in python using the syntax x[2][3]=1 but instead of just changing the item in the 2nd row 3rd column, it ends up changing the values of all of the 3rd column. My code is below:
population = [[0]*20]*5
population[2][3] = 1
for row in population:
print(row)
This outputs
[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
but I only want
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
How would I index the item such that it only changes the 2nd row and 3rd column?
I'm using python 3.7.4 on repl.it
Link here: https://repl.it/#ajqe/2d-array-test

Use :
population = [[0]*20 for _ in range(5)]
to generate the lists instead. The method you are using is referencing the same object 5 times, instead of creating 5 separate lists. To check this you can use the is operator:
>>> population = [[0]*20]*5
>>> population[0] is population[1]
True
>>> population = [[0]*20 for _ in range(5)]
>>> population[0] is population[1]
False

Transform Pandas dataframe into frequency matrix

I'm trying to transform a pandas dataframe with three columns (Date, Start, End) into a frequency matrix. My input dataframe look like this:
Date, Start, End
2016-09-02 09:16:00 18 16
2016-09-02 16:14:10 16 1
2016-09-02 06:17:21 18 17
2016-09-02 05:51:07 23 17
2016-09-02 18:34:44 18 17
2016-09-02 05:44:44 20 4
2016-09-02 09:25:22 18 17
2016-09-02 22:27:44 18 17
2016-09-02 16:02:46 0 18
2016-09-02 15:35:07 17 17
2016-09-02 16:06:42 8 17
2016-09-02 14:47:04 16 23
2016-09-02 07:47:24 20 1
...
The values of 'Start' and 'End' are integers between 0 and 23 inclusive. The 'Date' is a datetime. The frequency matrix I'm trying to create is a 24 by 24 csv, where row i and column j is the number of times 'End'=i and 'Start'=j occurs in the input. For example, the above data would create:
0, 1, 2, 3, 4, 5, 6, 7, 8, 9,10,11,12,13,14,15,16,17,18,19,20,21,22,23
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0
2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0
5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
9, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
11, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
13, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
14, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
15, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
16, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0
17, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 4, 0, 0, 0, 0, 1
18, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
19, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
20, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
21, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
22, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
23, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0
For extra help, could this be done in a way that creates a separate matrix for every 15 minutes? That would be 672 matrices as this date range is one week.
I'm a self taught beginner, and I really can't think of how to solve this in a pythonic way, any solutions or advice would be greatly appreciated.

Create your matrix with a simple count and unstack one of one column:
mat = df.groupby(['Start', 'End']).count().unstack(level=0)
Clean up the Date level:
mat.columns = mat.columns.droplevel(0)
Now reindex rows and columns and cast into integers:
mat.reindex(*[range(0,24)]*2).fillna(0)
Detailed explanations
First, you count the number of occurences a given (start,end) couple appears. The result of groupby against these two columns actually brings back a multiindex.
df.groupby(['Start', 'End']).count()
Out[134]:
Date
Start End
0 18 1
8 17 1
16 1 1
23 1
17 17 1
18 16 1
17 4
20 1 1
4 1
23 17 1
What we want from that result is to get the Start index in columns. unstack does this:
df.groupby(['Start', 'End']).count().unstack(level=0)
Out[135]:
Date
Start 0 8 16 17 18 20 23
End
1 NaN NaN 1.0 NaN NaN 1.0 NaN
4 NaN NaN NaN NaN NaN 1.0 NaN
16 NaN NaN NaN NaN 1.0 NaN NaN
17 NaN 1.0 NaN 1.0 4.0 NaN 1.0
18 1.0 NaN NaN NaN NaN NaN NaN
23 NaN NaN 1.0 NaN NaN NaN NaN
The result of unstack is the Start column being moved as an additional column index level on top of the current Date column index (see below). That's why we drop the level 0 afterwards. Another way - depending on your current source code - could be to filter out the Date column upfront, then unstack would bring one level.
_.columns
Out[136]:
MultiIndex(levels=[['Date'], [0, 8, 16, 17, 18, 20, 23]],
labels=[[0, 0, 0, 0, 0, 0, 0], [0, 1, 2, 3, 4, 5, 6]],
names=[None, 'Start'])

Bit late but for anyone who's here:
There is a function explicitly for this called pd.crosstab()
https://pandas.pydata.org/docs/reference/api/pandas.crosstab.html
You will want to use it like:
output = pd.crosstab(df["Start"], df["End"])

Python Brainf*** – Bugs on while loops

I am a relative beginner to python, and in order to strengthen my skills, I am (attempting) to write a compiler for the Brainfu** language. All is good, except for the bracket [] loops. The program I am using to test my code is >++[>++<-]>+, which should set cell 2 to 5. When I run this, however, it does this:
0 [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] 0 >
1 [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] 1 +
2 [0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] 2 +
3 [0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] 2 [
4 [0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] 0 >
5 [0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] 1 +
6 [0, 2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] 2 +
7 [0, 2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] 2 <
8 [0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] 1 -
3 [0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] 1 [
10 [0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] 2 >
11 [0, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] 3 +
[0, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
(The lines are formatted in the iteration, then the list at that point, then the value it's focused on and then the character it's running.)
My current code is
def generateArray(code):
array = []
for i in range(0,20):
array.append(0);
return array
def run(code):
print code
data = generateArray(code)
chars = list(code)
pointer = 0
for i in range(0, len(chars)):
current = chars[i]
if(current == "+"):
data[pointer] += 1
if(current == ">"):
pointer += 1
if(current == "-"):
data[pointer] -= 1
if(current == "<"):
pointer -= 1
if(current == "."):
print str(chr(data[pointer]))
if(current == ","):
given = raw_input()
data[pointer] = ord( given )
if(current == "["):
posOfEnd = chars[i:len(chars)].index("]")
if(data[pointer] == 0):
i += posOfEnd+1
if(current == "]"):
posOfBegin = len(chars) - 1 - chars[::-1].index('[')
i = posOfBegin
print i, data, data[pointer], chars[i]
return data
print run(">++[>++<-]>+")
posOfEnd is trying to find out where the next bracket is, and posOfBegin is trying to find out where the previous bracket is.

I suppose the problem is your loop variable i which you modify during the loop:
i += posOfEnd+1
and
i = posOfBegin
However python for loops are different from their C/C++ counterparts. In python the variable i will be set to each element of the iterable you provide it, in this case range. range(n) evaluates to a list containing all numbers from 0 up to n-1. If you modify your loop variable during an iteration then this modification remains for only that iteration but for the next iteration the loop variable will be assigned the next element of the iterable (not preserving your modifications).
You might want to use a while loop instead.

Categories

Resources