My Data Looks Like this:
statnr datum ele h01 h02 h03 h04 h05 h06 h07 h08 h09 h10 h11 h12 h13 h14 h15 h16 h17 h18 h19 h20 h21 h22 h23 h24
----------- ----------- --- ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------
20101 20020401 D6K 103 126 115 114 105 101 118 118 130 129 126 128 132 133 131 130 130 131 130 130 125 117 122 124
20101 20020402 D6K 126 118 119 120 114 111 107 119 124 126 122 130 130 130 128 128 126 119 129 134 132 127 112 118
........
20101 20150909 D6K 72 82 75 76 82 93 91 96 99 101 108 108 103 100 94 90 82 92 88 79 77 89 94 92
20101 20020401 FLP 54 61 58 61 66 67 65 56 47 46 40 40 39 32 34 34 37 43 45 45 50 54 59 63
20101 20020402 FLP 64 61 67 66 68 69 67 56 50 46 42 39 33 32 33 34 39 48 55 58 61 62 65 68
........
20101 20150909 FLP 93 95 92 94 94 96 95 92 90 84 87 75 81 75 75 74 83 87 89 96 94 92 91 94
20101 20070906 GSE 32700 0 0 0 0 0 3 10 17 30 28 27 37 44 37 25 16 5 1 0 0 0 0 0
20101 20070907 GSE 0 0 0 0 0 0 11 48 72 107 257 264 290 216 255 178 122 57 6 0 0 0 0 0
........
20101 20150909 GSE 0 0 0 0 0 1 17 51 71 118 82 200 116 130 142 156 48 15 1 0 0 0 0 0
20101 20020101 SUV 0 0 0 0 0 0 0 0 9 10 10 10 10 10 10 10 2 0 0 0 0 0 0 0
........
20101 20150909 SUV 0 0 0 0 0 0 0 0 0 1 0 5 1 4 4 9 2 0 0 0 0 0 0 0
20101 20020401 TEX 30 18 21 18 9 10 18 42 69 91 114 117 126 135 133 127 114 87 58 47 39 33 27 24
........
20101 20150909 TEX 50 46 48 50 50 49 57 67 77 85 80 111 95 100 101 92 74 67 59 53 49 49 49 47
20101 20020401 QVX 6 10 9 8 13 25 19 15 16 19 24 24 19 23 24 22 24 23 19 13 12 16 16 18
........
20101 20150909 QVX 40 42 37 34 30 34 22 22 27 31 26 28 37 38 42 43 52 54 59 81 80 69 78 60
as you can see it is a huge sheet with a statnr Row, DateRow, ele stands for the parameter and than h01 - h24 are as you can imagine the hours.
I need to adjust the format from that Sheet to the Format of the other Files I'm working with (Plotting and processing reasons)
I'm currently trying to bring this FileSheet into this Format:
Date Time D6K FLP GSE SUV TEX QVX
01.04.2002 01:00 103 54 0 30 6
.....
09.09.2015 23:59 92 94 0 0 47 60
So what I'm trying to do is:
1) Get rid of row[0] (statnr)
2) Switch the Header with Row[2] so that all parameters are in the header and link them to the new Time Date fmt in the lines
3) Convert the time fmt from %H%M%D to %D%M&Y %H:%M
Since I'm new to python and coding I thought I'd ask if there's maybe a package out there that deals with that kind of Problem, and if there's a term for that Problem in general (switching header with lines) --> thanks (Peter Wood) I switched the Title to Transpose
Thanks for suggestions
For Clarification:
the ........ indicates that I left some rows out
the ----------- are in the file
Because you may have missing data, this isn't a simple case of transposing blocks. I think what you need to do is read the input file into a data structure from which you can then look up the values as required to generate your output. In Python you can use a dictionary whose key is a tuple of your element type, date, and hour:
mydict = {}
with open('F:\myfile.txt') as f:
z = f.readline() # discard headings
z = f.readline() # discard row of dashes
for line in f:
fields = line.split()
date = fields[1]
ele = fields[2]
for hour, value in enumerate(fields[3:27]):
mydict[(ele, date, hour)] = value
Now you have all the data in a big dictionary that's addressable by ele, date and hour. I'm going to guess that the ele values are fixed and you can hardcode them, but you'll want to build a list of the unique dates you actually found in the input file, and put them in ascending order:
dateset=set()
for k in mydict.keys():
dateset.add(k[1])
dates=list(dateset)
dates.sort()
Now you're ready to build your output file.
for date in dates:
for hour in range(24):
output = date + '\t' + hour
for ele in ['D6K', 'FLP', 'GSE', 'SUV', 'TEX', 'QVX']:
output = output + '\t' + mydict.get((ele, date, hour), '')
print(output)
Using the get method on the dictionary allows you to specify a default value to be returned if the key you supplied isn't in the dictionary.
I haven't dealt with the date formatting (note that 'hour' ranges from 0 to 23), or writing the output to a file, but the above should get you going.
Related
i am trying to convert code from matlab to python.
Can you please help me to convert this code from matlab to python?
in matlab code
z is list and z length is 121
z= 7.0502 5.8030 4.4657 3.0404 1.5416 0 -1.5416 -3.0404 -4.4657
-5.8030 -7.0502 7.5944 6.3059 4.8990 3.3662 1.7189 0 -1.7189 -3.3662 -4.8990 -6.3059 -7.5944 8.2427 6.9282 5.4611 3.8122 1.9735 0 -1.9735 -3.8122 -5.4611 -6.9282 -8.2427 9.0135 7.7027 6.2075 4.4590 2.3803 0 -2.3803 -4.4590 -6.2075 -7.7027 -9.0135 9.9185 8.6576 7.2038 5.4466 3.1530 0 -3.1530 -5.4466 -7.2038 -8.6576 -9.9185 10.9545 9.7980 8.4853 6.9282 4.8990 0 -4.8990 -6.9282 -8.4853 -9.7980 -10.9545 12.0986 11.0885 9.9947 8.8128 7.6119 -6.9282 -7.6119 -8.8128 -9.9947 -11.0885 -12.0986 13.3133 12.4632 11.5988 10.7649 10.0829 -9.7980 -10.0829 -10.7649 -11.5988 -12.4632 -13.3133 14.5583 13.8564 13.1842 12.5910 12.1612 -12.0000 -12.1612 -12.5910 -13.1842 -13.8564 -14.5583 15.8011 15.2238 14.6969 14.2594 13.9626 -13.8564 -13.9626 -14.2594 -14.6969 -15.2238 -15.8011 17.0207 16.5431 16.1227 15.7875 15.5684 -15.4919 -15.5684 -15.7875 -16.1227 -16.5431 -17.0207
Matlab code : [z,index]=sort(abs(z));
after the code
z = 0 0 0 0 0 0 1.5416 1.5416 1.7189 1.7189 1.9735 1.9735 2.3803 2.3803 3.0404 3.0404 3.1530 3.1530 3.3662 3.3662 3.8122 3.8122 4.4590 4.4590 4.4657 4.4657 4.8990 4.8990 4.8990 4.8990 5.4466 5.4466 5.4611 5.4611 5.8030 5.8030 6.2075 6.2075 6.3059 6.3059 6.9282 6.9282 6.9282 6.9282 6.9282 7.0502 7.0502 7.2038 7.2038 7.5944 7.5944 7.6119 7.6119 7.7027 7.7027 8.2427 8.2427 8.4853 8.4853 8.6576 8.6576 8.8128 8.8128 9.0135 9.0135 9.7980 9.7980 9.7980 9.9185 9.9185 9.9947 9.9947 10.0829 10.0829 10.7649 10.7649 10.9545 10.9545 11.0885 11.0885 11.5988 11.5988 12.0000 12.0986 12.0986 12.1612 12.1612 12.4632 12.4632 12.5910
12.5910 13.1842 13.1842 13.3133 13.3133 13.8564 13.8564 13.8564 13.9626 13.9626 14.2594 14.2594 14.5583 14.5583 14.6969 14.6969 15.2238 15.2238 15.4919 15.5684 15.5684 15.7875 15.7875 15.8011 15.8011 16.1227 16.1227 16.5431 16.5431 17.0207 17.0207
and index is
index = 6 17 28 39 50 61 5 7 16 18 27 29 38 40 4 8 49 51 15 19 26 30 37 41 3 9 14 20 60 62 48 52 25 31 2 10 36 42 13 21 24 32 59 63 72 1 11 47 53 12 22 71 73 35 43 23 33 58 64 46 54 70 74 34 44 57 65 83 45 55 69 75 82 84 81 85 56 66 68 76 80 86 94 67 77 93 95 79 87 92 96 91 97 78 88 90 98 105 104 106 103 107 89 99 102 108 101 109 116 115 117 114 118 100 110 113 119 112 120 111 121
so what is the [z,index] in python ?
Do you need to return the index? If you don't, you could use:
z = abs(z)
new_list = sorted(map(abs, z))
index = sorted(range(len(z)), key=lambda k: z[k])
where x is the output and z is the list.
EDIT:
Try that now
I am trying to rotate some images for data augmentation to train a network for image segmentation task. After searching a lot, the best candidate for rotating each image and its corresponding mask was to use the scipy.ndimage.rotate function, but the problem with this is that after rotating the mask image numpy array ( which includes only 0 and 255 values for pixel values) the rotated mask has got all the values from 0 to 255 while I expect the mask array to have only 0 and 255 as its pixel values.
Here is the code:
from scipy.ndimage import rotate
import numpy as np
ample = dataset[1]
print(np.unique(sample['image']))
print(np.unique(sample['mask']))
print(sample['image'].shape)
print(sample['mask'].shape)
rot_image = rotate(sample['image'], 60, reshape = False)
rot_mask = rotate(sample['mask'], 60, reshape = False)
print(np.unique(rot_image))
print(np.unique(rot_mask))
print(rot_image.shape)
print(rot_mask.shape)
Here are the results:
[ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53
54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89
90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107
108 109 110 111 112 113 114 115 118 119 120 121 125 139]
[ 0 255]
(512, 512, 1)
(512, 512, 1)
[ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53
54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89
90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107
108 109 110 111 112 113 115 117 118 124 125 132 135]
[ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 17 18 20
24 25 26 28 31 34 35 38 39 41 42 43 45 46 48 49 50 51
52 58 59 62 66 67 68 73 75 76 79 80 82 85 86 88 90 96
98 101 108 109 111 114 116 118 119 123 124 125 127 128 130 138 140 142
146 148 151 156 157 158 161 164 165 166 168 169 176 180 184 185 188 189
194 196 197 198 199 201 203 204 205 207 208 210 211 213 216 217 218 219
220 221 222 225 228 229 230 231 233 234 235 237 239 240 241 242 243 244
245 246 247 248 249 250 251 252 253 254 255]
(512, 512, 1)
(512, 512, 1)
It seems to be a simple problem to rotate image array, but I'm searching for days and I didn't find any solution to this problem. I am really confused how to prevent mask array values( 0 and 255) to take all values from 0 to 255 after rotation. I mean something like this:
x = np.unique(sample['mask'])
rot_mask = rotate(sample['mask'], 30, reshape = False)
x_rot = np.unique(rot_mask)
print(np.unique(x - x_rot))
[ 0]
Since you are using numpy arrays to represent images, why not using numpy functions? This library has all sorts of array manipulations. Try the rot90 function
so I have webscraped data from an mlb betting site aggregator and have my data points in two lists. The first list is all of the teams. The way it is formatted is that teamlist[1] and teamlist[2] are playing each other, then teamlist[3] and teamlist[4] play each other and so on. Each row index is a team, and each column index is a betting site.
site1|site2|site3|site4|...
team1
team2
team3
team4
...
This outlines the general form.
I have figured out the pattern I need to put each betting odd I need to put it into, but I cannot figure out the way to input them properly.
I apologize, I do not have to reputation to post the actual image so I must do a link instead.This outlines the structure I need to index. The data points are the index I need to go there. As you can see df[0,0] = moneylines[0], and df[0,1]= moneylines[1]. My Primary issue is once I make it through the first two rows (which are done in the same loop) and it tries to go to the third row, it reindexes over the first two rows.link
Here is the code I am currently using to populate the DataFrame. moneylines is the list of betting odds I am trying to populate the dataframe with, and teams is the row index:
ctr = 0
for t in range(0,int(len(teams)/2)):
for m in range(14):
df.ix[m,t] = moneylines[ctr]
df.ix[m,t+1] = moneylines[ctr+1]
ctr = ctr + 2
Please let me know if there is anything else I can include to help solve this question.
Your issue is due to your first for loop. You increment it one by one so:
first loop :
t = 0
you fill line 0 and line 1
then
t = 1
you fill line 1 and line 2
and so on...
You should use instead of :
for t in range(0,int(len(teams)/2)):
this:
for t in range(0, len(teams), 2)
NB :You can also multiply t by 2 in the index but it's not as logic as using the above solution
I hope it helps,
I'm posting an alternative to looping over the values of a dataframe, which you can avoid pretty easily here, because doing so loses the efficiency boost of using a dataframe in the first place.
It's not entirely clear to me what the formatting of your starting data is, but if, say, you have a series s with values 0 through 195:
s = pd.Series(range(196))
Then, using numpy.reshape you could get the pairings:
>>>s.values.reshape((len(s)//2, 2))
array([[ 0, 1],
[ 2, 3],
[ 4, 5],
...,
[190, 191],
[192, 193],
[194, 195]])
And using it again you could get the desired output:
>>>pd.DataFrame(s.values.reshape((len(s)//2, 2)).T.reshape((len(s)//14, 14))).sort_values(0)
0 1 2 3 4 5 6 7 8 9 10 11 12 13
0 0 2 4 6 8 10 12 14 16 18 20 22 24 26
7 1 3 5 7 9 11 13 15 17 19 21 23 25 27
1 28 30 32 34 36 38 40 42 44 46 48 50 52 54
8 29 31 33 35 37 39 41 43 45 47 49 51 53 55
2 56 58 60 62 64 66 68 70 72 74 76 78 80 82
9 57 59 61 63 65 67 69 71 73 75 77 79 81 83
3 84 86 88 90 92 94 96 98 100 102 104 106 108 110
10 85 87 89 91 93 95 97 99 101 103 105 107 109 111
4 112 114 116 118 120 122 124 126 128 130 132 134 136 138
11 113 115 117 119 121 123 125 127 129 131 133 135 137 139
5 140 142 144 146 148 150 152 154 156 158 160 162 164 166
12 141 143 145 147 149 151 153 155 157 159 161 163 165 167
6 168 170 172 174 176 178 180 182 184 186 188 190 192 194
13 169 171 173 175 177 179 181 183 185 187 189 191 193 195
How do i use a loop nested within a loop to create this output:
100
101 102
103 104 105
106 107 108 109
You can do this using a while loop with certain other variables:
>>> st, end, length = 100, 110, 1
>>> while st < end:
... print (" ".join(map(lambda x: "%3d" % x, range(st,st+length))))
... st += length
... length += 1
...
100
101 102
103 104 105
106 107 108 109
Note that I have used a lambda instead of str in the map, this is so that different width numbers don't break up the indentation.
You can similarly base this on the width of the final line as below (in which case there is no need to keep track of the end variable I was doing above):
>>> st, width = 1, 1
>>> while width <= 15:
... print (" ".join(map(lambda x: "%3d" % x, range(st, st + width))))
... st += width
... width += 1
...
1
2 3
4 5 6
7 8 9 10
11 12 13 14 15
16 17 18 19 20 21
22 23 24 25 26 27 28
29 30 31 32 33 34 35 36
37 38 39 40 41 42 43 44 45
46 47 48 49 50 51 52 53 54 55
56 57 58 59 60 61 62 63 64 65 66
67 68 69 70 71 72 73 74 75 76 77 78
79 80 81 82 83 84 85 86 87 88 89 90 91
92 93 94 95 96 97 98 99 100 101 102 103 104 105
106 107 108 109 110 111 112 113 114 115 116 117 118 119 120
Something like this???
start = 100
lines = 5
for i in range(lines):
stop = start + i + 1
for j in range(a, stop):
print j,
start = stop
print
Just use two-level for loop:
x = 100
for i in range(1, 5):
for j in range(i):
print(x, end=" ")
x += 1
print("")
I have the following dataframe:
Obj BIT BIT BIT GAS GAS GAS OIL OIL OIL
Date
2007-01-03 18 7 0 184 35 2 52 14 0
2007-01-09 43 3 0 249 35 2 68 11 1
2007-01-16 60 6 0 254 35 5 72 13 1
2007-01-23 69 11 1 255 43 2 81 6 0
2007-01-30 74 8 0 263 29 4 69 9 0
2007-02-06 78 6 1 259 34 2 79 6 0
2007-02-14 76 9 1 263 24 2 70 10 1
2007-02-20 85 7 0 241 20 6 72 4 0
2007-02-27 79 6 0 242 35 3 68 7 0
2007-03-06 68 14 0 225 26 2 57 10 1
How can I sum each of the 9 columns into 3 columns. "BIT","GAS" and "OIL"
This is the code for the dataframe which basically just gets me a cross section from a larger df I want:
ABrigsA = ndfAB.xs(['BIT','GAS','OIL'],axis=1)
Any suggestions?
Assuming that you want to sum similarly-named columns, you can use groupby [tutorial docs]:
>>> df.groupby(level=0, axis='columns').sum()
Obj BIT GAS OIL
Date
2007-01-03 25 221 66
2007-01-09 46 286 80
2007-01-16 66 294 86
2007-01-23 81 300 87
2007-01-30 82 296 78
2007-02-06 85 295 85
2007-02-14 86 289 81
2007-02-20 92 267 76
2007-02-27 85 280 75
2007-03-06 82 253 68