I have a shared variable persistent_vis_chain which is being updated by a theano function where it gets its function from a theano.scan, But thats not the problem just back story.
My shared variable looks like D = [image1, ... , imageN] where each images is [x1,x2,...,x784].
What I want to do is take the average of all the images and put them into the last imageN. That is I want to sum all the values in each image except the last 1, which will result in [s1,s2,...,s784] then I want to set imageN = [s1/len(D),s2/len(D),...s784/len(D)]
So my problem is I do not know how to do this with theano.shared and may be with my understanding of theano functions and doing this computation with symbolic variables. Any help would be greatly appreciated.
If you have N images, each of shape 28x28=784 then, presumably, your shared variable has shape (N,28,28) or (N,784)? This method should work with either shape.
Given D is your shared variable containing your image data. If you want to get the average image then D.mean(keepdims=True) will give it to you symbolically.
It's unclear if you want to change the final image to equal the mean image (sounds like a strange thing to do), or if you want to add a further N+1'th image to the shared variable. For the former you could do something like this:
D = theano.shared(load_D_data())
D_update_expression = do_something_with_scan_to_get_D_update_expression(D)
updates = [(D, T.concatenate(D_update_expression[:-1],
D_update_expression.mean(keepdims=True)))]
f = theano.function(..., updates=updates)
If you want to do the latter (add an additional image), change the updates line as follows:
updates = [(D, T.concatenate(D_update_expression,
D_update_expression.mean(keepdims=True)))]
Note that this code is intended as a guide. It may not work as it stands (e.g. you may need to mess with the axis= parameter in the T.concatenate command).
The point is that you need to construct a symbolic expression explaining what the new value for D looks like. You want it to be a combination of the updates from scan plus this additional average thing. T.concatenate allows you to combine those two parts together.
Related
I'm currently working on a project researching properties of some gas mixtures. Testing my code with different inputs, I came upon a bug(?) which I fail to be able to explain. Basically, it's concerning a computation on a numpy array in a for loop. When it computed the for-loop, it yields a different (and wrong) result as opposed to the manual construction of the result, using the same exact code snippets as in the for-loop, but indexing manually. I have no clue, why it is happening and whether it is my own mistake, or a bug within numpy.
It's super weird, that certain instances of the desired input objects run through the whole for loop without any problem, while others run perfectly up to a certain index and others fail to even compute the very first loop.
For instance, one input always stopped at index 16, throwing a:
ValueError: could not broadcast input array from shape (25,) into shape (32,)
Upon further investigation I could confirm, that the previous 15 loops threw the correct results, the results in loop of index 16 were wrong and not even of the correct size. When running loop 16 manually through the console, no errors occured...
The lower array shows the results for index 16, when it's running in the loop.
These are the results for index 16, when running the code in the for loop manually in the console. These are, what one would expect to get.
The important part of the code is really only the np.multiply() in the for loop - I left the rest of it for context but am pretty sure it shouldn't interfere with my intentions.
def thermic_dissociation(input_gas, pressure):
# Copy of the input_gas object, which may not be altered out of scope
gas = copy.copy(input_gas)
# Temperature range
T = np.logspace(2.473, 4.4, 1000)
# Matrix containing the data over the whole range of interest
moles = np.zeros((gas.gas_cantera.n_species, len(T)))
# Array containing other property of interest
sum_particles = np.zeros(len(T))
# The troublesome for-loop:
for index in range(len(T)):
print(str(index) + ' start')
# Set temperature and pressure of the gas
gas.gas_cantera.TP = T[index], pressure
# Set gas mixture to a state of chemical equilibrium
gas.gas_cantera.equilibrate('TP')
# Sum of particles = Molar Density * Avogadro constant for every temperature
sum_particles[index] = gas.gas_cantera.density_mole * ct.avogadro
#This multiplication is doing the weird stuff, printed it to see what's computed before it puts it into the result matrix and throwing the error
print(np.multiply(list(gas.gas_cantera.mole_fraction_dict().values()), sum_particles[index]))
# This is where the error is thrown, as the resulting array is of smaller size, than it should be and thus resulting in the error
moles[:, index] = np.multiply(list(gas.gas_cantera.mole_fraction_dict().values()), sum_particles[index])
print(str(index) + ' end')
# An array helping to handle the results
molecule_order = list(gas.gas_cantera.mole_fraction_dict().keys())
return [moles, sum_particles, T, molecule_order]
Help will be very appreciated!
If you want the array of all species mole fractions, you should use the X property of the cantera.Solution object, which always returns that full array directly. You can see the documentation for that method: cantera.Solution.X`.
The mole_fraction_dict method is specifically meant for cases where you want to refer to the species by name, rather than their order in the Solution object, such as when relating two different Solution objects that define different sets of species.
This particular issue is not related to numpy. The call to mole_fraction_dict returns a standard python dictionary. The number of elements in the dictionary depends on the optional threshold argument, which has a default value of 0.0.
The source code of Cantera can be inspected to see what happens exactly.
mole_fraction_dict
getMoleFractionsByName
In other words, a value ends up in the dictionary if x > threshold. Maybe it would make more sense if >= was used here instead of >. And maybe this would have prevented the unexpected outcome in your case.
As confirmed in the comments, you can use mole_fraction_dict(threshold=-np.inf) to get all of the desired values in the dictionary. Or -float('inf') can also be used.
In your code you proceed to call .values() on the dictionary but this would be problematic if the order of the values is not guaranteed. I'm not sure if this is the case. It might be better to make the order explicit by retrieving values out of the dict using their key.
I have a code which returns some variables that I would like to later on use in another program. However, the output doesn't look like I want it to. The rows are within brackets [], and I would like to have them removed.
I have found the following question that deals with something similar; however, I am saving the variable, not printing it on the screen:
how to remove characters from printed output in python
This is where the variable is defined, the libraries used are chaospy and numpy.
nodes, weights = cp.generate_quadrature(order, dist, rule="G", sparse=True)
nodes_trans = nodes.transpose()
And this is where the variable is saved
with open('nodes_smolyak_trans.txt', 'w') as ndsT:
for itemndsT in nodes_trans:
ndsT.write("%s\n" % itemndsT)
ndsT.close()
Also, dist, mentioned above is defined as
dist = cp.J(wX, wY, pX, pY)
And all it's compoents are defined equally as
windX = cp.Uniform(0, 100)
Now, all rows of my output looks like this:
[50. 50. 50. 21.13248654]
I would like them to be instead simply
21.13248654
If the number of rows is small, I can manually remove them by hand, but when the output contains hundreds, I waste too much time, so any sugestion to remove the brackets manually is welcome.
My suspicion is that itemndsT is a list type, which is why when you write it to the file it includes the square brackets. You'll need to format it into a string yourself before writing it. There are a few ways you can do this, but using the join string function is one of the simplest:
for itemndsT in nodes_trans:
item_str = " ".join(map(str,itemndsT))
ndsT.write("%s\n" % item_str)
With the below code I'm trying to update the column df_test['placed'] to = 1 when the if statement is triggered and a prediction is placed. I haven't been able to get this to update correctly though, the code compiles but doesn't update to = 1 for the respective predictions placed.
df_test['placed'] = np.zeros(len(df_test))
for i in set(df_test['id']) :
mask = df_test['id']==i
predictions = lm.predict(X_test[mask])
j = np.argmax(predictions)
if predictions[j] > 0 :
df_test['placed'][mask][j] = 1
print(df_test['placed'][mask][j])
Answering your question
Edit: changed suggestion based on comments
The assignment part of your code, df_test['placed'][mask][j] = 1, uses what is called chained indexing. In short, your assignment only changes a temporary copy of the DataFrame that gets immediately thrown away, and never changes the original DataFrame.
To avoid this, the rule of thumb when doing assignment is: use only one set of square braces on a single DataFrame. For your problem, that should look like:
df_test.loc[mask.nonzero()[0][j], 'placed'] = 1
(I know the mask.nonzero() uses two sets of square brackets; actually nonzero() returns a tuple, and the first element of that tuple is an ndarray. But the dataframe only uses one set, and that's the important part.)
Some other notes
There are a couple notes I have on using pandas (& numpy).
Pandas & NumPy both have a feature called broadcasting. Basically, if you're assigning a single value to an entire array, you don't need to make an array of the same size first; you can just assign the single value, and pandas/NumPy automagically figures out for you how to apply it. So the first line of your code can be replaced with df_test['placed'] = 0, and it accomplishes the same thing.
Generally speaking when working with pandas & numpy objects, loops are bad; usually you can find a way to use some combination of broadcasting, element-wise operations and boolean indexing to do what a loop would do. And because of the way those features are designed, it'll run a lot faster too. Unfortunately I'm not familiar enough with the lm.predict method to say, but you might be able to avoid the whole for-loop entirely for this code.
I am using OpenCV to detect BRISK keypoints this way:
img = cv2.imread(image_path)
gray= cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
kp, descriptors = cv2.BRISK_create().detectAndCompute(gray,None)
kp is a list, however, it is also a kind of struct. It has parameters or "sublists" such as kp.pt values, kp. response values and so on. I am a little rookie in Python so I would like to know the following.
1-Is there a specific name for this list? it does not seem to be the list I am used to using in Python. How to get more information about this kind of list?
2- How can I know what are the other parameters (or "sublists") of kp? can I print them in Python?
3- How can I save in a variable all elements of, for example, kp.response? I can only index individual elements like this kp[1].response
4-How can I sort kp.response values, getting the indices of the N highest kp.response values?
1- type(kp) This gets the class that the object is an instance of.
2- dir(kp) This should list the attributes of the object. For certain very dynamic types that generate their attributes on the fly in response to access attempts, this might not work. (But it still might, since dir can also be generated dynamically.)
In that case, if you can find the __getattr__ method in the source code of the class, it might give you a clue where those dynamic attributes are actually stored.
3- [e.response for e in kp] probably. Usually if it's indexible like that it's also iterable.
4- print([i for _, i in sorted((e.response, i) for i, e in enumerate(kp))[-N:]]) haven't tested this.
Also try help(kp), it might explain how to use it better. Use help on anything you don't understand. It might have a docstring.
Now I just need to figure out how to use the resulting list as kp indexes
Maybe something like [kp[i] for i in indexes]?
I am playing around with images and the random number generator in Python, and I would like to understand the strange output picture my script is generating.
The larger script iterates through each pixel (left to right, then next row down) and changes the color. I used the following function to offset the given input red, green, and blue values by a randomly determined integer between 0 and 150 (so this formula is invoked 3 times for R, G, and B in each iteration):
def colCh(cVal):
random.seed()
rnd = random.randint(0,150)
newVal = max(min(cVal - 75 + rnd,255),0)
return newVal
My understanding is that random.seed() without arguments uses the system clock as the seed value. Given that it is invoked prior to the calculation of each offset value, I would have expected a fairly random output.
When reviewing the numerical output, it does appear to be quite random:
Scatter plot of every 100th R value as x and R' as y:
However, the picture this script generates has a very peculiar grid effect:
Output picture with grid effect hopefully noticeable:
Furthermore, fwiw, this grid effect seems to appear or disappear at different zoom levels.
I will be experimenting with new methods of creating seed values, but I can't just let this go without trying to get an answer.
Can anybody explain why this is happening? THANKS!!!
Update: Per Dan's comment about possible issues from JPEG compression, the input file format is .jpg and the output file format is .png. I would assume only the output file format would potentially create the issue he describes, but I admittedly do not understand how JPEG compression works at all. In order to try and isolate JPEG compression as the culprit, I changed the script so that the colCh function that creates the randomness is excluded. Instead, it merely reads the original R,G,B values and writes those exact values as the new R,G,B values. As one would expect, this outputs the exact same picture as the input picture. Even when, say, multiplying each R,G,B value by 1.2, I am not seeing the grid effect. Therefore, I am fairly confident this is being caused by the colCh function (i.e. the randomness).
Update 2: I have updated my code to incorporate Sascha's suggestions. First, I moved random.seed() outside of the function so that it is not reseeding based on the system time in every iteration. Second, while I am not quite sure I understand how there is bias in my original function, I am now sampling from a positive/negative distribution. Unfortunately, I am still seeing the grid effect. Here is my new code:
random.seed()
def colCh(cVal):
rnd = random.uniform(-75,75)
newVal = int(max(min(cVal + rnd,255),0))
return newVal
Any more ideas?
As imgur is down for me right now, some guessing:
Your usage of PRNGs is a bit scary. Don't use time-based seeds in very frequently called loops. It's very much possible, that the same seeds are generated and of course this will generate patterns. (granularity of time + number of random-bits used matter here)
So: seed your PRNG once! Don't do this every time, don't do this for every channel. Seed one global PRNG and use it for all operations.
There should be no pattern then.
(If there is: also check the effect of interpolation = image-size change)
Edit: As imgur is on now, i recognized the macro-block like patterns, like Dan mentioned in the comments. Please change your PRNG-usage first before further analysis. And maybe show more complete code.
It may be possible, that you recompressed the output and JPEG-compression emphasized the effects observed before.
Another thing is:
newVal = max(min(cVal - 75 + rnd,255),0)
There is a bit of a bias here (better approach: sample from symmetric negative/positive distribution and clip between 0,255), which can also emphasize some effect (what looked those macroblocks before?).