So, i'm currently working with a pyomo model with multiple instances that are being solved in parallel. Issue is, solving them takes pyomo quite a long time (like 2 to 3 secs, even though the solving part by gurobi takes about 0.08s). I've found out that, by exporting a pyomo instance into an .mps file and then giving it to gurobipy i can get like an increase of 30% in overall speed.
The problem comes later, when i want to work with the variables of the solved model, because ive noticed that, when exporting the original instance from pyomo into a .mps file, variable names get lost; they all get named "x" (so, for example, model.Delta, model.Pg, model.Alpha, etc get turned into x1, x2, ... ,x9999 instead of Delta[0], Delta[1], ... Alpha[99,99]).
Is there a way to keep the original variable name when exporting the model?
Managed to solve it!
To anyone who might find this useful, i passed a dictionary with "symbolic_solver_labels" as an io_options argument for the method, like this:
instance.write(filename = str(es_) + ".mps", io_options = {"symbolic_solver_labels":True})
Now my variables are correctly labeled in the .mps file!
Related
While following the Jupyter notebooks for the course
I hit upon an error when these lines are run.
I know that the cnn_learner line has got no errors whatsoever, The problem lies in the lr_find() part
It seems that learn.lr_find() does not want to return two values! Although its documentation says that it returns a tuple. That is my problem.
These are the lines of code:
learn = cnn_learner(dls, resnet34, metrics=error_rate)
lr_min,lr_steep = learn.lr_find()
The error says:
not enough values to unpack (expected 2, got 1)
for the second line.
Also, I get this graph with one 'marker' which I suppose is either one of the values of lr_min or lr_steep
This is the graph
When I run learn.lr_find() only, i.e. do not capture the output in lr_min, lr_steep; it runs well but then I do not get the min and steep learning rates (which is really important for me)
I read through what lr_find does and it is clear that it returns a tuple. Its docstring says
Launch a mock training to find a good learning rate and return suggestions based on suggest_funcs as a named tuple
I had duplicated the original notebook, and when I hit this error, I ran the original notebook, with the same results. I update the notebooks as well, but no change!
Wherever I have searched for this online, any sort of error hasn't popped up. The only relevant thing I found is that lr_find() returns different results of the learning rates after every run, which is perfectly fine.
I was having the same problem and I found that the lr_find() output's has updated. You can substitute the second line to lrs = learn.lr_find(suggest_funcs=(minimum, steep, valley, slide)), and then you just substitute where you using lr_min and lr_steep to lrs.minimum and lrs.steep respectively, this should work fine and solve your problem.
If you wanna read more about it, you can see this post that is in the fastai's forum.
I'm currently trying to learn more about Deep learning/CNN's/Keras through what I thought would be a quite simple project of just training a CNN to detect a single specific sound. It's been a lot more of a headache than I expected.
I'm currently reading through this ignoring the second section about gpu usage, the first part definitely seems like exactly what I'm needing. But when I go to run the script, (my script is pretty much totally lifted from the section in the link above that says "Putting the pieces together, you may end up with something like this:"), it gives me this error:
AttributeError: 'DataFrame' object has no attribute 'file_path'
I can't find anything in the pandas documentation about a DataFrame.file_path function. So I'm confused as to what that part of the code is attempting to do.
My CSV file contains two columns, one with the paths and then a second column denoting the file paths as either positive or negative.
Sidenote: I'm also aware that this entire guide just may not be the thing I'm looking for. I'm having a very hard time finding any material that is useful for the specific project I'm trying to do and if anyone has any links that would be better I'd be very appreciative.
The statement df.file_path denotes that you want access the file_path column in your dataframe table. It seams that you dataframe object does not contain this column. With df.head() you can check if you dataframe object contains the needed fields.
I'm coming from an R background where I didn't run into this issue.
Generally in the past I've made functions that act upon a dataframe and return some modified version of the dataframe. For example:
df=pd.DataFrame({"a":[1,2,3,4,5], "b":[6,7,8,9,10]})
def multiply_function(dataset):
dataset['output']=dataset.iloc[:,1] * dataset.iloc[:,0]
return(dataset)
new_df=multiply_function(df)
new_df # looks good!
df # I would expect that df stays the same and isn't updated with the new column
I'm trying to convert a good amount of functions or code from one language to another. I'd like to avoid having this issue happen so that df is NOT updated globally because of what happens inside a function.
This is particularly important when I'm re-running code or modifying code because a dataframe may not be valid to run through a function twice.
I have seen usage of
dataset = dataset.copy()
as the first line of code...but is this really ideal? Is there a better way around this? I was thinking that this would really blow up the amount of data in memory when working with large datasets?
Thank you!
I am trying to look at the running mean and running variance of a trained tensorflow model that is exported via GCMLE (saved_model.pb, assets/* & variables/*). Where are these values kept in the graph? I can access gamma/beta values from tf.GraphKeys.TRAINABLE_VARIABLES but I have not been able to find the running mean and running variance in any of the tf.GraphKeys.MODEL_VARIABLES. Are the running mean and running variance stored somewhere else?
I know that at test time (ie. Modes.EVAL), the running mean and running variance are used to normalize the incoming data, then the normalized data is scaled and shifted using gamma and beta. I am trying to look at all of the variables that I need at inference time, but I cannot find the running mean and running variance. Are these only used at test time and not at inference time (Modes.PREDICT)? If so, that would explain why I can't find them in the exported model, but I am expecting them to be there.
Based on tf.GraphKeys I have tried other things like tf.GraphKeys.MOVING_AVERAGE_VARIABLES but they are also empty. I also saw this line in the batch_normalization documentation "Note: when training, the moving_mean and moving_variance need to be updated. By default the update ops are placed in tf.GraphKeys.UPDATE_OPS, so they need to be added as a dependency to the train_op." so I then tried looking at tf.GraphKeys.UPDATE_OPS from my saved model and they contain an assign op batch_normalization/AssignMovingAvg:0 but still not clear where I would get the value from.
It appears that the moving mean and moving variance are stored within tf.GraphKeys.GLOBAL_VARIABLES and it looks like the reason nothing showed up in MODEL_VARIABLES is because you need to use tf.contrib.framework.local_variable
In addition to #reese0106's answer, if you'd like to take out the moving_mean, moving_variance for BatchNorm, you can index them with names as follows.
vars = tf.global_variables() # shows every variable being used.
vars_moving_mean_variance = []
for var in vars:
if ("moving_mean" in var.name) or ("moving_variance" in var.name):
vars_moving_mean_variance.append(var)
print(vars_moving_mean_variance)
p.s. Thanks for the question and the answer. I solved my own problem too.
I have a very large simulation in python with lots of modules. I call a lot of random functions. To keep the same random results I have a variable keep_seed_random.
As so:
import random
keep_seed_random = True
if keep_seed_random is False:
fixed_seed = random.Random(0)
else:
fixed_seed = random
Then I use fixed_seed all over the program, such as
fixed_seed.choice(['male', 'female'])
fixed_seed.randint()
fixed_seed.gammavariate(3, 3)
fixed_seed.random()
fixed_seed.randrange(20, 40)
and so on...
It used to work well.
But now, that the programme is too large, there is something else interfering and the results are no longer identical, even when I choose keep_seed_random = False
My question is whether there is any other source of randomness in Python that I am missing?
P.S. I import random just once.
EDITED
We have been trying to pinpoint the exact moment when the program turned from exact same results to different results. It seemed to be when we introduced a lot of reading of databases with no connection to random modules.
The results now ALTERNATE among two similar results.
That is, I run main.py once get a result of 8148.78 for GDP
I run again I get 7851.49
Again 8148.78 back
Again 7851.49
Also for the working version, before the change, the first result (when we create instances and pickle save them) I get one result. Then, from the second onwards the results are the same. So, I am guessing it is related to pickle reading/loading.
The question remains!
2nd EDITED
We partially found the problem.
The problem is when we create instances and pickle dump and then pickle load.
We still cannot have the exact same results for creating and just loading.
However, when loading repeatedly the results are exact.
Thus, the problem is in PICKLE
Some randomization may occur when dumping and loading (I guess).
Thanks,
This is difficult to diagnose without a good reproduce case as #mart0903 mentions. However, in general, there are several sources of randomness that can occur. A few things come to mind:
If for example you are using the multiprocessing and/or subprocess packages to spawn several parallel processes, you may be experiencing a race condition. That is, different processes finishing at different times each time you run the program. Perhaps you are combining the result in some way that is dependent on these threads executing in a particular order.
Perhaps you are simply looping over a dictionary and expecting the keys to be in a certain order, when in fact, dictionaries are not ordered. For example run the following a couple times in a row (I'm using Python 3.5 in case it matters) and you'll notice that the key-value pairs print out in a different order each time:
if __name__=='__main__':
data = dict()
data['a'] = 6
data['b'] = 7
data['c'] = 42
for key in data:
print(key + ' : ' + str(data[key]))
You might even be looking at time-stamps or set some value, or perhaps generating a uuid somewhere that you are using in a calculation.
The possibilities could go on. But again, difficult to nail down without a simple reproduce case. It may just take some good-ol breakpoints and a lot of stepping through code.
Good luck!