Develop Reference

MSEn of data error Mobj unable to be identified

'def MSEn(dataset_mus, Mobj, Scales=3, Methodx='coarse', RadNew=0, Plotx=False):
class MSobject:
x=(5,2,[3.2])
Mobj = MSobject(x)
dataset_mus=np.squeeze(dataset_mus)
Func2 = globals()[Methodx.lower()]
MSx = np.zeros(Scales)
for T in range(1,Scales+1):
print(' .', end='')
Temp = Func2(Sig,T)
MSx[T-1] = Temp2
CI = sum(MSx)
if np.any(np.isnan(MSx)):
print('Some entropy values may be undefined.')
if Plotx:
figure()
ax1 = axes()
ax1.plot(np.arange(1,Scales+1), MSx, color=(8/255, 63/255, 77/255), linewidth=3)
ax1.scatter(np.arange(1,Scales+1), MSx, 60, color=(1,0,1))
ax1.set_xlabel('Scale Factor',fontsize=12,fontweight='bold',color=(7/255, 54/255, 66/255))
ax1.set_ylabel('Entropy Value',fontsize=12,fontweight='bold',color=(7/255, 54/255, 66/255))
ax1.set_title('Multiscale %s (%s-graining method)'%(Mobj.Func.__name__,Methodx),
fontsize=16,fontweight='bold',color=(7/255, 54/255, 66/255))
show()
return MSx, CI
MSEn(dataset_mus,Mobj.x)'
error
'---------------------------------------------------------------------------
NameError Traceback (most recent call last)
/var/folders/pm/zc3gg3ts2x17sm0r11qd66k00000gn/T/ipykernel_89686/2723706301.py in
30 show()
31 return MSx, CI
---> 32 MSEn(dataset_mus,Mobj.x)
NameError: name 'Mobj' is not defined' '

Mobj needs to be defined.
Mobj= EH.MSobject('SampEn',m=2,r=0.15)
this will create a sample entropy object with the given parameters and then you can call MSen function to calculate multiscale entropy.
I am working on the same problem right now. But my code works for a large data but throws error for smalller data.

Why "NameError: name 'product_id_list' is not defined"=

I write this and i don't know why product_id_list is not defined if i have defined it like 4 lines before.
Any suggestions? I thin identation is alright so I don't have any more ideas and I also searched around without luck.
Thank you!!
def make_dataSet_rowWise(reorder_product):
print('unique Product in dataset = ', len(reorder_product.product_id.unique()))
print('unique order_id in dataset = ', len(reorder_product.order_id.unique()))
product_id_list = reorder_product.product_id.unique().tolist()
product_id_list.append("order_id")
product_id_dict = {}
i = 0
for prod_id in product_id_list:
product_id_dict[prod_id] = i
i = i+1
product_id_df = pd.Dataframe(columns = product_id_list)
row_list_all = []
order_id_list = reorder_product.order_id.unique()
i = 1
for id in order_id_list:
#print(i)
i = i+1
np_zeros = np.zeros(shape = [len(product_id_list)-1])
ordered_product_list = reorder_product.loc[reorder_product.order_id == id]["product_id"].tolist()
for order_prod in ordered_product_list:
np_zeros[product_id_dict.get(order_prod)] = 1
row_list = np_zeros.tolist()
row_list.append(id)
row_list_all.append(row_list)
return (row_list_all, product_id_list)
df_row_wise = make_dataSet_rowWise(reorder_product_99Pct)
product_id_df = pd.DataFrame(df_row_wise[0], columns = df_row_wise[1])
product_id_df.head()
The error I have is this one:
NameError Traceback (most recent call last)
<ipython-input-343-07bcac1b3b48> in <module>
7 i = 0
8
----> 9 for prod_id in product_id_list:
10 product_id_dict[prod_id] = i
11 i = i+1
NameError: name 'product_id_list' is not defined

As already mentioned by the other answers, your indentation is wrong.
My recommendation is that you use a IDE like VSCode, there is also a free web version https://vscode.dev/
With such kind of IDE you can see that your indentation is wrong, check screenshot and line 27
There are also wrong indentations with the 3 for loops. The correct indentation should be as the following

I think your indentation may be wrong, the for-loops and return statement is out of the function (with your indentation) so I indented it so that it would still be part of the function...
def make_dataSet_rowWise(reorder_product):
print('unique Product in dataset = ', len(reorder_product.product_id.unique()))
print('unique order_id in dataset = ', len(reorder_product.order_id.unique()))
product_id_list = reorder_product.product_id.unique().tolist()
product_id_list.append("order_id")
product_id_dict = {}
i = 0
for prod_id in product_id_list:
product_id_dict[prod_id] = i
i = i+1
product_id_df = pd.Dataframe(columns = product_id_list)
row_list_all = []
order_id_list = reorder_product.order_id.unique()
i = 1
for id in order_id_list:
#print(i)
i = i+1
np_zeros = id.zeros(shape = [len(product_id_list)-1])
ordered_product_list = reorder_product.loc[reorder_product.order_id == id]["product_id"].tolist()
for order_prod in ordered_product_list:
np_zeros[product_id_dict.get(order_prod)] = 1
row_list = np_zeros.tolist()
row_list.append(id)
row_list_all.append(row_list)
return (row_list_all, product_id_list)

I'm new here, but i think you either need to define the variable out of the scope of
def make_dataSet_rowWise(reorder_product):
OR indent the for loops to be inside
make_dataSet_rowWise

How to prevents a AssertionError from stop Python code?

I want to do a system using a Autorules fuzzy controller where i create rules from process real data.
My problem is when i use the new data to simulate the fuzzy controller with the rules extracted of the older data, i get a error about the crisp output that cannot be calculated because do not exist rules enough, what is totally normal because my fuzzy system needs more rules and that's my point! I want to implement a new routine that analyses the crisp input/output and create new rules from this data, after that, i want to go back in my code and simulate again.
Is there a function that blocks the AssertionError from stop the code and redirect to another def or to a previously code line?
I tried to find some lib with a function that allows me to redirect the error steady to stop the code but no sucess. I think i will have to change the skfuzzy defuzz def code to allow to make it.
Thank you very much.
''' python
step = stp_ini
k = 0
delay = stp_ini
end = stp_fim - stp_ini
Tout_sim = pd.DataFrame(columns=['val'])
Vent_sim = pd.DataFrame(columns=['val'])
start = timeit.default_timer()
for step in range(end-k):
clear_output(wait=True)
simulation.input['PCA1'] = comp1n[step+delay]
simulation.input['PCA2'] = comp2n[step+delay]
simulation.input['PCA3'] = comp3n[step+delay]
simulation.input['PCA4'] = comp4n[step+delay]
simulation.input['Vent'] = dataoutf.NumVentOn[step+delay]
simulation.compute()
Tout_sim = Tout_sim.append({'val':simulation.output['Tout']},ignore_index=True)
stop = timeit.default_timer()
if ((step/(stp_fim-k-1))*100) < 5:
expected_time = "Calculating..."
else:
time_perc = timeit.default_timer()
expected_time = np.round( ( (time_perc-start)/(step/(end-k-1)) )/60,2)
'''
~\AppData\Local\Continuum\anaconda3\lib\site-packages\skfuzzy\control\controlsystem.py in defuzz(self)
587 self.var.defuzzify_method)
588 except AssertionError:
--> 589 raise ValueError("Crisp output cannot be calculated, likely "
590 "because the system is too sparse. Check to "
591 "make sure this set of input values will "
ValueError: Crisp output cannot be calculated, likely because the system is too sparse. Check to make sure this set of input values will activate at least one connected Term in each Antecedent via the current set of Rules.
edit:
I try to wrap the line code ValueError by try but the ValueError is activated yet
def defuzz(self):
"""Derive crisp value based on membership of adjective(s)."""
if not self.sim._array_inputs:
ups_universe, output_mf, cut_mfs = self.find_memberships()
if len(cut_mfs) == 0:
raise ValueError("No terms have memberships. Make sure you "
"have at least one rule connected to this "
"variable and have run the rules calculation.")
try:
return defuzz(ups_universe, output_mf,
self.var.defuzzify_method)
except AssertionError:
try:
new_c1 = []
new_c2 = []
new_c3 = []
new_c4 = []
new_vent = []
new_tout = []
newcondition1 = []
newcondition2 = []
newcondition3 = []
newcondition4 = []
newcondition5 = []
newcondition6 = []
#input
n = 0
for n in range(len(namespca)):
new_c1.append(fuzz.interp_membership(PCA1.universe, PCA1[namespcapd.name.loc[n]].mf, comp1n[step]))
new_c2.append(fuzz.interp_membership(PCA2.universe, PCA2[namespcapd.name.loc[n]].mf, comp2n[step]))
new_c3.append(fuzz.interp_membership(PCA3.universe, PCA3[namespcapd.name.loc[n]].mf, comp3n[step]))
new_c4.append(fuzz.interp_membership(PCA4.universe, PCA4[namespcapd.name.loc[n]].mf, comp4n[step]))
n = 0
for n in range(len(namesvent)):
new_vent.append(fuzz.interp_membership(Vent.universe, Vent[namesventpd.name.loc[n]].mf, dataoutf.NumVentOn[step]))
#output
n = 0
for n in range(len(namestemp)):
new_tout.append(fuzz.interp_membership(Tout.universe, Tout[namestemppd.name.loc[n]].mf, dataoutf.TsaidaHT[step]))
#new_c1 = np.transpose(new_c1)
new_c1_conv = pd.DataFrame(new_c1)
#new_c2 = np.transpose(new_c2)
new_c2_conv = pd.DataFrame(new_c2)
#new_c3 = np.transpose(new_c3)
new_c3_conv = pd.DataFrame(new_c3)
#new_c4 = np.transpose(new_c4)
new_c4_conv = pd.DataFrame(new_c4)
#new_vent = np.transpose(new_vent)
new_vent_conv = pd.DataFrame(new_vent)
#new_tout = np.transpose(new_tout)
new_tout_conv = pd.DataFrame(new_tout)
i=0
for i in range(pcamf):
newcondition1.append([new_c1_conv.idxmax(axis=0) == i])
newcondition2.append([new_c2_conv.idxmax(axis=0) == i])
newcondition3.append([new_c3_conv.idxmax(axis=0) == i])
newcondition4.append([new_c4_conv.idxmax(axis=0) == i])
i=0
for i in range(ventmf):
newcondition5.append([new_vent_conv.idxmax(axis=0) == i])
i=0
for i in range(tempmf):
newcondition6.append([new_tout_conv.idxmax(axis=0) == i])
choicelistpca = namespca
choicelistvent = namesvent
choicelisttout = namestemp
new_c1_rules = np.select(newcondition1, choicelistpca)
new_c2_rules = np.select(newcondition2, choicelistpca)
new_c3_rules = np.select(newcondition3, choicelistpca)
new_c4_rules = np.select(newcondition4, choicelistpca)
new_vent_rules = np.select(newcondition5, choicelistvent)
new_tout_rules = np.select(newcondition6, choicelisttout)
new_rules = np.vstack([new_c1_rules,new_c2_rules,new_c3_rules,new_c4_rules,new_vent_rules,new_tout_rules])
new_rules = new_rules.T
new_rulespd = pd.DataFrame(new_rules,columns=['PCA1','PCA2','PCA3','PCA4','Vent','Tout'])
#Checar se a nova regra está dentro do conjunto de regras fuzzy atual
if pd.merge(new_rulespd,AutoRules, on=['PCA1','PCA2','PCA3','PCA4','Vent','Tout'],how='inner').empty:
print('Nova regra não encontrada no conjunto atual de regras fuzzy!')
else:
pd.merge(new_rulespd,AutoRules, on=['PCA1','PCA2','PCA3','PCA4','Vent','Tout'],how='inner')
"""except AssertionError:
raise ValueError("Crisp output cannot be calculated, likely "
"because the system is too sparse. Check to "
"make sure this set of input values will "
"activate at least one connected Term in each "
"Antecedent via the current set of Rules.")"""
else:
# Calculate using array-aware version, one cut at a time.
output = np.zeros(self.sim._array_shape, dtype=np.float64)
it = np.nditer(output, ['multi_index'], [['writeonly', 'allocate']])
for out in it:
universe, mf = self.find_memberships_nd(it.multi_index)
out[...] = defuzz(universe, mf, self.var.defuzzify_method)
return output

Wrap the line of code that raises ValueError in a try. And decide what to do in its except ValueError: clause. Perhaps continue-ing on to the next iteration might be reasonable.

Need Suggestion.

How to debug "NameError: global name 'X' is not defined" in Python? I am pretty much new in Python. I am using jupyter_notebook with Python 2.7 to execute code. I am facing following error.
My code:
logFile = "NASAlog.txt"
def parseLogs():
parsed_logs=(sc
.textFile(logFile)
.map(parseApacheLogLine)
.cache())
access_logs = (parsed_logs
.filter(lambda s: s[1] == 1)
.map(lambda s: s[0])
.cache())
failed_logs = (parsed_logs
.filter(lambda s: s[1] == 0)
.map(lambda s: s[0]))
failed_logs_count = failed_logs.count()
if failed_logs_count > 0:
print 'Number of invalid logline: %d' % failed_logs.count()
for line in failed_logs.take(20):
print 'Invalid logline: %s' % line
print 'Read %d lines, successfully parsed %d lines, failed to parse %d lines' % (parsed_logs.count(), access_logs.count(), failed_logs.count())
return parsed_logs, access_logs, failed_logs
parsed_logs, access_logs, failed_logs = parseLogs()
ERROR
> NameError Traceback (most recent call last)
> <ipython-input-18-b365aa793252> in <module>()
> 24 return parsed_logs, access_logs, failed_logs
> 25
> ---> 26 parsed_logs, access_logs, failed_logs = parseLogs()
>
> <ipython-input-18-b365aa793252> in parseLogs()
> 2
> 3 def parseLogs():
> ----> 4 parsed_logs=(sc
> 5 .textFile(logFile)
> 6 .map(parseApacheLogLine)
>
> NameError: global name 'sc' is not defined

The problem is that you did never define sc. Therefore python can't find it. (Makes sense, doesn't it?)
Now there are several possible reasons:
- python is case-sensitive. Did you somewhere define SC instead of sc? ... Or Sc instead of sc?
You defined sc in another function (-> you defined it in a function outside parseLogs()). If you only define it there the variable will be local and just be available to the code inside the function. Add the line global sc to the first line of your function to make it accessible everywhere in you whole code.
You simply did not define sc.

Runtime Exception. Exception in python callback function evaluation:

I am working on an assignment for Coursera's Machine Learning: Regression course. I am using the kc_house_data.gl/ dataset and GraphLab Create. I am adding new variables to train_data and test_data that are combinations of old variables. Then I take the mean of all these variables. These are the variables I am adding:
bedrooms_squared = bedrooms * bedrooms
bed_bath_rooms = bedrooms*bathrooms
log_sqft_living = log(sqft_living)
lat_plus_long = lat + long
Here is my code:
train_data['bedrooms_squared'] = train_data['bedrooms'].apply(lambda x: x**2)
test_data['bedrooms_squared'] = test_data['bedrooms'].apply(lambda x: x**2)
# create the remaining 3 features in both TEST and TRAIN data
train_data['bed_bath_rooms'] = train_data.apply(lambda row: row['bedrooms'] * row['bathrooms'])
test_data['bed_bath_rooms'] = test_data.apply(lambda row: row['bedrooms'] * row['bathrooms'])
train_data['log_sqft_living'] = train_data['sqft_living'].apply(lambda x: log(x))
test_data['log_sqft_living'] = test_data['bedrooms'].apply(lambda x: log(x))
train_data['lat_plus_long'] = train_data.apply(lambda row: row['lat'] + row['long'])
train_data['lat_plus_long'] = train_data.apply(lambda row: row['lat'] + row['long'])
test_data['bedrooms_squared'].mean()
test_data['bed_bath_rooms'].mean()
test_data['log_sqft_living'].mean()
test_data['lat_plus_long'].mean()
This is the error I'm getting:
RuntimeError: Runtime Exception. Exception in python callback function evaluation:
ValueError('math domain error',):
Traceback (most recent call last):
File "graphlab\cython\cy_pylambda_workers.pyx", line 426, in graphlab.cython.cy_pylambda_workers._eval_lambda
File "graphlab\cython\cy_pylambda_workers.pyx", line 169, in graphlab.cython.cy_pylambda_workers.lambda_evaluator.eval_simple
File "<ipython-input-13-1cdbcd5f5d9b>", line 5, in <lambda>
ValueError: math domain error
I have no idea what this means. Any idea on what caused it and how I fix it? Thanks.

Your problem is that log is receiving a negative number.
log is defined only for numbers greater than zero.
You need to check your values.

Please add/learn exceptions to make your code more robust:
try:
train_data['log_sqft_living'] = train_data['sqft_living'].apply(lambda x: log(x))
test_data['log_sqft_living'] = test_data['bedrooms'].apply(lambda x: log(x))
train_data['lat_plus_long'] = train_data.apply(lambda row: row['lat'] + row['long'])
train_data['lat_plus_long'] = train_data.apply(lambda row: row['lat'] + row['long'])
test_data['bedrooms_squared'].mean()
test_data['bed_bath_rooms'].mean()
test_data['log_sqft_living'].mean()
test_data['lat_plus_long'].mean()
except e as Exception:
print "ERROR in function:", e