Develop Reference

Predicting new data with statsmodels gives ValueError: shapes

I built a multiple regression model using Python statsmodels.
X = df[['var1','var2','var3','var4']]
X = sm.add_constant(X) ## let's add an intercept (beta_0) to our model
y = df['target_trait']
model = sm.OLS(y, X).fit() #argument order: sm.OLS(output, input), see (https://towardsdatascience.com/simple-and-multiple-linear-regression-in-python-c928425168f9)
predictions = model.predict(X)
model.summary()
Now, I want to predict new data. the dataframe for my new data has 4 columns (var1, var2, var3, var4) and 143 rows. Below is how I proceeded.
X_new = df_new[['var1','var2','var3','var4']] #df_new has other variables not to be used. I am extracting the relevant variables.
y_new = model.predict(X_new)
y_new
Running the code above gave me ValueError: shapes (143,4) and (5,) not aligned: 4 (dim 1) != 5 (dim 0).
I am not sure how to fix it. I really would appreciate your help. Thank you in advance for your time

I think I found the issue. When fitting the model I added a constant to the X matrix by doing X = sm.add_constant(X). By doing the same to X_new, the algorithm worked. Anyway, thank you for taking a look.

Can't transform list of arrays into 3D array

I have 3 classifiers that run over 288 samples. All of them are sklearn.neural_network.MLPClassifier structures. Here is the code i am using:
list_of_clfs = [MLPClassifier(...), MLPClassifier(...), MLPClassifier(...)]
probas_list = []
for clf in list_of_clfs:
probas_list.append(clf.predict_proba(X_test))
Each predict_proba(X_test) will return a 2D array with shape (n_samples, n_classes). Then, i am creating a 3D array that will contain all predict_proba() in one single place:
proba = np.array(probas_list) #this should return a (3, n_samples, n_classes) array
This should work fine, but i get an error:
ValueError: could not broadcast input array from shape (288,4) into shape (288)
I don't know why, but this works with dummy examples but not with my dataset.
update: it seems like one of the predict_proba() calls is returning an array of shape (288, 2) but my problem has 4 classes. All classifiers are being tested on the same dataset, so i don't know what this comes from.
Thanks in advance

ValueError: shapes (2,100) and (2,1) not aligned: 100 (dim 1) != 2 (dim 0)

I am doing a machine learning homework and I am making the Logistic Regression Descent Gradient and Logistic Regression Cost.
My functions are like this:
def calcLogRegressionCost(X, y, theta):
#X is the feature vector
#Y is the target vector/ output vector
#theta is the weight vector
observations = len(y)
predictions = sigmoid(np.dot(X, theta))
#Take the error when label=1
class1_cost = -y*np.log(predictions)
#Take the error when label=0
class2_cost = (1-y)*np.log(1-predictions)
#Take the sum of both costs
cost = class1_cost + class2_cost
#Take the average cost
cost = cost.sum() / observations
return cost
def logRegressionGradientDescent(X, y, theta0, alpha):
#X is the feature vector
#Y is the target vector/ output vector
#theta0 is the weight vector
#alpha is the learning rate
#iteration is the steps you want to take
#Start you code from here\
N = len(X)
#1 - Get Predictions
predictions = sigmoid(np.dot(X, theta0))
#2 Transpose features from (100, 2) to (2, 100)
# So we can multiply w the (100,1) cost matrix.
# Returns a (2,1) matrix holding 3 partial derivatives --
# one for each feature -- representing the aggregate
# slope of the cost function across all observations
gradient = np.dot(X.T, predictions - y)
#3 Take the average cost derivative for each feature
gradient /= N
#4 - Multiply the gradient by our learning rate
gradient *= lr
#5 - Subtract from our weights to minimize cost
weights -= gradient
#you should return theta or loss or the both depending on your way
#of implementation
return weights
They ask me to run the Gradient Descent Algorith to fit my parameters theta to my training set. I did a train function which is the following:
W1 = 0.0
W2 = 0.0
weights = np.array([
[W1],
[W2]
])
def train(features, labels, weights, lr, iters):
cost_history = []
for i in range(iters):
weights = logRegressionGradientDescent(features, labels, weights, lr)
#Calculate error for auditing purposes
cost = cost_function(features, labels, weights)
cost_history.append(cost)
# Log Progress
if i % 1000 == 0:
print ("iter: " +str(i) + " cost: "+str(cost))
return weights, cost_history
train([data['First Exam Score'], data['Second Exam Score']], data['Admitted'], weights, 0.00001, 1000)
When I call the function train with my data, it gives me the following error:
ValueError: shapes (2,100) and (2,1) not aligned: 100 (dim 1) != 2 (dim 0)
I am not sure how to make the parameters fit with my dataset. The dataset is a 100 x 3 dataframe. The first 2 columns are data about the grades 100 students obtained in the First and Second Exam respetively. The third columns shows whether they got admitted or not in their desired university, depending on their grades. It is represented by 0 or 1.

When I call the function train with my data, it gives me the following
error:
ValueError: shapes (2,100) and (2,1) not aligned: 100 (dim 1) != 2
(dim 0)
One thing that you must remember as a programmer is that error messages are invaluable for debugging. They give you valuable information about where your logic or code is prone to failure, or is already failing.
If you read the error message, you can note the following things:
As the error mentions misaligned shapes, and we know shapes are associated with vectors and matrices, the problem seems to be related to dimensions of feature matrix and weight matrix being passed into your logistic regression function.
Error message mentions misalignment, which indicates problem possibly with matrix multiplication because misaligned matrices are expected to throw this error if the dimensions of the matrices are not compatible for multiplication or the order in which they are being multiplied leaves the operation infeasible.
By now, you probably would have realized that the error is pointing towards the Numpy dot product of the feature matrix X and the weight vector θ.
In order to fix this error, you will have to ensure two things: shapes of matrices are compatible for carrying out matrix multiplication, and the order of multiplication is correct. Remember that in logistic regression, you need one scalar output for each observation in the feature matrix, which can be further passed as an argument into a probability mapping like sigmoid function to give you the probability of that certain instance belonging to a given class.
Solution to error
To solve this problem, transpose the feature matrix X so that its shape changes to (100,2). After taking the transpose of the feature matrix, the dot product should become feasible, consequently solving the error you are encountering.
It is recommended to create a separate feature matrix, matrix X, which contains only the feature columns and not the target column, which is the last column in your data. It is also recommended to create a label vector y, which stores only the labels or target class column. If I was doing this, I would be doing everything in Pandas but since you are working with Numpy, here's how you can do it.
X = np.transpose([(data['First Exam Score'], data['Second Exam Score']]) #Reshapes the feature matrix from (2,100) to (100,2)
y = data['Admitted']
train(X, y, weights, 0.00001, 1000)
As you can notice, code becomes more readable this way and chances of encountering errors is reduced.
Hope this helps.

How to fix Values error in multiple output regression sent in python

I am trying to get the summary of the OLS regression of a single input parameter/multiple output parameters from my dataset done but I get a value error
I have a dataset in Excel and I want to check how variations in input, i.e.Temperature (t), can affect the 4 outputs, energy consumed or produced in four different components.
If I run the model for one input and just one output, I was able to see the OLS summary. But I need to see how this variation in temperature affects all 4 outputs at the same time.
The dataset has 1000 observations done from a Monte Carlo simulation and printed in an excel file.
X1 = pd.DataFrame({'t': dataset['t']})
Y1 = pd.DataFrame({'E3': dataset['E3'],
'E4': dataset['E4'],
'E5': dataset['E5'],
'E6': dataset['E6']})
model1 = smf.ols('Y1 ~ X1', data=dataset).fit()
model1.summary()
I am supposed to see the OLS regression summary but instead I see ValueError: shapes (1000,4) and (1000,4) not aligned: 4 (dim 1) != 1000 (dim 0)
I am not sure if this is the way to go about it as what I am looking for, is a multi-output regression (single input vs multiple outputs)

Keras predict getting incorrect shape?

I'm new to Keras and am trying to test out a model I've just trained.
I'm using Tensorflow backend and Python 3.
However, the shape my input has and the shape Keras says it has in an error are completely different. Here's my code:
testnote = np.zeros((3,))
testnote[0] = 70
testnote[1] = 70
print(testnote.shape)
pred = model.predict(testnote)
print(pred)
My consistent output is "(3,)" for the shape of testnote and then an error for my predict line: "ValueError: Error when checking input: expected dense_1_input to have shape (3,) but got array with shape (1,)"
How is it that Keras reads testnote as having shape (1,) when I've just confirmed that the shape is (3,)? Is it using some sort of different standard for what "shape" means? I've tried reshaping and adding brackets and a bunch of other things, but I don't really know what the problem is.
For additional context, the model takes in an array with 3 scalar input (representing pitch, velocity, and instrument class) and outputs an array with 1025 scalar outputs. I am carefully not using the word "dimension" since I think this is where I'm getting confused, and technically both are only 1 dimension. I'm sure there are many problems with my model which I will have to fix following this. However, I'd like to just get this prediction function working so I can understand what my output looks like.
Thanks in advance for any help.

A Keras Model implicitly expects that your data (passed as a np array) has a dimension for the batch size. Currently, your model is interpreting testnote as being 3 examples of shape 1. Try adding the batch dimension to 'testnote' as follows:
testnote = testnote.reshape(1,-1)
This will reshape testnote to shape (1, 3), so that you explicitly define the batch size to be 1.