Fitting a Multiple Linear Regression Curve

In Data Science Discovery we briefly touched upon how a linear regression model finds a "line of best fit" for our training dataset. In the case of our example, we are seeking to find the best intercept and slopes $\hat{\beta}_0,\hat{\beta}_1,...,\hat{\beta}_8$ for our linear regression model.

$\hat{y}=\hat{\beta}_0+\hat{\beta}_1 x_1+\hat{\beta}_2 x_2+⋯+\hat{\beta}_8 x_8$

But how do we know when we've selected the best intercept and slopes $\hat{\beta}_0,\hat{\beta}_1,...,\hat{\beta}_8$? There are actually many ways that a certain set of intercepts and slopes may be deemed best.

However, one of the most popular methods is the ordinary least squares (OLS) method which says that the best intercept and slopes are the ones that minimize the sum of the square of each of our residuals in the training dataset. Or in other words,

$f(\hat{\beta}_0,\hat{\beta}_1,...,\hat{\beta}_8)$

$=\sum_{i=1}^n(residual_i)^2 =(residual_1)^2+...+(residual_n)^2$

$=\sum_{i=1}^n(observed\_ y_i - predicted\_ y_i)^2 =(observed\_ y_1 - predicted\_ y_1)^2+...+(observed\_ y_n - predicted\_ y_n)^2$

$=(y_1 - \hat{y}_1)^2+...+(y_n - \hat{y}_n)^2$

$=\left(y_1 - (\hat{\beta}_0 +\hat{\beta}_1x_{1,1}+...+\hat{\beta}_8x_{8,1})\right)^2+...+ \left(y_n - (\hat{\beta}_0+\hat{\beta}_1x_{1,n}+...+\hat{\beta}_8x_{8,n}) \right)^2$

For illustrative purposes, let’s temporarily switch gears to the task of predicting price with just a single explanatory variable beds. Thus our linear regression curve that we are trying to fit would be a simple linear regression model, because we have just one explanatory variable.

$\hat{price}=\hat{\beta}_0+\hat{\beta}_1 beds$

df_train[['price','beds']]

This means that our goal would be to find the optimal values for $\hat{\beta}_0,\hat{\beta}_1$ that minimized a 3-d curve like the one shown below.

$f(\hat{\beta}_0,\hat{\beta}_1)$
$=(85 - (\hat{\beta}_0+\hat{\beta}_1(2)))^2+...+(50 - (\hat{\beta}_0+\hat{\beta}_1(2))^2$

The process to find these optimal values of the intercept and slope requires multivariate calculus, which is beyond the scope of this course. However, the sci-kit learn LinearRegression() function will return these optimal values for us.

For instance, the optimal intercept was found to be $\hat{\beta}_0^*=40.86$ and the optimal slope was found to be $\hat{\beta}_1^*=66.61$.

df_train = X_train.copy()
df_train['price']=y_train
df_train.head()

df_test = X_test.copy()
df_test['price']=y_test
df_test.head()

from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(X_train[['beds']], y_train)
print('slope: ', model.coef_)
print('intercept: ', model.intercept_)

    slope:  [66.6196058]
    intercept:  40.859249687151646

Hence our fitted ordinary least squares simple regression model is the following:
$\hat{price}=40.86+66.61 beds$