📚 Introductory Econometrics with MORISTAT

A Practical Guide to Empirical Analysis

Naguib Lallmahomed · naglal@linux-mauritius.com · 2026-06-14

Chapter 11: Machine Learning for Econometrics

🎯 Learning Objectives

Upon completing this chapter, you will be able to:

Understand the key differences between econometrics and machine learning
Apply regularization (Ridge, Lasso, Elastic Net)
Understand cross-validation and its importance
Implement decision trees and random forests
Evaluate models using MSE, RMSE, MAE
Understand the bias-variance tradeoff

11.1 What is Machine Learning?

Machine learning (ML) is a branch of artificial intelligence that enables computers to learn from data without being explicitly programmed.

Key Insight: While econometrics focuses on causal inference and hypothesis testing, machine learning focuses on prediction and pattern recognition.

11.1.1 Econometrics vs Machine Learning

Aspect	Econometrics	Machine Learning
Goal	Causal inference, hypothesis testing	Prediction, pattern recognition
Interpretability	High (coefficients, p-values)	Low to moderate (black box)
Model Size	Small (few variables)	Large (many variables)
Data Requirements	Moderate	Large
Overfitting	Controlled via theory	Controlled via regularization

11.1.2 Supervised vs Unsupervised Learning

Supervised Learning: Model learns from labelled data (Y is known). Examples: regression, classification.
Unsupervised Learning: Model finds patterns in unlabelled data (Y is unknown). Examples: clustering, principal components.

11.2 Regularization: Ridge, Lasso, and Elastic Net

Regularization adds a penalty term to the OLS objective function to prevent overfitting.

11.2.1 Ridge Regression (L2 Penalty)

min Σ(Yᵢ − β₀ − β₁X₁ᵢ − ⋯ − βₖXₖᵢ)² + λΣβⱼ²

Shrinks coefficients toward zero (but never exactly zero)
Good when all variables are important
λ (lambda) controls the strength of the penalty

11.2.2 Lasso Regression (L1 Penalty)

min Σ(Yᵢ − β₀ − β₁X₁ᵢ − ⋯ − βₖXₖᵢ)² + λΣ|βⱼ|

Shrinks some coefficients to exactly zero
Performs variable selection
Good when many variables are irrelevant

11.2.3 Elastic Net (L1 + L2 Penalty)

min Σ(Yᵢ − β₀ − β₁X₁ᵢ − ⋯ − βₖXₖᵢ)² + λ₁Σ|βⱼ| + λ₂Σβⱼ²

Combines Ridge and Lasso
Handles correlated variables better than Lasso
α controls the mix (α = 0 → Ridge, α = 1 → Lasso)

11.3 Cross-Validation

Cross-validation (CV) is a technique for evaluating model performance on unseen data.

11.3.1 k-Fold Cross-Validation

Split data into k folds (typically 5 or 10)
Train on k−1 folds, validate on the remaining fold
Repeat k times, average the performance
Reduces overfitting and provides more robust estimates

Example: 5-Fold CV

Fold 1: Train on 80%, Validate on 20%
Fold 2: Train on 80%, Validate on 20% (different split)
...
Fold 5: Train on 80%, Validate on 20%
Average the 5 validation errors

11.4 Decision Trees and Random Forests

11.4.1 Decision Trees

Non-parametric model that splits data based on feature values
Easy to interpret (visual tree structure)
Prone to overfitting (if deep)

11.4.2 Random Forests

Ensemble of many decision trees
Each tree is trained on a random subsample of data and features
Reduces variance and prevents overfitting
Provides feature importance rankings

Key Insight: Random forests often outperform single decision trees and are widely used in applied economics and finance.

11.5 Model Evaluation Metrics

11.5.1 Regression Metrics

Metric	Formula	Interpretation
MSE	Σ(Yᵢ − Ŷᵢ)² / n	Average squared error (penalizes large errors)
RMSE	√MSE	Root MSE (in same units as Y)
MAE	Σ\|Yᵢ − Ŷᵢ\| / n	Average absolute error (robust to outliers)
R²	1 − SSE/SST	Proportion of variance explained

11.5.2 Classification Metrics

Accuracy: (TP + TN) / (Total)
Precision: TP / (TP + FP)
Recall (Sensitivity): TP / (TP + FN)
F1-Score: 2 × (Precision × Recall) / (Precision + Recall)
AUC-ROC: Area under the ROC curve (measures discrimination)

11.6 The Bias-Variance Tradeoff

Key Concept: There is a tradeoff between bias and variance in predictive models.

High Bias (Underfitting): Model is too simple, misses relationships. Example: OLS with few variables.
High Variance (Overfitting): Model is too complex, captures noise. Example: Deep decision tree.
Goal: Find the "sweet spot" that minimizes total error.

Total Error = Bias² + Variance + Irreducible Error

11.7 MORISTAT Examples

11.7.1 Preparing Data for ML

MORISTAT> LOAD ml_data.csv
MORISTAT> LIST
MORISTAT> SUMMARY
MORISTAT> CORR

11.7.2 Ridge Regression

MORISTAT> REGRESS Y ~ X1 X2 X3 X4 X5
MORISTAT> DIAG ALL

11.8 Practical Exercises

Exercise 11.1: Econometrics vs ML

You are asked to analyse the effect of education on wages. You have data on 10,000 individuals.

If you want to measure the causal effect of education on wages, which approach (econometrics or ML) is more appropriate?
If you want to predict wages for job applicants, which approach is more appropriate?
Explain why the goals are different.

Solution:

Causal inference: Econometrics (control for confounders, instrumental variables)
Prediction: Machine Learning (many features, non-linear relationships)
The goals differ: econometrics asks why (causality), ML asks what (prediction)

Exercise 11.2: Understanding Regularization

You have a dataset with 100 observations and 50 variables. OLS has high variance (overfitting).

Which regularization method (Ridge, Lasso, Elastic Net) would you use if you believe only a few variables are important?
Which method would you use if all variables are likely important?
What is the role of λ (lambda) in regularization?

Solution:

Lasso if only a few variables are important (performs variable selection).
Ridge if all variables are important (shrinks coefficients but doesn't zero them).
λ controls the strength of the penalty: larger λ → more shrinkage/regularization.

Exercise 11.3: Cross-Validation

You have 200 observations. You use 5-fold cross-validation.

How many observations are in each training set?
How many observations are in each validation set?
Why do we use cross-validation instead of a single train-test split?

Solution:

Training set per fold: 160 (80% of 200)
Validation set per fold: 40 (20% of 200)
Cross-validation reduces variance and provides a more robust estimate of model performance than a single split.

Exercise 11.4: Bias-Variance Tradeoff

You have three models:

Model A: OLS with 3 variables (high bias, low variance)
Model B: Random Forest (low bias, high variance)
Model C: Ridge Regression (moderate bias, moderate variance)

Which model is most likely to underfit?
Which model is most likely to overfit?
Which model would you choose for prediction?

Solution:

Underfitting: Model A (high bias, too simple)
Overfitting: Model B (high variance, too complex)
Best for prediction: Model C (Ridge) often balances bias and variance well.

Exercise 11.5: Evaluating Model Performance

You have two forecasting models with the following errors:

Model 1: Errors: 2, -3, 1, -2, 4
Model 2: Errors: 1, -1, 0, -1, 1

Calculate RMSE for both models.
Calculate MAE for both models.
Which model is better and why?

Solution:

Model 1 RMSE: √((4+9+1+4+16)/5) = √6.8 = 2.607
Model 2 RMSE: √((1+1+0+1+1)/5) = √0.8 = 0.894
Model 1 MAE: (2+3+1+2+4)/5 = 2.4
Model 2 MAE: (1+1+0+1+1)/5 = 0.8
Model 2 is better (lower RMSE and MAE)

11.9 Key Terms

Machine Learning Supervised Learning Unsupervised Learning Regularization Ridge Regression Lasso Regression Elastic Net Cross-Validation k-Fold CV Decision Tree Random Forest Bias-Variance Tradeoff MSE RMSE MAE Overfitting Underfitting Feature Importance AUC-ROC

11.10 Further Reading

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An Introduction to Statistical Learning (2nd ed.). Springer.
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.). Springer.
Mullainathan, S., & Spiess, J. (2017). "Machine Learning: An Applied Econometric Approach." Journal of Economic Perspectives, 31(2), 87–106.
Athey, S., & Imbens, G. W. (2019). "Machine Learning Methods That Economists Should Know About." Annual Review of Economics, 11, 685–725.

🔄 Ready to connect with MORISTAT?