Optimization for Data Science

Academic Year: 2024–2025

Lecture notes on optimization theory and algorithms for Optimization for Data Science at University of Pisa. Topics include convexity, gradient methods, nonsmooth optimization, duality, and constrained algorithms.

⚠️ Disclaimer: These notes are my personal interpretation of the course slides and lectures. They surely contain errors, omissions, or inaccuracies (they were written as a study support after understanding all topics, not as a starting point — please keep this in mind if you use them). They are not an official course resource and should not be treated as a substitute for the original material. Use them at your own risk.

---

📖 Topics Covered

1. Foundations
Problem formulation, reformulations, existence of optima, approximate solutions, multivariate and multi-objective optimization.

2. Linear Algebra and Quadratic Forms
Norms, scalar products, matrices, eigenvalues, spectral decomposition, matrix definiteness, quadratic function geometry.

3. Univariate Optimization
Lipschitz continuity, local optimality, golden ratio search, dichotomic search, quadratic interpolation, Newton's method.

4. Unconstrained Optimality and Convexity
Gradients, Hessians, optimality conditions, convexity, quasi-convexity, strong convexity, smoothness.

5. Gradient Methods for Quadratic Functions
Steepest descent, convergence analysis, conjugate gradient, preconditioning, Cholesky factorization.

6. Smooth Unconstrained Optimization
Gradient method, line search, Newton and quasi-Newton methods, trust region, deflected gradient, heavy ball, Nesterov acceleration.

7. Nonsmooth Unconstrained Optimization
Subgradients, subdifferential, subgradient methods, smoothing, bundle methods; applications to Lasso, SVM, SVR.

8. Constrained Optimality and Duality
Tangent cones, KKT conditions, second-order conditions, Lagrangian duality, conic programs, SVM/SVR duality.

9. Constrained Optimization Algorithms
Active set, projected gradient, Frank–Wolfe, dual methods, barrier methods, primal-dual interior point.

---

📄 Compiled PDF

A pre-compiled version is available as Optimization4DS.pdf in the repository root.

---

🔧 How to Edit

The easiest way to compile and edit the source is via Overleaf:

1. Download this repository as a .zip file (Code → Download ZIP)
2. Go to overleaf.com → New Project → Upload Project
3. Upload the .zip file
4. Set main.tex as the main document and compile

---

🤝 Contributing

Found a typo, an error, or have a suggestion? Contributions are welcome!
Feel free to open an Issue or submit a Pull Request.