Math405
Numerical Linear Algebra & Optimization
Faculty
Aleksandr Katrutsa
Research scientist, Skoltech Senior research scientist, Artificial Intelligence Research Institute
Course length
Duration
Total hours
Credits
Language
Course type
Fee for single course
Fee for degree students
Skills you’ll learn
Overview
This course is about modelling real-world problems with optimisation and numerical linear algebra tools and how to solve them. We will discuss the most important algorithms from NLA, classical convex optimisation, and stochastic optimisation methods, which are crucial for efficient training of large neural networks. The selected topics will be discussed from both theoretical and practical perspectives. In particular, practice sessions will include introducing the standard open-source packages for solving considered problems. Every block of the course will be equipped with multiple practical demos and home assignments that help students go deeper into the discussed topics.
Learning highlights
- The key objective of this course is to develop skills in modelling real-world problems with different optimisation and numerical linear algebra tools. Students will be able to analyse the given problem in terms of convexity and select the proper method, solver, or package to solve the given problem. In addition, students become familiar with the main ingredients for developing specific methods for new problems and their efficient implementation.
Course outline
15 classes
Session 1
NLA Intro: floating points, basic operations needed in neural networks. Intro to simple neural networks.
Practice: check pre-trained models in different formats and simple tasks for theory recap
Session 2
Dimensionality reduction, SVD (complete and partial), briefly on algorithms for computing SVD
Practice: NN compression
Session 3
Condition number and computations stability. Eigenvalue decomposition (complete and partial)
Practice: PageRank and Lipschitz Ness of neural networks
Session 4
Tensors and basic operations with them.
Practice: applications with tensors and their decompositions (Tucker, CP, TT)
Session 5
Automatic differentiation
and review of the discussed topics
Session 6
How to model real-world problems with optimisation tools?
Practice: converting text description to maths
Session 7
Convexity features recap: functions and optimality conditions
Practice: check functions for convexity and modelling tools
Session 8
Intro to numerical methods. GD recap and intro to the acceleration phenomenon
Practice: implementation and comparison of methods
Session 9
Newton method and quasi-Newton methods
Practice: implementation and comparison of methods + complexity analysis
Session 10
Methods for constrained optimisation problems: brief intro
Practice: processing of regularizers in neural networks (sparsity + low-rank)
Session 11
Stochastic gradient and why is it important?
Practice: comparison of GD and stochastic GD
Session 12
Adaptive first-order methods and why do they improve convergence
Practice: more experiments and discussion of issues arose in training neural networks
Session 13
Adversarial attacks and methods for their construction
Practice: Foolbox package tutorial
Session 14
Final exam
Session 15
Project presentations and discussion
Course materials
Books
Prerequisites
The list of required skills is provided below: Basic knowledge of linear algebra and calculus, Practice in Python, Understanding the notion of the asymptotic complexity of the algorithms, Basic probability concepts and distributions,
The minor gaps in the skills can be filled during the sessions or after-class discussions.
Methodology
The course will be delivered in multiple formats. The regular classes are a mix of lectures and practical sessions. Students will work on projects in groups and report their results in the final presentation session. The home assignments for individual work will be released after in-class discussions of the corresponding topics.
Grading
Aleksandr is a research scientist at Skoltech and a senior research scientist at the Artificial Intelligence Research Institute. He defended his PhD thesis in 2019 at the Moscow Institute of Physics and Technology (MIPT). The focus of his research is on the intersection of machine learning methods and numerical analysis. Aleksandr is the author of multiple papers published in top-tier journals and conferences. Also, he participated in industrial projects that were successfully deployed in production. Aleksandr created and delivered courses on numerical linear algebra and optimisation methods at Yandex School of Data Analysis, AI Masters, and MIPT.
See full profileApply for this course
Numerical Linear Algebra & Optimization
by Aleksandr Katrutsa
Total hours
45 Hours
Dates
Dec 02 - Dec 20, 2024
Fee for single course
€1500
Fee for degree students
€750
How to secure your spot
Complete the form below to kickstart your application
Schedule your Harbour.Space interview
If successful, get ready to join us on campus
FAQ
Will I receive a certificate after completion?
Yes. Upon completion of the course, you will receive a certificate signed by the director of the program your course belonged to.
Do I need a visa?
This depends on your case. Please check with the Spanish or Thai consulate in your country of residence about visa requirements. We will do our part to provide you with the necessary documents, such as the Certificate of Enrollment.
Can I get a discount?
Yes. The easiest way to enroll in a course at a discounted price is to register for multiple courses. Registering for multiple courses will reduce the cost per individual course. Please ask the Admissions Office for more information about the other kinds of discounts we offer and what you can do to receive one.