Learning highlights

• The course starts with a brief overview of a data-storage task for any software. Data storage tasks in general, with all possible solutions, are like files, in-memory services, or specialised applications (databases).

• We proceed with a list of requirements that proved to be essential for a data storage tool: ACID, transactions, availability of data access languages (SQL, etc.). Afterwards, we will illustrate why given requirements determined the market dominance of classical relational databases (Oracle, MS SQL, PostgreSQL, MySQL, etc.) at the end of the 20th century. Later, we’ll describe why the technological advances of the 21st century gave birth to a set of non-classical databases, such as in-memory storage, document storage, columnar storage, etc.

• The bulk of the remaining course focuses on the tradeoffs to be considered during technology selection and database design. We will discuss a Polyglot Persistence paradigm for combining multiple databases for different facets of an application, combining their strengths and mitigating their weaknesses. We discuss the balance between performance, complexity, and permitted data delay for various databases and architectural approaches, as well as the fundamental limitations of the CAP theorem. We emphasise the difference between OLAP (analytical) and OLTP tasks and modern data warehouse designs (Data Vault, Anchor Modelling, etc.).

• Plenty of hands-on examples and homework are given to demonstrate ideas and compare and contrast various approaches and technologies. The course wraps up with a discussion of modern state-of-the-art databases, like serverless cloud databases and global cloud tools, violating the CAP theorem. We will also discuss how modern LLM tools (ChatGPT) can be used to design data storage applications and how data storage can be used to benefit LLM (vector databases).

• During the course we will make a significant focus on the various open-source tools to avoid relying too heavily on particular vendors.