The course provides the foundational concepts, methods, languages, and systems for ingesting, processing, and analyzing data that flows to enable real-time decisions. The course aims to the tame velocity dimensions of Big Data without forgetting the volume and variety dimensions.
Topics covered and tentative scheduling TENTATIVE
- 13/09/2023 – Administrative items and course introduction
Streaming Data Engineering
- 14/09/2023 – From the foundations of streaming algorithms to real-world languages and systems
- 20/09/2023 – DSMS Theory/Practice – Languages for Data Stream Management Systems (DSMS) illustrated via EPL & Fire Alarm case study in EPL
- 21/09/2023 – CEP Theory/Practice – Languages for Complex Event Processing (CEP) illustrated with the every clause and the pattern guards in EPL & Fire Alarm case study in EPL
- 27/09/2023 – DSMS/CEP Practice – Robotic Arm case study in EPL
- 28/09/2023 – DSMS/CEP Practice – Bocce Game case study in EPL
- 04/10/2023 – Scaling stream ingestion with Apache Kafka
- 05/10/2023 – Lessons are suspended due to master’s degree exams
- 11/10/2023 – Scaling stream processing with Apache Spark Structured Streaming
- 12/10/2023 – Practice with Spark Structured Streaming
- 18/10/2023 – ksqlDB: cutting all from the same (kafka) molt
- 19/10/2023 – Practice with ksqlDB
- 25/10/2023 – Wrap up and preview of the questions and the exercises about Streaming Data Engineering
Streaming Data Science
This second part of the course covers Time-series Analytics (TSA), Streaming Machine Learning (SML), Continual Artificial Intelligence (Continual AI).
- 26/10/2023 – Introduction to Streaming Data Science
- 08/11/2023 – TSA Theory/Practice – Introduction to Time-series Analytics and the key concept of Stationarity
- 09/11/2023 – TSA Theory/Practice – Decomposing and detrending time-series with and without seasonality
- 15/11/2023 – TSA Theory/Practice -Time-series forecasting
- 16/11/2023 – TSA Theory/Practice – Temporal Dependence, ARMA/ARIMA/SARIMA/SARIMAX models & order estimation
- 22/11/2023 – TSA Therory/Practice – Recurrent Neural Networks
- 23/11/2023 – SML Theory/Practice – Foundations: learning one sample at a time, and prequential evaluation
- 29/11/2023 – SML Theory/Practice – Concept drift and Methods for streaming regression (a.k.a., forecasting) illustrated via River
- 30/11/2023 – SML Theory/Practice – Methods for streaming classification and Ensemble methods for streaming machine learning via River part 1/2
- 06/12/2023 – SML Theory/Practice – Ensemble methods for streaming machine learning via River part 2/2 and Wrap up
- 13/12/2023 – Continual AI Theory – An introduction
- 14/12/2023 – Continual AI Practice – Hands on Avalanche
- 20/12/2023 – Preview of the questions and the exercises about Streaming Data Science
- 21/12/2023 – Stream Reasoning: Artificial Intelligence + Stream Processing (BONUS LECTURE)
Please complete this form if you want a thesis on the course topic.
Students are expected to know the basics about: database management, SQL and Machine Learning.
For a refresh of SQL, I recommend https://www.w3schools.com/sql/. It is simple and comprehensive.
For a gentle introduction to Machine Learning, I recommend watching the following two videos by Luis Serrano:
- A Friendly Introduction to Machine Learning – https://youtu.be/IpGxLWOIZy4
- Machine Learning: Testing and Error Metrics – https://youtu.be/aDW44NPhNw0
Expected learning outcomes
|Knowledge and understanding||Students will learn how to identify problems that can be addressed with big data techniques tailored for velocity and apply the stream data analysis technologies for solving real-world problems|
|Applying knowledge and understanding||Given specific project cases, students will be able to define and implement a streaming data analysis solution for the problem, and apply it on real data streams from social media and IoT sensors|
|Making judgements||Given specific project cases, students will be able to learn how to decide which streaming data analysis solution to apply and how to evaluate this decision|
|Communication||Students will learn to write a report on a project describing and motivating the decisions taken and the results obtained, and oresent their work in front of their colleagues and teachers|
|Lifelong learning skills||Students will learn how to develop a realistic streaming data analysis project in all its phases|
The exam consists of a theoretical (written exam, with a possible oral discussion if necessary by the instructor) and an optional practical part (project work with oral presentation). Further contributions to the mark may come from optional continuous evaluations along the course using in-presence quizzes during the lessons and other interactive modalities in class (max one mark).
The written exam comprises a mix of theoretical questions regarding any course subjects and exercises regarding the technical content and how to apply it in practice. Students can get up to 30 in the written test. The whole exam is a closed-book evaluation. A minimum score on each part is requested.
The optional practical project requires using one or more of the technologies presented in the lectures. It solves a realistic streaming data analysis problem based on real or realistic datasets publicly available or provided by the teachers. Only students who get at least 27/30 in the written exam can opt for it. The maximum increment for the optional project is three marks.
The final grade is computed as follows: written text result + optional continuous evaluation result + optional practical project result. E.g., written text 27 + optional continuous evaluation 1 + optional practical project 3 = 30L
- Kreps, Jay, I Love logs: Event data, stream processing, and data integration., O’Reilly Media, Inc., 2014
- Event Processing Language (EPL)
- ksqlDB Documentation
- Spark Structured Streaming
- Flux language
- Geoff Holmes, Ricard Gavaldà, Albert Bifet, Bernhard Pfahringer, Machine Learning for Data Streams: With Practical Examples in MOA, MIT Press, 2018