252-3900-00L  Big Data for Engineers

SemesterFrühjahrssemester 2021
DozierendeG. Fourny
Periodizitätjährlich wiederkehrende Veranstaltung
LehrspracheEnglisch
KommentarThis course is not intended for Computer Science and Data Science MSc students!



Lehrveranstaltungen

NummerTitelUmfangDozierende
252-3900-00 VBig Data for Engineers
The lecturers will communicate the exact lesson times of ONLINE courses.
2 Std.
Di10:00-12:00ON LI NE »
G. Fourny
252-3900-00 UBig Data for Engineers
Gruppeneinteilung erfolgt über myStudies.
2 Std.
Mi14:15-16:00CAB G 57 »
16:15-18:00ML H 34.3 »
16:15-18:00NO C 44 »
16:15-18:00NO D 11 »
Fr16:15-18:00CAB G 52 »
16:15-18:00CAB G 56 »
16:15-18:00CAB G 57 »
G. Fourny
252-3900-00 ABig Data for Engineers1 Std.G. Fourny

Katalogdaten

KurzbeschreibungThis course is part of the series of database lectures offered to all ETH departments, together with Information Systems for Engineers. It introduces the most recent advances in the database field: how do we scale storage and querying to Petabytes of data, with trillions of records? How do we deal with heterogeneous data sets? How do we deal with alternate data shapes like trees and graphs?
LernzielThis lesson is complementary with Information Systems for Engineers as they cover different time periods of database history and practices -- you can even take both lectures at the same time.

The key challenge of the information society is to turn data into information, information into knowledge, knowledge into value. This has become increasingly complex. Data comes in larger volumes, diverse shapes, from different sources. Data is more heterogeneous and less structured than forty years ago. Nevertheless, it still needs to be processed fast, with support for complex operations.

This combination of requirements, together with the technologies that have emerged in order to address them, is typically referred to as "Big Data." This revolution has led to a completely new way to do business, e.g., develop new products and business models, but also to do science -- which is sometimes referred to as data-driven science or the "fourth paradigm".

Unfortunately, the quantity of data produced and available -- now in the Zettabyte range (that's 21 zeros) per year -- keeps growing faster than our ability to process it. Hence, new architectures and approaches for processing it were and are still needed. Harnessing them must involve a deep understanding of data not only in the large, but also in the small.

The field of databases evolves at a fast pace. In order to be prepared, to the extent possible, to the (r)evolutions that will take place in the next few decades, the emphasis of the lecture will be on the paradigms and core design ideas, while today's technologies will serve as supporting illustrations thereof.

After visiting this lecture, you should have gained an overview and understanding of the Big Data landscape, which is the basis on which one can make informed decisions, i.e., pick and orchestrate the relevant technologies together for addressing each business use case efficiently and consistently.
InhaltThis course gives an overview of database technologies and of the most important database design principles that lay the foundations of the Big Data universe.

It targets specifically students with a scientific or Engineering, but not Computer Science, background.

We take the monolithic, one-machine relational stack from the 1970s, smash it down and rebuild it on top of large clusters: starting with distributed storage, and all the way up to syntax, models, validation, processing, indexing, and querying. A broad range of aspects is covered with a focus on how they fit all together in the big picture of the Big Data ecosystem.

No data is harmed during this course, however, please be psychologically prepared that our data may not always be in normal form.

- physical storage: distributed file systems (HDFS), object storage(S3), key-value stores

- logical storage: document stores (MongoDB), column stores (HBase)

- data formats and syntaxes (XML, JSON, RDF, CSV, YAML, protocol buffers, Avro)

- data shapes and models (tables, trees)

- type systems and schemas: atomic types, structured types (arrays, maps), set-based type systems (?, *, +)

- an overview of functional, declarative programming languages across data shapes (SQL, JSONiq)

- the most important query paradigms (selection, projection, joining, grouping, ordering, windowing)

- paradigms for parallel processing, two-stage (MapReduce) and DAG-based (Spark)

- resource management (YARN)

- what a data center is made of and why it matters (racks, nodes, ...)

- underlying architectures (internal machinery of HDFS, HBase, Spark)

- optimization techniques (functional and declarative paradigms, query plans, rewrites, indexing)

- applications.

Large scale analytics and machine learning are outside of the scope of this course.
LiteraturPapers from scientific conferences and journals. References will be given as part of the course material during the semester.
Voraussetzungen / BesonderesThis course is not intended for Computer Science and Data Science students. Computer Science and Data Science students interested in Big Data MUST attend the Master's level Big Data lecture, offered in Fall.

Requirements: programming knowledge (Java, C++, Python, PHP, ...) as well as basic knowledge on databases (SQL). If you have already built your own website with a backend SQL database, this is perfect.

Attendance is especially recommended to those who attended Information Systems for Engineers last Fall, which introduced the "good old databases of the 1970s" (SQL, tables and cubes). However, this is not a strict requirement, and it is also possible to take the lectures in reverse order.

Leistungskontrolle

Information zur Leistungskontrolle (gültig bis die Lerneinheit neu gelesen wird)
Leistungskontrolle als Semesterkurs
ECTS Kreditpunkte6 KP
PrüfendeG. Fourny
FormSessionsprüfung
PrüfungsspracheEnglisch
RepetitionDie Leistungskontrolle wird in jeder Session angeboten. Die Repetition ist ohne erneute Belegung der Lerneinheit möglich.
Prüfungsmodusschriftlich 180 Minuten
Zusatzinformation zum PrüfungsmodusWorking on the exercises is rewarded in the sense of ETH's continuous performance assessment with up to 0.25 bonus points. In principle, it is expected that students solve all exercises. In order to control this, every week, 2 to 3 batches of small assignments (25 in total) will be marked as "bonus point eligible". Students can hand in their solution to these special assignments every week, and they will be graded (pass/fail). Each one of the 25 assignment batches gives 0.01 extra point at the exam, which means that up to 0.25 can be obtained as a bonus. Note that these extra points are added before rounding to a grade with a quarter-point precision and thus may or may not affect the final grade. The final exam may be computer-based.
Hilfsmittel schriftlichGeneral dictionaries are allowed.
Online-PrüfungDie Prüfung kann am Computer stattfinden.
FernprüfungDas Ablegen als Fernprüfung ist nicht möglich.
Diese Angaben können noch zu Semesterbeginn aktualisiert werden; verbindlich sind die Angaben auf dem Prüfungsplan.

Lernmaterialien

 
HauptlinkInformation
Es werden nur die öffentlichen Lernmaterialien aufgeführt.

Gruppen

252-3900-00 UBig Data for Engineers
GruppenG-01
Mi14:15-16:00CAB G 57 »
G-02
Mi16:15-18:00ML H 34.3 »
G-03
Mi16:15-18:00NO C 44 »
G-04
Mi16:15-18:00NO D 11 »
G-05
Fr16:15-18:00CAB G 56 »
G-06
Fr16:15-18:00CAB G 57 »
G-07
Fr16:15-18:00CAB G 52 »

Einschränkungen

Keine zusätzlichen Belegungseinschränkungen vorhanden.

Angeboten in

StudiengangBereichTyp
Computational Biology and Bioinformatics MasterTheorieWInformation
DAS in Data ScienceBig Data SystemsWInformation
Elektrotechnik und Informationstechnologie MasterVertiefungsfächerWInformation
Elektrotechnik und Informationstechnologie MasterEmpfohlene FächerWInformation
Informatik (Allgemeines Angebot)Informatik für NichtinformatikerZInformation
Integrated Building Systems MasterVertiefungsfächerWInformation
Rechnergestützte Wissenschaften BachelorWahlfächerWInformation
Rechnergestützte Wissenschaften MasterWahlfächerWInformation
Science, Technology, and Policy MasterDaten und InformationstechnologieWInformation
Statistik MasterStatistische und mathematische FächerWInformation
Statistik MasterFachbezogene WahlfächerWInformation