263-3010-00L  Big Data

SemesterAutumn Semester 2022
LecturersG. Fourny
Periodicityyearly recurring course
Language of instructionEnglish



Courses

NumberTitleHoursLecturers
263-3010-00 VBig Data3 hrs
Tue14:15-16:00CAB G 61 »
Wed09:15-10:00HG E 5 »
G. Fourny
263-3010-00 UBig Data
Groups are selected in myStudies.
2 hrs
Wed14:15-16:00CAB G 52 »
14:15-16:00CAB G 59 »
14:15-16:00HG E 33.1 »
14:15-16:00HG G 26.1 »
Fri14:15-16:00CAB G 52 »
14:15-16:00CHN D 46 »
G. Fourny
263-3010-00 ABig Data
Individual work to get hands-on experience with the technologies covered, no fixed presence required.
4 hrsG. Fourny

Catalogue data

AbstractThe key challenge of the information society is to turn data into information, information into knowledge, knowledge into value. This has become increasingly complex. Data comes in larger volumes, diverse shapes, from different sources. Data is more heterogeneous and less structured than forty years ago. Nevertheless, it still needs to be processed fast, with support for complex operations.
ObjectiveDo you want to be able to query your own data productively and efficiently in your future semester projects, master thesis, or PhD thesis? Are you looking for something beyond the Python+Pandas hype? This courses teaches you how to do so as well as the dos and don'ts.

"Big Data" refers to the case when the amount of data is very large (100 GB and more), or when the data is not completely structured (or messy). The Big Data revolution has led to a completely new way to do business, e.g., develop new products and business models, but also to do science -- which is sometimes referred to as data-driven science or the "fourth paradigm".

Unfortunately, the quantity of data produced and available -- now in the Zettabyte range (that's 21 zeros) per year -- keeps growing faster than our ability to process it. Hence, new architectures and approaches for processing it are needed. Harnessing them must involve a deep understanding of data not only in the large, but also in the small.

The field of databases evolves at a fast pace. In order to be prepared, to the extent possible, to the (r)evolutions that will take place in the next few decades, the emphasis of the lecture will be on the paradigms and core design ideas, while today's technologies will serve as supporting illustrations thereof.

After visiting this lecture, you should have gained an overview and understanding of the Big Data landscape, which is the basis on which one can make informed decisions, i.e., pick and orchestrate the relevant technologies together for addressing each one of your projects efficiently and consistently.
ContentThis course gives an overview of database technologies and of the most important database design principles that lay the foundations of the Big Data universe. We take the monolithic, one-machine relational stack from the 1970s, smash it down and rebuild it on top of large clusters: starting with distributed storage, and all the way up to syntax, models, validation, processing, indexing, and querying. A broad range of aspects is covered with a focus on how they fit all together in the big picture of the Big Data ecosystem.

No data is harmed during this course, however, please be psychologically prepared that our data may not always be in third normal form.

- physical storage: distributed file systems (HDFS), object storage(S3), key-value stores

- logical storage: document stores (MongoDB), column stores (HBase), graph databases (neo4j), data warehouses (ROLAP)

- data formats and syntaxes (XML, JSON, RDF, Turtle, CSV, XBRL, YAML, protocol buffers, Avro)

- data shapes and models (tables, trees, graphs, cubes)

- type systems and schemas: atomic types, structured types (arrays, maps), set-based type systems (?, *, +)

- an overview of functional, declarative programming languages across data shapes (SQL, XQuery, JSONiq, Cypher, MDX)

- the most important query paradigms (selection, projection, joining, grouping, ordering, windowing)

- paradigms for parallel processing, two-stage (MapReduce) and DAG-based (Spark)

- resource management (YARN)

- what a data center is made of and why it matters (racks, nodes, ...)

- underlying architectures (internal machinery of HDFS, HBase, Spark, neo4j)

- optimization techniques (functional and declarative paradigms, query plans, rewrites, indexing)

- applications.

Large scale analytics and machine learning are outside of the scope of this course.
LiteratureCourse textbook: https://ghislainfourny.github.io/big-data-textbook/

Papers from scientific conferences and journals. References will be given as part of the course material during the semester.
Prerequisites / NoticeThe lecture is hybrid, meaning you can attend with us in the lecture hall, or on Zoom, or watch the recordings on YouTube later. Exercise sessions are in presence.

This course, in the autumn semester, is only intended for:
- Computer Science students
- Data Science students
- CBB students with a Computer Science background

Mobility students in CS are also welcome and encouraged to attend. If you experience any issue while registering, please contact the study administration and you will be gladly added.

For students of all other departements interested in this fascinating topic: I would love to have you visit my lectures as well! So there is a series of two courses specially designed for you:
- "Information Systems for Engineers" (SQL, relational databases): this Fall
- "Big Data for Engineers" (similar to Big Data, but adapted for non Computer Scientists): Spring 2023
There is no hard dependency, so you can either them in any order, but it may be more enjoyable to start with Information Systems for Engineers.

Students who successfully completed Big Data for Engineers are not allowed to enrol in the course Big Data.
Fostered competenciesFostered competencies
Subject-specific CompetenciesConcepts and Theoriesassessed
Techniques and Technologiesassessed
Method-specific CompetenciesAnalytical Competenciesassessed
Decision-makingassessed
Media and Digital Technologiesnot assessed
Problem-solvingnot assessed
Social CompetenciesCommunicationnot assessed
Sensitivity to Diversitynot assessed
Negotiationnot assessed
Personal CompetenciesCreative Thinkingnot assessed
Critical Thinkingnot assessed
Integrity and Work Ethicsnot assessed

Performance assessment

Performance assessment information (valid until the course unit is held again)
Performance assessment as a semester course
ECTS credits10 credits
ExaminersG. Fourny
Typesession examination
Language of examinationEnglish
RepetitionThe performance assessment is only offered in the session after the course unit. Repetition only possible after re-enrolling.
Mode of examinationwritten 180 minutes
Additional information on mode of examinationWorking on the exercises is rewarded in the sense of ETH's continuous performance assessment with up to 0.25 bonus points. In principle, it is expected that students solve all exercises. In order to control this, every week, 2 to 3 batches of small assignments (25 in total) will be marked as "bonus point eligible". Students can hand in their solution to these special assignments every week, and they will be graded (pass/fail). Each one of the 25 assignment batches gives 0.01 extra point at the exam, which means that up to 0.25 can be obtained as a bonus. Note that these extra points are added before rounding/truncating to a grade with a quarter-point precision and thus may or may not affect the final grade.

It is not allowed to register for both the Big Data and the Big Data for Engineers exams, as they mostly contain the same material targeted at difference audiences.
Written aidsGeneral dictionaries are allowed. This includes general English dictionaries (with word definitions) as well as general bilingual dictionaries (English <-> other language). Specialized dictionaries are not allowed. Dictionaries cannot be annotated by hand.
Online examinationThe examination may take place on the computer.
Distance examinationIt is not possible to take a distance examination.
This information can be updated until the beginning of the semester; information on the examination timetable is binding.

Learning materials

 
Main linkInformation
RecordingAll lectures are recorded and made available on YouTube
LiteratureBig Data: textbook (in progress!)
Only public learning materials are listed.

Groups

263-3010-00 UBig Data
GroupsG-01
Wed14:15-16:00HG E 33.1 »
G-02
Wed14:15-16:00HG G 26.1 »
G-03
Wed14:15-16:00CAB G 52 »
G-04
Fri14:15-16:00CAB G 52 »
G-05
Fri14:15-16:00CHN D 46 »
G-06
Wed14:15-16:00CAB G 59 »

Restrictions

PriorityRegistration for the course unit is only possible for the primary target group
Primary target groupCyber Security MSc (260000)
Cyber Security MSc (EPFL) (260100)
Data Science MSc (261000)
Computational Biology and Bioinformatics MSc (262200)
Computational Biology and Bioinformatics MSc (262200)
Computer Science MSc (263000)
Doctorate Computer Science (264002)
DAS ETH in Data Science (266000)
CAS ETH in Computer Science (269000)
Computer Science (Mobility) (274000)

Offered in

ProgrammeSectionType
CAS in Computer ScienceFocus Courses and ElectivesWInformation
Computational Biology and Bioinformatics MasterTheoryWInformation
Cyber Security MasterCore CoursesWInformation
DAS in Data ScienceBig Data SystemsWInformation
Data Science MasterData ManagementWInformation
Doctorate Computer ScienceSubject SpecialisationWInformation
Computer Science MasterCore CoursesWInformation
Computer Science MasterMinor in Data ManagementWInformation