GP-GPU and High Performances Computing

Course Description

In this course, we will introduce parallel programming paradigms to the students in the context of applied mathematics. The students will learn to identify the parallel pattern in numerical algorithm. The key components that the course will focus on are : efficiency, scalability, parallel pattern, comparison of parallel algorithms, operational intensity and emerging programming paradigm. Trough different lab assignments, the students will apply the concepts of efficient parallel programming using distributed, shared memory programming and accelerator languages (OpenMP, MPI, CUDA, OpenCL). In the final project, the students will have the possibility to parallelize one of their own numerical application developed in a previous course.

Syllabus

• Introduction to parallelism
• Introduction to general context of parallelism
• Models of parallel programming
• Description of various model of parallelism
• Paradigm of parallelism
• Templates of parallelism
• Parallel architectures
• Programming tools: OpenMP, OpenMPI

Organisation

Every Monday 8:15am - 9:45pm.

Every Friday 8:15am - 9:45am.

In class session will alternate between general lectures and practical sessions.

If required, some Office Hours can be organized.

Prerequisites

Basic knowledge in computer science to write and understand a non-trivial program, preferably in C or C++.

Knowledge in linear algebra and numerical methods is necessary to understand some examples.

Grading

Course grades will be based on a major open ended term project.

In the term project, you will investigate some aspect of high performance computing in the context of applied mathematics or apply high performance computing techniques to a problem in applied mathematics that interests you.

How you should approach this class

The majority of the students have some knowledge in scientific computing, but by no means everyone, and you all bring a variety of backgrounds. Students new to the subject are introduced to a set of tools and ideas that are used by many engineers and scientists in an astonishing variety of areas.

Students who have seen some set of parallel of techniques in other courses benefit from a course that puts together some formalism in this approaches.

Students also benefit by seeing a greater variety of applications at a deeper level. Keep in mind the different backgrounds and interests of your fellow students, keep an open mind to the variety of the material, and let’s all cut each other some slack.

Lectures

• Lecture 1: Introduction, Parallelism context
• Lecture 2: GPU architecture
• Lecture 3: Programming on GP-GPU (1)
• Lecture 4: Programming on GP-GPU (2)
• Lecture 5: Parallel patterns (1)
• Lecture 6: Parallel patterns (2)

• Lecture 8: Parallel design
• Lecture 10: Parallel pattern: sparse matrix

• Lecture 9: Parallel performances

Exam Sample

• 2021-2022 : Exam Training - Exam Correction

Example of codes

• Reduction
• Bitonic Sort
• Radix Sort
• Transposition

Practical sessions

• To wake up a computer , you may use the command

mailx -s 1 -r firsname.lastname@grenoble-inp.fr ensipc-wake@ensimag.fr << FIN
ensipcXXX
FIN

• Lab 1

  • Introduction to Cuda
  • Elements of code

• Lab 2

  • Matrix multiplication with CUDA (Updated)
  • Elements of code

• Tutorial 1: Radix sort

• Tutorial 2: Hierarchal Clustering

About the project

• Choosing a project
• Writing a report

Project examples

• Swarming
• Stencil method for the finite differences
• Parallelism of numerical methods : dense or sparse matrix solvers, Wavelet, Fast Fourier Transform, Multigrid
• Model of cardiac excitation
• Molecular dynamics
• Clustering algorithm
• Meshing
• k-maximum subarray problem

Reading materials

• Programming A64FX : video
• Programming A64FX : slides
• OpenMP Tutorial
• MPI Tutorial
• TOP 500