Departemen Ilmu Komputer IPB Komputasi Paralel Kuliah 01: Pendahuluan Yeni Herdiyeni http://www.cs.ipb.ac.id/~yeni/paralel Departemen Ilmu Komputer IPB Semester Genap 2010

Deskripsi Membahas kebutuhan dan klasifikasi mesin paralel (SISD, SIMD, MISD, MIMD, SPMD), komunikasi antar prosesor, memori persekutuan (shared memory), pengiriman pesan (message passing), jaringan interkoneksi (interconnection network), Desain algoritma paralel, efisiensi dan percepatan pemrosesan paralel, dan contoh aplikasi pemprosesan paralel. Perangkat lunak yang digunakan : MPI (Message Passing Interface)

Pengajar Dr. Yeni Herdiyeni, S.Si, M.Komp Hendra Rahmawan, S.Si, M.T Endang Purnama, S.Si, M.Komp Komponen Penilaian UTS UAS Tugas Quiz Project

Materi Kuliah Pendahuluan Definisi dan motivasi pemrosesan paralel Arsitektur system, Shared memory multiprocessor system, Message passing multicomputer distributed, Shared memory dan klasifikasi memori persekutuan MIMD dan SIMD Topologi Network Paradigma pengiriman pesan dengan menggunakan MPI Prinsip-prinsip Desain Algoritme Paralel Analisis kinerja Pemrosesan paralel UTS

Materi Kuliah #2 Pemrograman Paralel : Distributed Memory Tinjauan ulang critical section dengan menggunakan Pthread, siknronisasi dengan Semaphore, Implementasi Semaphore dilingkungan MPI Sorting Dense Matrix Algorithm Aplikasi pemrosesan (shared memory): problema produsen-konsumen, problema writer reader, problema dining philosophy Presentasi/diskusi proyek UAS Materi Kuliah dapat diakses di http://www.cs.ipb.ac.id/kulon

Buku Ajar Grama, Ananth., Gupta, Anshul., Karypis, George., Kumar, Vipin. 2003. Introduction to Parallel Computing. Second Edition. Pearson Addision Wesley. Quinn, Michael J. 2003. Parallel Programming in C with MPI and OpenMP . International Edition, McGraw-Hill. Wilkinson, Barry & Allen, Michael. 2005. Parallel Programming . 2nd Edition,Pearson Educational International. Jordan, Harry F., Alaghband Gita. 2003. Fundamentals of Parallel Processing. Prentice Hall.

Motivation : Classical Science Nature Observation Physical Experimentation Theory

Modern Scientific Method Nature Observation Numerical Simulation Physical Experimentation Theory

Modern Parallel Architectures Two basic architectural scheme: Distributed Memory Shared Memory Now most computers have a mixed architecture

What is Parallel and Distributed computing? Solving a single problem faster using multiple CPUs Parallel = Shared Memory among all CPUs Distributed = Local Memory/CPU Common Issues: Partition, Synchronization, Dependencies

Distributed Memory NETWORK CPU CPU CPU CPU CPU CPU memory memory node

Shared Memory memory CPU CPU CPU CPU CPU

Seeking Concurrency Data dependence graphs Data parallelism Functional parallelism Task Parallelism Pipelining

Interconnection Networks Uses of interconnection networks Connect processors to shared memory Connect processors to each other Interconnection media types Shared medium Switched medium

Most Common Networks switched switch Cube, hypercube, n-cube Torus in 1,2,...,N Dim Fat Tree

Real Shared Memory banks System Bus CPU CPU CPU CPU CPU

Virtual Shared Network CPU CPU CPU CPU CPU CPU node node node node memory memory memory memory memory memory HUB HUB HUB HUB HUB HUB CPU CPU CPU CPU CPU CPU node node node node node node

Mixed Architectures NETWORK CPU CPU CPU CPU CPU CPU memory memory node CPU CPU CPU CPU node node NETWORK

General MPI Program Structure MPI include file #include <mpi.h> void main (int argc, char *argv[]) { int np, rank, ierr; ierr = MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD,&rank); MPI_Comm_size(MPI_COMM_WORLD,&np); /* Do Some Works */ ierr = MPI_Finalize(); } variable declarations #include <mpi.h> void main (int argc, char *argv[]) { int np, rank, ierr; ierr = MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD,&rank); MPI_Comm_size(MPI_COMM_WORLD,&np); /* Do Some Works */ ierr = MPI_Finalize(); } Terminate MPI Environment #include <mpi.h> void main (int argc, char *argv[]) { int np, rank, ierr; ierr = MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD,&rank); MPI_Comm_size(MPI_COMM_WORLD,&np); /* Do Some Works */ ierr = MPI_Finalize(); } Initialize MPI environment #include <mpi.h> void main (int argc, char *argv[]) { int np, rank, ierr; ierr = MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD,&rank); MPI_Comm_size(MPI_COMM_WORLD,&np); /* Do Some Works */ ierr = MPI_Finalize(); } Do work and make message passing calls #include <mpi.h> void main (int argc, char *argv[]) { int np, rank, ierr; ierr = MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD,&rank); MPI_Comm_size(MPI_COMM_WORLD,&np); /* Do Some Works */ ierr = MPI_Finalize(); }

The Message-Passing Programming Paradigm

Message

Foster’s Design Methodology Partitioning Communication Agglomeration Mapping

Foster’s Methodology

Example program (1) Calculating the value of  by:

=3.141... Start calculation! OK! OK! Calculated by process 0

Sequential Algorithm 19 1 3 4 2 3 1 2 1 4 3 =  14 1 3 4 2 4 1 3 1 9 2 4 3 =  14 1 3 4 2 4 1 3 1 9 2 1 4 3 2 1 11 1 3 4 2 1 3 5 5 4 9 4 1 3 9 2 3 13 3 13 1 4 14 1 17 1 4 11 2 4 19 2 1 11 1

Phases of Parallel Algorithm b Row i of A b ci Inner product computation Row i of A Row i of A b c All-gather communication

Contoh 4x0 +6x1 +2x2 – 2x3 = 8 2x0 +5x2 4 –4x0 – 3x1 – 5x2 +4x3 1 8x0 40

Partitioning P0 P1

Communication P0 P1

Communication (CONT..) P0 P1

Communication (cont..) P0 P1

Communication (cont..) P0 P1

Shared memory multiprocessor using a single bus

Process “A program in a run”, a program in the memory. A high level view of a UNIX process

Threads A stream of control in a process. A high level view of threads in a UNIX process

Parallel Bubble Sort Iteration could start before previous iteration finished if does not overtake previous bubbling action: Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, @ 2004 Pearson Education Inc. All rights reserved.

Virtual Topology