Heterogeneous Parallel Programming

Brought by: Coursera

Overview

All computing systems, from mobile to supercomputers, are becoming heterogeneous, massively parallel computers for higher power efficiency and computation throughput. While the computing community is racing to build tools and libraries to ease the use of these systems, effective and confident use of these systems will always require knowledge about low-level programming in these systems. This course is designed for students to learn the essence of low-level programming interfaces and how to use these interfaces to achieve application goals. CUDA C, with its good balance between user control and verboseness, will serve as the teaching vehicle for the first half of the course. Students will then extend their learning into closely related programming interfaces such as OpenCL, OpenACC, and C++AMP.


The course is unique in that it is application oriented and only introduces the necessary underlying computer science and computer engineering knowledge for understanding. It covers the concept of data parallel execution models, memory models for managing locality, tiling techniques for reducing bandwidth consumption, parallel algorithm patterns, overlapping computation with communication, and a variety of heterogeneous parallel programming interfaces. The concepts learned in this course form a strong foundation for learning other types of parallel programming systems.


Syllabus

  • Week One: Introduction to Heterogeneous Computing, Overview of CUDA C, and Kernel-Based Parallel Programming, with lab tour and programming assignment of vector addition in CUDA C.
  •  Week Two: Memory Model for Locality, Tiling for Conserving Memory Bandwidth, Handling Boundary Conditions, and Performance Considerations, with programming assignment of simple matrix-matrix multiplication in CUDA C.
  • Week Three: Parallel Convolution Pattern, with programming assignment of tiled matrix-matrix multiplication in CUDA C.
  • Week Four: Parallel Scan Pattern, with programming assignment of parallel convolution in CUDA C.
  • Week Five: Parallel Histogram Pattern and Atomic Operations, with programming assignment of parallel scan in CUDA C.
  • Week Six: Data Transfer and Task Parallelism, with programming assignment of parallel histogram in CUDA C.
  • Week Seven: Introduction to OpenCL, Introduction to C++AMP, Introduction to OpenACC, with programming assignment of vector addition using streams in CUDA C.
  • Week Eight: Course Summary, Other Related Programming Models –Thrust, Bolt, and CUDA FORTRAN, with programming assignment of simple matrix-matrix multiplication in choice of OpenCL, C++AMP, or OpenACC.
  • Week Nine: complete any remaining lab assignments, with optional, bonus programming assignments in choice of OpenCL, C++AMP, or OpenACC.

Taught by

Wen-mei W. Hwu

Heterogeneous Parallel Programming
Go to course

Heterogeneous Parallel Programming

Brought by: Coursera

  • Coursera
  • Free
  • English
  • Certificate Not Available
  • Certain days
  • All
  • N/A
8.1.2PHP Version316msRequest Duration2MBMemory UsageGET en/courses/{slug}Route
    • Booting (205ms)
    • Application (111ms)
    • 1 x Booting (64.74%)
      204.72ms
      1 x Application (35.03%)
      110.77ms
      14 templates were rendered
      • public.courses.show (resources/views/public/courses/show.blade.php)3bladefile
        Params
        0
        course
        1
        links
        2
        config
      • public.courses.partials.breadcrumbs (resources/views/public/courses/partials/breadcrumbs.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.heading (resources/views/public/courses/partials/heading.blade.php)7bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        classes
      • public.courses.partials.details (resources/views/public/courses/partials/details.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.breadcrumbs (resources/views/public/courses/partials/breadcrumbs.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.heading (resources/views/public/courses/partials/heading.blade.php)7bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        classes
      • public.layouts.main (resources/views/public/layouts/main.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.layouts.partials.meta (resources/views/public/layouts/partials/meta.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.layouts.partials.navbar (resources/views/public/layouts/partials/navbar.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.auth.profile.partials.links (resources/views/public/auth/profile/partials/links.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.layouts.partials.flash-session (resources/views/public/layouts/partials/flash-session.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      uri
      GET en/courses/{slug}
      middleware
      web, localize:en
      controller
      App\Http\Controllers\CourseController@show
      as
      en.courses.show
      namespace
      prefix
      /en
      where
      file
      app/Http/Controllers/CourseController.php:17-35
      7 statements were executed6.57ms
      • select * from `courses` where `slug_en` = 'heterogeneous-parallel-programming' limit 1
        4.66ms/app/Http/Controllers/CourseController.php:20corspedia
        Metadata
        Bindings
        • 0. heterogeneous-parallel-programming
        Backtrace
        • 17. /app/Http/Controllers/CourseController.php:20
        • 18. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 19. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 20. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • update `courses` set `visitors` = `visitors` + 1, `courses`.`updated_at` = '2025-02-03 18:18:32' where `id` = 1719
        310μs/app/Http/Controllers/CourseController.php:21corspedia
        Metadata
        Bindings
        • 0. 2025-02-03 18:18:32
        • 1. 1719
        Backtrace
        • 17. /app/Http/Controllers/CourseController.php:21
        • 18. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 19. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 20. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select `id`, `name_en`, `name_ar`, `topic_id`, `slug_en`, `slug_ar` from `subjects` where `subjects`.`id` in (6)
        250μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 20. /app/Http/Controllers/CourseController.php:23
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 22. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 23. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 24. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select `id`, `name_en`, `name_ar`, `slug_en`, `slug_ar` from `topics` where `topics`.`id` in (1)
        210μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 25. /app/Http/Controllers/CourseController.php:23
        • 26. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 27. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 28. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 29. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select * from `institutions` where `institutions`.`id` in (15) and `institutions`.`deleted_at` is null
        240μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 20. /app/Http/Controllers/CourseController.php:23
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 22. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 23. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 24. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select * from `providers` where `providers`.`id` in (2) and `providers`.`deleted_at` is null
        250μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 20. /app/Http/Controllers/CourseController.php:23
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 22. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 23. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 24. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select * from `html_files` where `html_files`.`id` = 1710 limit 1
        650μs/app/Models/Course.php:84corspedia
        Metadata
        Bindings
        • 0. 1710
        Backtrace
        • 21. /app/Models/Course.php:84
        • 28. view::public.courses.show:29
        • 30. /vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php:125
        • 31. /vendor/laravel/framework/src/Illuminate/View/Engines/PhpEngine.php:58
        • 32. /vendor/laravel/framework/src/Illuminate/View/Engines/CompilerEngine.php:72
      App\Models\HtmlFile
      1
      App\Models\Provider
      1
      App\Models\Institution
      1
      App\Models\Topic
      1
      App\Models\Subject
      1
      App\Models\Course
      1
        _token
        pCY6LG1fj2KU6fCcCfBODZB82WfRFANSDsQOXG5r
        locale
        en
        _previous
        array:1 [ "url" => "https://www.corspedia.com/en/courses/heterogeneous-parallel-programming" ]
        _flash
        array:2 [ "old" => [] "new" => [] ]
        PHPDEBUGBAR_STACK_DATA
        []
        path_info
        /en/courses/heterogeneous-parallel-programming
        status_code
        200
        
        status_text
        OK
        format
        html
        content_type
        text/html; charset=UTF-8
        request_query
        []
        
        request_request
        []
        
        request_headers
        0 of 0
        array:24 [ "sec-ch-ua-mobile" => array:1 [ 0 => "?0" ] "sec-ch-ua" => array:1 [ 0 => ""HeadlessChrome";v="129", "Not=A?Brand";v="8", "Chromium";v="129"" ] "cache-control" => array:1 [ 0 => "no-cache" ] "pragma" => array:1 [ 0 => "no-cache" ] "cdn-loop" => array:1 [ 0 => "cloudflare; loops=1" ] "priority" => array:1 [ 0 => "u=0, i" ] "upgrade-insecure-requests" => array:1 [ 0 => "1" ] "user-agent" => array:1 [ 0 => "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" ] "cf-connecting-ip" => array:1 [ 0 => "3.22.120.179" ] "accept" => array:1 [ 0 => "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" ] "sec-fetch-site" => array:1 [ 0 => "none" ] "cf-visitor" => array:1 [ 0 => "{"scheme":"https"}" ] "sec-fetch-mode" => array:1 [ 0 => "navigate" ] "sec-fetch-user" => array:1 [ 0 => "?1" ] "x-forwarded-proto" => array:1 [ 0 => "https" ] "cf-ipcountry" => array:1 [ 0 => "US" ] "accept-encoding" => array:1 [ 0 => "gzip, br" ] "sec-fetch-dest" => array:1 [ 0 => "document" ] "sec-ch-ua-platform" => array:1 [ 0 => ""Windows"" ] "x-forwarded-for" => array:1 [ 0 => "3.22.120.179" ] "cf-ray" => array:1 [ 0 => "90c46c8c1f7361f0-ORD" ] "host" => array:1 [ 0 => "www.corspedia.com" ] "content-length" => array:1 [ 0 => "" ] "content-type" => array:1 [ 0 => "" ] ]
        request_server
        0 of 0
        array:50 [ "USER" => "www-data" "HOME" => "/var/www" "HTTP_SEC_CH_UA_MOBILE" => "?0" "HTTP_SEC_CH_UA" => ""HeadlessChrome";v="129", "Not=A?Brand";v="8", "Chromium";v="129"" "HTTP_CACHE_CONTROL" => "no-cache" "HTTP_PRAGMA" => "no-cache" "HTTP_CDN_LOOP" => "cloudflare; loops=1" "HTTP_PRIORITY" => "u=0, i" "HTTP_UPGRADE_INSECURE_REQUESTS" => "1" "HTTP_USER_AGENT" => "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" "HTTP_CF_CONNECTING_IP" => "3.22.120.179" "HTTP_ACCEPT" => "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" "HTTP_SEC_FETCH_SITE" => "none" "HTTP_CF_VISITOR" => "{"scheme":"https"}" "HTTP_SEC_FETCH_MODE" => "navigate" "HTTP_SEC_FETCH_USER" => "?1" "HTTP_X_FORWARDED_PROTO" => "https" "HTTP_CF_IPCOUNTRY" => "US" "HTTP_ACCEPT_ENCODING" => "gzip, br" "HTTP_SEC_FETCH_DEST" => "document" "HTTP_SEC_CH_UA_PLATFORM" => ""Windows"" "HTTP_X_FORWARDED_FOR" => "3.22.120.179" "HTTP_CF_RAY" => "90c46c8c1f7361f0-ORD" "HTTP_HOST" => "www.corspedia.com" "REDIRECT_STATUS" => "200" "SERVER_NAME" => "corspedia.com" "SERVER_PORT" => "443" "SERVER_ADDR" => "141.95.147.152" "REMOTE_USER" => "" "REMOTE_PORT" => "42534" "REMOTE_ADDR" => "172.69.6.157" "SERVER_SOFTWARE" => "nginx/1.18.0" "GATEWAY_INTERFACE" => "CGI/1.1" "HTTPS" => "on" "REQUEST_SCHEME" => "https" "SERVER_PROTOCOL" => "HTTP/2.0" "DOCUMENT_ROOT" => "/var/www/corspedia/public" "DOCUMENT_URI" => "/index.php" "REQUEST_URI" => "/en/courses/heterogeneous-parallel-programming" "SCRIPT_NAME" => "/index.php" "CONTENT_LENGTH" => "" "CONTENT_TYPE" => "" "REQUEST_METHOD" => "GET" "QUERY_STRING" => "" "SCRIPT_FILENAME" => "/var/www/corspedia/public/index.php" "PATH_INFO" => "" "FCGI_ROLE" => "RESPONDER" "PHP_SELF" => "/index.php" "REQUEST_TIME_FLOAT" => 1738606711.964 "REQUEST_TIME" => 1738606711 ]
        request_cookies
        []
        
        response_headers
        0 of 0
        array:5 [ "content-type" => array:1 [ 0 => "text/html; charset=UTF-8" ] "cache-control" => array:1 [ 0 => "no-cache, private" ] "date" => array:1 [ 0 => "Mon, 03 Feb 2025 18:18:32 GMT" ] "set-cookie" => array:2 [ 0 => "XSRF-TOKEN=eyJpdiI6IlJ6Q3FkMG1LVE41bE1FaWpabmNCQVE9PSIsInZhbHVlIjoidUNsMHVPck1neWhBZ2N5c1IvQ0w4NjRMbDhnVzZMWGJLcjcwZGZTUDRNYURyRDRwTjhOWno0SzBFNThFV2pnV3dLQnBkUHk3ZDZxai9yZU5NcDBEeGFnVXovR2RwMVg5WXgzK2ZMNmszVTRTdVE0UktZNHlhSUVKWHI4RHVKZXMiLCJtYWMiOiJlMjNkZjFmNzkxM2FhY2RiNzA0NmU3NmZkMjkyOWQ4NTI4Nzk2YTczZGExNjkwMDFiYWVhZTc2MTQxMDFlNjBiIiwidGFnIjoiIn0%3D; expires=Mon, 03 Feb 2025 20:18:32 GMT; Max-Age=7200; path=/; samesite=laxXSRF-TOKEN=eyJpdiI6IlJ6Q3FkMG1LVE41bE1FaWpabmNCQVE9PSIsInZhbHVlIjoidUNsMHVPck1neWhBZ2N5c1IvQ0w4NjRMbDhnVzZMWGJLcjcwZGZTUDRNYURyRDRwTjhOWno0SzBFNThFV2pnV3dLQnBkU" 1 => "laravel_session=eyJpdiI6Ik83OUlpamI4RjBzT2tXUEpnZjZLd0E9PSIsInZhbHVlIjoicHZnbjh2S3MrcmdxbkVSZnZWa2NhVmhLaDE4Vi85ZFpQQ1Y3RGxMbEVkQkZPeUNSTE1BQzlGN0IyZ094WXNxcUVaM2R2U0FoYVNPKy9CUzB4YUhBYVhDSGlkTGl4bHRJWTF1d3lwM2w3TWpIbzRndU80YVN3dnNDTFZIVExQRUUiLCJtYWMiOiI1YTFlYTJlZDI1Njg2OGVhNDk3MDAxOWEwZjA0OGUxZmViY2E5M2NiNmE1OGIxNzJlMGQ4NjY0YTY1ODM0M2JlIiwidGFnIjoiIn0%3D; expires=Mon, 03 Feb 2025 20:18:32 GMT; Max-Age=7200; path=/; httponly; samesite=laxlaravel_session=eyJpdiI6Ik83OUlpamI4RjBzT2tXUEpnZjZLd0E9PSIsInZhbHVlIjoicHZnbjh2S3MrcmdxbkVSZnZWa2NhVmhLaDE4Vi85ZFpQQ1Y3RGxMbEVkQkZPeUNSTE1BQzlGN0IyZ094WXNxcUVa" ] "Set-Cookie" => array:2 [ 0 => "XSRF-TOKEN=eyJpdiI6IlJ6Q3FkMG1LVE41bE1FaWpabmNCQVE9PSIsInZhbHVlIjoidUNsMHVPck1neWhBZ2N5c1IvQ0w4NjRMbDhnVzZMWGJLcjcwZGZTUDRNYURyRDRwTjhOWno0SzBFNThFV2pnV3dLQnBkUHk3ZDZxai9yZU5NcDBEeGFnVXovR2RwMVg5WXgzK2ZMNmszVTRTdVE0UktZNHlhSUVKWHI4RHVKZXMiLCJtYWMiOiJlMjNkZjFmNzkxM2FhY2RiNzA0NmU3NmZkMjkyOWQ4NTI4Nzk2YTczZGExNjkwMDFiYWVhZTc2MTQxMDFlNjBiIiwidGFnIjoiIn0%3D; expires=Mon, 03-Feb-2025 20:18:32 GMT; path=/XSRF-TOKEN=eyJpdiI6IlJ6Q3FkMG1LVE41bE1FaWpabmNCQVE9PSIsInZhbHVlIjoidUNsMHVPck1neWhBZ2N5c1IvQ0w4NjRMbDhnVzZMWGJLcjcwZGZTUDRNYURyRDRwTjhOWno0SzBFNThFV2pnV3dLQnBkU" 1 => "laravel_session=eyJpdiI6Ik83OUlpamI4RjBzT2tXUEpnZjZLd0E9PSIsInZhbHVlIjoicHZnbjh2S3MrcmdxbkVSZnZWa2NhVmhLaDE4Vi85ZFpQQ1Y3RGxMbEVkQkZPeUNSTE1BQzlGN0IyZ094WXNxcUVaM2R2U0FoYVNPKy9CUzB4YUhBYVhDSGlkTGl4bHRJWTF1d3lwM2w3TWpIbzRndU80YVN3dnNDTFZIVExQRUUiLCJtYWMiOiI1YTFlYTJlZDI1Njg2OGVhNDk3MDAxOWEwZjA0OGUxZmViY2E5M2NiNmE1OGIxNzJlMGQ4NjY0YTY1ODM0M2JlIiwidGFnIjoiIn0%3D; expires=Mon, 03-Feb-2025 20:18:32 GMT; path=/; httponlylaravel_session=eyJpdiI6Ik83OUlpamI4RjBzT2tXUEpnZjZLd0E9PSIsInZhbHVlIjoicHZnbjh2S3MrcmdxbkVSZnZWa2NhVmhLaDE4Vi85ZFpQQ1Y3RGxMbEVkQkZPeUNSTE1BQzlGN0IyZ094WXNxcUVa" ] ]
        session_attributes
        0 of 0
        array:5 [ "_token" => "pCY6LG1fj2KU6fCcCfBODZB82WfRFANSDsQOXG5r" "locale" => "en" "_previous" => array:1 [ "url" => "https://www.corspedia.com/en/courses/heterogeneous-parallel-programming" ] "_flash" => array:2 [ "old" => [] "new" => [] ] "PHPDEBUGBAR_STACK_DATA" => [] ]