Optimize ML Models and Deploy Human-in-the-Loop Pipelines

Brought by: Coursera

Overview

In the third course of the Practical Data Science Specialization, you will learn a series of performance-improvement and cost-reduction techniques to automatically tune model accuracy, compare prediction performance, and generate new training data with human intelligence. After tuning your text classifier using Amazon SageMaker Hyper-parameter Tuning (HPT), you will deploy two model candidates into an A/B test to compare their real-time prediction performance and automatically scale the winning model using Amazon SageMaker Hosting. Lastly, you will set up a human-in-the-loop pipeline to fix misclassified predictions and generate new training data using Amazon Augmented AI and Amazon SageMaker Ground Truth.

Practical data science is geared towards handling massive datasets that do not fit in your local hardware and could originate from multiple sources. One of the biggest benefits of developing and running data science projects in the cloud is the agility and elasticity that the cloud offers to scale up and out at a minimum cost.

The Practical Data Science Specialization helps you develop the practical skills to effectively deploy your data science projects and overcome challenges at each step of the ML workflow using Amazon SageMaker. This Specialization is designed for data-focused developers, scientists, and analysts familiar with the Python and SQL programming languages and want to learn how to build, train, and deploy scalable, end-to-end ML pipelines - both automated and human-in-the-loop - in the AWS cloud.

Syllabus

  • Week 1: Advanced model training, tuning and evaluation
    • Train, tune, and evaluate models using data-parallel and model-parallel strategies and automatic model tuning.
  • Week 2: Advanced model deployment and monitoring
    • Deploy models with A/B testing, monitor model performance, and detect drift from baseline metrics.
  • Week 3: Data labeling and human-in-the-loop pipelines
    • Label data at scale using private human workforces and build human-in-the-loop pipelines.

Taught by

Antje Barth, Shelbee Eigenbrode, Sireesha Muppala and Chris Fregly

Optimize ML Models and Deploy Human-in-the-Loop Pipelines
Go to course

Optimize ML Models and Deploy Human-in-the-Loop Pipelines

Brought by: Coursera

  • Coursera
  • Free
  • English
  • Certificate Available
  • Available at any time
  • All
  • English, Arabic, French, Portuguese, German, Thai, Indonesian, Spanish, Kazakh, Hindi, Russian, Swedish, Korean, Greek, Italian, Chinese, Ukrainian, Japanese, Polish, Dutch, Turkish
8.1.2PHP Version228msRequest Duration2MBMemory UsageGET en/courses/{slug}Route
    • Booting (138ms)
    • Application (89.11ms)
    • 1 x Booting (60.65%)
      138.19ms
      1 x Application (39.1%)
      89.11ms
      14 templates were rendered
      • public.courses.show (resources/views/public/courses/show.blade.php)3bladefile
        Params
        0
        course
        1
        links
        2
        config
      • public.courses.partials.breadcrumbs (resources/views/public/courses/partials/breadcrumbs.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.heading (resources/views/public/courses/partials/heading.blade.php)7bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        classes
      • public.courses.partials.details (resources/views/public/courses/partials/details.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.breadcrumbs (resources/views/public/courses/partials/breadcrumbs.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.heading (resources/views/public/courses/partials/heading.blade.php)7bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        classes
      • public.layouts.main (resources/views/public/layouts/main.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.layouts.partials.meta (resources/views/public/layouts/partials/meta.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.layouts.partials.navbar (resources/views/public/layouts/partials/navbar.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.auth.profile.partials.links (resources/views/public/auth/profile/partials/links.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.layouts.partials.flash-session (resources/views/public/layouts/partials/flash-session.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      uri
      GET en/courses/{slug}
      middleware
      web, localize:en
      controller
      App\Http\Controllers\CourseController@show
      as
      en.courses.show
      namespace
      prefix
      /en
      where
      file
      app/Http/Controllers/CourseController.php:17-35
      7 statements were executed4.26ms
      • select * from `courses` where `slug_en` = 'optimize-ml-models-and-deploy-human-in-the-loop-pipelines' limit 1
        2.65ms/app/Http/Controllers/CourseController.php:20corspedia
        Metadata
        Bindings
        • 0. optimize-ml-models-and-deploy-human-in-the-loop-pipelines
        Backtrace
        • 17. /app/Http/Controllers/CourseController.php:20
        • 18. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 19. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 20. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • update `courses` set `visitors` = `visitors` + 1, `courses`.`updated_at` = '2025-04-15 18:44:58' where `id` = 295
        520μs/app/Http/Controllers/CourseController.php:21corspedia
        Metadata
        Bindings
        • 0. 2025-04-15 18:44:58
        • 1. 295
        Backtrace
        • 17. /app/Http/Controllers/CourseController.php:21
        • 18. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 19. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 20. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select `id`, `name_en`, `name_ar`, `topic_id`, `slug_en`, `slug_ar` from `subjects` where `subjects`.`id` in (4)
        260μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 20. /app/Http/Controllers/CourseController.php:23
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 22. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 23. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 24. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select `id`, `name_en`, `name_ar`, `slug_en`, `slug_ar` from `topics` where `topics`.`id` in (1)
        230μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 25. /app/Http/Controllers/CourseController.php:23
        • 26. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 27. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 28. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 29. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select * from `institutions` where `institutions`.`id` in (34) and `institutions`.`deleted_at` is null
        230μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 20. /app/Http/Controllers/CourseController.php:23
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 22. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 23. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 24. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select * from `providers` where `providers`.`id` in (2) and `providers`.`deleted_at` is null
        170μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 20. /app/Http/Controllers/CourseController.php:23
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 22. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 23. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 24. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select * from `html_files` where `html_files`.`id` = 289 limit 1
        200μs/app/Models/Course.php:84corspedia
        Metadata
        Bindings
        • 0. 289
        Backtrace
        • 21. /app/Models/Course.php:84
        • 28. view::public.courses.show:29
        • 30. /vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php:125
        • 31. /vendor/laravel/framework/src/Illuminate/View/Engines/PhpEngine.php:58
        • 32. /vendor/laravel/framework/src/Illuminate/View/Engines/CompilerEngine.php:72
      App\Models\HtmlFile
      1
      App\Models\Provider
      1
      App\Models\Institution
      1
      App\Models\Topic
      1
      App\Models\Subject
      1
      App\Models\Course
      1
        _token
        FnFZcIuGpQkXYYZi7Wlha3LML5ygjsQAUsUuLoLZ
        locale
        en
        _previous
        array:1 [ "url" => "https://www.corspedia.com/en/courses/optimize-ml-models-and-deploy-human-in-th...
        _flash
        array:2 [ "old" => [] "new" => [] ]
        PHPDEBUGBAR_STACK_DATA
        []
        path_info
        /en/courses/optimize-ml-models-and-deploy-human-in-the-loop-pipelines
        status_code
        200
        
        status_text
        OK
        format
        html
        content_type
        text/html; charset=UTF-8
        request_query
        []
        
        request_request
        []
        
        request_headers
        0 of 0
        array:24 [ "cf-ipcountry" => array:1 [ 0 => "US" ] "cf-connecting-ip" => array:1 [ 0 => "3.133.113.237" ] "cdn-loop" => array:1 [ 0 => "cloudflare; loops=1" ] "x-forwarded-proto" => array:1 [ 0 => "https" ] "cf-visitor" => array:1 [ 0 => "{"scheme":"https"}" ] "sec-fetch-site" => array:1 [ 0 => "none" ] "accept" => array:1 [ 0 => "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" ] "user-agent" => array:1 [ 0 => "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" ] "upgrade-insecure-requests" => array:1 [ 0 => "1" ] "sec-ch-ua-platform" => array:1 [ 0 => ""Windows"" ] "sec-ch-ua-mobile" => array:1 [ 0 => "?0" ] "sec-ch-ua" => array:1 [ 0 => ""HeadlessChrome";v="129", "Not=A?Brand";v="8", "Chromium";v="129"" ] "cache-control" => array:1 [ 0 => "no-cache" ] "pragma" => array:1 [ 0 => "no-cache" ] "sec-fetch-dest" => array:1 [ 0 => "document" ] "cf-ray" => array:1 [ 0 => "930d98e7bcb2124b-ORD" ] "accept-encoding" => array:1 [ 0 => "gzip, br" ] "priority" => array:1 [ 0 => "u=0, i" ] "sec-fetch-user" => array:1 [ 0 => "?1" ] "sec-fetch-mode" => array:1 [ 0 => "navigate" ] "x-forwarded-for" => array:1 [ 0 => "3.133.113.237" ] "host" => array:1 [ 0 => "www.corspedia.com" ] "content-length" => array:1 [ 0 => "" ] "content-type" => array:1 [ 0 => "" ] ]
        request_server
        0 of 0
        array:50 [ "USER" => "www-data" "HOME" => "/var/www" "HTTP_CF_IPCOUNTRY" => "US" "HTTP_CF_CONNECTING_IP" => "3.133.113.237" "HTTP_CDN_LOOP" => "cloudflare; loops=1" "HTTP_X_FORWARDED_PROTO" => "https" "HTTP_CF_VISITOR" => "{"scheme":"https"}" "HTTP_SEC_FETCH_SITE" => "none" "HTTP_ACCEPT" => "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" "HTTP_USER_AGENT" => "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" "HTTP_UPGRADE_INSECURE_REQUESTS" => "1" "HTTP_SEC_CH_UA_PLATFORM" => ""Windows"" "HTTP_SEC_CH_UA_MOBILE" => "?0" "HTTP_SEC_CH_UA" => ""HeadlessChrome";v="129", "Not=A?Brand";v="8", "Chromium";v="129"" "HTTP_CACHE_CONTROL" => "no-cache" "HTTP_PRAGMA" => "no-cache" "HTTP_SEC_FETCH_DEST" => "document" "HTTP_CF_RAY" => "930d98e7bcb2124b-ORD" "HTTP_ACCEPT_ENCODING" => "gzip, br" "HTTP_PRIORITY" => "u=0, i" "HTTP_SEC_FETCH_USER" => "?1" "HTTP_SEC_FETCH_MODE" => "navigate" "HTTP_X_FORWARDED_FOR" => "3.133.113.237" "HTTP_HOST" => "www.corspedia.com" "REDIRECT_STATUS" => "200" "SERVER_NAME" => "corspedia.com" "SERVER_PORT" => "443" "SERVER_ADDR" => "141.95.147.152" "REMOTE_USER" => "" "REMOTE_PORT" => "43186" "REMOTE_ADDR" => "172.69.59.35" "SERVER_SOFTWARE" => "nginx/1.18.0" "GATEWAY_INTERFACE" => "CGI/1.1" "HTTPS" => "on" "REQUEST_SCHEME" => "https" "SERVER_PROTOCOL" => "HTTP/2.0" "DOCUMENT_ROOT" => "/var/www/corspedia/public" "DOCUMENT_URI" => "/index.php" "REQUEST_URI" => "/en/courses/optimize-ml-models-and-deploy-human-in-the-loop-pipelines" "SCRIPT_NAME" => "/index.php" "CONTENT_LENGTH" => "" "CONTENT_TYPE" => "" "REQUEST_METHOD" => "GET" "QUERY_STRING" => "" "SCRIPT_FILENAME" => "/var/www/corspedia/public/index.php" "PATH_INFO" => "" "FCGI_ROLE" => "RESPONDER" "PHP_SELF" => "/index.php" "REQUEST_TIME_FLOAT" => 1744742698.4529 "REQUEST_TIME" => 1744742698 ]
        request_cookies
        []
        
        response_headers
        0 of 0
        array:5 [ "content-type" => array:1 [ 0 => "text/html; charset=UTF-8" ] "cache-control" => array:1 [ 0 => "no-cache, private" ] "date" => array:1 [ 0 => "Tue, 15 Apr 2025 18:44:58 GMT" ] "set-cookie" => array:2 [ 0 => "XSRF-TOKEN=eyJpdiI6InJFaUdQYnowTm42UkpOU2wzbldvaGc9PSIsInZhbHVlIjoiYjVaR0N5aUFFcXR5eXgwaEk1enpNMU51dU8zN0w4eFpuUmhkKy9Hd3cvSjZZUFdySUJIUGRnUG92cXpqdFdobi9sUll4VU4rUnVWSjdXaW5ERFpZUTZJRE1xcGkxOGNqWXk0UlpzS0dNci9YeXJLUlZZYW9LNFpGZVAyREhSVEIiLCJtYWMiOiIzMjRkZWE2ZDg0NjQ5Y2NlY2JkNDI0ODIwZjk0MGMwMjRmNGY1NDUyNGQ1ZmQ1MGEyZDRmYzMyMTdlMjUyOTQ2IiwidGFnIjoiIn0%3D; expires=Tue, 15 Apr 2025 20:44:58 GMT; Max-Age=7200; path=/; samesite=laxXSRF-TOKEN=eyJpdiI6InJFaUdQYnowTm42UkpOU2wzbldvaGc9PSIsInZhbHVlIjoiYjVaR0N5aUFFcXR5eXgwaEk1enpNMU51dU8zN0w4eFpuUmhkKy9Hd3cvSjZZUFdySUJIUGRnUG92cXpqdFdobi9sUll4V" 1 => "laravel_session=eyJpdiI6IklKWERhQmU0SllSd1J4Nk0xMDdFRHc9PSIsInZhbHVlIjoiODJsZjRYaTZMUjRZalZGRnZkVHpGc2lKeWpycjQydnBmYVlUb1NPTHkycFU0Mi94ZmhTQmIweWVMZVFITW1zMGtWSit5NGlsMzRPaUU4ais3RllYdVo2OFU3QVh0SjdNK1pOby9SWHNndXZjTXc1d0RRbHJ6cG80MDdMcU8xTDEiLCJtYWMiOiI5YzY3MmQ0YWMzMTgyNmZiNGVmYzNlYTdjZmMwZDYzZGUzNTlkZDMwNzk0MmRjNzJhZDlmNTQwMWJmZGY1MGY3IiwidGFnIjoiIn0%3D; expires=Tue, 15 Apr 2025 20:44:58 GMT; Max-Age=7200; path=/; httponly; samesite=laxlaravel_session=eyJpdiI6IklKWERhQmU0SllSd1J4Nk0xMDdFRHc9PSIsInZhbHVlIjoiODJsZjRYaTZMUjRZalZGRnZkVHpGc2lKeWpycjQydnBmYVlUb1NPTHkycFU0Mi94ZmhTQmIweWVMZVFITW1zMGtW" ] "Set-Cookie" => array:2 [ 0 => "XSRF-TOKEN=eyJpdiI6InJFaUdQYnowTm42UkpOU2wzbldvaGc9PSIsInZhbHVlIjoiYjVaR0N5aUFFcXR5eXgwaEk1enpNMU51dU8zN0w4eFpuUmhkKy9Hd3cvSjZZUFdySUJIUGRnUG92cXpqdFdobi9sUll4VU4rUnVWSjdXaW5ERFpZUTZJRE1xcGkxOGNqWXk0UlpzS0dNci9YeXJLUlZZYW9LNFpGZVAyREhSVEIiLCJtYWMiOiIzMjRkZWE2ZDg0NjQ5Y2NlY2JkNDI0ODIwZjk0MGMwMjRmNGY1NDUyNGQ1ZmQ1MGEyZDRmYzMyMTdlMjUyOTQ2IiwidGFnIjoiIn0%3D; expires=Tue, 15-Apr-2025 20:44:58 GMT; path=/XSRF-TOKEN=eyJpdiI6InJFaUdQYnowTm42UkpOU2wzbldvaGc9PSIsInZhbHVlIjoiYjVaR0N5aUFFcXR5eXgwaEk1enpNMU51dU8zN0w4eFpuUmhkKy9Hd3cvSjZZUFdySUJIUGRnUG92cXpqdFdobi9sUll4V" 1 => "laravel_session=eyJpdiI6IklKWERhQmU0SllSd1J4Nk0xMDdFRHc9PSIsInZhbHVlIjoiODJsZjRYaTZMUjRZalZGRnZkVHpGc2lKeWpycjQydnBmYVlUb1NPTHkycFU0Mi94ZmhTQmIweWVMZVFITW1zMGtWSit5NGlsMzRPaUU4ais3RllYdVo2OFU3QVh0SjdNK1pOby9SWHNndXZjTXc1d0RRbHJ6cG80MDdMcU8xTDEiLCJtYWMiOiI5YzY3MmQ0YWMzMTgyNmZiNGVmYzNlYTdjZmMwZDYzZGUzNTlkZDMwNzk0MmRjNzJhZDlmNTQwMWJmZGY1MGY3IiwidGFnIjoiIn0%3D; expires=Tue, 15-Apr-2025 20:44:58 GMT; path=/; httponlylaravel_session=eyJpdiI6IklKWERhQmU0SllSd1J4Nk0xMDdFRHc9PSIsInZhbHVlIjoiODJsZjRYaTZMUjRZalZGRnZkVHpGc2lKeWpycjQydnBmYVlUb1NPTHkycFU0Mi94ZmhTQmIweWVMZVFITW1zMGtW" ] ]
        session_attributes
        0 of 0
        array:5 [ "_token" => "FnFZcIuGpQkXYYZi7Wlha3LML5ygjsQAUsUuLoLZ" "locale" => "en" "_previous" => array:1 [ "url" => "https://www.corspedia.com/en/courses/optimize-ml-models-and-deploy-human-in-the-loop-pipelines" ] "_flash" => array:2 [ "old" => [] "new" => [] ] "PHPDEBUGBAR_STACK_DATA" => [] ]