Cloud Computing Applications, Part 2: Big Data and Applications in the Cloud

Brought by: Coursera

Overview

Welcome to the Cloud Computing Applications course, the second part of a two-course series designed to give you a comprehensive view on the world of Cloud Computing and Big Data!

In this second course we continue Cloud Computing Applications by exploring how the Cloud opens up data analytics of huge volumes of data that are static or streamed at high velocity and represent an enormous variety of information. Cloud applications and data analytics represent a disruptive change in the ways that society is informed by, and uses information. We start the first week by introducing some major systems for data analysis including Spark and the major frameworks and distributions of analytics applications including Hortonworks, Cloudera, and MapR. By the middle of week one we introduce the HDFS distributed and robust file system that is used in many applications like Hadoop and finish week one by exploring the powerful MapReduce programming model and how distributed operating systems like YARN and Mesos support a flexible and scalable environment for Big Data analytics. In week two, our course introduces large scale data storage and the difficulties and problems of consensus in enormous stores that use quantities of processors, memories and disks. We discuss eventual consistency, ACID, and BASE and the consensus algorithms used in data centers including Paxos and Zookeeper. Our course presents Distributed Key-Value Stores and in memory databases like Redis used in data centers for performance. Next we present NOSQL Databases. We visit HBase, the scalable, low latency database that supports database operations in applications that use Hadoop. Then again we show how Spark SQL can program SQL queries on huge data. We finish up week two with a presentation on Distributed Publish/Subscribe systems using Kafka, a distributed log messaging system that is finding wide use in connecting Big Data and streaming applications together to form complex systems. Week three moves to fast data real-time streaming and introduces Storm technology that is used widely in industries such as Yahoo. We continue with Spark Streaming, Lambda and Kappa architectures, and a presentation of the Streaming Ecosystem. Week four focuses on Graph Processing, Machine Learning, and Deep Learning. We introduce the ideas of graph processing and present Pregel, Giraph, and Spark GraphX. Then we move to machine learning with examples from Mahout and Spark. Kmeans, Naive Bayes, and fpm are given as examples. Spark ML and Mllib continue the theme of programmability and application construction. The last topic we cover in week four introduces Deep Learning technologies including Theano, Tensor Flow, CNTK, MXnet, and Caffe on Spark.

Syllabus

  • Course Orientation
    • You will become familiar with the course, your classmates, and our learning environment. The orientation will also help you obtain the technical skills required for the course.
  • Module 1: Spark, Hortonworks, HDFS, CAP
    • In Module 1, we introduce you to the world of Big Data applications. We start by introducing you to Apache Spark, a common framework used for many different tasks throughout the course. We then introduce some Big Data distro packages, the HDFS file system, and finally the idea of batch-based Big Data processing using the MapReduce programming paradigm.
  • Module 2: Large Scale Data Storage
    • In this module, you will learn about large scale data storage technologies and frameworks. We start by exploring the challenges of storing large data in distributed systems. We then discuss in-memory key/value storage systems, NoSQL distributed databases, and distributed publish/subscribe queues.
  • Module 3: Streaming Systems
    • This module introduces you to real-time streaming systems, also known as Fast Data. We talk about Apache Storm in length, Apache Spark Streaming, and Lambda and Kappa architectures. Finally, we contrast all these technologies as a streaming ecosystem.
  • Module 4: Graph Processing and Machine Learning
    • In this module, we discuss the applications of Big Data. In particular, we focus on two topics: graph processing, where massive graphs (such as the web graph) are processed for information, and machine learning, where massive amounts of data are used to train models such as clustering algorithms and frequent pattern mining. We also introduce you to deep learning, where large data sets are used to train neural networks with effective results.

Taught by

Reza Farivar and Roy H. Campbell

Cloud Computing Applications, Part 2: Big Data and Applications in the Cloud
Go to course

Cloud Computing Applications, Part 2: Big Data and Applications in the Cloud

Brought by: Coursera

  • Coursera
  • Free
  • English
  • Certificate Available
  • Available at any time
  • All
  • Arabic, French, Portuguese, Italian, German, Russian, English, Spanish, Korean, Thai, Indonesian, Kazakh, Hindi, Swedish, Greek, Chinese, Ukrainian, Japanese, Polish, Dutch, Turkish
8.1.2PHP Version286msRequest Duration2MBMemory UsageGET en/courses/{slug}Route
    • Booting (174ms)
    • Application (111ms)
    • 1 x Booting (60.92%)
      174.44ms
      1 x Application (38.84%)
      111.21ms
      14 templates were rendered
      • public.courses.show (resources/views/public/courses/show.blade.php)3bladefile
        Params
        0
        course
        1
        links
        2
        config
      • public.courses.partials.breadcrumbs (resources/views/public/courses/partials/breadcrumbs.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.heading (resources/views/public/courses/partials/heading.blade.php)7bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        classes
      • public.courses.partials.details (resources/views/public/courses/partials/details.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.breadcrumbs (resources/views/public/courses/partials/breadcrumbs.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.heading (resources/views/public/courses/partials/heading.blade.php)7bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        classes
      • public.layouts.main (resources/views/public/layouts/main.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.layouts.partials.meta (resources/views/public/layouts/partials/meta.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.layouts.partials.navbar (resources/views/public/layouts/partials/navbar.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.auth.profile.partials.links (resources/views/public/auth/profile/partials/links.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.layouts.partials.flash-session (resources/views/public/layouts/partials/flash-session.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      uri
      GET en/courses/{slug}
      middleware
      web, localize:en
      controller
      App\Http\Controllers\CourseController@show
      as
      en.courses.show
      namespace
      prefix
      /en
      where
      file
      app/Http/Controllers/CourseController.php:17-35
      7 statements were executed3.8ms
      • select * from `courses` where `slug_en` = 'cloud-computing-applications,-part-2:-big-data-and-applications-in-the-cloud' limit 1
        2.25ms/app/Http/Controllers/CourseController.php:20corspedia
        Metadata
        Bindings
        • 0. cloud-computing-applications,-part-2:-big-data-and-applications-in-the-cloud
        Backtrace
        • 17. /app/Http/Controllers/CourseController.php:20
        • 18. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 19. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 20. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • update `courses` set `visitors` = `visitors` + 1, `courses`.`updated_at` = '2025-02-03 18:13:04' where `id` = 126
        360μs/app/Http/Controllers/CourseController.php:21corspedia
        Metadata
        Bindings
        • 0. 2025-02-03 18:13:04
        • 1. 126
        Backtrace
        • 17. /app/Http/Controllers/CourseController.php:21
        • 18. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 19. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 20. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select `id`, `name_en`, `name_ar`, `topic_id`, `slug_en`, `slug_ar` from `subjects` where `subjects`.`id` in (10)
        260μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 20. /app/Http/Controllers/CourseController.php:23
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 22. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 23. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 24. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select `id`, `name_en`, `name_ar`, `slug_en`, `slug_ar` from `topics` where `topics`.`id` in (1)
        210μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 25. /app/Http/Controllers/CourseController.php:23
        • 26. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 27. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 28. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 29. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select * from `institutions` where `institutions`.`id` in (15) and `institutions`.`deleted_at` is null
        250μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 20. /app/Http/Controllers/CourseController.php:23
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 22. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 23. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 24. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select * from `providers` where `providers`.`id` in (2) and `providers`.`deleted_at` is null
        240μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 20. /app/Http/Controllers/CourseController.php:23
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 22. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 23. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 24. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select * from `html_files` where `html_files`.`id` = 126 limit 1
        230μs/app/Models/Course.php:84corspedia
        Metadata
        Bindings
        • 0. 126
        Backtrace
        • 21. /app/Models/Course.php:84
        • 28. view::public.courses.show:29
        • 30. /vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php:125
        • 31. /vendor/laravel/framework/src/Illuminate/View/Engines/PhpEngine.php:58
        • 32. /vendor/laravel/framework/src/Illuminate/View/Engines/CompilerEngine.php:72
      App\Models\HtmlFile
      1
      App\Models\Provider
      1
      App\Models\Institution
      1
      App\Models\Topic
      1
      App\Models\Subject
      1
      App\Models\Course
      1
        _token
        L0XpZ6lEHyyWvwusai20mhA9f04gbBQDisMD6U0r
        locale
        en
        _previous
        array:1 [ "url" => "https://www.corspedia.com/en/courses/cloud-computing-applications,-part-2:-big...
        _flash
        array:2 [ "old" => [] "new" => [] ]
        PHPDEBUGBAR_STACK_DATA
        []
        path_info
        /en/courses/cloud-computing-applications,-part-2:-big-data-and-applications-in-the-cloud
        status_code
        200
        
        status_text
        OK
        format
        html
        content_type
        text/html; charset=UTF-8
        request_query
        []
        
        request_request
        []
        
        request_headers
        0 of 0
        array:24 [ "sec-ch-ua-mobile" => array:1 [ 0 => "?0" ] "sec-ch-ua" => array:1 [ 0 => ""HeadlessChrome";v="129", "Not=A?Brand";v="8", "Chromium";v="129"" ] "cache-control" => array:1 [ 0 => "no-cache" ] "pragma" => array:1 [ 0 => "no-cache" ] "cdn-loop" => array:1 [ 0 => "cloudflare; loops=1" ] "priority" => array:1 [ 0 => "u=0, i" ] "upgrade-insecure-requests" => array:1 [ 0 => "1" ] "user-agent" => array:1 [ 0 => "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" ] "cf-connecting-ip" => array:1 [ 0 => "18.191.217.79" ] "accept" => array:1 [ 0 => "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" ] "sec-fetch-site" => array:1 [ 0 => "none" ] "cf-visitor" => array:1 [ 0 => "{"scheme":"https"}" ] "sec-fetch-mode" => array:1 [ 0 => "navigate" ] "sec-fetch-user" => array:1 [ 0 => "?1" ] "x-forwarded-proto" => array:1 [ 0 => "https" ] "cf-ipcountry" => array:1 [ 0 => "US" ] "accept-encoding" => array:1 [ 0 => "gzip, br" ] "sec-fetch-dest" => array:1 [ 0 => "document" ] "sec-ch-ua-platform" => array:1 [ 0 => ""Windows"" ] "x-forwarded-for" => array:1 [ 0 => "18.191.217.79" ] "cf-ray" => array:1 [ 0 => "90c4648c7a2572eb-ORD" ] "host" => array:1 [ 0 => "www.corspedia.com" ] "content-length" => array:1 [ 0 => "" ] "content-type" => array:1 [ 0 => "" ] ]
        request_server
        0 of 0
        array:50 [ "USER" => "www-data" "HOME" => "/var/www" "HTTP_SEC_CH_UA_MOBILE" => "?0" "HTTP_SEC_CH_UA" => ""HeadlessChrome";v="129", "Not=A?Brand";v="8", "Chromium";v="129"" "HTTP_CACHE_CONTROL" => "no-cache" "HTTP_PRAGMA" => "no-cache" "HTTP_CDN_LOOP" => "cloudflare; loops=1" "HTTP_PRIORITY" => "u=0, i" "HTTP_UPGRADE_INSECURE_REQUESTS" => "1" "HTTP_USER_AGENT" => "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" "HTTP_CF_CONNECTING_IP" => "18.191.217.79" "HTTP_ACCEPT" => "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" "HTTP_SEC_FETCH_SITE" => "none" "HTTP_CF_VISITOR" => "{"scheme":"https"}" "HTTP_SEC_FETCH_MODE" => "navigate" "HTTP_SEC_FETCH_USER" => "?1" "HTTP_X_FORWARDED_PROTO" => "https" "HTTP_CF_IPCOUNTRY" => "US" "HTTP_ACCEPT_ENCODING" => "gzip, br" "HTTP_SEC_FETCH_DEST" => "document" "HTTP_SEC_CH_UA_PLATFORM" => ""Windows"" "HTTP_X_FORWARDED_FOR" => "18.191.217.79" "HTTP_CF_RAY" => "90c4648c7a2572eb-ORD" "HTTP_HOST" => "www.corspedia.com" "REDIRECT_STATUS" => "200" "SERVER_NAME" => "corspedia.com" "SERVER_PORT" => "443" "SERVER_ADDR" => "141.95.147.152" "REMOTE_USER" => "" "REMOTE_PORT" => "42546" "REMOTE_ADDR" => "172.70.126.160" "SERVER_SOFTWARE" => "nginx/1.18.0" "GATEWAY_INTERFACE" => "CGI/1.1" "HTTPS" => "on" "REQUEST_SCHEME" => "https" "SERVER_PROTOCOL" => "HTTP/2.0" "DOCUMENT_ROOT" => "/var/www/corspedia/public" "DOCUMENT_URI" => "/index.php" "REQUEST_URI" => "/en/courses/cloud-computing-applications,-part-2:-big-data-and-applications-in-the-cloud" "SCRIPT_NAME" => "/index.php" "CONTENT_LENGTH" => "" "CONTENT_TYPE" => "" "REQUEST_METHOD" => "GET" "QUERY_STRING" => "" "SCRIPT_FILENAME" => "/var/www/corspedia/public/index.php" "PATH_INFO" => "" "FCGI_ROLE" => "RESPONDER" "PHP_SELF" => "/index.php" "REQUEST_TIME_FLOAT" => 1738606384.3405 "REQUEST_TIME" => 1738606384 ]
        request_cookies
        []
        
        response_headers
        0 of 0
        array:5 [ "content-type" => array:1 [ 0 => "text/html; charset=UTF-8" ] "cache-control" => array:1 [ 0 => "no-cache, private" ] "date" => array:1 [ 0 => "Mon, 03 Feb 2025 18:13:04 GMT" ] "set-cookie" => array:2 [ 0 => "XSRF-TOKEN=eyJpdiI6Ilc2WDYzalNpR09VZ2JGOGNrZkxOWWc9PSIsInZhbHVlIjoiaUR3eXRqODVUTFpsR3U0Znlib1pkTTlMOGNmSVRUT1Y0ZTFPbXVMMm9DRFZKVFJlM2dZWTFRalN6dkl1Y1g5ejJjdEphWENFQUN3NGxoaVZPajlseEYzQ2Zucys0cHR5WE9pclN4VGxaNmtObEw2bUwvdm4xb1pPaE9VSUNqTFkiLCJtYWMiOiI4MGQ1ODU2MGRmNDliNWExOGQyNDA5Yzc2Yjg1MDg0ZGY4ZTA3MDEyYmVmOGY2MGJkNDk0YTk4N2VmYzY4ZGJhIiwidGFnIjoiIn0%3D; expires=Mon, 03 Feb 2025 20:13:04 GMT; Max-Age=7200; path=/; samesite=laxXSRF-TOKEN=eyJpdiI6Ilc2WDYzalNpR09VZ2JGOGNrZkxOWWc9PSIsInZhbHVlIjoiaUR3eXRqODVUTFpsR3U0Znlib1pkTTlMOGNmSVRUT1Y0ZTFPbXVMMm9DRFZKVFJlM2dZWTFRalN6dkl1Y1g5ejJjdEphW" 1 => "laravel_session=eyJpdiI6IjMxdU9rUnI4Z09HMzkwYlVOSzd6S2c9PSIsInZhbHVlIjoiK0N5aVowWllyc2tMZWlDMFRoSHUwTkI1cXBvMm03SmtYTFRFbWcwZENzM2xyL0lPUENFNFZ3dldBZEdvWHZBeVdJU1Y1Vjk4cUxNU0FiWERsR2Y5OFNsODZNTW5vUzIyZE0vUG1WN3F6cWRadnllTDdMcFBCS25GQjRyOVVNL1ciLCJtYWMiOiI1ZmEyMDhkODY3MjJiNTE4MTVmNWI4YTBhZDRmN2NmZTQ5ZDBmMGExYWZhMGUwYjRhOTZiMDQxOTdkMmRhNTFkIiwidGFnIjoiIn0%3D; expires=Mon, 03 Feb 2025 20:13:04 GMT; Max-Age=7200; path=/; httponly; samesite=laxlaravel_session=eyJpdiI6IjMxdU9rUnI4Z09HMzkwYlVOSzd6S2c9PSIsInZhbHVlIjoiK0N5aVowWllyc2tMZWlDMFRoSHUwTkI1cXBvMm03SmtYTFRFbWcwZENzM2xyL0lPUENFNFZ3dldBZEdvWHZBeVdJ" ] "Set-Cookie" => array:2 [ 0 => "XSRF-TOKEN=eyJpdiI6Ilc2WDYzalNpR09VZ2JGOGNrZkxOWWc9PSIsInZhbHVlIjoiaUR3eXRqODVUTFpsR3U0Znlib1pkTTlMOGNmSVRUT1Y0ZTFPbXVMMm9DRFZKVFJlM2dZWTFRalN6dkl1Y1g5ejJjdEphWENFQUN3NGxoaVZPajlseEYzQ2Zucys0cHR5WE9pclN4VGxaNmtObEw2bUwvdm4xb1pPaE9VSUNqTFkiLCJtYWMiOiI4MGQ1ODU2MGRmNDliNWExOGQyNDA5Yzc2Yjg1MDg0ZGY4ZTA3MDEyYmVmOGY2MGJkNDk0YTk4N2VmYzY4ZGJhIiwidGFnIjoiIn0%3D; expires=Mon, 03-Feb-2025 20:13:04 GMT; path=/XSRF-TOKEN=eyJpdiI6Ilc2WDYzalNpR09VZ2JGOGNrZkxOWWc9PSIsInZhbHVlIjoiaUR3eXRqODVUTFpsR3U0Znlib1pkTTlMOGNmSVRUT1Y0ZTFPbXVMMm9DRFZKVFJlM2dZWTFRalN6dkl1Y1g5ejJjdEphW" 1 => "laravel_session=eyJpdiI6IjMxdU9rUnI4Z09HMzkwYlVOSzd6S2c9PSIsInZhbHVlIjoiK0N5aVowWllyc2tMZWlDMFRoSHUwTkI1cXBvMm03SmtYTFRFbWcwZENzM2xyL0lPUENFNFZ3dldBZEdvWHZBeVdJU1Y1Vjk4cUxNU0FiWERsR2Y5OFNsODZNTW5vUzIyZE0vUG1WN3F6cWRadnllTDdMcFBCS25GQjRyOVVNL1ciLCJtYWMiOiI1ZmEyMDhkODY3MjJiNTE4MTVmNWI4YTBhZDRmN2NmZTQ5ZDBmMGExYWZhMGUwYjRhOTZiMDQxOTdkMmRhNTFkIiwidGFnIjoiIn0%3D; expires=Mon, 03-Feb-2025 20:13:04 GMT; path=/; httponlylaravel_session=eyJpdiI6IjMxdU9rUnI4Z09HMzkwYlVOSzd6S2c9PSIsInZhbHVlIjoiK0N5aVowWllyc2tMZWlDMFRoSHUwTkI1cXBvMm03SmtYTFRFbWcwZENzM2xyL0lPUENFNFZ3dldBZEdvWHZBeVdJ" ] ]
        session_attributes
        0 of 0
        array:5 [ "_token" => "L0XpZ6lEHyyWvwusai20mhA9f04gbBQDisMD6U0r" "locale" => "en" "_previous" => array:1 [ "url" => "https://www.corspedia.com/en/courses/cloud-computing-applications,-part-2:-big-data-and-applications-in-the-cloud" ] "_flash" => array:2 [ "old" => [] "new" => [] ] "PHPDEBUGBAR_STACK_DATA" => [] ]