big_data

Big Data for beginners

Explore a variety of tutorials and interactive demonstrations focused on Big Data technologies like Hadoop, Spark, and more, primarily presented in the format of Jupyter notebooks. Most notebooks are self-contained, with instructions for installing all required services. They can be run on Google Colab or in a virtual Ubuntu machine/container.

Setting Up Hadoop: Single-Node Configuration

• Hadoop_Setting_up_a_Single_Node_Cluster.ipynb Set up a single-node Hadoop cluster on Google Colab and run some basic HDFS and MapReduce examples
• Hadoop_single_node_cluster_setup_Python.ipynb Set up a single-node Hadoop cluster on Google Colab using Python
• Hadoop_minicluster.ipynb Deploy a test Hadoop Cluster with a single command and no need for configuration.

Running Apache Spark in Standalone Mode

• Setting_up_Spark_Standalone_on_Google_Colab.ipynb Set up a single-node Spark server on Google Colab and estimate „π“ with a Montecarlo method
• Setting_up_Spark_Standalone_on_Google_Colab_BigtopEdition.ipynb Set up a single-node Spark server on Google Colab using the Bigtop distribution and utilities, estimate „π“ with a Montecarlo method and run another Java ML example.
• Run_Spark_on_Google_Colab.ipynb Set up a single-node standalone Spark server on Google Colab including Web UI and History Server - compact version
• Spark_Standalone_Architecture_on_Google_Colab.ipynb Explore the Spark architecture through the immersive experience of deploying a standalone setup.
• SparkOnSLURM.ipynb Spark on SLURM: Single-Node Parallelism Guide.

MapReduce Tutorials

• MapReduce_Primer_HelloWorld.ipynb A MapReduce Primer with “Hello, World!”
• MapReduce_Primer_HelloWorld_bash.ipynb A MapReduce Primer with “Hello, World! in Bash with just a few lines of code”
• mapreduce_with_bash.ipynb An introduction to MapReduce using MapReduce Streaming and bash to create mapper and reducer
• simplest_mapreduce_bash_wordcount.ipynb A very basic MapReduce wordcount example
• mrjob_wordcount.ipynb This demonstration showcases MapReduce using the mrjob library for performing the classic WordCount algorithm, which counts the frequency of every word in a text dataset.
• getting_started_with_mrjob.ipynb Getting started with mrjob. This demonstration showcases the power and flexibility of the mrjob Python framework for developing and executing scalable data processing jobs, supporting both MapReduce and Spark across different execution backends, culminating in a hybrid approach utilizing Spark on YARN.
• Hadoop_spilling.ipynb Hadoop spilling explained

PySpark Tutorials

• PySpark_On_Google_Colab.ipynb Explore the inner workings of PySpark on Google Colab
• PySpark_miscellanea.ipynb Tips, tricks, and insights related to PySpark.
• getting_started_with_mrjob.ipynb Getting started with mrjob. This demonstration showcases the power and flexibility of the mrjob Python framework for developing and executing scalable data processing jobs, supporting both MapReduce and Spark across different execution backends, culminating in a hybrid approach utilizing Spark on YARN.
• demoSparkSQLPython.ipynb A hands-on demo showcasing the fundamentals of PySpark SQL — how to create DataFrames, register temporary views, and query data using SQL syntax.
• ngrams_with_pyspark.ipynb Basic example of n-grams extraction with PySpark
• Ngrams.ipynb This notebook demonstrates the power of Apache Spark (PySpark) for large-scale data processing through the computation of N-grams. It showcases fundamental Spark concepts, including transformations and actions, using two distinct and massive datasets: Common Crawl (Web Data) and Genomic Data
• generate_data_with_Faker.ipynb Fake It Till You Make It: Generate Test Data with Faker. Create customizable fake data for testing and development using the Faker library. Useful for populating databases, simulating user activity, or prototyping applications without relying on real data.
• Encoding_dataframe_columns.ipynb DataFrame Column Encoding with PySpark and Parquet Format
• Apache_Sedona_with_PySpark.ipynb Apache Sedona™ is a high-performance cluster computing system for processing large-scale spatial data, extending the capabilities of Apache Spark for advanced geospatial analytics. Run a basic example with PySpark on Google Colab

Miscellaneous Tutorials

• GutenbergBooks.ipynb Explore and download books from the Gutenberg books collection.
• TestDFSio.ipynb Demo of TestDFSio for benchmarking Hadoop clusters
• Unicode.ipynb Exploring Unicode categories
• polynomial_regression.ipynb Worked out example of polynomial regression with numpy and matplotlib
• downloadSpark.ipynb How to download and verify the Spark distribution. Learn about the Apache CDN distribution and visualize the Spark releases timeline.
• SLURM.ipynb SLURM (Simple Linux Utility for Resource Management) is an open-source workload manager used to schedule and run jobs on shared compute resources—even on a single machine. This walkthrough shows how to set up and use SLURM, a popular workload manager, on a single Ubuntu virtual machine inside Google Colab.

Virtualization and Cloud Automation

• docker_for_beginners.md Docker for beginners: an introduction to the world of containers
• Terraform for beginners.md Getting started with Terraform
• Terraform in 5 minutes A short introduction to Terraform, the powerful and popular tool for infrastructure provisioning and management

Big Data Learning Pathways

• online_resources.md Online resources for learning Big Data

About this repository

Notebooks Testing and CI

All Jupyter notebooks are tested on an Ubuntu virtual machine through a GitHub automated workflow. The executions results are saved in: report.md.

💡 Keep Learning, Keep Sharing

If you find these tutorials useful, feel free to leave a ⭐ on the repository.