etl-job 开源项目推荐（110 个）

AlexIoannides /pyspark-example-project

为PySpark ESL作业和应用程序实施最佳实践。

san089 /goodreads_etl_pipeline

用于构建数据湖、数据仓库和分析平台的端到端 GoodReads 数据管道。

paillave /Etl.Net

Mass processing data with a complete ETL for .net developers

jbogard /bulk-writer

为快速 ETL 作业提供指导，提供用于 SqlBulkCopy（或 MySql、Oracle 的等效实现）的 IDataReader 实现，该实现包装 IEnumerable，并提供用于将实体映射到表列的库。

visiologyofficial /vixtract

该项目在 GitHub 上没有填写描述

cloudposse /terraform-aws-glue

Terraform modules for provisioning and managing AWS Glue resources

nsphung /pyspark-template

A Python PySpark Projet with Poetry

felipefrizzo /terraform-aws-kinesis-firehose

This code creates a Kinesis Firehose in AWS to send CloudWatch log data to S3.

ktnsh24 /DataModelling

This repo will guide you step-by-step method to create star schema dimensional model.

kishlayjeet /Twitter-Data-Pipeline-using-Airflow-and-AWS-S3

An end-to-end Twitter Data Pipeline that extracts data from Twitter and loads it into AWS S3.

Joshua-omolewa /Retailstore_ETL_pipeline_project

Built a Data Pipeline for a Retail store using AWS services that collects data from its transactional database (OLTP) in Snowflake and transforms the raw data (ETL process) using Apache spark to meet business requirements and also enables Data Analyst create Data Visualization using Superset. Airflow is used to orchestrate the pipeline

michaelbironneau /analyst

A declarative, SQL-like DSL for data integration tasks.

ankiano /etl

Extract transform load CLI tool for extracting small and middle data volume from sources (databases, csv files, xls files, gspreadsheets) to target (databases, csv files, xls files, gspreadsheets) in free combination.

yennanliu /AirflowJob

Airflow POC demo : 1) env set up 2) airflow DAG 3) Spark/ML pipeline | #DE

SourceWatcher /source-watcher-core

PHP ETL engine with pluggable steps: extractors, transformers, loaders

san089 /airflow-training

Introduction to the data pipeline management with Airflow. Airflow schedule and maintain numerous ETL processes running on a large scale Enterprise Data Warehouse.

swadhinbiswas /eu-air-traffic

Live and historical European air traffic: a VPS collector streams ADS-B, OpenSky movements, schedules and weather through Kafka into a Bronze/Silver lake, a dbt warehouse and a read-only serving copy behind a public dashboard. Budget-aware polling and atomic publishes keep it correct while unattended.

ShihWen /tpe-mrt-traffic-etl

A data pipeline from source to data warehouse using Taipei Metro Hourly Traffic data

2298-Software /Mambo

A simple in-memory, configuration driven, data processing pipeline for Apache Spark.

san089 /pyspark-example-project

Example project and best practices for Python-based Spark ETL jobs and applications.