yuhexiong/airflow-dag-kafka-flink-doris-python 开源项目中文解读：技术栈、适用场景与上手建议

项目介绍

yuhexiong/airflow-dag-kafka-flink-doris-python 是一个托管在 GitHub 上的开源项目，主要使用 Python 编写，目前获得 3 个 Star、0 次 Fork。

仓库全名: yuhexiong/airflow-dag-kafka-flink-doris-python

README 摘要

Airflow Dag Kafka/Flink/Doris use airflow to maintain operation of pipeline and database. Overview Tool: Apache Airflow v2.9.1 Language: Python v3.8 Run ENV setting config/airflow.cfg Run Docker UI Page Open http://localhost:8080 to run DAGs in UI. DAGs 1. Check Kafka Updates confirm that new data comes in every day in kafka's topic. setting variables 2. Check Flink Job State check all jobs on flink are running…

完整内容见 GitHub 原始 README 或本页。

常见问题

airflow-dag-kafka-flink-doris-python 用什么语言开发？

主要使用 Python 编写，目前有 3 个 Star。

如何获取 airflow-dag-kafka-flink-doris-python 的源码？

可以直接克隆仓库：git clone https://github.com/yuhexiong/airflow-dag-kafka-flink-doris-python.git，默认分支为 main；安装与运行步骤见仓库 README。

airflow-dag-kafka-flink-doris-python 最近还在维护吗？

仓库最近一次更新为 2025年4月24日，当前没有未关闭的 Issue；Star 数为 3，可作为社区活跃度的参考。

关于

Airflow to maintain operation of Flink pipeline, Doris database and Kafka.

airflow airflow-dags airflow-docker apache-airflow apache-doris apache-flink apache-kafka doris flink kafka

3 Stars
0 Forks
1 Watching

描述取自 GitHub 官方 API，中文内容由 GitRead 中文社区编辑团队整理。

简要分析

热度分

价值分

活跃状态

活跃

GitHub

语言占比

Dockerfile

Python

贡献者

yuhexiong

Airflow Dag Kafka/Flink/Doris

use airflow to maintain operation of pipeline and database.

Overview

Tool: Apache Airflow v2.9.1
Language: Python v3.8

Run

ENV

setting config/airflow.cfg

smtp_host = smtp.gmail.com
smtp_starttls = True
smtp_ssl = False
smtp_user = [email protected]
smtp_password = pass
smtp_port = 587
smtp_mail_from = [email protected]

Run Docker

docker compose up -d

UI Page

Open http://localhost:8080 to run DAGs in UI.

DAGs

1. Check Kafka Updates

confirm that new data comes in every day in kafka’s topic.

setting variables

bootstrap_servers: localhost:9093
topic_name: topic1
log_file: local log file path to collect errors and send them once.

2. Check Flink Job State

check all jobs on flink are running every day.

setting variables

flink_url: localhost:8081
log_file: local log file path to collect errors and send them once.

3. Check Flink Job And Upload

check the jobs are running normally according to the configuration file yaml every day.
If there are errors, cancel and rerun them.

setting variables

log_file: local log file path to collect errors and send them once.

flink_pipeline.yaml

host: http://localhost:8081
jars:
  - jar_id: xxx_xxx_flink-1.0.jar
    jobs:
      - entry_class: com.examples.entry.KafkaToKafka
        program_file: https://storage.googleapis.com/xxx/KafkaToKafka.yaml
        job_name: kafka(topic-1) -> kafka(topic-2)
      - entry_class: com.examples.entry.KafkaToDoris
        program_file: https://storage.googleapis.com/xxx/KafkaToDoris.yaml
        job_name: kafka(topic-1) -> doris(table-1)

4. Check Doris Updates

check doris has new data coming in every day.

setting variables

doris_host: localhost
username: root
password: password
database: database
doris_table: doris_table
doris_timestamp_column: timestamp
log_file: local log file path to collect errors and send them once.

5. Send Exception Email

check there is any data in the log file every day. If there is any data in the log file, send an email.

setting variables

log_file: local log file path to collect errors and send them once.

暂无评论

airflow-dag-kafka-flink-doris-python