ai-storage 开源项目推荐（17 个）

rustfs /rustfs

RustFS 是一个开源、兼容 S3 的高性能对象存储系统，支持与其他 S3 兼容平台（如 MinIO 和 Ceph）迁移和共存。

achtungsoftware /alarik

高性能分布式 S3 兼容对象存储，专注于速度，旨在成为 MinIO 和 RustFS 的开源替代方案。

russfellows /s3dlio

Part of the sai3 project that delivers multi-protocol storage access for AI/ML workflows, supporting Pytorch, Tensorflow and Jax. This project provides a CLI, along with Rust and Python libraries for AI/ML storage workflows. Supporting S3, File, Azure Blob and GCS using the latest Rust SDKs.

HaManhDong /storage-galaxy

该项目在 GitHub 上没有填写描述

zegang /cppio

Hybrid Storage with freely combine

Develop-Packt /Handling-Big-Data-File-Formats

You will look at the different formats for storing, transmitting and organizing very large collections of data. You will learn how to choose the right data formats as well as look at compression.

Develop-Packt /Working-with-Data-Stores-SQL-and-NoSQL-Databases

We will look at SQL and NoSQL databases and decide the ideal database to be used based on the format of data. Then you will implement storing and querying data in different databases such as MySQL, MongoDB, and Cassandra.

Hoangstephen /shelby-aptos-provenance

High-performance AI data storage pipeline utilizing Shelby's global object storage and Aptos blockchain for verifiable data provenance. Optimized for low-latency AI workloads.

Develop-Packt /Introduction-to-the-Ethics-of-AI-Data-Storage

You will look at several case studies, examining everything from AI being used to manipulate elections, to AI displaying racial and sexist prejudices. Implement a simple sentiment classifier to differentiate between positive and negative words and sentences. You'll observe how this works in many cases, and display the problematic biases and huma…

zettalane-systems /open-pnfs-cloud

Open source, cloud native Parallel NFS (pNFS) on AWS, Azure and GCP — quick deploy into your own cloud account. Free Community Edition.

vladimir-fonseca /platform-secure-sds-infra-gitops

GitOps-managed secure software-defined storage platform: multi-backend SDS, multi-plane CNI, DPU/SR-IOV offload, Vault KMS, and staged catalog-to-deployment orchestration.

zettalane-systems /open-lustre-cloud

Open Cloud-native Lustre Community Edition AWS, Azure and GCP — quick deploy into your own cloud account. Free Community Edition

lwwgo /fstorex

Fast distributed file storage for AI workloads — Raft-based HA metadata, POSIX-like semantics, FUSE mount. Pure Go, cross-platform(Linux/MacOS).

Develop-Packt /Introduction-to-Analytics-Engine-Spark-for-Big-Data

Discover the fundamentals of Apache Spark, including its architecture, transformations and actions. Learn why the design choices of RDD were made and how this enhances the Hadoop and MapReduce construct.

vladimir-fonseca /ai-data-pipelines-storage-architecture-for-the-ai-factory

A technical publication defining storage architecture, data movement, tiering, governance, and infrastructure requirements for AI data pipelines and AI factories.

api-evangelist /weka

Weka — independent third-party profile of a public API surface, by API Evangelist. WEKA (WekaIO, Inc.) builds NeuralMesh, a software-defined, high-performance data platform for AI, machine learning, and HPC workloads. Its container-native storage system delivers a unified namespace across NVMe SSD and object storage, running on bare metal and acros

api-evangelist /minio

MinIO — independent third-party profile of a public API surface, by API Evangelist. MinIO is a high-performance, S3-compatible object storage system built for large-scale AI, analytics, and cloud-native data infrastructure. Its API is a drop-in implementation of the Amazon S3 REST API (bucket and object operations, multipart uploads, versioning, ob