摘要：一个RDD就是一个分布式对象集合，本质上是一个只读的分区记录集合。Spark Core建立在统一的抽象RDD之上，使得Spark的各个组件可以无缝进行集成，在同一个应用程序中完成大数据计算任务。

分享兴趣，传播快乐，增长见闻，留下美好！

亲爱的您，这里是LearningYard新学苑。

今天小编为大家带来文章

“小h漫谈（20）：RDD的特性”

欢迎您的访问。

Share interest, spread happiness, increase knowledge, leave a beautiful!

Dear, this is LearningYard Academy.

Today Xiaobian brings you an article

"Xiaoh's Ramblings (20):Characteristics of RDDs"

Welcome to your visit.

一、思维导图（Mind mapping）

二、精读内容（Intensive Reading Content）

一个RDD就是一个分布式对象集合，本质上是一个只读的分区记录集合。Spark Core建立在统一的抽象RDD之上，使得Spark的各个组件可以无缝进行集成，在同一个应用程序中完成大数据计算任务。

An RDD (Resilient Distributed Dataset) is a collection of distributed objects, essentially a read-only collection of partitioned records. Spark Core is built on the unified abstraction of RDDs, enabling seamless integration among various Spark components to accomplish big data computing tasks within a single application.

RDD的特性包括：

The characteristics of RDDs include:

（1）高容错性。现有的分布式共享内存、键值存储、内存数据库等，为了实现容错，必须在集群节点之间进行数据复制或者记录日志，也就是在节点之间会发生大量的数据传输，这对于数据密集型应用而言会带来很大的开销。在RDD的设计中，数据只读，不可修改。如果需要修改数据，则必须从父RDD转换到子RDD,由此在不同RDD之间建立血缘关系。

High fault tolerance. Existing distributed shared memory, key-value stores, and in-memory databases, in order to achieve fault tolerance, must replicate data or log records among cluster nodes, resulting in a large amount of data transfer between nodes. This can incur significant overhead for data-intensive applications. In the design of RDDs, data is read-only and immutable. If data modification is required, it must be achieved through a transformation from a parent RDD to a child RDD, thereby establishing a lineage relationship between different RDDs.

因此，RDD是一种天生具有容错机制的特殊集合，不需要通过数据冗余的方式(如检查点)实现容错，而只需通过RDD父子依赖(血缘)关系重新计算得到丢失的分区来实现容错，无须回滚整个系统。

Therefore, RDDs are a special type of collection that inherently possess fault-tolerance mechanisms. They do not need to rely on data redundancy (such as checkpointing) for fault tolerance. Instead, they can recover lost partitions by recalculating them based on the parent-child dependencies (lineage) between RDDs, without the need to roll back the entire system.

这样就避免了数据复制的高开销，而且重算过程可以在不同节点之间并行进行，实现了高容错。此外，RDD提供的转换操作都是一些粗粒度的操作(如map、filter和join ), RDD依赖关系只需要记录这种粗粒度的转换操作，而不需要记录具体的数据和各种细粒度操作的日志(如对哪个数据项进行了修改)，这就大大降低了数据密集型应用中的容错开销。

This approach avoids the high overhead of data replication, and the recalculation process can be performed in parallel across different nodes, achieving high fault tolerance. Moreover, the transformation operations provided by RDDs are coarse-grained (such as map, filter, and join). The dependencies between RDDs only need to record these coarse-grained transformations, rather than logging specific data and various fine-grained operations (such as which data item was modified). This significantly reduces the fault-tolerance overhead in data-intensive applications.

（2）中间结果持久化到内存。数据在内存中的多个RDD操作之间进行传递，不需要“落地”到磁盘上，避免了不必要的读写磁盘的开销。

Persistence of intermediate results in memory. Data is passed between multiple RDD operations in memory without the need to "spill" to disk, avoiding unnecessary disk read/write overhead.

（3）存放的数据可以是Java对象，避免了不必要的对象序列化和反序列化开销。

The data stored can be Java objects, eliminating the need for unnecessary object serialization and deserialization overhead.

今天的分享就到这里了

如果您对今天的文章有独特的想法

欢迎给我们留言

让我们相约明天

祝您今天过得开心快乐！

That's all for today's sharing.

If you have a unique idea about the article

please leave us a message

and let us meet tomorrow

I wish you a nice day!

文案|小h

排版|小h

审核|ls

参考资料：

文字：《Spark编程基础》

图片：CSDN

翻译：Kimi.ai

本文由LearningYard新学苑整理并发出，如有侵权请在后台留言！

来源：LearningYard学苑