Paper Reading on NoneBack

Borg: Large-scale Cluster Management at Google with Borg

Mon, 19 Feb 2024 11:12:16 +0800

Borg is a cluster management system, similar to the closed-source version of Kubernetes (k8s).

It achieves high utilization through admission control, efficient task packing, overcommitment, machine sharing, and process-level performance isolation.
It provides runtime features to reduce failure recovery time for high-availability applications and scheduling policies that reduce the probability of correlated failures.
It offers a declarative job description language, DNS integration, real-time job monitoring, and tools for analyzing and simulating system behavior, simplifying usage for end-users.

The paper aims to introduce the system design and share the experiences Google has gained behind it. This blog mainly focuses on system design, specifically the services Borg offers in terms of SLA, its abstraction of workloads, resources, and scheduling.

System Abstraction

Borg manages two primary workloads: long-running services and batch jobs, corresponding to two types of jobs (prod/non-prod). A job consists of several tasks, and different jobs have different priorities.

In terms of deployment architecture, a Borg cluster consists of several cells, each containing multiple machines.

For task scheduling, all physical or logical units on machines are treated as resources, including CPU, memory, IO, etc.

System Architecture

Borg uses a master-slave architecture, consisting of a BorgMaster and several Borglet nodes. The scheduler is an independent service.

BorgMaster is a logical node responsible for interacting with both external components and Borglets, as well as maintaining the internal state of the cluster. It uses Paxos to achieve multi-replication and high availability.

Borglet is the Borg proxy on each machine in the cell. It is responsible for starting/stopping tasks, managing node physical resources, and reporting status.

Scheduler is the service responsible for task scheduling. It uses the state recorded by the master to asynchronously handle task scheduling and informs the master for a secondary check.

Resource Scheduling

The scheduler is a key service in Borg. The quality of the scheduling algorithm directly affects resource utilization and is closely related to cost efficiency.

Basic Process

The scheduling algorithm has two parts:

Feasibility Check: Finds a set of machines capable of running the task.
Scoring: Selects the most suitable machine from that set.

During the feasibility check, the scheduler finds a set of machines that meet task constraints and have enough available resources. Available resources include those already allocated to lower-priority tasks that can be preempted.

During the scoring phase, the scheduler determines the suitability of each feasible machine. Scoring considers user-specific preferences but primarily depends on built-in criteria, such as minimizing the number and priority of preempted tasks, selecting machines that already have the task package, distributing tasks across different power and failure domains, and optimizing packing quality (mixing high- and low-priority tasks on a single machine to allow high-priority tasks to expand during load spikes).

The scheduler uses a cached copy of the cell state and performs the following steps repeatedly:

Retrieves state changes (including assigned and pending jobs) from the elected master and updates its local copy.
Runs a round of scheduling to assign tasks and sends assignment information to the master.
The master accepts and applies the assignments, but if they are unsuitable (e.g., based on outdated state), it waits for the scheduler’s next round.

Additional Aspects

The paper also discusses how to provide oversubscription and handle performance contention, though these are not the focus of this blog. Readers can refer to the original paper for more details.

References

https://www.cnblogs.com/hellojamest/p/16526159.html

Percolator: Large-scale Incremental Processing Using Distributed Transactions and Notifications

Thu, 28 Sep 2023 10:43:23 +0800

It has been a while since I last studied, and I wanted to learn something interesting. This time, I’ll be covering Percolator, a distributed transaction system. I won’t translate the paper or delve into detailed algorithms; I’ll just document my understanding.

Percolator and 2PC

2PC

The Two-Phase Commit (2PC) protocol involves two types of roles: Coordinator and Participant. The coordinator manages the entire process to ensure multiple participants reach a unanimous decision. Participants respond to the coordinator’s requests, completing prepare operations and commit/abort operations based on those requests.

The 2PC protocol ensures the atomicity (ACD) of a transaction but does not implement isolation (I), relying instead on single-node transactions for ACD. The coordinator is clearly a critical point, which can become a bottleneck or cause blocking if it fails.

text

 Coordinator Participant
QUERY TO COMMIT
-------------------------------->
VOTE YES/NO prepare*/abort*
<-------------------------------
commit*/abort* COMMIT/ROLLBACK
-------------------------------->
ACKNOWLEDGMENT commit*/abort*
<--------------------------------
end

Percolator

Percolator can be seen as an optimized version of 2PC, with some improvements such as:

Optimizing the use of locks by introducing primary-secondary dual-level locks, which eliminates the reliance on a coordinator.
Providing full ACID semantics and supporting MVCC (Multi-Version Concurrency Control) through a timestamp service.

Percolator Protocol Details

The Percolator system consists of three main components:

Client: The client initiating a transaction. It acts as the control center for the entire protocol and is the coordinator of the two-phase commit process.
TO (Time Observer): Responsible for assigning timestamps, providing unique and incrementing timestamps to implement MVCC.
Bigtable: Provides single-row transactions, storing data as well as some attributes for transactional control.

lock + write + data: for transactions, where lock indicates that a cell is held by a transaction, and write represents the data visibility.

notify + ack: for watcher or notifier mechanisms.

Externally, Percolator is provided to businesses through an SDK, offering transactions and R/W operations. The model is similar to Begin Txn → Sets of RW Operations → Commit or Abort or Rollback. Bigtable acts as the persistent component, hiding details about Tablet Server data sharding. Each write operation (including read-then-write) in the transaction is treated as a participant in a distributed transaction and may be dispatched to multiple Tablet Server nodes.

Algorithm Workflow

All writes in a transaction are cached on the client before being written during the commit phase. The commit phase itself is a standard two-phase commit consisting of prewrite and commit stages.

Prewrite

Obtain a timestamp from TO as the start time of the transaction.
Lock the data, marking it as held by the current transaction. If locking fails, it means the data is held by another transaction, and the current transaction fails.

The locking process utilizes the primary-secondary mechanism, where one write is chosen as the primary and all others as secondary. The secondary locks point to the primary.

Clearly, data in the prewrite phase is invisible to other transactions.

Commit

Attempt to commit the data prewritten. The commit starts by committing the primary record, whose commit time will serve as the commit time for the entire transaction. First, the lock record is checked. If the lock does not exist, it indicates that the lock from the prewrite phase has been cleaned by another transaction, causing the current transaction to fail. If the lock exists, the write column is updated to indicate that the data is visible to the system.

In an asynchronous network, single-node failures and network delays are common. The algorithm must detect and clean up these locks to avoid deadlocks. Therefore, in the commit phase, if a lock is found to be missing, it means that an issue occurred with a participant, and the current transaction must be cleaned.

After successfully committing, clean up the lock record. Lock cleanup can be done asynchronously.

These designs eliminate the dependency on a centralized coordinator. Previously, a centralized service was required to maintain information about all transaction participants. In this algorithm, the primary-secondary lock and the write column achieve the same goal. The write column indicates the visibility and version chain of the data, while the lock column shows which transaction holds the data. The primary-secondary locks record the logical relationship among participants. Thus, committing the primary record becomes the commit point for the entire transaction. Once the primary is committed, all secondary records can be asynchronously committed by checking the corresponding primary record’s write column.

Snapshot Isolation

Two-phase commit ensures the atomicity of a transaction. On top of that, Percolator also provides snapshot isolation. In simple terms, snapshot isolation requires that committed transactions do not cause data conflicts and that read operations within a transaction satisfy snapshot reads. By leveraging the transaction start time and the primary commit time, a total ordering among transactions can be maintained, solving these issues naturally.

Deadlock Issues in Asynchronous Networks

As mentioned earlier, in an asynchronous network, single-node failures and network delays are common. The algorithm must clean up locks to prevent deadlocks when such failures are detected. The failure detection strategy can be as simple as a timeout, causing the current transaction to fail. When a node fails and then recovers, its previous transaction has already failed, and the relevant lock records must be cleaned up. Lock cleanup can be asynchronous; for example, during the prewrite phase, if a record’s lock column is found to be non-empty, its primary lock can be checked. If the primary lock is not empty, it means the transaction is incomplete, and the lock can be cleaned up; if empty, the transaction has committed, and the data should be committed and the lock cleaned (RollForward).

Notification Mechanism

A notification mechanism is crucial for state observation and linkage in asynchronous systems, but it is not the focus of this article.

Percolator in TiDB

Based on our analysis above, Percolator is an optimized 2PC distributed transaction implementation, relying on a storage engine that supports single-node transactions.

Let’s briefly look at how TiDB uses Percolator to implement distributed transactions.

The architecture of TiDB and TiKV is shown above. Data from relational tables in TiDB is ultimately mapped to KV pairs in TiKV. TiKV is a distributed KV store based on Raft and RocksDB. RocksDB supports transactional operations on KV pairs.

Thus, the transaction path in TiDB is as follows: a relational table transaction is converted into a set of KV transactions, which are executed based on Percolator to achieve relational table transaction operations.

Of course, it cannot provide the same transactional semantics and performance guarantees as a single-node TP database. However, a shared-nothing architecture has its own advantages, which may make this trade-off acceptable.

References

Engineering Practice of Two-Phase Commit

PolarDB Database Kernel Monthly Report

Percolator: Online Incremental Processing System (Chinese Translation)

Percolator: Online Incremental Processing System (Chinese Translation) | A Small Bird

Two-Phase Commit - Wikipedia

Percolator and TiDB Transaction Algorithm

Two-Phase Commit | OceanBase Learning Guide

TiDB Architecture Overview

Dynamo: Amazon’s Highly Available Key-value Store

Tue, 01 Aug 2023 16:15:29 +0800

An old paper by AWS, Dynamo has been in the market for a long time, and the architecture has likely evolved since the paper’s publication. Despite this, the paper was selected as one of the SIGMOD best papers of the year, and there are still many valuable lessons to learn.

Design

Dynamo is a NoSQL product that provides a key-value storage interface. It emphasizes high availability rather than consistency, which leads to differences in architectural design and technical choices compared to other systems.

Technical Details

Dynamo has many aspects that may be considered problematic from a technical perspective, such as the NWR (N-W-R) approach. However, given Dynamo’s long track record in production, these issues may have been resolved over time, though the paper is not explicit about this. For now, let’s discuss some of the aspects I found noteworthy:

Data Partitioning

Dynamo uses a consistent hashing algorithm. Traditional consistent hashing employs a hash ring to address the problem of extensive rehashing when nodes are added or removed, but it cannot avoid issues like data skew and performance imbalance caused by heterogeneous machines. In practice, Dynamo introduces virtual nodes into the hash ring, which elegantly solves these problems.

Data Write Challenges

Most storage systems ensure a certain level of consistency during writes, trading off lower write performance for reduced read complexity. However, Dynamo takes a different approach.

Dynamo’s design goal is to provide a highly available key-value store that ensures always writable operations while only guaranteeing eventual consistency. To achieve this, Dynamo pushes data conflict resolution to the read operation, ensuring that writes are never rejected.

There are two key issues to consider here:

Data Conflict Resolution: Concurrent reads and writes to the same key by multiple clients can easily lead to data conflicts. Since Dynamo only provides eventual consistency, data on different nodes in the Dynamo ring might be inconsistent.
- Dynamo uses vector clocks to keep track of data versions and merges them during reads to resolve conflicts.
Replica Data Gaps: Since Dynamo employs the NWR gossip protocol, it is theoretically possible that none of the nodes hold the complete data set, requiring synchronization between replicas.
- Dynamo uses an anti-entropy process to address this, employing Merkle Trees to efficiently detect inconsistencies between replicas and minimize the amount of data transferred.

The table in the paper clearly shows the aspects considered during Dynamo’s development and the corresponding technical choices. For more information, refer to the original paper.

References

Dynamo’s Implementation and Decentralization

Dynamo’s Flawed Architecture (Translation) by Tim Yang

Dynamo: A Flawed Architecture | Hacker News

MIT6.824 AuroraDB

Tue, 01 Aug 2023 16:11:54 +0800

This article introduces the design considerations of AWS’s database product, Aurora, including storage-compute separation, single-writer multi-reader architecture, and quorum-based NRW consistency protocol. The article also mentions how PolarDB was inspired by Aurora, with differences in addressing network bottlenecks and system call overhead.

Aurora is a database product provided by AWS, primarily aimed at OLTP business scenarios.

In terms of design, there are several aspects worth noting:

The design premise of Aurora is that with databases moving to the cloud, thanks to advancements in cloud infrastructure, the biggest bottleneck for databases has shifted from compute and storage to the network. This was an important premise for AWS when designing Aurora. Based on this premise, Aurora revisits the concept of “Log is Database”, pushing only the RedoLog down to the storage layer.
Storage-compute separation: The database storage layer interfaces with a distributed storage system, which provides reliability and security guarantees. The compute and storage layers can scale independently. The storage system provides a unified data view to the upper layers, significantly improving the efficiency of core functions and operations (such as backup, data recovery, and high availability).
Interesting reliability guarantees: For example, quorum-based NRW consistency protocol, where read and write operations on storage nodes require majority voting, ensures dual-AZ level fault tolerance. Sharding is used to reduce failure handling time, improving the SLA. Most reads occur during database recovery when the current state needs to be restored.
Single-writer multi-reader: Unlike NewSQL products with a shared-nothing architecture, Aurora provides only a single write node. This simplifies data consistency guarantees since the single write node can use the RedoLog LSN as a logical clock to maintain the partial order of data updates. By pushing the RedoLog to all nodes and applying these operations in order, consistency can be achieved.
Transaction implementation: Since the storage system provides a unified file view to the upper layer, Aurora’s transaction implementation is almost the same as that of a single-node transaction algorithm and can provide similar transaction semantics. NewSQL transactions are generally implemented via distributed transactions based on 2PC.
Background acceleration for foreground processing: Similar to the approach in LevelDB, storage nodes try to make some operations asynchronous (such as log apply) to improve user-perceived performance. These asynchronous operations maintain progress using various LSNs, such as VLSN, commit-LSN, etc.

Interestingly, although PolarDB’s design was inspired by Aurora, its architectural design considers that the network is not the bottleneck but rather that various system calls through the OS slow down overall speed. Given the instability of Alibaba Cloud’s storage system at that time, PolarStore was introduced, using hardware and FUSE-based storage technology to bypass or optimize system calls. Now that Pangu has improved significantly in terms of both stability and performance, it makes sense to weaken the role of PolarStore. I think this reasoning makes sense.

Additionally, why did they choose to use NRW instead of a consensus protocol like Raft? For now, it seems that NRW has one less round of network communication compared to Raft, which might be the reason.

References

MIT6.824 Chain Replication

Wed, 08 Feb 2023 23:05:57 +0800

This post provides a brief overview of the Chain Replication (CR) paper, which introduces a simple but effective algorithm for providing linearizable consistency in storage services. For those interested in the detailed design, it’s best to refer directly to the original paper.

Introduction

In short, the Chain Replication (CR) paper presents a replicated state machine algorithm designed for storage services that require linearizable consistency. It uses a chain replication method to improve throughput and relies on multiple replicas to ensure service availability.

The design of the algorithm is both simple and elegant. CR splits the replication workload across all nodes in the chain, with each node responsible for forwarding updates to its successor. Write requests are propagated from the head node to the tail, while read requests are served by the tail node.

To maintain relationships between nodes in the chain, Chain Replication introduces a Master service responsible for managing node configurations and handling node failures.

Failure Handling

Head Failure: If the head node fails, any pending or unprocessed requests are lost, but linearizable consistency remains unaffected. The second node in the chain is promoted to the new head.
Tail Failure: If the tail node fails, the second-to-last node becomes the new tail, and pending requests from the original tail are committed.
Middle Node Failure: When a middle node fails, the chain is reconnected in a manner similar to linked list operations. The previous node (Node_pre) is linked directly to the next node (Node_next). To ensure that no requests are lost during this failure, each CR node maintains a SendReqList that records all requests forwarded to its successor. Since requests are propagated from head to tail, Node_pre only needs to send Node_next any missing data. When the tail node receives a request, it marks it as committed, and an acknowledgment (Ack(req)) is sent back from the tail to the head, removing the request from each node’s SendReqList as the acknowledgment propagates.

Pros and Cons

The main advantages of Chain Replication include:

High Throughput: By distributing the workload across all nodes, CR effectively increases the throughput of a single node.
Balanced Load: Each node has a similar workload, resulting in balanced utilization.
Simplicity: The overall design is clean and straightforward, making it easier to implement.

However, there are some clear disadvantages:

Bottlenecks: If a node in the chain processes requests slowly, it will delay the entire chain’s processing.
Read Limitations: Only the head and tail nodes can serve requests efficiently. The data in the middle nodes is mostly there for replication purposes and not directly utilized for serving requests. However, the CRAQ (Chain Replication with Asynchronous Queries) variant allows middle nodes to serve read-only requests, similar to Raft’s Read Index, which can help alleviate this limitation.

References

MIT6.824-ZooKeeper

Tue, 03 Jan 2023 23:49:41 +0800

This article mainly discusses the design and practical considerations of the ZooKeeper system, such as wait-free and lock mechanisms, consistency choices, system-provided APIs, and specific semantic decisions. These trade-offs are the most insightful aspects of this article.

Positioning

ZooKeeper is a wait-free, high-performance coordination service for distributed applications. It supports the coordination needs of distributed applications by providing coordination primitives (specific APIs and data models).

Design

Keywords

There are two key phrases in ZooKeeper’s positioning: high performance and distributed application coordination service.

ZooKeeper’s high performance is achieved through wait-free design, local reads from multiple replicas, and the watch mechanism:

Wait-free requests are handled asynchronously, which may lead to request reordering, making the state machine different from the real-time sequence. ZooKeeper provides FIFO client order guarantees to manage this. Additionally, asynchronous handling is conducive to batch processing and pipelining, further improving performance.
The watch mechanism notifies clients of updates when a znode changes, reducing the overhead of clients querying local caches.
Local reads from multiple replicas: ZooKeeper uses the ZAB protocol to achieve data consensus, ensuring that write operations are linearizable. Read requests, however, are served locally from replicas without going through the ZAB consensus protocol, which provides serializability and might return stale data, improving performance.

The distributed application coordination service refers to the data model and API semantics provided by ZooKeeper, allowing distributed applications to freely use them to fulfill coordination needs such as group membership and distributed locking.

Data Model and API

ZooKeeper provides an abstraction of data nodes called znodes, which are organized through a hierarchical namespace. ZooKeeper offers two types of znodes: regular and ephemeral. Each znode stores data and is accessed using standard UNIX filesystem paths.

In practice, znodes are not designed for general data storage. Instead, znodes map to abstractions in client applications, often corresponding to metadata used for coordination.

In other words, when coordinating through ZooKeeper, utilize the metadata associated with znodes instead of treating them as mere data storage. For example, znodes associate metadata with timestamps and version counters, allowing clients to track changes to the znodes and perform conditional updates based on the znode version.

Essentially, this data model is a simplified file system API that supports full data reads and writes. Users implement distributed application coordination using the semantics provided by ZooKeeper.

The difference between regular and ephemeral znodes is that ephemeral nodes are automatically deleted when the session ends.

Clients interact with ZooKeeper through its API, and ZooKeeper manages client connections through sessions. In a session, clients can observe state changes that reflect their operations.

CAP Guarantees

ZooKeeper provides CP (Consistency and Partition Tolerance) guarantees. For instance, during leader election, ZooKeeper will stop serving requests until a new leader is elected, ensuring consistency.

Implementation

ZooKeeper uses multiple replicas to achieve high availability.

In simple terms, ZooKeeper’s upper layer uses the ZAB protocol to handle write requests, ensuring linearizability across replicas. Reads are processed locally, ensuring sequential consistency. The underlying data state machine is stored in the replicated database (in-memory) and Write-Ahead Log (WAL) on ZooKeeper cluster machines, with periodic snapshots to ensure durability. The entire in-memory database uses fuzzy snapshots and WAL replay to ensure crash safety and fast recovery after a crash.

The advantage of fuzzy snapshots is that they do not block online requests.

Interaction with Clients

Update operations will notify and clear the relevant znode’s watch.
Read requests are processed locally, and the partial order of write requests is defined by zxid. Sequential consistency is ensured, but reads may be stale. ZooKeeper provides the sync operation, which can mitigate this to some extent.
When a client connects to a new ZooKeeper server, the maximum zxid is compared. The outdated ZooKeeper server will not establish a session with the client.
Clients maintain sessions through heartbeats, and the server handles requests idempotently.

References

ZooKeeper Paper

MIT6.824-ZooKeeper FAQ

MIT6.824-RaftKV

Fri, 15 Apr 2022 10:49:57 +0800

Earlier, I looked at the code of Casbin-Mesh because I wanted to try GSOC. Casbin-Mesh is a distributed Casbin application based on Raft. This RaftKV in MIT6.824 is quite similar, so I took the opportunity to write this blog.

Lab Overview

Lab 03 involves building a distributed KV service based on Raft. We need to implement the server and client for this service.

The structure of RaftKV and the interaction between its modules are shown below:

Compared to the previous lab, the difficulty is significantly lower. For implementation, you can refer to this excellent implementation, so I won’t elaborate too much.

Let’s talk about Raft and its interactions with clients.

Routing and Linearizability

To build a service that allows client access on top of Raft, the issues of routing and linearizability must first be addressed.

Routing

Raft is a Strong Leader consensus algorithm, and read and write requests usually need to be executed by the Leader. When a client queries the Raft cluster, it typically randomly selects a node. If that node is not the Leader, it returns the Leader information to the client, and the client redirects the request to the Leader.

Linearizability

Currently, Raft only supports At Least Once semantics. For a single client request, the Raft state machine may apply the command multiple times, which is particularly unsuitable for consensus-based systems.

To achieve linearizability, it is clear that requests need to be made idempotent.

A basic approach is for the client to assign a unique UID to each request, and the server maintains a session using this UID to cache the response of successful requests. When a duplicate request arrives at the server, it can respond directly using the cached response, thus achieving idempotency.

Of course, this introduces the issue of session management, but that is not the focus of this article.

Read-Only Optimization

After solving the above two problems, we have a usable Raft-based service.

However, we notice that whether it’s a read or write request, our application needs to go through a round of AppendEntries communication initiated by the Leader. It also requires successful quorum ACKs and additional disk write operations before the log is committed, after which the result can be returned to the client.

Write operations change the state machine, so these are necessary steps for write requests. However, read operations do not change the state machine, and we can optimize read requests to bypass the Raft log, reducing the overhead of synchronous write operations on disk IO.

The problem is that without additional measures, read-only query results that bypass the Raft log may become stale.

For example, if the old cluster Leader and a new Leader’s cluster are partitioned, queries made to the old Leader could be outdated.

The Raft paper mentions two methods to bypass the Raft log and optimize read-only requests: Read Index and Lease Read.

Read Index

The Read Index approach needs to address several issues:

Committed logs from the old term

For example, if the old Leader commits a log but crashes before sending heartbeats, other nodes will elect a new Leader. According to the Raft paper, the new Leader does not proactively commit logs from the old Leader.

To solve this, a no-op log is committed after a new Leader is elected to commit the old log.

Gap between commitIndex and appliedIndex

Introduce a readIndex variable, where the Leader saves the current commitIndex in a local variable called readIndex. This acts as a boundary for applying the log, and when a read-only request arrives, the log must be applied up to the position recorded by readIndex before the Leader can query the state machine to provide read services.

Ensure no Leader change when providing read-only services

To achieve this, after receiving a read request, the Leader first sends a heartbeat and needs to receive quorum ACKs to ensure there is no other Leader with a higher term, thus ensuring that readIndex is the highest committed index in the cluster.

For the specific process and optimizations like Batch and Follower Read, refer to the author’s PhD dissertation on Raft.

Lease Read

The Read Index approach only optimizes the overhead of disk IO, but still requires a round of network communication. However, this overhead can also be optimized, leading to the Lease Read approach.

The core idea of Lease Read is to use the fact that a Leader Election requires at least one ElectionTimeout time period. During this period, the system will not conduct a new election, thereby avoiding Leader changes when providing read-only services. We can use clocks to optimize network IO.

Implementation

To let the clock replace network communication, we need an additional lease mechanism. Once the Leader’s Heartbeat is approved by a quorum, the Leader can assume that no other node can become Leader during the ElectionTimeout period, and it can extend its lease accordingly. While holding the lease, the Leader can directly serve read-only queries without extra network communication.

However, there may be clock drift among servers, which means Followers cannot ensure that the Leader will not time out during the lease. This introduces the critical design for Lease Read: what strategy should be used to extend the lease?

The paper assumes that $ClockDrift$ is bounded, and when a heartbeat successfully updates the lease, the lease is extended to $start + rac{ElectionTimeout}{ClockDriftBound}$.

$ClockDriftBound$ represents the limit of clock drift in the cluster, but discovering and maintaining this limit is challenging due to many real-time factors that cause clock drift.

For instance, garbage collection (GC), virtual machine scheduling, cloud machine scaling, etc.

In practice, some safety is usually sacrificed for Lease Read performance. Generally, the lease is extended to $StartTime + ElectionTimeout - \Delta{t}$, where $\Delta{t}$ is a positive value. This reduces the lease extension time compared to ElectionTimeout, trading off between network IO overhead and safety.

Summary

When building a Raft-based service, it is crucial to design routing and idempotency mechanisms for accessing the service.

For read-only operations, there are two main optimization methods: Read Index and Lease Read. The former optimizes disk IO during read operations, while the latter uses clocks to optimize network IO.

References

Implementation Doc

Raft Paper

MIT6.824 Official

Consensus: Bridging Theory and Practice - zh

Tikv Lease-Read

MIT6.824-Raft

Mon, 21 Feb 2022 01:26:46 +0800

Finally, I managed to complete Lab 02 during this winter break, which had been on hold for quite some time. I was stuck on one of the cases in Test 2B for a while. During the winter break, I revisited the implementations from experts, and finally completed all the tasks, so I decided to document them briefly.

Algorithm Overview

The basis of consensus algorithms is the replicated state machine, which means that executing the same deterministic commands in the same order will eventually lead to a consistent state. Raft is a distributed consensus algorithm that serves as an alternative to Paxos, making it easier to learn and understand compared to Paxos.

The core content of the Raft algorithm can be divided into three parts: $Leader Election + Log Replication + Safety$.

Initially, all nodes in the cluster start as Followers. If a Follower does not receive a heartbeat from the Leader within a certain period, it becomes a Candidate and triggers an election, requesting votes from the other Followers. The Candidate that receives a majority of votes becomes the Leader.

Raft is a strong leader and strongly consistent distributed consensus algorithm. It uses Terms as a logical clock, and only one Leader can exist in each term. The Leader needs to send heartbeats periodically to maintain its status and to handle log replication.

When replicating logs, the Leader first replicates the log to other Followers. Once a majority of the Followers successfully replicate the log, the Leader commits the log.

Safety mainly consists of five parts, with two core elements relevant to the implementation. One is the leader’s append-only rule, which means it cannot modify committed logs. The other is election safety, preventing split-brain scenarios and ensuring that the new Leader has the most up-to-date log.

For more details, please refer to the original paper.

Implementation Ideas

The implementation largely follows an excellent blog post (see references), and many algorithm details are also provided in Figure 2 of the original paper, so I will only focus on aspects that need attention when implementing each function.

Leader Election

Triggering Election + Handling Election Results

The election is initiated by launching multiple goroutines to send RPC requests to other nodes in the background. Therefore, when handling RPC responses, it is necessary to confirm that the current node is a Candidate and that the request is not outdated, i.e., rf.state == Candidate && req.Term == rf.currentTerm. If the election is successful, the node should immediately send heartbeats to notify other nodes of the election result.

If a failed response is received with resp.Term > rf.currentTerm, the node should switch to the Follower state, update the term, and reset voting information.

In fact, whenever the term is updated, the voting information needs to be reset. If the votedFor information is not reset, some tests will fail.

Request Vote RPC

First, filter outdated requests with req.Term < rf.currentTerm and ignore duplicate voting requests for the current term. Then, follow the algorithm’s logic to process the request. Note that if the node successfully grants the vote, it should reset the election timer.

Resetting the election timeout only when granting a vote helps with liveness in leader elections under unstable network conditions.

State Transition

When switching roles, be mindful of handling the state of different timers (stop or reset). When switching to Leader, reset the values of matchIndex and nextIndex.

Log Replication

Log replication is the core of the Raft algorithm, and it requires careful attention.

My implementation uses multiple replicator and applier threads for asynchronous replication and application.

Log Replication RPC

First, filter outdated requests with req.Term < rf.currentTerm. Then, handle log inconsistencies, log truncation, and duplicate log entries before replicating logs and processing commitIndex.

Trigger Log Replication + Handle Request Results

Determine whether to replicate logs directly or send a snapshot before initiating replication.

The key point in handling request results is how to update matchIndex, nextIndex, and commitIndex.

matchIndex is used to record the latest log successfully replicated on other nodes, while nextIndex records the next log to be sent to other nodes. commitIndex is updated by sorting matchIndex and determining whether to trigger the applier to update appliedIndex.

If the request fails, nextIndex should be decremented, or the node should switch to the Follower state.

Asynchronous Apply

This is essentially a background goroutine controlled by condition variables and uses channels for communication. Each time it is triggered, it sends log[lastApplied:commitIndex] to the upper layer and updates appliedIndex.

Persistence

Persist the updated attributes that need to be saved to disk in a timely manner.

Install Snapshot

The main components are the Snapshot triggered by the Leader and the corresponding RPC. When applying a Snapshot, determine its freshness and update log[0], appliedIndex, and commitIndex.

Pitfalls

Defer

The first pitfall is related to the defer keyword in Go. I like to use the defer keyword at the beginning of an RPC to directly print some data from the node: defer Dprintf("%+v", raft.currentTerm). This way, the log can be printed at the end of the call. However, the content to be printed is fixed at the time the defer statement is executed. The correct usage should be defer func(ID int) { Dprintf("%+v", id) }().

Log Dummy Header

It is best to reserve a spot in the log for storing the index and term of the snapshot to avoid a painful refactor of the Snapshot section later.

Lock

Refer to the guidance on locking suggestions. Use a single coarse-grained lock rather than multiple locks. Correctness of the algorithm is more important than performance. Avoid holding locks while sending RPCs or using channels, as it may lead to timeouts.

References

Raft Wikipedia

Raft Official Website

Raft Paper

MIT6.824 Official

Potato’s Implementation Doc

DFS-Haystack

Wed, 06 Oct 2021 22:44:01 +0800

The primary project in my group is a distributed file system (DFS) that provides POSIX file system semantics. The approach to handle “lots of small files” (LOSF) is inspired by Haystack, which is specifically designed for small files. I decided to read through the Haystack paper and take some notes as a learning exercise.

These notes are not an in-depth analysis of specific details but rather a record of my thoughts on the problem and design approach.

Introduction

Haystack is a storage system designed by Facebook for small files. In traditional DFS, file addressing typically involves using caches to store metadata, reducing disk interaction and improving lookup efficiency. For each file, a separate set of metadata must be maintained, with the volume of metadata depending on the number of files. In high-concurrency scenarios, metadata is cached in memory to reduce disk I/O.

With a large number of small files, the volume of metadata becomes significant. Considering the maintenance overhead of in-memory metadata, this approach becomes impractical. Therefore, Haystack was developed specifically for small files, with the core idea of aggregating multiple small files into a larger one to reduce metadata.

Background

The “small files” in the paper specifically refer to image data.

Facebook, as a social media company, deals heavily with image uploads and retrieval. As the business scaled, it became necessary to have a dedicated service to handle the massive, high-concurrency requests for image reads and writes.

In the social networking context, this type of data is characterized as written once, read often, never modified, and rarely deleted. Based on this, Facebook developed Haystack to support image sharing services.

Design

Traditional Design

The paper describes two historical designs: CDN-based and NAS-based solutions.

CDN-based Solution

The core of this solution is to use CDN (Content Delivery Network) to cache hot image data, reducing network transmission.

This approach optimizes access to hot images but also has some issues. Firstly, CDN is expensive and has limited capacity. Secondly, image sharing includes many less popular images, which leads to the long tail effect, slowing down access.

CDNs are generally used to serve static data and are often pre-warmed before an event, making them unsuitable as an image cache service. Many less popular images do not enter the CDN, leading to the long tail effect.

NAS-based Solution

This was Facebook’s initial design and is essentially a variation of the CDN-based solution.

They introduced NAS (Network Attached Storage) for horizontal storage expansion, incorporating file system semantics, but disk I/O remained an issue. Similar to local files, reading uncached data requires at least three disk I/O operations:

Read directory metadata into memory
Load the inode into memory
Read the content of the file

PhotoStore was used as a caching layer to store some metadata like file handles to speed up the addressing process.

The NAS-based design did not solve the fundamental issue of excessive metadata that could not be fully cached. When the number of files reaches a certain threshold, disk I/O becomes inevitable.

The fundamental issue is the one-to-one relationship between files and addressing metadata, causing the volume of metadata to change with the number of files.

Thus, the key to optimization is changing the one-to-one relationship between files and metadata, reducing the frequency of disk I/O during addressing.

Haystack-based Solution

The core idea of Haystack is to aggregate multiple small files into a larger one, maintaining a single piece of metadata for the large file. This changes the mapping between metadata and files, making it feasible to keep all metadata in memory.

Metadata is maintained only for the aggregated file, and the position of small files within the large file is maintained separately.

Implementation

Haystack mainly consists of three components: Haystack Directory, Haystack Cache, and Haystack Store.

File Mapping and Storage

File data is ultimately stored on logical volumes, each of which corresponds to multiple physical volumes across machines.

Users first access the Directory to obtain access paths and then use the URL generated by the Directory to access other components to retrieve the required data.

Components

Haystack Directory

This is Haystack’s access layer, responsible for file addressing and access control.

Read and write requests first go through the Directory. For read requests, the Directory generates an access URL containing the path: http://{cdn}/{cache}/{machine id}/{logicalvolume,Photo}. For write requests, it provides a volume to write into.

The Directory has four main functions:

Load balancing for read and write requests.
Determine request access paths (e.g., CDN or direct access) and generate access URLs.
Metadata and mapping management, e.g., logical attributes to volume mapping.
Logical volume read/write management, where volumes can be read-only or write-enabled.

This design is based on the data characteristics: “write once, read more.” This setup improves concurrency.

The Directory stores metadata such as file-to-volume mappings, logical-to-physical mappings, and volume attributes (size, owner, etc.). It relies on a distributed key-value store and a cache service to ensure low latency and high availability.

Proxy, Metadata Mapping, Access Control

Haystack Cache

The Cache layer optimizes addressing and image retrieval. The core design is the Cache Rule, which determines what data should be cached and how to handle cache misses.

Images are cached if they meet these criteria:

The request is directly from a user, not from a CDN.
The photo is retrieved from a write-enabled store machine.

If a cache miss occurs, the Cache fetches the image from the Store and pushes it to both the user and the CDN.

The caching policy is based on typical access patterns.

Haystack Store

The Store layer is responsible for data storage operations.

The addressing abstraction is: filename + offset => logical volume id + offset => data.

Multiple physical volumes constitute a logical volume. In the Store, small files are encapsulated as Needles managed by physical volumes.

Needles represent a way to encapsulate small files and manage volume blocks.

Store data is accessed at the Needle level. To speed up addressing, a memory map is used: key/alternate key => needle's flag/offset/other attributes.

These maps are persisted in Index Files on disk to provide a checkpoint for quick metadata recovery after a crash.

Each volume maintains its own in-memory mapping and index file.

When updating the in-memory mapping (e.g., adding or modifying a file), the index file is updated asynchronously. Deleted files are only marked as deleted, not removed from the index file.

The index serves as a lookup aid. Needles without an index can still be addressed, making the asynchronous update and index retention strategy feasible.

Workloads

Read

(Logical Volume ID, key, alternate key, cookies) => photo

For a read request, Store queries the in-memory mapping for the corresponding Needle. If found, it fetches the data from the volume and verifies the cookie and integrity; otherwise, it returns an error.

Cookies are randomly generated strings that prevent malicious attacks.

Write

(Logical Volume ID, key, alternate key, cookies, data) => result

Haystack only supports appending data rather than overwriting. When a write request is received, Store asynchronously appends data to a Needle and updates the in-memory mapping. If it’s an existing file, the Directory updates its metadata to point to the latest version.

Older volumes are frozen as read-only, and new writes are appended, so a larger offset indicates a newer version.

Delete

Deletion is handled using Mark Delete + Compact GC.

Fault Tolerance

Store ensures fault tolerance through monitoring + hot backup. Directory and Cache use Raft-like consistency algorithms for data replication and availability.

Optimization

The main optimizations include: Compaction, Batch Load, and In-Memory processing.

Summary

Key abstraction optimizations include asynchronous processing, batch operations, and caching.
Identifying the core issues, such as metadata management burden for a large number of small files, is crucial.

References

Finding a needle in Haystack: Facebook’s photo storage

MIT6.824 Bigtable

Thu, 16 Sep 2021 22:54:59 +0800

I recently found a translated version of the Bigtable paper online and saved it, but hadn’t gotten around to reading it. Lately, I’ve noticed that Bigtable shares many design similarities with a current project in our group, so I took some time over the weekend to read through it.

This is the last of Google’s three foundational distributed system papers, and although it wasn’t originally part of the MIT6.824 reading list, I’ve categorized it here for consistency.

As with previous notes, I won’t dive deep into the technical details but will instead focus on the design considerations and thoughts on the problem.

Introduction

Bigtable is a distributed structured data storage system built on top of GFS, designed to store large amounts of structured and semi-structured data. It is a NoSQL data store that emphasizes scalability and performance, as well as reliable fault tolerance through GFS.

Design Goal: Wide Applicability, Scalability, High Performance, High Availability

Data Model

Bigtable’s data model is No Schema and provides a simple model. It treats all data as strings, with encoding and decoding handled by the application layer.

Bigtable is essentially a sparse, distributed, persistent multidimensional sorted Map. The index of the Map is composed of Row Key, Column Key, and TimeStamp, and the value is an unstructured byte array.

go


// Mapping abstraction
(row:string, column:string, time:int64) -> string
// A Row Key is essentially a multi-dimensional structure composed of {Row, Column, Timestamp}.

The paper describes the data model as follows:

A Bigtable is a sparse, distributed, persistent multidimensional sorted map.

Sparse means that columns in the same table can be null, which is quite common.

Row	Columns
Row1	{ID, Name, Phone}
Row2	{ID, Name, Phone, Address}
Row3	{ID, Name, Phone, Email}

Distributed refers to scalability and fault tolerance, i.e., Replication and Sharding. Bigtable leverages GFS replicas for fault tolerance and uses Tablet for partitioning data to achieve scalability.

Persistent Multidimensional Sorted indicates data is eventually persisted, and Bigtable optimizes write and read latency with WAL and LSM.

The open-source implementation of Bigtable is HBase, a row and column database.

Rows

Bigtable organizes data using lexicographic order of row keys. A Row Key can be any string, and read and write operations are atomic at the row level.

Lexicographic ordering helps aggregate related row records. MySQL achieves atomic row operations using an undo log.

Column Family

A set of column keys forms a Column Family, where the data often shares the same type.

A column key is composed of Column Family : Qualifier. The column family’s name must be a printable string, whereas the qualifier name can be any string.

The paper mentions:

Access control and both disk and memory accounting are performed at the column-family level.

This is because business users tend to retrieve data by columns, e.g., reading webpage content. In practice, column data is often compressed for storage. Thus, the Column Family level is a more suitable level for access control and resource accounting than rows.

TimeStamp

The timestamp is used to maintain different versions of the same data, serving as a logical clock. It is also used as an index to query data versions.

Typically, timestamps are sorted in reverse chronological order. When the number of versions is low, a pointer to the previous version is used to maintain data versioning; when the number of versions increases, an index structure is needed. TimeStamp indexing inherently requires range queries, so a sortable data structure is appropriate for indexing. Extra version management increases maintenance overhead, usually handled by limiting the number of data versions and garbage collecting outdated versions.

Tablet

Bigtable uses a range-based data sharding strategy, and Tablet is the basic unit for data sharding and load balancing.

A tablet is a collection of rows, managed by a Tablet Server. Rows in Bigtable are ultimately stored in a tablet, which is split or merged for load balancing among Tablet Servers.

Range-based sharding is beneficial for range queries, compared to hash-based sharding.

SSTable

SSTable is a persistent, sorted, immutable Map. Both keys and values are arbitrary byte arrays.

A tablet in Bigtable is stored in the form of SSTable files.

SSTable is organized into data blocks (typically 64KB each), with an index for fast data lookup. Data is read by first reading the index, searching the index, and then reading the data block.

API

The paper provides an API that highlights the differences from RDBMS.

cpp


// Writing to Bigtable
// Open the table
Table *T = OpenOrDie("/bigtable/web/webtable");
// Write a new anchor and delete an old anchor
RowMutation r1(T, "com.cnn.www");
r1.Set("anchor:www.c-span.org", "CNN");
r1.Delete("anchor:www.abc.com");
Operation op;
Apply(&op, &r1);

// Reading from Bigtable
Scanner scanner(T);
ScanStream *stream;
stream = scanner.FetchColumnFamily("anchor");
stream->SetReturnAllVersions();
scanner.Lookup("com.cnn.www");
for (; !stream->Done(); stream->Next()) {
 printf("%s %s %lld %s\n",
 scanner.RowName(),
 stream->ColumnName(),
 stream->MicroTimestamp(),
 stream->Value());
}

Architecture Design

External Components

Bigtable is built on top of other components in Google’s ecosystem, which significantly simplifies Bigtable’s design.

GFS

GFS is Bigtable’s underlying storage, providing replication and fault tolerance.

Refer to the previous notes for details.

Chubby

Chubby is a highly available distributed lock service that provides a namespace, where directories and files can serve as distributed locks.

High availability means maintaining multiple service replicas, with consistency ensured via Paxos. A lease mechanism prevents defunct Chubby clients from holding onto locks indefinitely.

Why Chubby? What is its role?

Stores Column Family information
Stores ACL (Access Control List)
Stores root metadata for the Root Tablet location, which is essential for Bigtable startup.

Bigtable uses a three-layer B+ tree-like structure for metadata. The Root Tablet location is in Chubby, which helps locate other metadata tablets, which in turn store user Tablet locations.
Tablet Server lifecycle monitoring

Each Tablet Server creates a unique file in a designated directory in Chubby and acquires an exclusive lock on it. The server is considered offline if it loses the lock.

In summary, Chubby’s functionality can be categorized into two parts. One is to store critical metadata as a highly available node, while the other is to manage the lifecycle of storage nodes (Tablet Servers) using distributed locking.

In GFS, these responsibilities are handled by the Master. By offloading them to Chubby, Bigtable simplifies the Master design and reduces its load.

Conceptually, Chubby can be seen as part of the Master node.

Internal Components

Master

Bigtable follows a Master-Slave architecture, similar to GFS and MapReduce. However, unlike GFS, Bigtable relies on Chubby and Tablet Servers to store metadata, with the Master only responsible for orchestrating the process and not storing tablet locations.

Responsibilities include Tablet allocation, garbage collection, monitoring Tablet Server health, load balancing, and metadata updates. The Master requires:

All Tablet information to determine allocation and distribution.

Tablet Server status information to decide on allocations.

Tablet Server

Tablet Servers manage tablets, handling reads and writes, splitting and merging tablets when necessary.

Metadata is not stored by the Master. Clients interact directly with Chubby and Tablet Servers for reading data. Tablets are split by Tablet Servers, and Master may not be notified instantly. WAL+retry mechanisms should be employed to ensure operations aren’t lost.

Client SDK

The client SDK is the entry point for businesses to access Bigtable. To minimize metadata lookup overhead, caching and prefetching are used to reduce the frequency of network interactions, making use of temporal and spatial locality.

Caching may introduce inconsistency issues, which require appropriate solutions, such as retries during inconsistent states.

Storage Design

Mapping and Addressing

Bigtable data is uniquely determined by a (Table, Row, Column) tuple, stored in tablets, which in turn are stored in SSTable format on GFS.

Tablets are logical representations of Bigtable’s on-disk entity, managed by Tablet Servers.

Bigtable uses Root Tablet + METADATA Table for addressing. The Root Tablet location is stored in Chubby, while the METADATA Table is maintained by Tablet Servers.

The Root Tablet stores the location of METADATA Tablets, and each METADATA Tablet contains the location of user tablets.

METADATA Table Row: (TableID, encoding of last row in Tablet) => Tablet Location

The system uses a B+ tree-like three-layer structure to maintain tablet location information.

Scheduling and Monitoring

Scheduling

Scheduling involves Tablet allocation and load balancing.

A Tablet can only be assigned to one Tablet Server at any given time. The Master maintains Tablet Server states and sends allocation requests as needed.

The Master does not maintain addressing information but holds Tablet Server states (including tablet count, status, and available resources) for scheduling.

Monitoring

Monitoring is carried out by Chubby and the Master.

Each Tablet Server creates a unique file in a Chubby directory and acquires an exclusive lock. When the Tablet Server disconnects and loses its lease, the lock is released.

The unique file determines whether a Tablet Server is active, and the Master may delete the file as needed. In cases of network disconnection, the Tablet Server will try to re-acquire the exclusive lock if the file still exists. If the file doesn’t exist, the disconnected Tablet Server should automatically leave the cluster.

The Master ensures its uniqueness by acquiring an exclusive lock on a unique file in Chubby, and monitors a specific directory for Tablet Server files.

Once it detects a failure, it deletes the Tablet Server’s Chubby file and reallocates its tablets to other Tablet Servers.

Compaction

Bigtable provides read and write services and uses an LSM-like structure to optimize write performance. For each write operation, the ACL information is first retrieved from Chubby to verify permissions. The write is then logged in WAL and stored in Memtable before eventually being persisted in SSTable.

When Memtable grows to a certain size, it triggers a Minor Compaction to convert Memtable to SSTable and write it to GFS.

Memtable is first converted into an immutable Memtable before becoming SSTable. This intermediate step ensures that Minor Compaction does not interfere with incoming writes.

Bigtable uses Compaction to accelerate writes, converting random writes into sequential writes and writing data in the background. Compaction occurs in three types:

Minor Compaction: Converts Memtable to SSTable, discarding deleted data and retaining only the latest version.
Merge Compaction: Combines Memtable and SSTable into a new SSTable.
Major Compaction: Combines multiple SSTables into one.

For reads, data aggregation is required across Memtable and multiple SSTables, as data may be distributed across these structures. Second-level caching and Bloom filters are used to speed up reads.

Tablet Servers have two levels of caching:

Scan Cache: Caches frequently read key-value pairs.
Block Cache: Caches SSTable blocks.

Bloom filters are also employed to reduce the number of SSTable lookups by indicating whether a key is not present.

Optimization

Locality

High-frequency columns can be grouped together into one SSTable, reducing the time to fetch related data.

Space is traded for time, leveraging locality principles.

Compression

SSTable blocks are compressed to reduce network bandwidth and latency during transfers.

Compression is performed in blocks to reduce encoding/decoding time and improve parallelism.

CommitLog Design

Tablet Servers maintain one Commit Log each, instead of one per Tablet, to minimize disk seeks and enable batch operations. During recovery, log entries must be sorted by (Table, Row, Log Seq Num) to facilitate recovery.

Summary

Keep it simple: Simple is better than complex.
Cluster monitoring is crucial for distributed services. Google’s three papers emphasize cluster monitoring and scheduling.
Do not make assumptions about other systems in your design. Issues may range from common network issues to unexpected operational problems.
Leverage background operations to accelerate user-facing actions, such as making writes fast and using background processes for cleanups.

References

MIT6.824 GFS

Thu, 09 Sep 2021 00:44:24 +0800

This article introduces the Google File System (GFS) paper published in 2003, which proposed a distributed file system designed to store large volumes of data reliably, meeting Google’s data storage needs. This write-up reflects on the design goals, trade-offs, and architectural choices of GFS.

Introduction

GFS is a distributed file system developed by Google to meet the needs of data-intensive applications, using commodity hardware to provide a scalable and fault-tolerant solution.

Background

Component Failures as the Norm: In GFS, component failures are treated as normal events rather than exceptions.

GFS uses inexpensive hardware to build a reliable service. Each machine has a certain probability of failure, resulting in a binomial distribution of overall system failures. The key challenge is to ensure the system remains available through redundancy and rapid failover.

Massive Files: Files in GFS can be extremely large, ranging from several hundred megabytes to tens of gigabytes.

GFS favors large files rather than many small files. Managing a large number of small files in a distributed system can lead to increased metadata overhead, inefficient caching, and greater inode usage.

Sequential Access: Most file modifications append data to the end of files rather than random modifications, and reads are generally sequential.

GFS is optimized for sequential writes, especially for appending data. Random writes are not well-supported and do not guarantee consistency.

Collaborative Design: The API and file system are designed collaboratively to improve efficiency and flexibility.

GFS provides an API similar to POSIX but includes additional optimizations to better match Google’s workload.

Design Goals

Storage Capacity

GFS is designed to manage millions of files, most of which are at least 100 MB in size. Files of several gigabytes are common, but GFS also supports smaller files without specific optimization.

Workload

Read Workload

Large-Scale Sequential Reads: Large-scale sequential data retrieval using disk I/O.
Small-Scale Random Reads: Small-scale random data retrieval, optimized through techniques such as request batching.

Write Workload

Primarily large-scale sequential writes, typically appending data to the end of files. GFS supports concurrent data appends from multiple clients, with atomic guarantees and synchronization.

Bandwidth vs. Latency

High sustained bandwidth is prioritized over low latency, given the typical workloads of GFS.

Fault Tolerance

GFS continuously monitors its state to detect and recover from component failures, which are treated as common occurrences.

Operations and Interfaces

GFS provides traditional file system operations such as file creation, deletion, and reading, along with features like snapshots and atomic record append.

Snapshots create file or directory copies, while atomic record append guarantees that data is appended atomically.

Architecture

The architecture of GFS follows a Master-Slave design, consisting of a single Master node and multiple Chunk Servers.

The Master and Chunk Servers are logical concepts and do not necessarily refer to specific physical machines.

GFS provides a client library (SDK) that allows clients to access the system, abstracting the underlying complexity. File data is divided into chunks and stored across multiple Chunk Servers, with replication for reliability. The Master manages metadata such as namespace, chunk locations, and more.

Component Overview

Client

Clients in GFS are application processes that use the GFS SDK for seamless integration. Key functionalities of the client include:

Caching: Cache metadata obtained from the Master to reduce communication overhead.
Encapsulation: Encapsulate retries, request splitting, and checksum validation.
Optimization: Perform request batching, load balancing, and caching to enhance efficiency.
Mapping: Map file operations to chunk-based ones, such as converting (filename, offset) into (chunk index, offset).

Master

The Master maintains all metadata, including the namespace, file-to-chunk mappings, and chunk versioning. Key functionalities include:

Monitoring: Track Chunk Server status and data locations using heartbeats.
Directory Tree Management: Manage the hierarchical file system structure with efficient locking mechanisms.
Mapping Management: Maintain mappings between files and chunks for fast lookups.
Fault Tolerance: Utilize checkpointing and Raft-style multi-replica backups to recover from Master failures.
System Scheduling: Manage chunk replication, garbage collection, lease distribution, and primary Chunk Server selection.

Metadata is stored in memory for performance reasons, resulting in a simplified design, but making checkpointing and logging crucial to ensure recovery.

Chunk Server

Chunk Servers are responsible for storing data, with each file chunk being saved as a Linux file. Chunk Servers also perform data integrity checks and report health information to the Master regularly.

Key Concepts and Mechanisms

Chunk Size

Chunks are the logical units for storing data in GFS, with each chunk typically sized at 64 MB. The chunk size balances metadata overhead, caching efficiency, data locality, and fault tolerance.

Small chunks increase metadata load on the Master, whereas larger chunks can create data hot spots and fragmentation.

Lease Mechanism

GFS uses a lease mechanism to ensure consistency between chunk replicas. When concurrent write requests occur, the Master selects a Chunk Server to be the primary. The primary node assigns an order to client operations, ensuring concurrent operations are executed consistently.

This mechanism reduces the coordination load on the Master and allows data to be appended atomically.

Chunk Versioning

The versioning system is used to ensure that only the latest chunk version is valid. The Master increments the version whenever a lease is granted, and a new version number is committed after acknowledgment from the primary.

Versioning helps determine the freshness of data during recoveries.

Control Flow vs. Data Flow

GFS separates control flow and data flow to optimize data transfers. Control commands are issued separately from data transfers, enabling efficient utilization of network topology.

Data is sent using a pipeline approach between Chunk Servers, which minimizes network overhead and uses cache effectively.

Data Integrity

Chunks are split into 64 KB blocks, each with a corresponding checksum for data integrity. These checksums are used to verify data during read operations.

Checksums are stored separately from the data, providing an additional layer of reliability.

Fault Tolerance and Replication

Chunks are stored in multiple replicas across different Chunk Servers for reliability. The Master detects Chunk Server failures via heartbeats and manages replication to meet desired redundancy levels.

Data integrity failures or Chunk Server disconnections trigger replication to maintain availability.

Consistency

GFS has a relaxed consistency model. It provides eventual consistency and does not guarantee strong consistency.

In practice, operations such as atomic record append ensure data integrity during appends but may not eliminate duplicate writes. Random writes are not consistently managed.

Summary

GFS demonstrates how practical design trade-offs, driven by specific business needs, can lead to an efficient and scalable distributed file system. It focuses on resilience, fault tolerance, and high throughput, making it ideal for Google’s data processing needs.

In distributed systems, scalability is often more important than single-node performance. GFS embraces this principle through large file management, redundancy, and workload distribution.

References

MIT6.824-MapReduce

Fri, 22 Jan 2021 17:02:44 +0800

The third year of university has been quite intense, leaving me with little time to continue my studies on 6.824, so my progress stalled at Lab 1. With a bit more free time during the winter break, I decided to continue. Each paper or experiment will be recorded in this article.

This is the first chapter of my Distributed System study notes.

About the Paper

The core content of the paper is the proposed MapReduce distributed computing model and the approach to implementing the Distributed MapReduce System, including the Master data structure, fault tolerance, and some refinements.

MapReduce Computing Model

The model takes a series of key-value pairs as input and outputs a series of key-value pairs as a result. Users can use the MapReduce System by designing Map and Reduce functions.

Map: Takes input data and generates a set of intermediate key-value pairs
Reduce: Takes intermediate key-value pairs as input, combines all data with the same key, and outputs the result.

cpp


map(String key, String value):
 // key: document name
 // value: document contents
 for each word w in value:
 EmitIntermediate(w, "1");


reduce(String key, Iterator values):
 // key: a word
 // values: a list of counts
 int result = 0;
 for each v in values:
 result += ParseInt(v);
 Emit(AsString(result));

MapReduce Execution Process

The Distributed MapReduce System adopts a master-slave design. During the MapReduce computation, there is generally one Master and several Workers.

Master: Responsible for creating, assigning, and scheduling Map and Reduce tasks
Worker: Responsible for executing Map and Reduce tasks

A more detailed description is as follows:

The entire MapReduce execution process includes M Map Tasks and R Reduce Tasks, divided into two phases: Map Phase and Reduce Phase.
The input file is split into M splits, and the computation enters the Map Phase. The Master assigns Map Tasks to idle Workers. The assigned Worker reads the corresponding split data and executes the Task. When all Map Tasks are completed, the Map Phase ends. The Partition function (generally hash(key) mod R) is used to generate R sets of intermediate key-value pairs, which are stored in files and reported to the Master for subsequent Reduce Task operations.
The computation enters the Reduce Phase. The Master assigns Reduce Tasks, and each Worker reads the corresponding intermediate key-value file and executes the Task. Once all Reduce tasks are completed, the computation is finished, and the results are stored in result files.

MapReduce Fault Tolerance Mechanism

Since Google MapReduce heavily relies on the distributed atomic file read/write operations provided by Google File System, the fault tolerance mechanism of the MapReduce cluster is much simpler and primarily focuses on recovering from unexpected task interruptions.

Worker Fault Tolerance

In the cluster, the Master periodically sends Ping signals to each Worker. If a Worker does not respond for a period of time, the Master considers the Worker unavailable.

Any Map task assigned to that Worker, whether running or completed, must be reassigned by the Master to another Worker, as the Worker being unavailable also means the intermediate results stored on that Worker’s local disk are no longer available. The Master will also notify all Reducers about the retry, and Reducers that fail to obtain complete intermediate results from the original Mapper will start fetching data from the new Mapper.

If a Reduce task is assigned to that Worker, the Master will select any unfinished Reduce tasks and reassign them to other Workers. Since the results of completed Reduce tasks are stored in Google File System, the availability of these results is ensured by Google File System, and the MapReduce Master only needs to handle unfinished Reduce tasks.

If there is a Worker in the cluster that takes an unusually long time to complete the last few Map or Reduce tasks, the entire MapReduce computation time will be prolonged, and such a Worker becomes a straggler.

Once the MapReduce computation reaches a certain completion level, any remaining tasks are backed up and assigned to other idle Workers, and the task is considered completed once one of the Workers finishes it.

Master Fault Tolerance

There is only one Master node in the entire MapReduce cluster, so Master failures are relatively rare.

During operation, the Master node periodically saves the current state of the cluster as a checkpoint to disk. After the Master process terminates, a restarted Master process can use the data stored on disk to recover to the state of the last checkpoint.

Partition Function

Used during the Map Phase to assign intermediate key-value pairs to R files according to certain rules.

Combiner

In some situations, the user-defined Map task may generate a large number of duplicate intermediate keys. The Combiner function performs a partial merge of the intermediate results to reduce the amount of data that needs to be transmitted between Mapper and Reducer.

Experiment

The experiment involves designing and implementing the Master and Worker to complete the main functionality of a Simple MapReduce System.

In the experiment, the single Master and multiple Worker model was implemented through RPC calls, and different applications were formed by running Map and Reduce functions via Go Plugins.

Master & Worker Functionality

Master

Task creation and scheduling
Worker registration and task assignment
Receiving the current state of the Worker
Monitoring task status

Worker

Registering with the Master
Getting tasks and processing them
Reporting status

Note: The Master provides corresponding functions to Workers via RPC calls

Main Data Structures

The design of data structures is the main task, and good design helps in implementing functionality. The relevant code is shown here; for the specific implementation, see GitHub.

Master

go


type Master struct {
 // Your definitions here.
 nReduce int
 taskQueue chan Task
 tasksContext []TaskContext
 lock sync.Mutex
 files []string
 phase PhaseKind
 done bool
 workerID int
}

Worker

go


type worker struct {
 ID int
 mapf func(string, string) []KeyValue
 reducef func(string, []string) string
 nReduce int
 nMap int
}

Task & TaskContext

go


type Task struct {
 ID int
 Filename string
 Phase PhaseKind
}

type TaskContext struct {
 t *Task
 state ContextState
 workerID int
 startTime time.Time
}

Rpc Args & Reply

go


type RegTaskArgs struct {
 WorkerID int
}

type RegTaskReply struct {
 T Task
 HasT bool
}

type ReportTaskArgs struct {
 WorkerID int
 TaskID int
 State ContextState
}
type ReportTaskReply struct {
}

type RegWorkerArgs struct {
}

type RegWorkerReply struct {
 ID int
 NReduce int
 NMap int
}

Constant & Type

go


const (
 RUNNING ContextState = iota
 FAILED
 READY
 IDEL
 COMPLETE
)

const (
 MAX_PROCESSING_TIME = time.Second * 5
 SCHEDULE_INTERVAL = time.Second
)

const (
 MAP PhaseKind = iota
 REDUCE
)

type ContextState int

type PhaseKind int

Running and Testing

Running

bash


# In main directory
cd ./src/main
# Master
go run ./mrmaster.go pg*.txt
# Worker
go build -buildmode=plugin ../mrapps/wc.go && go run ./mrworker.go ./wc.so

Testing

bash

cd ./src/main

sh ./test-mr.sh

Optimization

These optimizations are some designs I thought of that could be improved when reviewing my code after completing the experiment.

Hotspot Issue

The hotspot issue here refers to a scenario where a particular data item appears frequently in the dataset. The intermediate key-value pairs generated during the Map phase can lead to a situation where one key appears frequently, resulting in excessive disk IO and network IO for a few machines during the shuffle step.

The essence of this issue is that the Shuffle step in MapReduce is highly dependent on the data.

The design purpose of Shuffle is to aggregate intermediate results to facilitate processing during the Reduce phase. Consequently, if the data is extremely unbalanced, hotspot issues will naturally arise.

In fact, the core problem is that a large number of keys are assigned to a single disk file after being hashed, serving as input for the subsequent Reduce phase.

The hash value for the same key should be identical, so the question becomes: How can we assign the same key’s hash value to different machines?

The solution I came up with is to add a random salt to the key in the Shuffle’s hash calculation so that the hash values are different, thereby reducing the probability of keys being assigned to the same machine and solving the hotspot issue.

Fault Tolerance

The paper already proposes some solutions for fault tolerance. The scenario in question is: a Worker node crashes unexpectedly and reconnects after a reboot. The Master observes the crash and reassigns its tasks to other nodes, but the reconnected Worker continues executing its original tasks, resulting in duplicate result files.

The potential issue here is that these two files may cause incorrect results. Furthermore, the reconnected Worker continuing to execute its original tasks wastes CPU and IO resources.

Based on this, we need to mark the newly generated result files, ensuring only the latest files are used as results, thus resolving the file conflict. Additionally, we should add an RPC interface for Worker nodes so that when they reconnect, the Master can call it to clear out any original tasks.

Straggler Issue

The straggler issue refers to a Task taking a long time to complete, delaying the overall MapReduce computation. Essentially, it is a hotspot issue and Worker crash handling problem, which can be addressed by referring to the above sections.

References

MIT6.824 Distributed System

Lab Official Site

MapReduce: Simplified Data Processing on Large Clusters

Detailed Explanation of Google MapReduce Paper

Paper Reading on NoneBack

Borg: Large-scale Cluster Management at Google with Borg

System Abstraction

System Architecture

Resource Scheduling

Basic Process

Additional Aspects

References

Percolator: Large-scale Incremental Processing Using Distributed Transactions and Notifications

Percolator and 2PC

2PC

Percolator

Percolator Protocol Details

Algorithm Workflow

Prewrite

Commit

Snapshot Isolation

Deadlock Issues in Asynchronous Networks

Notification Mechanism

Percolator in TiDB

References

Dynamo: Amazon’s Highly Available Key-value Store

Design

Technical Details

Data Partitioning

Data Write Challenges

References

MIT6.824 AuroraDB

References

MIT6.824 Chain Replication

Introduction

Failure Handling

Pros and Cons

References

MIT6.824-ZooKeeper

Positioning

Design

Keywords

Data Model and API

CAP Guarantees

Implementation

Interaction with Clients

References

MIT6.824-RaftKV

Lab Overview

Raft-Related Topics

Routing and Linearizability

Routing

Linearizability

Read-Only Optimization

Read Index

Lease Read

Implementation

Summary

References

MIT6.824-Raft

Algorithm Overview

Implementation Ideas

Leader Election

Triggering Election + Handling Election Results

Request Vote RPC

State Transition

Log Replication

Log Replication RPC

Trigger Log Replication + Handle Request Results

Asynchronous Apply

Persistence

Install Snapshot

Pitfalls

Defer

Log Dummy Header

Lock

References

DFS-Haystack

Introduction

Background

Design

Traditional Design

CDN-based Solution

NAS-based Solution