February 8, 2025 · 502 words · 3 min · LevelDB MVCC Storage
LevelDB implements concurrent sstable read/write operations and snapshot reads through MVCC. Let’s examine its implementation.
Sequence Number LevelDB uses Sequence Numbers as logical clocks to maintain a total order of KV write operations. The Sequence Number is encoded in the last few bytes of the InternalKey. This encoding ensures data ordering during memory writes.
December 31, 2024 · 4802 words · 10 min · Prometheus TSDB Storage
Recently got promoted, I took a moment to summarize some of my previous work. A significant part of my job was building large-scale database observability systems, which are quite different from cloud-native monitoring solutions like Prometheus. Now, I’m diving into the standard open-source monitoring system.
August 1, 2023 · 425 words · 2 min · Distributed System Storage
An old paper by AWS, Dynamo has been in the market for a long time, and the architecture has likely evolved since the paper’s publication. Despite this, the paper was selected as one of the SIGMOD best papers of the year, and there are still many valuable lessons to learn.
October 10, 2022 · 1056 words · 5 min · Big Data Lakehouse Stream Compute Storage
The Iceberg community provides an official Flink Connector, and this chapter’s source code analysis is based on that.
Flink writes data through RowData -> distributeStream -> WriterStream -> CommitterStream. Before data is committed, it is stored as intermediate files, which become visible to the system after being committed (through writing manifest, snapshot, and metadata files).
October 5, 2022 · 565 words · 3 min · Column Store Big Data Storage
Iceberg supports both ORC and Parquet columnar formats. Compared to Parquet, ORC offers advantages in query performance and ACID support. Considering the future data lakehouse requirements for query performance and ACID compliance, we are researching ORC to support a future demo involving Flink, Iceberg, and ORC.