Lakehouse on NoneBack

Flink-Iceberg-Connector Write Process

Mon, 10 Oct 2022 10:43:38 +0800

The Iceberg community provides an official Flink Connector, and this chapter’s source code analysis is based on that.

Overview of the Write Submission Process

Flink writes data through RowData -> distributeStream -> WriterStream -> CommitterStream. Before data is committed, it is stored as intermediate files, which become visible to the system after being committed (through writing manifest, snapshot, and metadata files).

java


private <T> DataStreamSink<T> chainIcebergOperators() {
 Preconditions.checkArgument(inputCreator != null,
 "Please use forRowData() or forMapperOutputType() to initialize the input DataStream.");
 Preconditions.checkNotNull(tableLoader, "Table loader shouldn't be null");

 DataStream<RowData> rowDataInput = inputCreator.apply(uidPrefix);

 if (table == null) {
 tableLoader.open();
 try (TableLoader loader = tableLoader) {
 this.table = loader.loadTable();
 } catch (IOException e) {
 throw new UncheckedIOException("Failed to load iceberg table from table loader: " + tableLoader, e);
 }
 }

 List<Integer> equalityFieldIds = checkAndGetEqualityFieldIds();

 RowType flinkRowType = toFlinkRowType(table.schema(), tableSchema);

 DataStream<RowData> distributeStream = distributeDataStream(
 rowDataInput, table.properties(), equalityFieldIds, table.spec(), table.schema(), flinkRowType);

 SingleOutputStreamOperator<WriteResult> writerStream = appendWriter(distributeStream, flinkRowType,
 equalityFieldIds);

 SingleOutputStreamOperator<Void> committerStream = appendCommitter(writerStream);

 return appendDummySink(committerStream);
}

Write Process Source Code Analysis

WriteStream

java


private SingleOutputStreamOperator<WriteResult> appendWriter(DataStream<RowData> input, RowType flinkRowType,
 List<Integer> equalityFieldIds) {
 boolean upsertMode = upsert || PropertyUtil.propertyAsBoolean(table.properties(),
 UPSERT_ENABLED, UPSERT_ENABLED_DEFAULT);

 if (upsertMode) {
 Preconditions.checkState(!overwrite,
 "OVERWRITE mode shouldn't be enabled when configuring to use UPSERT data stream.");
 Preconditions.checkState(!equalityFieldIds.isEmpty(),
 "Equality field columns shouldn't be empty when configuring to use UPSERT data stream.");
 if (!table.spec().isUnpartitioned()) {
 for (PartitionField partitionField : table.spec().fields()) {
 Preconditions.checkState(equalityFieldIds.contains(partitionField.sourceId()),
 "In UPSERT mode, partition field '%s' should be included in equality fields: '%s'",
 partitionField, equalityFieldColumns);
 }
 }
 }

 IcebergStreamWriter<RowData> streamWriter = createStreamWriter(table, flinkRowType, equalityFieldIds, upsertMode);

 int parallelism = writeParallelism == null ? input.getParallelism() : writeParallelism;
 SingleOutputStreamOperator<WriteResult> writerStream = input
 .transform(operatorName(ICEBERG_STREAM_WRITER_NAME), TypeInformation.of(WriteResult.class), streamWriter)
 .setParallelism(parallelism);
 if (uidPrefix != null) {
 writerStream = writerStream.uid(uidPrefix + "-writer");
 }
 return writerStream;
}

The WriterStream operator is transformed from the distributeStream, with RowData as input and WriteResult as output. The transformation logic is encapsulated in the IcebergStreamWriter, which processes each element using processElement:

java


private transient TaskWriter<T> writer;
@Override
public void processElement(StreamRecord<T> element) throws Exception {
 writer.write(element.getValue());
}

IcebergStreamWriter delegates the writing to a TaskWriter created by TaskWriterFactory. The specific type could be PartitionedDeltaWriter or UnpartitionedWriter:

java


public TaskWriter<RowData> create() {
 Preconditions.checkNotNull(outputFileFactory,
 "The outputFileFactory shouldn't be null if we have invoked the initialize().");

 if (equalityFieldIds == null || equalityFieldIds.isEmpty()) {
 if (spec.isUnpartitioned()) {
 return new UnpartitionedWriter<>(spec, format, appenderFactory, outputFileFactory, io, targetFileSizeBytes);
 } else {
 return new RowDataPartitionedFanoutWriter(spec, format, appenderFactory, outputFileFactory,
 io, targetFileSizeBytes, schema, flinkSchema);
 }
 } else {
 if (spec.isUnpartitioned()) {
 return new UnpartitionedDeltaWriter(spec, format, appenderFactory, outputFileFactory, io,
 targetFileSizeBytes, schema, flinkSchema, equalityFieldIds, upsert);
 } else {
 return new PartitionedDeltaWriter(spec, format, appenderFactory, outputFileFactory, io,
 targetFileSizeBytes, schema, flinkSchema, equalityFieldIds, upsert);
 }
 }
}

CommitterStream

The CommitterStream receives WriteResult as input with no output. WriteResult contains the data files produced by WriteStream:

java


public class WriteResult implements Serializable {
 private DataFile[] dataFiles;
 private DeleteFile[] deleteFiles;
 private CharSequence[] referencedDataFiles;
 ...
}

The core logic for processing data file submissions is encapsulated in IcebergFilesCommitter. The IcebergFilesCommitter maintains a list of files that need to be committed for each checkpoint. Once a checkpoint completes, it tries to commit those files to Iceberg.

java


class IcebergFilesCommitter extends AbstractStreamOperator<Void>
 implements OneInputStreamOperator<WriteResult, Void>, BoundedOneInput {
 ...

 private final NavigableMap<Long, byte[]> dataFilesPerCheckpoint = Maps.newTreeMap();
 private final List<WriteResult> writeResultsOfCurrentCkpt = Lists.newArrayList();
 private transient ListState<SortedMap<Long, byte[]>> checkpointsState;
 ...
}

The processElement method stores WriteResult from upstream in writeResultsOfCurrentCkpt:

java


@Override
public void processElement(StreamRecord<WriteResult> element) {
 this.writeResultsOfCurrentCkpt.add(element.getValue());
}

During checkpointing (snapshotState), it saves the current checkpoint’s data in dataFilesPerCheckpoint. Later, once the checkpoint is completed (notifyCheckpointComplete), it commits the files:

java


public void snapshotState(StateSnapshotContext context) throws Exception {
 long checkpointId = context.getCheckpointId();
 LOG.info("Start to flush snapshot state to state backend, table: {}, checkpointId: {}", table, checkpointId);

 dataFilesPerCheckpoint.put(checkpointId, writeToManifest(checkpointId));
 checkpointsState.clear();
 checkpointsState.add(dataFilesPerCheckpoint);

 jobIdState.clear();
 jobIdState.add(flinkJobId);

 writeResultsOfCurrentCkpt.clear();
}

@Override
public void notifyCheckpointComplete(long checkpointId) throws Exception {
 if (checkpointId > maxCommittedCheckpointId) {
 commitUpToCheckpoint(dataFilesPerCheckpoint, flinkJobId, checkpointId);
 this.maxCommittedCheckpointId = checkpointId;
 }
}

The commit logic is handled by commitUpToCheckpoint, which generates a new snapshot and adds it to Iceberg’s metadata:

java


private void commitUpToCheckpoint(NavigableMap<Long, byte[]> deltaManifestsMap,
 String newFlinkJobId,
 long checkpointId) throws IOException {
 NavigableMap<Long, byte[]> pendingMap = deltaManifestsMap.headMap(checkpointId, true);
 List<ManifestFile> manifests = Lists.newArrayList();
 NavigableMap<Long, WriteResult> pendingResults = Maps.newTreeMap();
 for (Map.Entry<Long, byte[]> e : pendingMap.entrySet()) {
 if (Arrays.equals(EMPTY_MANIFEST_DATA, e.getValue())) {
 continue;
 }

 DeltaManifests deltaManifests = SimpleVersionedSerialization
 .readVersionAndDeSerialize(DeltaManifestsSerializer.INSTANCE, e.getValue());
 pendingResults.put(e.getKey(), FlinkManifestUtil.readCompletedFiles(deltaManifests, table.io()));
 manifests.addAll(deltaManifests.manifests());
 }

 int totalFiles = pendingResults.values().stream()
 .mapToInt(r -> r.dataFiles().length + r.deleteFiles().length).sum();
 continuousEmptyCheckpoints = totalFiles == 0 ? continuousEmptyCheckpoints + 1 : 0;
 if (totalFiles != 0 || continuousEmptyCheckpoints % maxContinuousEmptyCommits == 0) {
 if (replacePartitions) {
 replacePartitions(pendingResults, newFlinkJobId, checkpointId);
 } else {
 commitDeltaTxn(pendingResults, newFlinkJobId, checkpointId);
 }
 continuousEmptyCheckpoints = 0;
 }
 pendingMap.clear();

 for (ManifestFile manifest : manifests) {
 try {
 table.io().deleteFile(manifest.path());
 } catch (Exception e) {
 LOG.warn("The iceberg transaction has been committed, but we failed to clean the temporary flink manifests: {}",
 manifest.path(), e);
 }
 }
}

java


public void commit(TableMetadata base, TableMetadata metadata) {
 if (base != current()) {
 if (base != null) {
 throw new CommitFailedException("Cannot commit: stale table metadata");
 } else {
 throw new AlreadyExistsException("Table already exists: %s", tableName());
 }
 }
 if (base == metadata) {
 LOG.info("Nothing to commit.");
 return;
 }

 long start = System.currentTimeMillis();
 doCommit(base, metadata);
 deleteRemovedMetadataFiles(base, metadata);
 requestRefresh();

 LOG.info("Successfully committed to table {} in {} ms",
 tableName(),
 System.currentTimeMillis() - start);
}

Write Issues

1. Lots of Small Files

For streaming writes, new files are generated each time, resulting in a lot of small files. While object storage supports small files well, it may increase Iceberg metadata overhead, as metadata files need to keep track of each data file. This can cause metadata files to become large and impact performance.

Solution:

Iceberg Rewrite Action: Iceberg supports rewriting data and metadata files via Flink or Spark actions, which need to be triggered separately.
Snapshot Expiry: Configure snapshot expiration to periodically delete old snapshots.

java


import org.apache.iceberg.flink.actions.Actions;

TableLoader tableLoader = TableLoader.fromHadoopTable("hdfs://nn:8020/warehouse/path");
Table table = tableLoader.loadTable();
RewriteDataFilesActionResult result = Actions.forTable(table)
 .rewriteDataFiles()
 .execute();

Iceberg Flink Documentation Iceberg Maintenance Documentation

2. Performance Issues with High Concurrency

Iceberg’s writing process creates a new snapshot for each commit and uses optimistic concurrency control to handle conflicts. In high-concurrency scenarios, this can lead to many commits being retried, impacting performance.

Solution:

Batch Commit: Introduce a caching layer or additional service to batch commits to the data lake, reducing the number of concurrent commit operations. This cache layer can also compact multiple data files before committing.

References: Optimizing Iceberg Writes for High Concurrency InfoQ Article on Iceberg Optimization

3. Flink Iceberg Connector Limitations

The Flink Iceberg Connector does not support hidden partitions or preprocessing of partition fields.

Apache-Iceberg Quick Investigation

Wed, 05 Oct 2022 19:55:54 +0800

A table format for large-scale analysis of datasets.
A specification for organizing data files and metadata files.
A schema semantic abstraction between storage and computation.
Developed and open-sourced by Netflix to enhance scalability, reliability, and usability.

Background

Issues encountered when migrating HIVE to the cloud:

Dependency on List and Rename semantics makes it impossible to replace HDFS with cheaper OSS.
Scalability issues: Schema information in Hive is centrally stored in metastore, which can become a performance bottleneck.
Unsafe operations, CBO unfriendly, etc.

Features

Supports secure and efficient schema, partition changes, and evolution, self-defined schema, hidden partition.
- Abstracts its own schema, not tied to any computation engine schema; partition is maintained at the schema level. Partition and sort order provide transformer functions, such as date(timestamp).
Supports object storage with minimal dependency on FS semantics.
ACID semantics support, parallel reads, serialized write operations:
- Separation of read and write snapshots.
- Optimistic handling of write parallel conflicts, retry to ensure writes.
Snapshot support:
- Data rollback and time travel.
- Supports snapshot expiration (by default, data files are not deleted, but customizable deletion behavior is available) (related API doc).
- Incremental reading can be achieved by comparing snapshot differences.
Query optimization-friendly: predicate pushdown, data file statistics. Currently, compaction is not supported, but invalid files can be deleted during snapshot expiration (deleteWith).
High abstraction level, easy for modification, optimization, and extension. Catalog, read/write paths, file formats, storage dependencies are all pluggable. Iceberg’s design goal is to define a standard, open, and general data organization format while hiding differences in underlying data storage formats, providing a unified operational API for different engines to connect through its API.
Others: file-level encryption and decryption.

Ecosystem

Community support for OSS, Flink, Spark, and Presto:
- Flink (detail): Supports streaming reads and writes, incremental reads (based on snapshot), upsert write (0.13.0-release-notes).
- Presto: Iceberg connector.
- Aliyun OSS: # pr 3689.
Integration with other components:
- Integration with lower storage layers: Only relies on three semantics: In-place write, Seekable reads, Deletes, supports AliOSS (# pr 3689).
- Integration with other file formats: High abstraction level, currently supports Avro, Parquet, ORC.
- Catalog: Customizable (Doc: Custom Catalog Implementation), currently supports JDBC, Hive Metastore, Hadoop, etc.
- Integration with computation layer: Provides native JAVA & Python APIs, with a high level of abstraction, supporting most computation engines.
Open and neutral community, allowing contributions to improve influence.

Table Specification

Specification for organizing data files and metadata files.

Case: Spark + Iceberg + Local FS

Iceberg supports Parquet, Avro, ORC file formats.

json


# Storage organization
test2
├── data
│ ├── 00000-1-ccff6767-12cc-481c-93fc-db9f1a57438c-00001.parquet
│ └── 00001-2-6c1e5a0b-89fe-4e77-b90a-1773a7fbbcc8-00001.parquet
└── metadata
├── 2c1dc0e8-1843-4cb9-9c55-ae43f800bf3f-m0.avro // manifest file
├── snap-8512048775051875497-1-2c1dc0e8-1843-4cb9-9c55-ae43f800bf3f.avro // manifest list file
├── v1.metadata.json // metadata file
├── v2.metadata.json
 └── version-hint.text // catalog

DataFile

Data files in columnar format: Parquet, ORC.

There are three types of Data Files: data file, partition delete file, equality delete file.

Manifest File

Indexes data files, including statistics and partition information.

json


[
 {
 "status":1,
 "snapshot_id":{
 "long":1274364374047997583
 },
 "data_file":{
 "file_path":"/tmp/warehouse/db/test3/data/id=1/00000-31-401a9d2e-d501-434c-a38f-5df5f08ebbd7-00001.parquet",
 "file_format":"PARQUET",
 "partition":{
 "id":{
 "long":1
 }
 },
 "record_count":1,
 "file_size_in_bytes":643,
 "block_size_in_bytes":67108864,
 "column_sizes":{
 "array":[
 {
 "key":1,
 "value":46
 },
 {
 "key":2,
 "value":48
 }
 ]
 },
 "value_counts":{
 "array":[
 {
 "key":1,
 "value":1
 },
 {
 "key":2,
 "value":1
 }
 ]
 },
 "null_value_counts":{
 "array":[
 {
 "key":1,
 "value":0
 },
 {
 "key":2,
 "value":0
 }
 ]
 },
 "nan_value_counts":{
 "array":[

 ]
 },
 "lower_bounds":{
 "array":[
 {
 "key":1,
 "value":"\u0001\u0000\u0000\u0000\u0000\u0000\u0000\u0000"
 },
 {
 "key":2,
 "value":"a"
 }
 ]
 },
 "upper_bounds":{
 "array":[
 {
 "key":1,
 "value":"\u0001\u0000\u0000\u0000\u0000\u0000\u0000\u0000"
 },
 {
 "key":2,
 "value":"a"
 }
 ]
 },
 "key_metadata":null,
 "split_offsets":{
 "array":[
 4
 ]
 },
 "sort_order_id":{
 "int":0
 }
 }
 },
 {
 // another data file meta
 }
]

Snapshot

Represents the state of a Table at a specific point in time, saved via a Manifest List File.
A new Snapshot is generated every time a data change is made to the Table.

Manifest List File

Contains information about all Manifest files in a Snapshot, as well as partition stats and data file count.
One Snapshot corresponds to one Manifest List File, and each submission generates a manifest list file.
Optimistic concurrency: when concurrent Snapshot submissions conflict, the later submission retries to ensure submission.

Each manifest list stores metadata about manifests, including partition stats and data file counts.

json


[
 {
 "manifest_path":"/tmp/warehouse/db/test3/metadata/f22b748f-a7bc-4e4c-ad6c-3e335c1c0c2b-m0.avro",
 "manifest_length":6019,
 "partition_spec_id":0,
 "added_snapshot_id":{
 "long":1274364374047997583
 },
 "added_data_files_count":{
 "int":2
 },
 "existing_data_files_count":{
 "int":0
 },
 "deleted_data_files_count":{
 "int":0
 },
 "partitions":{
 "array":[
 {
 "contains_null":false,
 "contains_nan":{
 "boolean":false
 },
 "lower_bound":{
 "bytes":"\u0001\u0000\u0000\u0000\u0000\u0000\u0000\u0000"
 },
 "upper_bound":{
 "bytes":"\u0002\u0000\u0000\u0000\u0000\u0000\u0000\u0000"
 }
 }
 ]
 },
 "added_rows_count":{
 "long":2
 },
 "existing_rows_count":{
 "long":0
 },
 "deleted_rows_count":{
 "long":0
 }
 },
 {
 // another manifest file
 }
]

Metadata File

Tracks the state of the table. When the state changes, a new metadata file is generated and replaces the previous one, ensuring atomicity.

The table metadata file tracks the table schema, partitioning config, custom properties, and snapshots of the table contents.

json


{
 "format-version":1,
 "table-uuid":"175d0b61-8507-40b2-9c19-3338b05f3d48",
 "location":"/tmp/warehouse/db/test3",
 "last-updated-ms":1653387947819,
 "last-column-id":2,
 "schema":{
 "type":"struct",
 "schema-id":0,
 "fields":[
 {
 "id":1,
 "name":"id",
 "required":false,
 "type":"long"
 },
 Object{...}
 ]
 },
 "current-schema-id":0,
 "schemas":[
 {
 "type":"struct",
 "schema-id":0,
 "fields":[
 {
 "id":1,
 "name":"id",
 "required":false,
 "type":"long"
 },
 Object{...}
 ]
 }
 ],
 "partition-spec":[
 {
 "name":"id",
 "transform":"identity",
 "source-id":1,
 "field-id":1000
 }
 ],
 "default-spec-id":0,
 "partition-specs":[
 {
 "spec-id":0,
 "fields":[
 {
 "name":"id",
 "transform":"identity",
 "source-id":1,
 "field-id":1000
 }
 ]
 }
 ],
 "last-partition-id":1000,
 "default-sort-order-id":0,
 "sort-orders":[
 {
 "order-id":0,
 "fields":[

 ]
 }
 ],
 "properties":{
 "owner":"chenlan"
 },
 "current-snapshot-id":1274364374047997700,
 "snapshots":[
 {
 "snapshot-id":1274364374047997700,
 "timestamp-ms":1653387947819,
 "summary":{
 "operation":"append",
 "spark.app.id":"local-1653381214613",
 "added-data-files":"2",
 "added-records":"2",
 "added-files-size":"1286",
 "changed-partition-count":"2",
 "total-records":"2",
 "total-files-size":"1286",
 "total-data-files":"2",
 "total-delete-files":"0",
 "total-position-deletes":"0",
 "total-equality-deletes":"0"
 },
 "manifest-list":"/tmp/warehouse/db/test3/metadata/snap-1274364374047997583-1-f22b748f-a7bc-4e4c-ad6c-3e335c1c0c2b.avro",
 "schema-id":0
 }
 ],
 "snapshot-log":[
 {
 "timestamp-ms":1653387947819,
 "snapshot-id":1274364374047997700
 }

 ],
 "metadata-log":[
 {
 "timestamp-ms":1653387937345,
 "metadata-file":"/tmp/warehouse/db/test3/metadata/v1.metadata.json"
 }
 ]
}

Catalog

Records the latest metadata file path.

Features

ACID semantics guarantee: Atomic table state changes + snapshot-based reads and writes.
Flexible partition management: hidden partition, seamless partition changes.
Supports incremental reads: incremental read of each change using snapshots.
Multi-version data: beneficial for data rollback.
No side effects, safe schema, and partition changes.

Data Types

Data files in different formats define different types.

Nested Types:
- struct: A tuple of typed values.
- list: A collection of values with an element type.
- map: A collection of key-value pairs with a key type and a value type.
Primitive Types:

Primitive type	Description	Requirements
boolean	True or false
int	32-bit signed integers	Can promote to `long`
long	64-bit signed integers
float	32-bit IEEE 754 floating point	Can promote to double
double	64-bit IEEE 754 floating point
decimal(P,S)	Fixed-point decimal; precision P, scale S	Scale is fixed [1], precision must be 38 or less
date	Calendar date without timezone or time
time	Time of day without date, timezone	Microsecond precision [2]
timestamp	Timestamp without timezone	Microsecond precision [2]
timestamptz	Timestamp with timezone	Stored as UTC [2]
string	Arbitrary-length character sequences	Encoded with UTF-8 [3]
uuid	Universally unique identifiers	Should use 16-byte fixed
fixed(L)	Fixed-length byte array of length L
binary	Arbitrary-length byte array

Read & Write Paths

select: catalog -> manifest list file -> manifest file -> data file -> data group.

insert: reverse (catalog -> manifest list file -> manifest file -> data file -> data group).

update: delete & insert, data file + partition delete file + equality delete file.

Using Partition delete file transaction: issue of repeatedly inserting and deleting the same row within a transaction.

delete: row-level delete.

References

Iceberg Spec

Flink+Iceberg Data Lake Construction

Construction Practice of Real-time Data Warehouse with Flink + Iceberg (Chinese)

Iceberg Aliyun OSS

Iceberg Flink Support

Building Enterprise-grade Real-time Data Lake with Flink + Iceberg

How Flink Analyzes CDC Data in Iceberg Real-time Data Lake

Iceberg GitHub

Alluxio POSIX API

Comparison of Delta, Iceberg, and Hudi Open-source Data Lake Solutions