org.apache.flink.runtime.io.network.partition

Class SortBasedDataBuffer

java.lang.Object

org.apache.flink.runtime.io.network.partition.SortBasedDataBuffer

All Implemented Interfaces:

DataBuffer

@NotThreadSafe
public class SortBasedDataBuffer
extends Object
implements DataBuffer

A DataBuffer implementation which sorts all appended records only by subpartition index. Records of the same subpartition keep the appended order.

It maintains a list of MemorySegments as a joint buffer. Data will be appended to the joint buffer sequentially. When writing a record, an index entry will be appended first. An index entry consists of 4 fields: 4 bytes for record length, 4 bytes for Buffer.DataType and 8 bytes for address pointing to the next index entry of the same channel which will be used to index the next record to read when coping data from this DataBuffer. For simplicity, no index entry can span multiple segments. The corresponding record data is seated right after its index entry and different from the index entry, records have variable length thus may span multiple segments.

Constructor Summary

Constructors

Constructor and Description
SortBasedDataBuffer(BufferPool bufferPool, int numSubpartitions, int bufferSize, int numGuaranteedBuffers, int[] customReadOrder)

Method Summary

Modifier and Type	Method and Description
boolean	append(ByteBuffer source, int targetChannel, Buffer.DataType dataType) No partial record will be written to this SortBasedDataBuffer, which means that either all data of target record will be written or nothing will be written.
void	finish() Finishes this DataBuffer which means no record can be appended any more.
BufferWithChannel	getNextBuffer(org.apache.flink.core.memory.MemorySegment transitBuffer) Copies data in this DataBuffer to the target MemorySegment in channel index order and returns BufferWithChannel which contains the copied data and the corresponding channel index.
boolean	hasRemaining() Returns true if not all data appended to this DataBuffer is consumed.
boolean	isFinished() Whether this DataBuffer is finished or not.
boolean	isReleased() Whether this DataBuffer is released or not.
long	numTotalBytes() Returns the total number of bytes written to this DataBuffer.
long	numTotalRecords() Returns the total number of records written to this DataBuffer.
void	release() Releases this DataBuffer which releases all resources.
void	reset() Resets this DataBuffer to be reused for data appending.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SortBasedDataBuffer

public SortBasedDataBuffer(BufferPool bufferPool,
                           int numSubpartitions,
                           int bufferSize,
                           int numGuaranteedBuffers,
                           @Nullable
                           int[] customReadOrder)

Method Detail

append

public boolean append(ByteBuffer source,
                      int targetChannel,
                      Buffer.DataType dataType)
               throws IOException

No partial record will be written to this SortBasedDataBuffer, which means that either all data of target record will be written or nothing will be written.

Specified by:

append in interface DataBuffer

Throws:

IOException

getNextBuffer

public BufferWithChannel getNextBuffer(org.apache.flink.core.memory.MemorySegment transitBuffer)

Description copied from interface: DataBuffer

Copies data in this DataBuffer to the target MemorySegment in channel index order and returns BufferWithChannel which contains the copied data and the corresponding channel index.

Specified by:

getNextBuffer in interface DataBuffer

numTotalRecords

public long numTotalRecords()

Description copied from interface: DataBuffer

Returns the total number of records written to this DataBuffer.

Specified by:

numTotalRecords in interface DataBuffer

numTotalBytes

public long numTotalBytes()

Description copied from interface: DataBuffer

Returns the total number of bytes written to this DataBuffer.

Specified by:

numTotalBytes in interface DataBuffer

hasRemaining

public boolean hasRemaining()

Description copied from interface: DataBuffer

Returns true if not all data appended to this DataBuffer is consumed.

Specified by:

hasRemaining in interface DataBuffer

reset

public void reset()

Description copied from interface: DataBuffer

Resets this DataBuffer to be reused for data appending.

Specified by:

reset in interface DataBuffer

finish

public void finish()

Description copied from interface: DataBuffer

Finishes this DataBuffer which means no record can be appended any more.

Specified by:

finish in interface DataBuffer

isFinished

public boolean isFinished()

Description copied from interface: DataBuffer

Whether this DataBuffer is finished or not.

Specified by:

isFinished in interface DataBuffer

release

public void release()

Description copied from interface: DataBuffer

Releases this DataBuffer which releases all resources.

Specified by:

release in interface DataBuffer

isReleased

public boolean isReleased()

Description copied from interface: DataBuffer

Whether this DataBuffer is released or not.

Specified by:

isReleased in interface DataBuffer