org.apache.flink.connector.file.sink.writer

Class FileWriter<IN>

java.lang.Object

org.apache.flink.connector.file.sink.writer.FileWriter<IN>

Type Parameters:

IN - The type of input elements.

All Implemented Interfaces:

AutoCloseable, org.apache.flink.api.connector.sink.Sink.ProcessingTimeService.ProcessingTimeCallback, org.apache.flink.api.connector.sink.SinkWriter<IN,FileSinkCommittable,FileWriterBucketState>

@Internal
public class FileWriter<IN>
extends Object
implements org.apache.flink.api.connector.sink.SinkWriter<IN,FileSinkCommittable,FileWriterBucketState>, org.apache.flink.api.connector.sink.Sink.ProcessingTimeService.ProcessingTimeCallback

A SinkWriter implementation for FileSink.

It writes data to and manages the different active buckes in the FileSink.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.api.connector.sink.SinkWriter

org.apache.flink.api.connector.sink.SinkWriter.Context

Constructor Summary

Constructors

Constructor and Description

FileWriter(org.apache.flink.core.fs.Path basePath, org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner<IN,String> bucketAssigner, FileWriterBucketFactory<IN> bucketFactory, org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<IN,String> bucketWriter, org.apache.flink.streaming.api.functions.sink.filesystem.RollingPolicy<IN,String> rollingPolicy, org.apache.flink.streaming.api.functions.sink.filesystem.OutputFileConfig outputFileConfig, org.apache.flink.api.connector.sink.Sink.ProcessingTimeService processingTimeService, long bucketCheckInterval) A constructor creating a new empty bucket manager.

Method Summary

Modifier and Type	Method and Description
void	close()
void	initializeState(List<FileWriterBucketState> bucketStates) Initializes the state after recovery from a failure.
void	onProcessingTime(long time)
List<FileSinkCommittable>	prepareCommit(boolean flush)
List<FileWriterBucketState>	snapshotState()
void	write(IN element, org.apache.flink.api.connector.sink.SinkWriter.Context context)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FileWriter

public FileWriter(org.apache.flink.core.fs.Path basePath,
                  org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner<IN,String> bucketAssigner,
                  FileWriterBucketFactory<IN> bucketFactory,
                  org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<IN,String> bucketWriter,
                  org.apache.flink.streaming.api.functions.sink.filesystem.RollingPolicy<IN,String> rollingPolicy,
                  org.apache.flink.streaming.api.functions.sink.filesystem.OutputFileConfig outputFileConfig,
                  org.apache.flink.api.connector.sink.Sink.ProcessingTimeService processingTimeService,
                  long bucketCheckInterval)

A constructor creating a new empty bucket manager.

Parameters:

basePath - The base path for our buckets.

bucketAssigner - The BucketAssigner provided by the user.

bucketFactory - The FileWriterBucketFactory to be used to create buckets.

bucketWriter - The BucketWriter to be used when writing data.

rollingPolicy - The RollingPolicy as specified by the user.

Method Detail

initializeState

public void initializeState(List<FileWriterBucketState> bucketStates)
                     throws IOException

Initializes the state after recovery from a failure.

During this process:

1. we set the initial value for part counter to the maximum value used before across all tasks and buckets. This guarantees that we do not overwrite valid data,
2. we commit any pending files for previous checkpoints (previous to the last successful one from which we restore),
3. we resume writing to the previous in-progress file of each bucket, and
4. if we receive multiple states for the same bucket, we merge them.

Parameters:

bucketStates - the state holding recovered state about active buckets.

Throws:

IOException - if anything goes wrong during retrieving the state or restoring/committing of any in-progress/pending part files

write

public void write(IN element,
                  org.apache.flink.api.connector.sink.SinkWriter.Context context)
           throws IOException

Specified by:

write in interface org.apache.flink.api.connector.sink.SinkWriter<IN,FileSinkCommittable,FileWriterBucketState>

Throws:

IOException

prepareCommit

public List<FileSinkCommittable> prepareCommit(boolean flush)
                                        throws IOException

Specified by:

prepareCommit in interface org.apache.flink.api.connector.sink.SinkWriter<IN,FileSinkCommittable,FileWriterBucketState>

Throws:

IOException

snapshotState

public List<FileWriterBucketState> snapshotState()
                                          throws IOException

Specified by:

snapshotState in interface org.apache.flink.api.connector.sink.SinkWriter<IN,FileSinkCommittable,FileWriterBucketState>

Throws:

IOException

close

public void close()

Specified by:

close in interface AutoCloseable

onProcessingTime

public void onProcessingTime(long time)
                      throws IOException

Specified by:

onProcessingTime in interface org.apache.flink.api.connector.sink.Sink.ProcessingTimeService.ProcessingTimeCallback

Throws:

IOException