Package io.smallrye.reactive.messaging.kafka.companion

Class ConsumerBuilder<K,V>

java.lang.Object
io.smallrye.reactive.messaging.kafka.companion.ConsumerBuilder<K,V>
Type Parameters:
K - The record key type
V - The record value type
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.kafka.clients.consumer.ConsumerRebalanceListener
public class ConsumerBuilder<K,V> extends Object implements org.apache.kafka.clients.consumer.ConsumerRebalanceListener, Closeable
Kafka consumer wrapper for creating tasks (ConsumerTask) that consume records.

The wrapped Kafka consumer is created lazily when a ConsumerTask is created and started. Until the consumer is created builder methods prefixed with with- act to provide additional configuration.

The created consumer is closed automatically when the task is terminated. The consumer uses an underlying ExecutorService for polling records. As Kafka consumer is not thread-safe, other operations accessing the underlying consumer are also run inside the ExecutorService

  • Constructor Details

    • ConsumerBuilder

      public ConsumerBuilder(Map<String,Object> props, Duration kafkaApiTimeout, org.apache.kafka.common.serialization.Deserializer<K> keyDeser, org.apache.kafka.common.serialization.Deserializer<V> valueDeser)
      Creates a new ConsumerBuilder.

      Note that on producer creation Deserializer.configure(java.util.Map<java.lang.String, ?>, boolean) methods will NOT be called.

      Parameters:
      props - the initial properties for consumer creation
      kafkaApiTimeout - the timeout for api calls to Kafka
      keyDeser - the deserializer for record keys
      valueDeser - the deserializer for record values

    • ConsumerBuilder

      public ConsumerBuilder(Map<String,Object> props, Duration kafkaApiTimeout, String keyDeserializerType, String valueDeserializerType)
      Creates a new ConsumerBuilder.

      On consumer creation, key and value deserializers will be created and Deserializer.configure(java.util.Map<java.lang.String, ?>, boolean) methods will be called.

      Parameters:
      props - the initial properties for consumer creation
      kafkaApiTimeout - the timeout for api calls to Kafka
      keyDeserializerType - the deserializer class name for record keys
      valueDeserializerType - the deserializer class name for record values

  • Method Details

    • unwrap

      public org.apache.kafka.clients.consumer.KafkaConsumer<K,V> unwrap()
      Returns:
      the underlying KafkaConsumer, may be null if producer is not yet created.

    • onPartitionsRevoked

      public void onPartitionsRevoked(Collection<org.apache.kafka.common.TopicPartition> partitions)
      Specified by:
      onPartitionsRevoked in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener

    • onPartitionsAssigned

      public void onPartitionsAssigned(Collection<org.apache.kafka.common.TopicPartition> partitions)
      Specified by:
      onPartitionsAssigned in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener

    • close

      public void close()
      Close the underlying KafkaConsumer.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable

    • terminate

      public void terminate(org.apache.kafka.clients.consumer.KafkaConsumer<K,V> consumer, Throwable throwable)

    • clientId

      public String clientId()
      Returns:
      the client.id property of this consumer

    • groupId

      public String groupId()
      Returns:
      the group.id property of this consumer

    • withProp

      public ConsumerBuilder<K,V> withProp(String key, String value)
      Add property to the configuration to be used when the consumer is created.
      Parameters:
      key - the key for property
      value - the value for property
      Returns:
      this ConsumerBuilder

    • withProps

      public ConsumerBuilder<K,V> withProps(Map<String,Object> properties)
      Add properties to the configuration to be used when the consumer is created.
      Parameters:
      properties - the properties
      Returns:
      this ConsumerBuilder

    • withClientId

      public ConsumerBuilder<K,V> withClientId(String clientId)
      Add configuration property for client.id
      Parameters:
      clientId - the client id
      Returns:
      this ConsumerBuilder

    • withGroupId

      public ConsumerBuilder<K,V> withGroupId(String groupId)
      Add configuration property for group.id
      Parameters:
      groupId - the group id
      Returns:
      this ConsumerBuilder

    • withOffsetReset

      public ConsumerBuilder<K,V> withOffsetReset(org.apache.kafka.clients.consumer.OffsetResetStrategy offsetResetStrategy)
      Add configuration property for auto.offset.reset
      Parameters:
      offsetResetStrategy - the offset reset strategy
      Returns:
      this ConsumerBuilder

    • withIsolationLevel

      public ConsumerBuilder<K,V> withIsolationLevel(org.apache.kafka.common.IsolationLevel isolationLevel)
      Set isolation level
      Parameters:
      isolationLevel - the isolation level
      Returns:
      this ConsumerBuilder

    • withPollTimeout

      public ConsumerBuilder<K,V> withPollTimeout(Duration timeout)
      Set poll timeout
      Parameters:
      timeout - the poll timeout
      Returns:
      this ConsumerBuilder

    • withAutoCommit

      public ConsumerBuilder<K,V> withAutoCommit()
      Enable auto commit
      Returns:
      this ConsumerBuilder

    • withCommitAsyncWhen

      public ConsumerBuilder<K,V> withCommitAsyncWhen(Predicate<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>> commitPredicate)
      If set the given function is used to determine when to commit async
      Parameters:
      commitPredicate - the commit async predicate
      Returns:
      this ConsumerBuilder

    • withCommitAsync

      public ConsumerBuilder<K,V> withCommitAsync(Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>,Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata>> commitFunction, org.apache.kafka.clients.consumer.OffsetCommitCallback callback)
      If set the given function is used to determine when and which offsets to commit async
      Parameters:
      commitFunction - the function to call for commit async
      callback - the callback to set for commit async
      Returns:
      this ConsumerBuilder

    • withCommitSyncWhen

      public ConsumerBuilder<K,V> withCommitSyncWhen(Predicate<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>> commitPredicate)
      If set the given function is used to determine when to commit sync
      Parameters:
      commitPredicate - the commit sync predicate
      Returns:
      this ConsumerBuilder

    • withCommitSync

      public ConsumerBuilder<K,V> withCommitSync(Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>,Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata>> commitFunction)
      If set the given function is used to determine when and which offsets to commit sync
      Parameters:
      commitFunction - the function to call
      Returns:
      this ConsumerBuilder

    • withOnPartitionsAssigned

      public ConsumerBuilder<K,V> withOnPartitionsAssigned(Consumer<Collection<org.apache.kafka.common.TopicPartition>> onPartitionsAssigned)
      Parameters:
      onPartitionsAssigned - the callback to be called on partitions assigned
      Returns:
      this ConsumerBuilder

    • withOnPartitionsRevoked

      public ConsumerBuilder<K,V> withOnPartitionsRevoked(Consumer<Collection<org.apache.kafka.common.TopicPartition>> onPartitionsRevoked)
      Parameters:
      onPartitionsRevoked - the callback to be called on partitions revoked
      Returns:
      this ConsumerBuilder

    • withOnTermination

      public ConsumerBuilder<K,V> withOnTermination(BiConsumer<org.apache.kafka.clients.consumer.KafkaConsumer<K,V>,Throwable> onTermination)
      Parameters:
      onTermination - the callback to be called on termination
      Returns:
      this ConsumerBuilder

    • groupMetadata

      public org.apache.kafka.clients.consumer.ConsumerGroupMetadata groupMetadata()
      Returns:
      the consumer group metadata

    • currentAssignment

      public Set<org.apache.kafka.common.TopicPartition> currentAssignment()
      Returns:
      the current partition assignments of this consumer

    • assignment

      public Set<org.apache.kafka.common.TopicPartition> assignment()
      Returns:
      the partition assignment of this consumer

    • waitForAssignment

      public io.smallrye.mutiny.Uni<Set<org.apache.kafka.common.TopicPartition>> waitForAssignment()
      Returns:
      the Uni for waiting on the assignments of partitions

    • position

      public long position(org.apache.kafka.common.TopicPartition partition)
      Parameters:
      partition - the topic partition
      Returns:
      the position of this consumer for the given topic partition

    • position

      public Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> position()
      Returns:
      the position of this consumer for all assigned topic partitions

    • resetToLastCommittedPositions

      public void resetToLastCommittedPositions()
      Reset to the last committed position

    • committed

      public org.apache.kafka.clients.consumer.OffsetAndMetadata committed(org.apache.kafka.common.TopicPartition partition)
      Parameters:
      partition - the topic partition
      Returns:
      the map of committed offsets for the topic partition

    • committed

      public Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> committed(org.apache.kafka.common.TopicPartition... partitions)
      Parameters:
      partitions - the topic partitions
      Returns:
      the map of committed offsets for topic partitions

    • committed

      public Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> committed()
      Returns:
      the map of committed offsets for topic partitions assigned to this consumer

    • pause

      public void pause()
      Pause the consumer

    • resume

      public void resume()
      Resume the consumer

    • fromOffsets

      public ConsumerTask<K,V> fromOffsets(Map<org.apache.kafka.common.TopicPartition,Long> offsets, Function<io.smallrye.mutiny.Multi<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>>,io.smallrye.mutiny.Multi<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>>> plugFunction)
      Create ConsumerTask for consuming records starting from the given offsets.

      The plug function is used to modify the Multi generating the records, ex. limiting the number of records produced.

      The task will run until a failure occurs or is explicitly stopped (subscription to the Multi cancelled).

      Parameters:
      offsets - the map of offsets to start consuming records
      plugFunction - the function to apply to the resulting multi
      Returns:
      the ConsumerTask

    • fromOffsets

      public ConsumerTask<K,V> fromOffsets(Map<org.apache.kafka.common.TopicPartition,Long> offsets)
      Create ConsumerTask for consuming records starting from the given offsets.

      The task will run until a failure occurs or is explicitly stopped (subscription to the Multi cancelled).

      Parameters:
      offsets - the map of offsets to start consuming records
      Returns:
      the ConsumerTask

    • fromOffsets

      public ConsumerTask<K,V> fromOffsets(Map<org.apache.kafka.common.TopicPartition,Long> offsets, Duration during)
      Create ConsumerTask for consuming records starting from the given offsets.

      The resulting ConsumerTask will be already started. The task will run until the Multi is terminated (with completion or failure) or is explicitly stopped (subscription to the Multi cancelled).

      Parameters:
      offsets - the map of offsets to start consuming records
      during - the duration of the producing task to run
      Returns:
      the ConsumerTask

    • fromOffsets

      public ConsumerTask<K,V> fromOffsets(Map<org.apache.kafka.common.TopicPartition,Long> offsets, long numberOfRecords)
      Create ConsumerTask for consuming records starting from the given offsets.

      The resulting ConsumerTask will be already started. The task will run for generating the given number of records.

      Parameters:
      offsets - the map of offsets to start consuming records
      numberOfRecords - the number of records to produce
      Returns:
      the ConsumerTask

    • fromTopics

      public ConsumerTask<K,V> fromTopics(Set<String> topics, Function<io.smallrye.mutiny.Multi<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>>,io.smallrye.mutiny.Multi<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>>> plugFunction)
      Create ConsumerTask for consuming records from the given topics.

      The plug function is used to modify the Multi generating the records, ex. limiting the number of records produced.

      The task will run until a failure occurs or is explicitly stopped (subscription to the Multi cancelled).

      Parameters:
      topics - the set of topics to subscribe
      plugFunction - the function to apply to the resulting multi
      Returns:
      the ConsumerTask

    • fromTopics

      public ConsumerTask<K,V> fromTopics(String topic, Function<io.smallrye.mutiny.Multi<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>>,io.smallrye.mutiny.Multi<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>>> plugFunction)
      Create ConsumerTask for consuming records from the given topic.

      The plug function is used to modify the Multi generating the records, ex. limiting the number of records produced.

      The resulting ConsumerTask will be already started. The task will run until the Multi is terminated (with completion or failure) or is explicitly stopped (subscription to the Multi cancelled).

      Parameters:
      topic - the topic to subscribe
      plugFunction - the function to apply to the resulting multi
      Returns:
      the ConsumerTask

    • fromTopics

      public ConsumerTask<K,V> fromTopics(String... topics)
      Create ConsumerTask for consuming records from the given topics.

      The resulting ConsumerTask will be already started. The task will run until a failure occurs or is explicitly stopped (subscription to the Multi cancelled).

      Parameters:
      topics - the set of topics to subscribe
      Returns:
      the ConsumerTask

    • fromTopics

      public ConsumerTask<K,V> fromTopics(Set<String> topics)
      Create ConsumerTask for consuming records from the given topics.

      The resulting ConsumerTask will be already started. The task will run until a failure occurs or is explicitly stopped (subscription to the Multi cancelled).

      Parameters:
      topics - the set of topics to subscribe
      Returns:
      the ConsumerTask

    • fromTopics

      public ConsumerTask<K,V> fromTopics(Set<String> topics, long numberOfRecords)
      Create ConsumerTask for consuming records from the given topics.

      The resulting ConsumerTask will be already started. The task will run until the given number of records consumed.

      Parameters:
      topics - the set of topics to subscribe
      numberOfRecords - the number of records to produce
      Returns:
      the ConsumerTask

    • fromTopics

      public ConsumerTask<K,V> fromTopics(Set<String> topics, Duration during)
      Create ConsumerTask for consuming records from the given topics.

      The resulting ConsumerTask will be already started. The task will run during the given duration.

      Parameters:
      topics - the set of topics to subscribe
      during - the duration of the producing task to run
      Returns:
      the ConsumerTask

    • fromTopics

      public ConsumerTask<K,V> fromTopics(String topic, long numberOfRecords)
      Create ConsumerTask for consuming records from the given topic.

      The resulting ConsumerTask will be already started. The task will run until the given number of records consumed.

      Parameters:
      topic - the topic to subscribe
      numberOfRecords - the number of records to produce
      Returns:
      the ConsumerTask

    • fromTopics

      public ConsumerTask<K,V> fromTopics(String topic, Duration during)
      Create ConsumerTask for consuming records from the given topic.

      The resulting ConsumerTask will be already started. The task will run during the given duration.

      Parameters:
      topic - the topic to subscribe
      during - the duration of the producing task to run
      Returns:
      the ConsumerTask

    • fromTopics

      public ConsumerTask<K,V> fromTopics(String topic, long numberOfRecords, Duration during)
      Create ConsumerTask for consuming records from the given topic.

      The resulting ConsumerTask will be already started. The task will run until the given number of records consumed or during the given duration.

      Parameters:
      topic - the topic to subscribe
      numberOfRecords - the number of records to produce
      during - the duration of the producing task to run
      Returns:
      the ConsumerTask

    • fromPrevious

      public ConsumerTask<K,V> fromPrevious(KafkaTask<?,?> task, Function<io.smallrye.mutiny.Multi<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>>,io.smallrye.mutiny.Multi<org.apache.kafka.clients.consumer.ConsumerRecord<K,V>>> plugFunction)

    • fromPrevious

      public ConsumerTask<K,V> fromPrevious(KafkaTask<?,?> task)

    • fromPrevious

      public ConsumerTask<K,V> fromPrevious(KafkaTask<?,?> task, long numberOfRecords)

    • fromPrevious

      public ConsumerTask<K,V> fromPrevious(KafkaTask<?,?> task, Duration during)

    • fromPrevious

      public ConsumerTask<K,V> fromPrevious(KafkaTask<?,?> task, long numberOfRecords, Duration during)

    • commitAndClose

      public void commitAndClose(Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> offsets)
      Commit the given offset and close the consumer
      Parameters:
      offsets - the map of offsets to commit

    • commitAndClose

      public void commitAndClose()
      Commit the current offset and close the consumer