Kafka Source Connector Configuration Reference for Confluent Platformï
This topic includes Kafka Connect source connector configuration properties available for Confluent Platform. The parameters are organized by order of importance, ranked from high to low.
Search by configuration property nameï
Enter a string to search and filter by configuration property name.
nameï
Globally unique name to use for this connector.
Type: | string |
Default: | |
Valid Values: | non-empty string without ISO control characters |
Importance: | high |
connector.classï
Name or alias of the class for this connector. Must be a subclass of org.apache.kafka.connect.connector.Connector. If the connector is org.apache.kafka.connect.file.FileStreamSinkConnector, you can either specify this full name, or use âFileStreamSinkâ or âFileStreamSinkConnectorâ to make the configuration a bit shorter
Type: | string |
Default: | |
Valid Values: | |
Importance: | high |
connector.plugin.versionï
Version of the connector.
Type: | string |
Default: | null |
Valid Values: | org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@161cd475 |
Importance: | medium |
tasks.maxï
Maximum number of tasks to use for this connector.
Type: | int |
Default: | 1 |
Valid Values: | [1,â¦] |
Importance: | high |
tasks.max.enforceï
(Deprecated) Whether to enforce that the tasks.max property is respected by the connector. By default, connectors that generate too many tasks will fail, and existing sets of tasks that exceed the tasks.max property will also be failed. If this property is set to false, then connectors will be allowed to generate more than the maximum number of tasks, and existing sets of tasks that exceed the tasks.max property will be allowed to run. This property is deprecated and will be removed in an upcoming major release.
Type: | boolean |
Default: | true |
Valid Values: | |
Importance: | low |
key.converterï
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the keys in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro.
Type: | class |
Default: | null |
Valid Values: | A concrete subclass of org.apache.kafka.connect.storage.Converter, A class with a public, no-argument constructor |
Importance: | low |
key.converter.plugin.versionï
Version of the key converter.
Type: | string |
Default: | null |
Valid Values: | org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@532760d8 |
Importance: | low |
value.converterï
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro.
Type: | class |
Default: | null |
Valid Values: | A concrete subclass of org.apache.kafka.connect.storage.Converter, A class with a public, no-argument constructor |
Importance: | low |
value.converter.plugin.versionï
Version of the value converter.
Type: | string |
Default: | null |
Valid Values: | org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@57fa26b7 |
Importance: | low |
header.converterï
HeaderConverter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the header values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. By default, the SimpleHeaderConverter is used to serialize header values to strings and deserialize them by inferring the schemas.
Type: | class |
Default: | null |
Valid Values: | A concrete subclass of org.apache.kafka.connect.storage.HeaderConverter, A class with a public, no-argument constructor |
Importance: | low |
header.converter.plugin.versionï
Version of the header converter.
Type: | string |
Default: | null |
Valid Values: | org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@5f8ed237 |
Importance: | low |
config.action.reloadï
The action that Connect should take on the connector when changes in external configuration providers result in a change in the connectorâs configuration properties. A value of ânoneâ indicates that Connect will do nothing. A value of ârestartâ indicates that Connect should restart/reload the connector with the updated configuration properties.The restart may actually be scheduled in the future if the external configuration provider indicates that a configuration value will expire in the future.
Type: | string |
Default: | restart |
Valid Values: | [none, restart] |
Importance: | low |
transformsï
Aliases for the transformations to be applied to records.
Type: | list |
Default: | ââ |
Valid Values: | non-null string, unique transformation aliases |
Importance: | low |
predicatesï
Aliases for the predicates used by transformations.
Type: | list |
Default: | ââ |
Valid Values: | non-null string, unique predicate aliases |
Importance: | low |
errors.retry.timeoutï
The maximum duration in milliseconds that a failed operation will be reattempted. The default is 0, which means no retries will be attempted. Use -1 for infinite retries.
Type: | long |
Default: | 0 |
Valid Values: | |
Importance: | medium |
errors.retry.delay.max.msï
The maximum duration in milliseconds between consecutive retry attempts. Jitter will be added to the delay once this limit is reached to prevent thundering herd issues.
Type: | long |
Default: | 60000 (1 minute) |
Valid Values: | |
Importance: | medium |
errors.toleranceï
Behavior for tolerating errors during connector operation. ânoneâ is the default value and signals that any error will result in an immediate connector task failure; âallâ changes the behavior to skip over problematic records.
Type: | string |
Default: | none |
Valid Values: | [none, all] |
Importance: | medium |
errors.log.enableï
If true, write each error and the details of the failed operation and problematic record to the Connect application log. This is âfalseâ by default, so that only errors that are not tolerated are reported.
Type: | boolean |
Default: | false |
Valid Values: | |
Importance: | medium |
errors.log.include.messagesï
Whether to include in the log the Connect record that resulted in a failure. For sink records, the topic, partition, offset, and timestamp will be logged. For source records, the key and value (and their schemas), all headers, and the timestamp, Kafka topic, Kafka partition, source partition, and source offset will be logged. This is âfalseâ by default, which will prevent record keys, values, and headers from being written to log files.
Type: | boolean |
Default: | false |
Valid Values: | |
Importance: | medium |
topic.creation.groupsï
Groups of configurations for topics created by source connectors
Type: | list |
Default: | ââ |
Valid Values: | non-null string, unique topic creation groups |
Importance: | low |
exactly.once.supportï
Permitted values are requested, required. If set to ârequiredâ, forces a preflight check for the connector to ensure that it can provide exactly-once semantics with the given configuration. Some connectors may be capable of providing exactly-once semantics but not signal to Connect that they support this; in that case, documentation for the connector should be consulted carefully before creating it, and the value for this property should be set to ârequestedâ. Additionally, if the value is set to ârequiredâ but the worker that performs preflight validation does not have exactly-once support enabled for source connectors, requests to create or validate the connector will fail.
Type: | string |
Default: | requested |
Valid Values: | (case insensitive) [REQUIRED, REQUESTED] |
Importance: | medium |
transaction.boundaryï
Permitted values are: poll, interval, connector. If set to âpollâ, a new producer transaction will be started and committed for every batch of records that each task from this connector provides to Connect. If set to âconnectorâ, relies on connector-defined transaction boundaries; note that not all connectors are capable of defining their own transaction boundaries, and in that case, attempts to instantiate a connector with this value will fail. Finally, if set to âintervalâ, commits transactions only after a user-defined time interval has passed.
Type: | string |
Default: | poll |
Valid Values: | (case insensitive) [INTERVAL, POLL, CONNECTOR] |
Importance: | medium |
transaction.boundary.interval.msï
If âtransaction.boundaryâ is set to âintervalâ, determines the interval for producer transaction commits by connector tasks. If unset, defaults to the value of the worker-level âoffset.flush.interval.msâ property. It has no effect if a different transaction.boundary is specified.
Type: | long |
Default: | null |
Valid Values: | [0,â¦] |
Importance: | low |
offsets.storage.topicï
The name of a separate offsets topic to use for this connector. If empty or not specified, the workerâs global offsets topic name will be used. If specified, the offsets topic will be created if it does not already exist on the Kafka cluster targeted by this connector (which may be different from the one used for the workerâs global offsets topic if the bootstrap.servers property of the connectorâs producer has been overridden from the workerâs). Only applicable in distributed mode; in standalone mode, setting this property will have no effect.
Type: | string |
Default: | null |
Valid Values: | non-empty string |
Importance: | low |
Note
This website includes content developed at the Apache Software Foundation under the terms of the Apache License v2.