Kafka connect

Extensibility is one of the important design principle followed rigorously by Kafka. The Kafka Connect tool makes Kafka extensible. The tool enables Kafka to connect with external systems and helps bring data into it, and also out from it to other systems. It has a common framework, using which custom connectors can be written. More details on Kafka connect can be found in Kafka documentation in https://kafka.apache.org/documentation.html#connect.

The following figure shows how Kafka Connect works.

Figure 13: Kafka connect working

The Kafka connectors are categorized into two:

  • Source Connectors: Connectors which bring data into Kafka topics.
  • Sink Connectors: Connectors which take data away from topics into other external systems

There are a huge list of connectors available, catering to various external systems, using which Kafka can hook onto them. These existing connectors are again categorized broadly into two:

  • Certified connectors: Connectors which are written using the Kafka Connect framework and have already passed the best coding practice. They are usually created and managed by vendors. These vendors write these connectors to enable Kafka to connect and integrate into their tool.
  • Other connectors (non-certified): These are connectors which are not yet certified and are maintained and managed by the community.

A full list of certified and non-certified connectors can be found at the link https://www.confluent.io/product/connectors/.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.145.172.146