Streaming connectors

Flink has many connectors for easy integration that can be broadly categorized as follows:

  1. Built-in Connectors: Connectors that are already there along with the Flink installation and developed by various providers and are already supported and maintained come under this category:
    • File: Reads file from a specified file path. It can monitor a directory and as and when a file is added or changed can read and do the needful
    • Collection: Reads from elements and also is capable of reading from Java collections
    • Socket: Reads text socket from exposed port
    • Basic:
      • File
      • Socket
      • Standard output
    • Advanced:
      • Elasticsearch: Used to store and index JSON documents. We do have a dedicated chapter on this in which we will discuss this technology in a bit more detail. Flink supports both Elasticsearch 1.x and Elasticsearch 2.x
      • Cassandra
      • HDFS (Hadoop FileSystem)
    • RabbitMQ
    • Apache Kafka: Gives provision to convert a Kafka topic to DataStream and also can write to a DataStream. Flink has good integration support with Kafka
    • Apache NiFi
    • Amazon Kinesis Streams
    • Sources: Connectors that are configured as source in Flink comes in this.
    • Sinks: Connectors which are configured as sink in Flink are categorized in this
    • Source and Sink: Connectors that behave as both source and sink in the Flink ecosystem come into this category
  2. Custom Connectors: If these built-in connectors suit your chosen use case, Flink does give provision to implement your own connector. Flink has a toolbox using which you could do this with ease. Using the toolbox (specification interfaces), both custom source and sink can be implemented.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.142.166.31