On November 5, 2014, DataTorrent, a company founded by ex-Yahoo!, announced a new project to bring the fault-tolerant, high-performance, scalable Apache Kafka messaging system to YARN.

The so-called Kafka on YARN (KOYA) project plans to leverage YARN for Kafka broker management, automatic broker recovery, and more. Planned features include a fully-HA ApplicationMaster, sticky allocation of containers (so that a restart can access local data), a web interface for Kafka, and more.

The expected release to the open source community is somewhere in Q2 2015.

More information is available at https://www.datatorrent.com/introducing-koya-apache-kafka-on-apache-hadoop-2-0-yarn/.