Kafka Connect is the primary method for doing integration with Apache Kafka and other systems. It includes Single Message Transform (SMT) which can be used to modify messages as they pass through the pipeline in or out of Kafka.
Just like the `RegExRouter`, the `TimeStampRouter` can be used to modify the topic name of messages as they pass through Kafka Connect. Since the topic name is usually the basis for the naming of the object to which messages are written in a sink connector, this is a great way to achieve time-based partitioning of those objects if required. For example, instead of streaming messages from Kafka to an Elasticsearch index called `cars`, they can be routed to monthly indices e.g. `cars_2020-10`, `cars_2020-11`, `cars_2020-12`, etc.
The `TimeStampRouter` takes two arguments; the format of the final topic name to generate, and the format of the timestamp to put in the topic name.
---
👾 Demo code: github.com/confluentinc/demo-...
🗒️ SMT reference: docs.confluent.io/platform/cu...
Learn more about Kafka Connect here:
🏃♂️ Quick: • Kafka Connect in 60 se...
🚶 More detail: rmoff.dev/kafka-connect-zero-...
---
☁️ Confluent Cloud: confluent.cloud/signup?...
💾Download Confluent Platform: www.confluent.io/download/?ut...
📺 Kafka Connect connector deep-dives: • Kafka Connect
✍️Kafka Connect documentation: docs.confluent.io/current/con...
🧩Confluent Hub: www.confluent.io/hub/?...
15 дек 2020
