Sending & Receiving Messages with Apache Kafka in Node.js

Step 1: Install Kafka Library for Node.js

Open a terminal in your Node.js project directory.

Run the following command to install the kafkajs library, a Node.js library for Apache Kafka: npm install kafkajs.

Step 2: Sending Messages with Producer in Node.js

Import the kafkajs library and define the Kafka Broker configuration:

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'your-client-id',
  brokers: ['broker1:port1', 'broker2:port2'], // Replace with actual addresses and ports
});

Create a producer to send messages, and send a message to a topic:

const producer = kafka.producer();

const sendMessage = async () => {
  await producer.connect();
  await producer.send({
    topic: 'your-topic',
    messages: [{ value: 'Hello Kafka!' }],
  });
  await producer.disconnect();
};

sendMessage();

Step 3: Receiving Messages with Consumer in Node.js

Import the kafkajs library and define the Kafka Broker configuration (if not done already):

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'your-client-id',
  brokers: ['broker1:port1', 'broker2:port2'], // Replace with actual addresses and ports
});

Create a consumer to receive messages from a specific topic:

const consumer = kafka.consumer({ groupId: 'your-group-id' });

const consumeMessages = async () => {
  await consumer.connect();
  await consumer.subscribe({ topic: 'your-topic', fromBeginning: true });

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      console.log(`Received message: ${message.value}`);
    },
  });
};

consumeMessages();

Note: Replace values like 'your-client-id', 'broker1:port1', 'your-topic', and 'your-group-id' with your actual project information.

Make sure to refer to the official documentation of Apache Kafka and the kafkajs library for more information about configuration options and functionalities.