Rancher with Kafka Cluster: Producing and consuming Kafka messages via JAVA

Hello everybody
I do have some questions on how the interplay between kafka, zookeeper and external java applications is supposed to be designed:

Situation:
I have a single machine with a rancher server running in a docker.
Additionally a single node zookeeper and a kafka cluster with three nodes have been added (from the catalog) to the same maschine.
Zookeeper and the kafka nodes have an internal IP from rancher network segment 10.42.XX.XX
Zookeeper and kafka nodes expose an ip to host (192.168.178.88)

By that the kafka nodes have the ip:port form outside: 192.168.178.88:9093,192.168.178.88:9094,192.168.178.88:9095
while the zookeeper node has the ip:port: 192.168.178.88:2181

From the console of each kafka node I am able to produce and consume messages; so everything seems to work ok.

However I can neither produce nor consume messages via java running on another machine.
Obviously the java producer initially communicates via bootstrap server with the kafka nodes, they subsequently communicate with zookeeper which finally returns the internal IPs of the kafka nodes which of course are not accessible from the outside world!
So no messages can be send to, or read from, kafka via Java.

This pattern seems a bit odd to me because it implies that no communication can be established
to a kafka cluster running in rancher from the outside world.
So I suppose I must have missunderstood something… :frowning:

any hints? Thanks in advance!

3 Likes

Hi there,

I have the same question - have you been able to figure out since posting this?

Regards
Jaco