MSK is one of the new services which just came out of Beta, it provides
the AWS users with the new Apache Kafka as a Service.

There is a lot in the documentation to read before you decide to use or
not this service in your environment so please make sure you go through
all the F.A.Q. because some of the usual stuff may not work as expected.
One of these is the auto-scaling of the cluster (does not have one) or
the fact that you cannot change the instance type in your cluster (for
the moment at least).

I just started working with it, and the first challenge was to create a
POC for the Dev team, needed to explore the capabilities of this product.
Being something that needs to be accessed directly from the developers’
laptop we needed this cluster to be publicly available to our network,
and this is how it started.

The new MSK service is not engineered to be available on the public
network and the workaround we found may not work for you, but I hope it
may be helpful to get an insight of how the service works.

First thing first I have created a new VPC over 3 availability zones, a
standard one with 3 Public subnets, 3 Private subnets, 1 NatGw, 3
InternetGw and a security group with only outgoing traffic allowed.
Over this VPC I have created a basic MSK cluster, no TLS auth, 10GB disk
space, 3 nodes one on each availability zone, each one in a public
subnet and assigned to the security group created before.

At this point, after 15 minutes, you will find yourself with a brand new
cluster and if you click the client details button you will get the
connection strings to use with your consumer/producer clients to reach
Kafka on the TLS or plain auth and ZooKeeper.
Problem being that this cluster is at the moment only reachable from the
inside of the VPC.

To make the cluster visible without the use of a VPN this is what I have
done:
– Request 3 EIP
– looking at the connection strings for the cluster resolve the names to
the private IP addresses
– Now look in the Network Interface section in the EC2 menu and
associate to each one of the cluster interface a public IP address (use
the IP address we found above to understand which of the network
interfaces are part of the cluster)
– Once done you will have a cluster with 3 public interfaces but not yet
reachable because the Security Group, so if we now modify the inbound
rules to accept connections on the right port for ZooKeper and Kafka.
– Now give it a try with telnet to verify that everything is open
– At this point you may try to connect but Kafka may throw an error at
you saying that ‘Connection to the cluster is working but the Kafka
cluster is not responsive’.
This is because we are not calling the Kafka nodes with the proper names.
– Edit your host file and insert for each node of the Kafka cluster an
entry with the public IP address and the name of that node; save it,
close it, try again.

Welcome to your new remote MKS cluster

Please note: every time you rebuild a cluster you will have to go
through all of this again, it may be a good idea to automate the
procedure if you think you will be destroying and rebuilding the cluster
over and over.
As well, remember that AWS will bill you for every IP address which is
not assigned, so please take care of cleaning up properly after you have
done with it.
For me this may be a one off job, but I may write a Cloudformation
template to couple Public IP addresses to Network Interfaces from the
Kafka nodes.
A VPN may work better, but this is just a POC and the less
infrastructure we put in place the less likely this POC may move from
POC to pre-prod or prod environment.