kafka_2.10-0.10.0.1 集群安装配置使用独立zookeeper集群

一、集群环境

• CentOS 6.6 X64
• zookeeper集群配置
• kafka_2.10-0.10.0.1.tgz
• 集群配置如下,之前已经部署过hadoop和zookeeper

master.hdp.imdst.com  zookeeper.0 kafka.0  
1.slave.hdp.imdst.com zookeeper.1 kafka.1  
2.slave.hdp.imdst.com zookeeper.2 kafka.2  

二、安装kafka

• 下载、解压

wget http://mirrors.cnnic.cn/apache/kafka/0.10.0.1/kafka_2.10-0.10.0.1.tgz  
tar zxf kafka_2.10-0.10.0.1.tgz -C /home/hadoop/src  
chown hadoop.hadoop /home/hadoop -R  

• 配置master

su hadoop  
cd ~/src && ln -s kafka_2.10-0.10.0.1 kafka  
vim kafka/config/server.properties  
------------------------------------
broker.id=0  
listeners=PLAINTEXT://:9092  
advertised.listeners=PLAINTEXT://:9092  
log.dirs=/data/kafka-logs  
zookeeper.connect=master.hdp.imdst.com:2181,1.slave.hdp.imdst.com:2181,2.slave.hdp.imdst.com:2181  
------------------------------------

• 配置slave 1和2

cd ~/src  
scp -r kafka 1.slave.hdp.imdst.com:/home/hadoop/src/  
scp -r kafka 2.slave.hdp.imdst.com:/home/hadoop/src/  
#分别修改server.properties
broker.id=1 和broker.id=2  

三、启动和关闭

• 启动

cd ~/src/kafka/bin  
 ./kafka-server-start.sh -daemon  ../config/server.properties 

• 关闭

cd ~/src/kafka/bin  
./kafka-server-stop.sh

四、在任意一台机器上测试

• 创建一个名为"mytest-topic"的topic，该topic切分为4份，每一份备份数为3

bin/kafka-topics.sh --create --zookeeper master.hdp.imdst.com:2181 --replication-factor 3 --partitions 4 --topic mytest-topic  

• 列出所有topic

./kafka-topics.sh --list --zookeeper master.hdp.imdst.com:2181,1.slave.hdp.imdst.com:2181,2.slave.hdp.imdst.com:2181

• 在zookeeper中可以查看到自己创建的topic

cd ~/src/zookeeper/bin  
./zkCli.sh
Connecting to localhost:2181  
[zk: localhost:2181(CONNECTED) 0] ls /brokers/topics
[mytest-topic, leoiceo-topic]