尚硅谷大數據技術之Kafka第2章 Kafka集羣部署

2.1 環境準備

2.1.1 集羣規劃

hadoop102           hadoop103           hadoop104html

zk                          zk                          zklinux

kafka                     kafka                    kafkaapache

2.1.2 jar下載

http://kafka.apache.org/downloads.html緩存

2.1.3 虛擬機準備

1準備3虛擬機服務器

2配置ip地址網絡

尚硅谷大數據技術之修改成靜態IPsession

 

3配置主機名稱app

尚硅谷大數據技術之修改主機名異步

 

43主機分別關閉防火牆socket

 

[root@hadoop102 atguigu]# chkconfig iptables off

 

[root@hadoop103 atguigu]# chkconfig iptables off

 

[root@hadoop104 atguigu]# chkconfig iptables off

 

2.1.4 安裝jdk

尚硅谷大數據技術之安裝jdk

 

2.1.5 安裝Zookeeper

 

0)集羣規劃

 

hadoop102hadoop103hadoop104三個節點上部署Zookeeper

 

1解壓安裝

 

1)解壓zookeeper安裝包到/opt/module/目錄下

 

[atguigu@hadoop102 software]$ tar -zxvf zookeeper-3.4.10.tar.gz -C /opt/module/

 

2)在/opt/module/zookeeper-3.4.10/這個目錄下建立zkData

 

mkdir -p zkData

 

3)重命名/opt/module/zookeeper-3.4.10/conf這個目錄下的zoo_sample.cfgzoo.cfg

 

mv zoo_sample.cfg zoo.cfg

 

2)配置zoo.cfg文件

 

1)具體配置

 

dataDir=/opt/module/zookeeper-3.4.10/zkData

 

增長以下配置

 

#######################cluster##########################

 

server.2=hadoop102:2888:3888

 

server.3=hadoop103:2888:3888

 

server.4=hadoop104:2888:3888

 

2)配置參數解讀

 

Server.A=B:C:D

 

A是一個數字,表示這個是第幾號服務器;

 

B是這個服務器的ip地址;

 

C是這個服務器與集羣中的Leader服務器交換信息的端口;

 

D萬一集羣中的Leader服務器掛了,須要一個端口來從新進行選舉,選出一個新的Leader,而這個端口就是用來執行選舉時服務器相互通訊的端口。

 

集羣模式下配置一個文件myid這個文件在dataDir目錄下,這個文件裏面有一個數據就是A的值,Zookeeper啓動時讀取此文件,拿到裏面數據與zoo.cfg裏面的配置信息比較從而判斷究竟是哪一個server

 

3集羣操做

 

1/opt/module/zookeeper-3.4.10/zkData目錄下建立一個myid的文件

 

touch myid

 

添加myid文件,注意必定要在linux裏面建立notepad++裏面極可能亂碼

 

2編輯myid文件

 

vi myid

 

文件中添加server的編號:如2

 

3)拷貝配置好的zookeeper到其餘機器上

 

scp -r zookeeper-3.4.10/ root@hadoop103.atguigu.com:/opt/app/

 

scp -r zookeeper-3.4.10/ root@hadoop104.atguigu.com:/opt/app/

 

分別修改myid文件中內容爲34

 

4分別啓動zookeeper

 

[root@hadoop102 zookeeper-3.4.10]# bin/zkServer.sh start

 

[root@hadoop103 zookeeper-3.4.10]# bin/zkServer.sh start

 

[root@hadoop104 zookeeper-3.4.10]# bin/zkServer.sh start

 

5查看狀態

 

[root@hadoop102 zookeeper-3.4.10]# bin/zkServer.sh status

 

JMX enabled by default

 

Using config: /opt/module/zookeeper-3.4.10/bin/../conf/zoo.cfg

 

Mode: follower

 

[root@hadoop103 zookeeper-3.4.10]# bin/zkServer.sh status

 

JMX enabled by default

 

Using config: /opt/module/zookeeper-3.4.10/bin/../conf/zoo.cfg

 

Mode: leader

 

[root@hadoop104 zookeeper-3.4.5]# bin/zkServer.sh status

 

JMX enabled by default

 

Using config: /opt/module/zookeeper-3.4.10/bin/../conf/zoo.cfg

 

Mode: follower

 

2.2 Kafka集羣部署

 

1解壓安裝包

 

[atguigu@hadoop102 software]$ tar -zxvf kafka_2.11-0.11.0.0.tgz -C /opt/module/

 

2修改解壓後的文件名稱

 

[atguigu@hadoop102 module]$ mv kafka_2.11-0.11.0.0/ kafka

 

3)在/opt/module/kafka目錄下建立logs文件夾

 

[atguigu@hadoop102 kafka]$ mkdir logs

 

4修改配置文件

 

[atguigu@hadoop102 kafka]$ cd config/

 

[atguigu@hadoop102 config]$ vi server.properties

 

輸入如下內容:

 

#broker的全局惟一編號,不能重複

broker.id=0

#刪除topic功能使能

delete.topic.enable=true

#處理網絡請求的線程數量

num.network.threads=3

#用來處理磁盤IO的現成數量

num.io.threads=8

#發送套接字的緩衝區大小

socket.send.buffer.bytes=102400

#接收套接字的緩衝區大小

socket.receive.buffer.bytes=102400

#請求套接字的緩衝區大小

socket.request.max.bytes=104857600

#kafka運行日誌存放的路徑

log.dirs=/opt/module/kafka/logs

#topic在當前broker上的分區個數

num.partitions=1

#用來恢復和清理data下數據的線程數量

num.recovery.threads.per.data.dir=1

#segment文件保留的最長時間,超時將被刪除

log.retention.hours=168

#配置鏈接Zookeeper集羣地址

zookeeper.connect=hadoop102:2181,hadoop103:2181,hadoop104:2181

 

5)配置環境變量

 

[root@hadoop102 module]# vi /etc/profile

 

#KAFKA_HOME

export KAFKA_HOME=/opt/module/kafka

export PATH=$PATH:$KAFKA_HOME/bin

 

[root@hadoop102 module]# source /etc/profile

 

6)分發安裝包

 

[root@hadoop102 etc]# xsync profile

 

[atguigu@hadoop102 module]$ xsync kafka/

 

7)分別在hadoop103hadoop104上修改配置文件/opt/module/kafka/config/server.propertiesbroker.id=1broker.id=2

 

broker.id不得重複

 

8啓動集羣

 

依次在hadoop102hadoop103hadoop104節點上啓動kafka

 

[atguigu@hadoop102 kafka]$ bin/kafka-server-start.sh config/server.properties &

 

[atguigu@hadoop103 kafka]$ bin/kafka-server-start.sh config/server.properties &

 

[atguigu@hadoop104 kafka]$ bin/kafka-server-start.sh config/server.properties &

 

9)關閉集羣

 

[atguigu@hadoop102 kafka]$ bin/kafka-server-stop.sh stop

 

[atguigu@hadoop103 kafka]$ bin/kafka-server-stop.sh stop

 

[atguigu@hadoop104 kafka]$ bin/kafka-server-stop.sh stop

 

2.3 Kafka命令行操做

 

1)查看當前服務器中的全部topic

 

[atguigu@hadoop102 kafka]$ bin/kafka-topics.sh --zookeeper hadoop102:2181 --list

 

2)建立topic

 

[atguigu@hadoop102 kafka]$ bin/kafka-topics.sh --zookeeper hadoop102:2181 --create --replication-factor 3 --partitions 1 --topic first

 

選項說明:

 

--topic 定義topic

 

--replication-factor  定義副本數

 

--partitions  定義分區數

 

3)刪除topic

 

[atguigu@hadoop102 kafka]$ bin/kafka-topics.sh --zookeeper hadoop102:2181 --delete --topic first

 

須要server.properties中設置delete.topic.enable=true不然只是標記刪除或者直接重啓。

 

4)發送消息

 

[atguigu@hadoop102 kafka]$ bin/kafka-console-producer.sh --broker-list hadoop102:9092 --topic first

 

>hello world

 

>atguigu  atguigu

 

5)消費消息

 

[atguigu@hadoop103 kafka]$ bin/kafka-console-consumer.sh --zookeeper hadoop102:2181 --from-beginning --topic first

 

--from-beginning會把first主題中以往全部的數據都讀取出來。根據業務場景選擇是否增長該配置。

 

6)查看某個Topic的詳情

 

[atguigu@hadoop102 kafka]$ bin/kafka-topics.sh --zookeeper hadoop102:2181 --describe --topic first

 

2.4 Kafka配置信息

 

2.4.1 Broker配置信息

 

屬性

默認值

描述

broker.id

 

必填參數,broker的惟一標識

log.dirs

/tmp/kafka-logs

Kafka數據存放的目錄。能夠指定多個目錄,中間用逗號分隔,當新partition被建立的時會被存放到當前存放partition最少的目錄。

port

9092

BrokerServer接受客戶端鏈接的端口號

zookeeper.connect

null

Zookeeper的鏈接串,格式爲:hostname1:port1,hostname2:port2,hostname3:port3。能夠填一個或多個,爲了提升可靠性,建議都填上。注意,此配置容許咱們指定一個zookeeper路徑來存放此kafka集羣的全部數據,爲了與其餘應用集羣區分開,建議在此配置中指定本集羣存放目錄,格式爲:hostname1:port1,hostname2:port2,hostname3:port3/chroot/path 。須要注意的是,消費者的參數要和此參數一致。

message.max.bytes

1000000

服務器能夠接收到的最大的消息大小。注意此參數要和consumermaximum.message.size大小一致,不然會由於生產者生產的消息太大致使消費者沒法消費。

num.io.threads

8

服務器用來執行讀寫請求的IO線程數,此參數的數量至少要等於服務器上磁盤的數量。

queued.max.requests

500

I/O線程能夠處理請求的隊列大小,若實際請求數超過此大小,網絡線程將中止接收新的請求。

socket.send.buffer.bytes

100 * 1024

The SO_SNDBUFF buffer the server prefers for socket connections.

socket.receive.buffer.bytes

100 * 1024

The SO_RCVBUFF buffer the server prefers for socket connections.

socket.request.max.bytes

100 * 1024 * 1024

服務器容許請求的最大值, 用來防止內存溢出,其值應該小於 Java heap size.

num.partitions

1

默認partition數量,若是topic在建立時沒有指定partition數量,默認使用此值,建議改成5

log.segment.bytes

1024 * 1024 * 1024

Segment文件的大小,超過此值將會自動新建一個segment,此值能夠被topic級別的參數覆蓋。

log.roll.{ms,hours}

24 * 7 hours

新建segment文件的時間,此值能夠被topic級別的參數覆蓋。

log.retention.{ms,minutes,hours}

7 days

Kafka segment log的保存週期,保存週期超過此時間日誌就會被刪除。此參數能夠被topic級別參數覆蓋。數據量大時,建議減少此值。

log.retention.bytes

-1

每一個partition的最大容量,若數據量超過此值,partition數據將會被刪除。注意這個參數控制的是每一個partition而不是topic。此參數能夠被log級別參數覆蓋。

log.retention.check.interval.ms

5 minutes

刪除策略的檢查週期

auto.create.topics.enable

true

自動建立topic參數,建議此值設置爲false,嚴格控制topic管理,防止生產者錯寫topic

default.replication.factor

1

默認副本數量,建議改成2

replica.lag.time.max.ms

10000

在此窗口時間內沒有收到followerfetch請求,leader會將其從ISR(in-sync replicas)中移除。

replica.lag.max.messages

4000

若是replica節點落後leader節點此值大小的消息數量,leader節點就會將其從ISR中移除。

replica.socket.timeout.ms

30 * 1000

replicaleader發送請求的超時時間。

replica.socket.receive.buffer.bytes

64 * 1024

The socket receive buffer for network requests to the leader for replicating data.

replica.fetch.max.bytes

1024 * 1024

The number of byes of messages to attempt to fetch for each partition in the fetch requests the replicas send to the leader.

replica.fetch.wait.max.ms

500

The maximum amount of time to wait time for data to arrive on the leader in the fetch requests sent by the replicas to the leader.

num.replica.fetchers

1

Number of threads used to replicate messages from leaders. Increasing this value can increase the degree of I/O parallelism in the follower broker.

fetch.purgatory.purge.interval.requests

1000

The purge interval (in number of requests) of the fetch request purgatory.

zookeeper.session.timeout.ms

6000

ZooKeeper session 超時時間。若是在此時間內server沒有向zookeeper發送心跳,zookeeper就會認爲此節點已掛掉。 此值過低致使節點容易被標記死亡;若過高,.會致使太遲發現節點死亡。

zookeeper.connection.timeout.ms

6000

客戶端鏈接zookeeper的超時時間。

zookeeper.sync.time.ms

2000

H ZK follower落後 ZK leader的時間。

controlled.shutdown.enable

true

容許broker shutdown。若是啓用,broker在關閉本身以前會把它上面的全部leaders轉移到其它brokers上,建議啓用,增長集羣穩定性。

auto.leader.rebalance.enable

true

If this is enabled the controller will automatically try to balance leadership for partitions among the brokers by periodically returning leadership to the 「preferred」 replica for each partition if it is available.

leader.imbalance.per.broker.percentage

10

The percentage of leader imbalance allowed per broker. The controller will rebalance leadership if this ratio goes above the configured value per broker.

leader.imbalance.check.interval.seconds

300

The frequency with which to check for leader imbalance.

offset.metadata.max.bytes

4096

The maximum amount of metadata to allow clients to save with their offsets.

connections.max.idle.ms

600000

Idle connections timeout: the server socket processor threads close the connections that idle more than this.

num.recovery.threads.per.data.dir

1

The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.

unclean.leader.election.enable

true

Indicates whether to enable replicas not in the ISR set to be elected as leader as a last resort, even though doing so may result in data loss.

delete.topic.enable

false

啓用deletetopic參數,建議設置爲true

offsets.topic.num.partitions

50

The number of partitions for the offset commit topic. Since changing this after deployment is currently unsupported, we recommend using a higher setting for production (e.g., 100-200).

offsets.topic.retention.minutes

1440

Offsets that are older than this age will be marked for deletion. The actual purge will occur when the log cleaner compacts the offsets topic.

offsets.retention.check.interval.ms

600000

The frequency at which the offset manager checks for stale offsets.

offsets.topic.replication.factor

3

The replication factor for the offset commit topic. A higher setting (e.g., three or four) is recommended in order to ensure higher availability. If the offsets topic is created when fewer brokers than the replication factor then the offsets topic will be created with fewer replicas.

offsets.topic.segment.bytes

104857600

Segment size for the offsets topic. Since it uses a compacted topic, this should be kept relatively low in order to facilitate faster log compaction and loads.

offsets.load.buffer.size

5242880

An offset load occurs when a broker becomes the offset manager for a set of consumer groups (i.e., when it becomes a leader for an offsets topic partition). This setting corresponds to the batch size (in bytes) to use when reading from the offsets segments when loading offsets into the offset manager’s cache.

offsets.commit.required.acks

-1

The number of acknowledgements that are required before the offset commit can be accepted. This is similar to the producer’s acknowledgement setting. In general, the default should not be overridden.

offsets.commit.timeout.ms

5000

The offset commit will be delayed until this timeout or the required number of replicas have received the offset commit. This is similar to the producer request timeout.

 

2.4.2 Producer配置信息

 

屬性

默認值

描述

metadata.broker.list

 

啓動時producer查詢brokers的列表,能夠是集羣中全部brokers的一個子集。注意,這個參數只是用來獲取topic的元信息用,producer會從元信息中挑選合適的broker並與之創建socket鏈接。格式是:host1:port1,host2:port2

request.required.acks

0

參見3.2節介紹

request.timeout.ms

10000

Broker等待ack的超時時間,若等待時間超過此值,會返回客戶端錯誤信息。

producer.type

sync

同步異步模式。async表示異步,sync表示同步。若是設置成異步模式,能夠容許生產者以batch的形式push數據,這樣會極大的提升broker性能,推薦設置爲異步。

serializer.class

kafka.serializer.DefaultEncoder

序列號類,.默認序列化成 byte[]

key.serializer.class

 

Key的序列化類,默認同上。

partitioner.class

kafka.producer.DefaultPartitioner

Partition類,默認對key進行hash

compression.codec

none

指定producer消息的壓縮格式,可選參數爲: 「none」, 「gzip」 and 「snappy」。關於壓縮參見4.1

compressed.topics

null

啓用壓縮的topic名稱。若上面參數選擇了一個壓縮格式,那麼壓縮僅對本參數指定的topic有效,若本參數爲空,則對全部topic有效。

message.send.max.retries

3

Producer發送失敗時重試次數。若網絡出現問題,可能會致使不斷重試。

retry.backoff.ms

100

Before each retry, the producer refreshes the metadata of relevant topics to see if a new leader has been elected. Since leader election takes a bit of time, this property specifies the amount of time that the producer waits before refreshing the metadata.

topic.metadata.refresh.interval.ms

600 * 1000

The producer generally refreshes the topic metadata from brokers when there is a failure (partition missing, leader not available…). It will also poll regularly (default: every 10min so 600000ms). If you set this to a negative value, metadata will only get refreshed on failure. If you set this to zero, the metadata will get refreshed after each message sent (not recommended). Important note: the refresh happen only AFTER the message is sent, so if the producer never sends a message the metadata is never refreshed

queue.buffering.max.ms

5000

啓用異步模式時,producer緩存消息的時間。好比咱們設置成1000時,它會緩存1秒的數據再一次發送出去,這樣能夠極大的增長broker吞吐量,但也會形成時效性的下降。

queue.buffering.max.messages

10000

採用異步模式時producer buffer 隊列裏最大緩存的消息數量,若是超過這個數值,producer就會阻塞或者丟掉消息。

queue.enqueue.timeout.ms

-1

當達到上面參數值時producer阻塞等待的時間。若是值設置爲0buffer隊列滿時producer不會阻塞,消息直接被丟掉。若值設置爲-1producer會被阻塞,不會丟消息。

batch.num.messages

200

採用異步模式時,一個batch緩存的消息數量。達到這個數量值時producer纔會發送消息。

send.buffer.bytes

100 * 1024

Socket write buffer size

client.id

「」

The client id is a user-specified string sent in each request to help trace calls. It should logically identify the application making the request.

 

2.4.3 Consumer配置信息

 

屬性

默認值

描述

group.id

 

Consumer的組ID,相同goup.idconsumer屬於同一個組。

zookeeper.connect

 

Consumerzookeeper鏈接串,要和broker的配置一致。

consumer.id

null

若是不設置會自動生成。

socket.timeout.ms

30 * 1000

網絡請求的socket超時時間。實際超時時間由max.fetch.wait + socket.timeout.ms 肯定。

socket.receive.buffer.bytes

64 * 1024

The socket receive buffer for network requests.

fetch.message.max.bytes

1024 * 1024

查詢topic-partition時容許的最大消息大小。consumer會爲每一個partition緩存此大小的消息到內存,所以,這個參數能夠控制consumer的內存使用量。這個值應該至少比server容許的最大消息大小大,以避免producer發送的消息大於consumer容許的消息。

num.consumer.fetchers

1

The number fetcher threads used to fetch data.

auto.commit.enable

true

若是此值設置爲trueconsumer會週期性的把當前消費的offset值保存到zookeeper。當consumer失敗重啓以後將會使用此值做爲新開始消費的值。

auto.commit.interval.ms

60 * 1000

Consumer提交offset值到zookeeper的週期。

queued.max.message.chunks

2

用來被consumer消費的message chunks 數量, 每一個chunk能夠緩存fetch.message.max.bytes大小的數據量。

auto.commit.interval.ms

60 * 1000

Consumer提交offset值到zookeeper的週期。

queued.max.message.chunks

2

用來被consumer消費的message chunks 數量, 每一個chunk能夠緩存fetch.message.max.bytes大小的數據量。

fetch.min.bytes

1

The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request.

fetch.wait.max.ms

100

The maximum amount of time the server will block before answering the fetch request if there isn’t sufficient data to immediately satisfy fetch.min.bytes.

rebalance.backoff.ms

2000

Backoff time between retries during rebalance.

refresh.leader.backoff.ms

200

Backoff time to wait before trying to determine the leader of a partition that has just lost its leader.

auto.offset.reset

largest

What to do when there is no initial offset in ZooKeeper or if an offset is out of range ;smallest : automatically reset the offset to the smallest offset; largest : automatically reset the offset to the largest offset;anything else: throw exception to the consumer

consumer.timeout.ms

-1

若在指定時間內沒有消息消費,consumer將會拋出異常。

exclude.internal.topics

true

Whether messages from internal topics (such as offsets) should be exposed to the consumer.

zookeeper.session.timeout.ms

6000

ZooKeeper session timeout. If the consumer fails to heartbeat to ZooKeeper for this period of time it is considered dead and a rebalance will occur.

zookeeper.connection.timeout.ms

6000

The max time that the client waits while establishing a connection to zookeeper.

zookeeper.sync.time.ms

2000

How far a ZK follower can be behind a ZK leader

 

本教程由尚硅谷教育大數據研究院出品,如需轉載請註明來源,歡迎你們關注尚硅谷公衆號(atguigu)瞭解更多。

相關文章
相關標籤/搜索