schema-registry HA 이슈

 

 

schema-registry는 distributed mode(clustering mode)로 실행 중인데.. 1대가 이유가 죽고 있다.


[2019-04-05 15:06:07,189] WARN [Consumer clientId=consumer-1, groupId=connect-cluster] 9 partitions have leader brokers without a matching listener, including [connect-offsets-0, connect-offsets-15, connect-offsets-9, connect-offsets-3, connect-offsets-24, connect-offsets-18, connect-offsets-12, connect-offsets-6, connect-offsets-21] (org.apache.kafka.clients.NetworkClient:1012)
[2019-04-05 15:06:07,279] ERROR Uncaught exception in herder work thread, exiting:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder:227)
org.apache.kafka.common.errors.TimeoutException: Failed to get offsets by times in 30001ms
[2019-04-05 15:06:07,281] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:65)
[2019-04-05 15:06:07,286] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:226)
[2019-04-05 15:06:07,292] INFO Stopped http_8083@724c5cbe{HTTP/1.1,[http/1.1]}{0.0.0.0:8083} (org.eclipse.jetty.server.AbstractConnector:341)
[2019-04-05 15:06:07,293] INFO node0 Stopped scavenging (org.eclipse.jetty.server.session:167)
[2019-04-05 15:06:07,310] INFO Stopped o.e.j.s.ServletContextHandler@3fd2322d{/,null,UNAVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:1040)
[2019-04-05 15:06:07,312] INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer:244)
[2019-04-05 15:06:07,312] INFO Herder stopping (org.apache.kafka.connect.runtime.distributed.DistributedHerder:398)
[2019-04-05 15:06:12,313] INFO Herder stopped (org.apache.kafka.connect.runtime.distributed.DistributedHerder:418)
[2019-04-05 15:06:12,313] INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect:70)

키프카 커넥트에서 사용하는 카프카  관리 topic에 이슈가 있어서 발생한 것이다.  topic 이름을 지우거나 새로 생성하거나 변경하니 문제가 발생하지 않는다.

config.storage.topic=connect-configs
offset.storage.topic=connect-offsets
status.storage.topic=connect-statuses

3가지 토픽은 카프카 커넥트에서 관리하는 토픽이다.
카프카 커넥트를 distributed 모드로 실행하면 분산 태스크는 카프카 토픽 정보에 커넥터와 태스크 설정, 커넥터 오프셋, 커넥터 상태 정보를 저장한다.

카프카 커넥트(distributed 모드)를 실행하려 할때, 세 개의 내부 토픽을 확인한다. 만약 토픽이 없으면 주어진 설정을 기반으로 생성한다. 토픽에 주어진 replication, partition 등이 설정으로 있으면 설정한다. 만약 아래 설정이 주어지지 않으면 카프카의 auto 생성 설정 규칙에 따라 생성된다.

config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1

offset.storage.partitions=25
status.storage.partitions=5