kafka 토픽 삭제 관련 예제

kafka(kafka_2.11-0.10.1.0)에서 delete.topic.enable을 설정하지 않으면(delete.topic.enable=false) 토픽을 삭제하지 못하는 데모를 보여준다. 

1. 로컬에서 간단히 실행하기

- 주키퍼를 실행한다

[/usr/local/kafka_2.11-0.10.1.0] ./bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

반대로 kafka 주키퍼 종료하려면 다음을 실행한다. 

[/usr/local/kafka_2.11-0.10.1.0] ./bin/zookeeper-server-stop.sh

2. kafka를 실행한다.

kafka를 실행하기 전에 먼저 설정이 delete.topic.enable 기본값(false)로 둔다.

[/usr/local/kafka_2.11-0.10.1.0] cat config/server.properties | grep delete.topic

#delete.topic.enable=true

[/usr/local/kafka_2.11-0.10.1.0] ./bin/kafka-server-start.sh  -daemon config/server.properties

x라는 토픽을 생성한다.

[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-topics.sh --create --zookeeper localhost --replication-factor 1 -partition 1 --topic x

Created topic "x".

[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-topics.sh --list --zookeeper localhost

x

토픽 잘 동작하는지 producer/consumer를 실행한다.

[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-console-producer.sh --broker-list localhost:9092 --topic x

aaa

[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic x --from-beginning

aaa

이제 x 토픽을 삭제하고 토픽 리스트를 보면 삭제라고 마크가 되어 있다.

[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-topics.sh --delete --zookeeper localhost --topic x

Topic x is marked for deletion.

Note: This will have no impact if delete.topic.enable is not set to true.

[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-topics.sh --list --zookeeper localhost

x - marked for deletion

없는 x 토픽에 추가하면, 다음과 같이 warning 에러가 많이 나온다. 

[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-console-producer.sh --broker-list localhost:9092 --topic x

aaa

[2017-12-06 14:59:35,645] WARN Error while fetching metadata with correlation id 0 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2017-12-06 14:59:35,846] WARN Error while fetching metadata with correlation id 1 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2017-12-06 14:59:35,951] WARN Error while fetching metadata with correlation id 2 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2017-12-06 14:59:36,059] WARN Error while fetching metadata with correlation id 3 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2017-12-06 14:59:36,168] WARN Error while fetching metadata with correlation id 4 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2017-12-06 14:59:36,277] WARN Error while fetching metadata with correlation id 5 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2017-12-06 14:59:36,383] WARN Error while fetching metadata with correlation id 6 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2017-12-06 14:59:36,492] WARN Error while fetching metadata with correlation id 7 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2017-12-06 14:59:36,599] WARN Error while fetching metadata with correlation id 8 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2017-12-06 14:59:36,706] WARN Error while fetching metadata with correlation id 9 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2017-12-06 14:59:36,815] WARN Error while fetching metadata with correlation id 10 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2017-12-06 14:59:36,924] WARN Error while fetching metadata with correlation id 11 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2017-12-06 14:59:37,031] WARN Error while fetching metadata with correlation id 12 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2017-12-06 14:59:37,140] WARN Error while fetching metadata with correlation id 13 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic x --from-beginning

[2017-12-06 14:59:22,928] WARN Error while fetching metadata with correlation id 1 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

2. delete.topic.enable=true로 설정한 후 kafka 실행하기 

[/usr/local/kafka_2.11-0.10.1.0] cat config/server.properties | grep delete.topic

delete.topic.enable=true

그리고, kafka를 재시작(Stop/start)한다.

토픽 만들고 다시 삭제해본다. 정상적이다.

[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-topics.sh --create --zookeeper localhost --replication-factor 1 -partition 1 --topic x

Created topic "x".

[/usr/local/kafka_2.11-0.10.1.0]  bin/kafka-topics.sh --list --zookeeper localhost

x

[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-topics.sh --delete --zookeeper localhost --topic x

Topic x is marked for deletion.

Note: This will have no impact if delete.topic.enable is not set to true.

[/usr/local/kafka_2.11-0.10.1.0]  bin/kafka-topics.sh --list --zookeeper localhost

test

3. 이슈

delete.topic.enable을 true로 설정했다 하더라도 토픽 삭제가 안되는 경우가 발생할 수 있다. 

예)

https://stackoverflow.com/questions/23976670/when-how-does-a-topic-marked-for-deletion-get-finally-removed

https://stackoverflow.com/questions/44564606/how-can-i-remove-kafka-topics-marked-for-deletion

이럴 때는 재시작을 하거나..  (실제로 재시작을 통해서 삭제된 경우가 있었음..)

설정의 log.dir (보통 /tmp 디렉토리)의 파일을 삭제하고 재시작하면 된다고 한다..ㅡㅡ;

4. 

참고로 한글 토픽은 생성할 수 없다. ASCII만 된다. 일부 특수 문자만 허용한다.

bin/kafka-topics.sh --create --zookeeper localhost --replication-factor 1 -partition 1 --topic 우와

Error while executing topic command : topic name 우와 is illegal,  contains a character other than ASCII alphanumerics, '.', '_' and '-'

[2017-12-06 15:02:33,492] ERROR org.apache.kafka.common.errors.InvalidTopicException: topic name 우와 is illegal,  contains a character other than ASCII alphanumerics, '.', '_' and '-'

 (kafka.admin.TopicCommand$)