kafka(kafka_2.11-0.10.1.0)에서 delete.topic.enable을 설정하지 않으면(delete.topic.enable=false) 토픽을 삭제하지 못하는 데모를 보여준다.
1. 로컬에서 간단히 실행하기
- 주키퍼를 실행한다
[/usr/local/kafka_2.11-0.10.1.0] ./bin/zookeeper-server-start.sh -daemon config/zookeeper.properties
반대로 kafka 주키퍼 종료하려면 다음을 실행한다.
[/usr/local/kafka_2.11-0.10.1.0] ./bin/zookeeper-server-stop.sh
2. kafka를 실행한다.
kafka를 실행하기 전에 먼저 설정이 delete.topic.enable 기본값(false)로 둔다.
[/usr/local/kafka_2.11-0.10.1.0] cat config/server.properties | grep delete.topic
#delete.topic.enable=true
[/usr/local/kafka_2.11-0.10.1.0] ./bin/kafka-server-start.sh -daemon config/server.properties
x라는 토픽을 생성한다.
[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-topics.sh --create --zookeeper localhost --replication-factor 1 -partition 1 --topic x
Created topic "x".
[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-topics.sh --list --zookeeper localhost
x
토픽 잘 동작하는지 producer/consumer를 실행한다.
[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-console-producer.sh --broker-list localhost:9092 --topic x
aaa
[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic x --from-beginning
aaa
이제 x 토픽을 삭제하고 토픽 리스트를 보면 삭제라고 마크가 되어 있다.
[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-topics.sh --delete --zookeeper localhost --topic x
Topic x is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-topics.sh --list --zookeeper localhost
x - marked for deletion
없는 x 토픽에 추가하면, 다음과 같이 warning 에러가 많이 나온다.
[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-console-producer.sh --broker-list localhost:9092 --topic x
aaa
[2017-12-06 14:59:35,645] WARN Error while fetching metadata with correlation id 0 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-12-06 14:59:35,846] WARN Error while fetching metadata with correlation id 1 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-12-06 14:59:35,951] WARN Error while fetching metadata with correlation id 2 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-12-06 14:59:36,059] WARN Error while fetching metadata with correlation id 3 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-12-06 14:59:36,168] WARN Error while fetching metadata with correlation id 4 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-12-06 14:59:36,277] WARN Error while fetching metadata with correlation id 5 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-12-06 14:59:36,383] WARN Error while fetching metadata with correlation id 6 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-12-06 14:59:36,492] WARN Error while fetching metadata with correlation id 7 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-12-06 14:59:36,599] WARN Error while fetching metadata with correlation id 8 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-12-06 14:59:36,706] WARN Error while fetching metadata with correlation id 9 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-12-06 14:59:36,815] WARN Error while fetching metadata with correlation id 10 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-12-06 14:59:36,924] WARN Error while fetching metadata with correlation id 11 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-12-06 14:59:37,031] WARN Error while fetching metadata with correlation id 12 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-12-06 14:59:37,140] WARN Error while fetching metadata with correlation id 13 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic x --from-beginning
[2017-12-06 14:59:22,928] WARN Error while fetching metadata with correlation id 1 : {x=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
2. delete.topic.enable=true로 설정한 후 kafka 실행하기
[/usr/local/kafka_2.11-0.10.1.0] cat config/server.properties | grep delete.topic
delete.topic.enable=true
그리고, kafka를 재시작(Stop/start)한다.
토픽 만들고 다시 삭제해본다. 정상적이다.
[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-topics.sh --create --zookeeper localhost --replication-factor 1 -partition 1 --topic x
Created topic "x".
[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-topics.sh --list --zookeeper localhost
x
[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-topics.sh --delete --zookeeper localhost --topic x
Topic x is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
[/usr/local/kafka_2.11-0.10.1.0] bin/kafka-topics.sh --list --zookeeper localhost
test
3. 이슈
delete.topic.enable을 true로 설정했다 하더라도 토픽 삭제가 안되는 경우가 발생할 수 있다.
예)
https://stackoverflow.com/questions/23976670/when-how-does-a-topic-marked-for-deletion-get-finally-removed
https://stackoverflow.com/questions/44564606/how-can-i-remove-kafka-topics-marked-for-deletion
이럴 때는 재시작을 하거나.. (실제로 재시작을 통해서 삭제된 경우가 있었음..)
설정의 log.dir (보통 /tmp 디렉토리)의 파일을 삭제하고 재시작하면 된다고 한다..ㅡㅡ;
4.
참고로 한글 토픽은 생성할 수 없다. ASCII만 된다. 일부 특수 문자만 허용한다.
bin/kafka-topics.sh --create --zookeeper localhost --replication-factor 1 -partition 1 --topic 우와
Error while executing topic command : topic name 우와 is illegal, contains a character other than ASCII alphanumerics, '.', '_' and '-'
[2017-12-06 15:02:33,492] ERROR org.apache.kafka.common.errors.InvalidTopicException: topic name 우와 is illegal, contains a character other than ASCII alphanumerics, '.', '_' and '-'
(kafka.admin.TopicCommand$)
