스톰(apache storm) 내부 구조에 대한 설명을 담은 글이다. 최신 1.0과 조금 다른 부분이 있자 개념은 비슷하다.



http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/


http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/




1.0 기준

* 로컬 서버에서 쓰레드간 통신 : LMAX Disruptor

* 네트워크 간 서버 통신 : Netty 




스톰의 내부 큐 구조



이전 글을 보면 buffer 사이즈가 많이 나오는데.. cpu/mem을 많이 사용하지 않은채 병목이 되기 때문에 기본 값을 잘 살펴보고 buffersize를 좀 고민해봐야할 수 있다.


https://github.com/apache/storm/blob/master/conf/defaults.yaml





전체적인 흐름을 보면 알겠지만, 파이프 수도관(pipeline), 물을 담는 통(ring buffer), 수도꼭지(thread) 처럼 되어 있다. 따라서 수도꼭지와 통을 통제하는, 즉 전체적은 흐름을 통제해야 하는 back pressure 개념이 필요한데. 바로 관련 내용은 아래에 있다.


https://issues.apache.org/jira/browse/STORM-886











그래서 병목이 발생하지 않도록 내부적으로 backpressure를 추가했는데..

이게 대용량일 때 문제가 될 수 있다.



트래픽이 많아지면 backpressure을 활성화하지 않도록 해서 문제가 해결되기도 한다.

https://stackoverflow.com/questions/38198961/storm-kafkaspout-stopped-to-consume-messages-from-kafka-topic



backpressure에 대한 튜닝 내용을 살펴보면. 

http://jobs.one2team.com/apache-storms/ 글을 보면, 기본 값이 병목임을 얘기하고 있다. 


executor.receive.buffer.size (1024)

executor.send.buffer.size (1024)

disruptor.highwatermark (default 0.9)

disruptor.lowwatermark (default 0.4)



The reason why the back pressure didn’t work out of the box with the default parameters is that we have a bolt that has a pretty long processing time, and usually takes 0.1 second to process a single message. During a peak of events, the spout was fast enough to fill the buffer of these slow bolts. When we kept the default size of 1024 messages, a bolt had to process more than 1000 messages in 0.1s each, which would add up to 100 seconds before the last message gets processed. That didn’t work well with the timeout of 30 seconds for a tuple in the topology, and it caused many messages to fail.

The main parameter we had to tune was the buffer size. Based on the assumptions that we don’t want to have too many fails, and that our topology is not latency sensitive, we agreed on the limit that a tuple shouldn’t wait more than 10 seconds in that executor receive buffer. This means we don’t want more than 100 messages in the buffer, then we can go for a buffer size of 64 or 128. As a rule of thumb, we align the value of the topology.executor.send.buffer.size with the one of the topology.executor.receive.buffer.size. For us, tuning the buffer to the adapted size was sufficient to get the backpressure to work. It throttles the spout when the bolts can’t keep up the pace.

We barely touched the watermark values, these seem to make sense only if you have specific considerations like:

  • When the spout is throttling at 0.9, it’s too late, some of the tuples are still filling the buffers, lets reduce the high watermark to 0.8
  • When the spout is throttled, and the messages are dropping under 0.4, the spout has some latency to fetch data and build new messages, that causes some bolts to be idle for a small moment, lets increase the low watermark to 0.5




backpressure을 사용하고 있다면 계속 이 부분을 튜닝하면서 운영해야 할 것 같다.


성능에 중요한 영향을 미친다.

'Apache Storm' 카테고리의 다른 글

[storm] 스톰 내부 구조와 back pressure 튜닝  (0) 2018.01.16
[storm] LMAX Disrupter 개념  (0) 2018.01.16
Posted by 김용환 '김용환'