fluentd 공부

기존에 ruby 2.1.0이 설치되어 있고, fluentd 0.14를 사용해서 fluentd를 테스트해본다.

설치 방법은 다음과 같다. 

$ gem install fluentd -v "~> 0.12.0" --no-ri --no-rdoc

Fetching: string-scrub-0.0.5.gem (100%)

Building native extensions.  This could take a while...

Successfully installed string-scrub-0.0.5

Fetching: fluentd-0.12.39.gem (100%)

Successfully installed fluentd-0.14.20

2 gems installed

버전을 확인한다. 

$ fluentd --version

fluentd 0.14.20

설정 파일을 생성하고 fluentd를 실행한다. 설정 파일과 플러그인 파일을 읽는 로그를 확인할 수 있다.

$ fluentd --setup ./fluent

Installed ./fluent/fluent.conf.

$ fluentd -c ./fluent/fluent.conf -vv

2017-08-16 16:18:25 +0900 [info]: fluent/log.rb:316:call: parsing config file is succeeded path="./fluent/fluent.conf"

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered output plugin 'stdout'

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered buffer plugin 'memory'

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered formatter plugin 'stdout'

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered formatter plugin 'json'

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered output plugin 'forward'

2017-08-16 16:18:25 +0900 [info]: fluent/log.rb:316:call: adding forwarding server '192.168.0.12:24224' host="192.168.0.12" port=24224 weight=60 plugin_id="object:3ff3daa51474"

2017-08-16 16:18:25 +0900 [info]: [forward_output] adding forwarding server '192.168.0.11:24224' host="192.168.0.11" port=24224 weight=60 plugin_id="forward_output"

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered input plugin 'forward'

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered parser plugin 'in_http'

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered input plugin 'http'

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered parser plugin 'msgpack'

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered parser plugin 'json'

2017-08-16 16:18:25 +0900 [info]: [http_input] Oj is not installed, and failing back to Yajl for json parser

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered input plugin 'monitor_agent'

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered input plugin 'debug_agent'

2017-08-16 16:18:25 +0900 [info]: fluent/log.rb:316:call: using configuration file: <ROOT>

  <source>

    @type forward

    @id forward_input

  </source>

  <source>

    @type http

    @id http_input

    port 8888

  </source>

  <source>

    @type monitor_agent

    @id monitor_agent_input

    port 24220

  </source>

  <source>

    @type debug_agent

    @id debug_agent_input

    bind "127.0.0.1"

    port 24230

  </source>

  <match debug.**>

    @type stdout

    @id stdout_output

  </match>

  <match system.**>

    @type forward

    @id forward_output

    <server>

      host "192.168.0.11"

    </server>

    <secondary>

      <server>

        host "192.168.0.12"

      </server>

    </secondary>

  </match>

</ROOT>

2017-08-16 16:18:25 +0900 [info]: fluent/log.rb:316:call: starting fluentd-0.14.20 pid=11375

2017-08-16 16:18:25 +0900 [info]: fluent/log.rb:316:call: spawn command to main:  cmdline=["/Users/samuel.kim/.story-spec/opt/ruby/bin/ruby", "-Eascii-8bit:ascii-8bit", "/Users/samuel.kim/.story-spec/opt/ruby/bin/fluentd", "-c", "./fluent/fluent.conf", "-vv", "--under-supervisor"]

2017-08-16 16:18:26 +0900 [info]: fluent/log.rb:316:call: gem 'fluentd' version '0.14.20'

2017-08-16 16:18:26 +0900 [info]: fluent/log.rb:316:call: gem 'fluentd' version '0.12.39'

2017-08-16 16:18:26 +0900 [info]: fluent/log.rb:316:call: adding match pattern="debug.**" type="stdout"

2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered output plugin 'stdout'

2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered buffer plugin 'memory'

2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered formatter plugin 'stdout'

2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered formatter plugin 'json'

2017-08-16 16:18:26 +0900 [info]: fluent/log.rb:316:call: adding match pattern="system.**" type="forward"

2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered output plugin 'forward'

2017-08-16 16:18:26 +0900 [info]: #0 fluent/log.rb:316:call: adding forwarding server '192.168.0.12:24224' host="192.168.0.12" port=24224 weight=60 plugin_id="object:3fe29618a3b8"

2017-08-16 16:18:26 +0900 [info]: #0 [forward_output] adding forwarding server '192.168.0.11:24224' host="192.168.0.11" port=24224 weight=60 plugin_id="forward_output"

2017-08-16 16:18:26 +0900 [info]: fluent/log.rb:316:call: adding source type="forward"

2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered input plugin 'forward'

2017-08-16 16:18:26 +0900 [info]: fluent/log.rb:316:call: adding source type="http"

2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered parser plugin 'in_http'

로그는 fluent/fluent.conf에 있다. 로그의 내용과 동일하다. 

$ cat fluent/fluent.conf

# In v1 configuration, type and id are @ prefix parameters.

# @type and @id are recommended. type and id are still available for backward compatibility

## built-in TCP input

## $ echo <json> | fluent-cat <tag>

<source>

  @type forward

  @id forward_input

</source>

## built-in UNIX socket input

#<source>

#  @type unix

#</source>

# HTTP input

# http://localhost:8888/<tag>?json=<json>

<source>

  @type http

  @id http_input

  port 8888

</source>

## File input

## read apache logs with tag=apache.access

#<source>

#  @type tail

#  format apache

#  path /var/log/httpd-access.log

#  tag apache.access

#</source>

# Listen HTTP for monitoring

# http://localhost:24220/api/plugins

# http://localhost:24220/api/plugins?type=TYPE

# http://localhost:24220/api/plugins?tag=MYTAG

<source>

  @type monitor_agent

  @id monitor_agent_input

  port 24220

</source>

# Listen DRb for debug

<source>

  @type debug_agent

  @id debug_agent_input

  bind 127.0.0.1

  port 24230

</source>

## match tag=apache.access and write to file

#<match apache.access>

#  @type file

#  path /var/log/fluent/access

#</match>

## match tag=debug.** and dump to console

<match debug.**>

  @type stdout

  @id stdout_output

</match>

# match tag=system.** and forward to another fluent server

<match system.**>

  @type forward

  @id forward_output

  <server>

    host 192.168.0.11

  </server>

  <secondary>

    <server>

      host 192.168.0.12

    </server>

  </secondary>

</match>

## match tag=myapp.** and forward and write to file

#<match myapp.**>

#  @type copy

#  <store>

#    @type forward

#    buffer_type file

#    buffer_path /var/log/fluent/myapp-forward

#    retry_limit 50

#    flush_interval 10s

#    <server>

#      host 192.168.0.13

#    </server>

#  </store>

#  <store>

#    @type file

#    path /var/log/fluent/myapp

#  </store>

#</match>

## match fluent's internal events

#<match fluent.**>

#  @type null

#</match>

## match not matched logs and write to file

#<match **>

#  @type file

#  path /var/log/fluent/else

#  compress gz

#</match>

fluent-cat를 이용해 동작 여부를 확인한다.

$ echo '{"json":"message"}' | fluent-cat debug.test

$ fluentd -c ./fluent/fluent.conf -vv

...

2017-08-16 19:52:03.309664000 +0900 debug.test: {"json":"message"}

fluent/fluent.conf 파일을 참조해서 실제로 동작하는지 테스트해본다.

 debug와 system 태그로 시작될 때에만 콘솔에 출력된다.

# HTTP input

# http://localhost:8888/<tag>?json=<json>

<source>

  @type http

  @id http_input

  port 8888

</source>
..

## match tag=debug.** and dump to console

<match debug.**>

  @type stdout

  @id stdout_output

</match>

# match tag=system.** and forward to another fluent server

<match system.**>

  @type forward

  @id forward_output

  <server>

    host 192.168.0.11

  </server>

  <secondary>

    <server>

      host 192.168.0.12

    </server>

  </secondary>

</match>

http로 사용한다.

$ curl -v http://localhost:8888/debug.test -F 'json={"hello":"world"}'

*   Trying 127.0.0.1...

* Connected to localhost (127.0.0.1) port 8888 (#0)

> POST /debug.http HTTP/1.1

> Host: localhost:8888

> User-Agent: curl/7.43.0

> Accept: */*

> Content-Length: 156

> Expect: 100-continue

> Content-Type: multipart/form-data; boundary=------------------------5bc2e845e167b296

>

< HTTP/1.1 100 Continue

< HTTP/1.1 200 OK

< Content-Type: text/plain

< Connection: Keep-Alive

< Content-Length: 0

<

* Connection #0 to host localhost left intact

fluent 로그에 전달한 json이 출력된다.

2017-08-16 19:55:47.099363000 +0900 debug.test: {"hello":"world"}

그러나 매개 변수가 json 규격에 잘 맞지 않으면 로그가 발생하지 않는다. 내부적으로 파싱하고 있는 듯 하다. 쌍따옴표를 없애면 파싱 구문만 제외하고 나온다

curl  -d 'json=afdafd'  http://localhost:8888/debug.test

(결과 안나옴)

curl  -d 'json={"hello":"world}'  http://localhost:8888/debug.test

(결과 : 2017-08-16 19:59:21.074761000 +0900 debug.test: "hello")

...

curl  -d 'json={"hello":"world}'  http://localhost:8888/debug1.test

(결과 : 

2017-08-16 20:00:39 +0900 [warn]: #0 fluent/log.rb:336:call: no patterns matched tag="debug1.test" )

fluent-logger를 설치하면 pry를 통해 fluentd에 전달할 수 있다. 

$ gem i fluent-logger --no-ri --no-rdoc

Fetching: fluent-logger-0.7.1.gem (100%)

Successfully installed fluent-logger-0.7.1

1 gem installed

$ pry

[1] pry(main)> require 'fluent-logger'

=> true

[2] pry(main)> Fluent::Logger::FluentLogger.open(nil, :host => 'localhost', :port => 24224)

=> #<Fluent::Logger::FluentLogger:0x007fc0892dbad0

 @buffer_overflow_handler=nil,

 @con=#<TCPSocket:fd 10>,

 @connect_error_history=[],

 @factory=#<MessagePack::Factory:0x007fc0892db918>,

 @host="localhost",

 @last_error={},

 @limit=8388608,

 @log_reconnect_error_threshold=13,

 @logged_reconnect_error=false,

 @logger=

  #<Logger:0x007fc0892db670

   @default_formatter=#<Logger::Formatter:0x007fc0892db648 @datetime_format=nil>,

   @formatter=nil,

   @level=1,

   @logdev=

    #<Logger::LogDevice:0x007fc0892db5a8

     @dev=#<IO:<STDERR>>,

     @filename=nil,

     @mutex=#<Logger::LogDevice::LogDeviceMutex:0x007fc0892db580 @mon_count=0, @mon_mutex=#<Mutex:0x007fc0892db530>, @mon_owner=nil>,

     @shift_age=nil,

     @shift_size=nil>,

   @progname=nil>,

 @mon=#<Monitor:0x007fc0892db760 @mon_count=0, @mon_mutex=#<Mutex:0x007fc0892db710>, @mon_owner=nil>,

 @nanosecond_precision=nil,

 @packer=#<MessagePack::Packer:0x007fc0892db8f0>,

 @pending=nil,

 @port=24224,

 @socket_path=nil,

 @tag_prefix=nil>

[3] pry(main)> Fluent::Logger.post("debug.test", { "id" => "1", "name" => "samuel" })

=> true

잘 나온다.

$ fluentd -c ./fluent/fluent.conf -vv

...

2017-08-16 20:03:59.000000000 +0900 debug.test: {"id":"1","name":"samuel"}

설정 파일 문법

https://docs.fluentd.org/articles/config-file#v1-format

tail log 문법

https://docs.fluentd.org/articles/in_tail

한글 착한 문서

http://blog.seulgi.kim/2014/04/fluentd-pluggable-log-collector.html

https://docs.google.com/presentation/d/1e6PXtPS_eQn7vY5hDhHlDVpCk1U6eKARoRqInoDiUkw/edit#slide=id.g271a4ef8f_018

과거에는 fluentd가 nanosecond 이슈가 있었다고 하나, 0.14부터는 nanosecond를 잘 보여주고 있다. 

https://www.fluentd.org/blog/fluentd-v0.14.9-has-been-released

time_as_integer true

out_forward

Since this version, time_as_integer parameter is now false. It means v0.14.9's out_forward can't forward data to v0.12's in_forward by default. You need to set time_as_integer true explicitly. We have already mentioned this point in v0.14.0 release, so we hope this change doesn't break your v0.14 to v0.12 forwarding.