기존에 ruby 2.1.0이 설치되어 있고, fluentd 0.14를 사용해서 fluentd를 테스트해본다.
설치 방법은 다음과 같다.
$ gem install fluentd -v "~> 0.12.0" --no-ri --no-rdoc
Fetching: string-scrub-0.0.5.gem (100%)
Building native extensions. This could take a while...
Successfully installed string-scrub-0.0.5
Fetching: fluentd-0.12.39.gem (100%)
Successfully installed fluentd-0.14.20
2 gems installed
버전을 확인한다.
$ fluentd --version
fluentd 0.14.20
설정 파일을 생성하고 fluentd를 실행한다. 설정 파일과 플러그인 파일을 읽는 로그를 확인할 수 있다.
$ fluentd --setup ./fluent
Installed ./fluent/fluent.conf.
$ fluentd -c ./fluent/fluent.conf -vv
2017-08-16 16:18:25 +0900 [info]: fluent/log.rb:316:call: parsing config file is succeeded path="./fluent/fluent.conf"
2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered output plugin 'stdout'
2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered buffer plugin 'memory'
2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered formatter plugin 'stdout'
2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered formatter plugin 'json'
2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered output plugin 'forward'
2017-08-16 16:18:25 +0900 [info]: fluent/log.rb:316:call: adding forwarding server '192.168.0.12:24224' host="192.168.0.12" port=24224 weight=60 plugin_id="object:3ff3daa51474"
2017-08-16 16:18:25 +0900 [info]: [forward_output] adding forwarding server '192.168.0.11:24224' host="192.168.0.11" port=24224 weight=60 plugin_id="forward_output"
2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered input plugin 'forward'
2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered parser plugin 'in_http'
2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered input plugin 'http'
2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered parser plugin 'msgpack'
2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered parser plugin 'json'
2017-08-16 16:18:25 +0900 [info]: [http_input] Oj is not installed, and failing back to Yajl for json parser
2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered input plugin 'monitor_agent'
2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered input plugin 'debug_agent'
2017-08-16 16:18:25 +0900 [info]: fluent/log.rb:316:call: using configuration file: <ROOT>
<source>
@type forward
@id forward_input
</source>
<source>
@type http
@id http_input
port 8888
</source>
<source>
@type monitor_agent
@id monitor_agent_input
port 24220
</source>
<source>
@type debug_agent
@id debug_agent_input
bind "127.0.0.1"
port 24230
</source>
<match debug.**>
@type stdout
@id stdout_output
</match>
<match system.**>
@type forward
@id forward_output
<server>
host "192.168.0.11"
</server>
<secondary>
<server>
host "192.168.0.12"
</server>
</secondary>
</match>
</ROOT>
2017-08-16 16:18:25 +0900 [info]: fluent/log.rb:316:call: starting fluentd-0.14.20 pid=11375
2017-08-16 16:18:25 +0900 [info]: fluent/log.rb:316:call: spawn command to main: cmdline=["/Users/samuel.kim/.story-spec/opt/ruby/bin/ruby", "-Eascii-8bit:ascii-8bit", "/Users/samuel.kim/.story-spec/opt/ruby/bin/fluentd", "-c", "./fluent/fluent.conf", "-vv", "--under-supervisor"]
2017-08-16 16:18:26 +0900 [info]: fluent/log.rb:316:call: gem 'fluentd' version '0.14.20'
2017-08-16 16:18:26 +0900 [info]: fluent/log.rb:316:call: gem 'fluentd' version '0.12.39'
2017-08-16 16:18:26 +0900 [info]: fluent/log.rb:316:call: adding match pattern="debug.**" type="stdout"
2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered output plugin 'stdout'
2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered buffer plugin 'memory'
2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered formatter plugin 'stdout'
2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered formatter plugin 'json'
2017-08-16 16:18:26 +0900 [info]: fluent/log.rb:316:call: adding match pattern="system.**" type="forward"
2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered output plugin 'forward'
2017-08-16 16:18:26 +0900 [info]: #0 fluent/log.rb:316:call: adding forwarding server '192.168.0.12:24224' host="192.168.0.12" port=24224 weight=60 plugin_id="object:3fe29618a3b8"
2017-08-16 16:18:26 +0900 [info]: #0 [forward_output] adding forwarding server '192.168.0.11:24224' host="192.168.0.11" port=24224 weight=60 plugin_id="forward_output"
2017-08-16 16:18:26 +0900 [info]: fluent/log.rb:316:call: adding source type="forward"
2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered input plugin 'forward'
2017-08-16 16:18:26 +0900 [info]: fluent/log.rb:316:call: adding source type="http"
2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered parser plugin 'in_http'
로그는 fluent/fluent.conf에 있다. 로그의 내용과 동일하다.
$ cat fluent/fluent.conf
# In v1 configuration, type and id are @ prefix parameters.
# @type and @id are recommended. type and id are still available for backward compatibility
## built-in TCP input
## $ echo <json> | fluent-cat <tag>
<source>
@type forward
@id forward_input
</source>
## built-in UNIX socket input
#<source>
# @type unix
#</source>
# HTTP input
# http://localhost:8888/<tag>?json=<json>
<source>
@type http
@id http_input
port 8888
</source>
## File input
## read apache logs with tag=apache.access
#<source>
# @type tail
# format apache
# path /var/log/httpd-access.log
# tag apache.access
#</source>
# Listen HTTP for monitoring
# http://localhost:24220/api/plugins
# http://localhost:24220/api/plugins?type=TYPE
# http://localhost:24220/api/plugins?tag=MYTAG
<source>
@type monitor_agent
@id monitor_agent_input
port 24220
</source>
# Listen DRb for debug
<source>
@type debug_agent
@id debug_agent_input
bind 127.0.0.1
port 24230
</source>
## match tag=apache.access and write to file
#<match apache.access>
# @type file
# path /var/log/fluent/access
#</match>
## match tag=debug.** and dump to console
<match debug.**>
@type stdout
@id stdout_output
</match>
# match tag=system.** and forward to another fluent server
<match system.**>
@type forward
@id forward_output
<server>
host 192.168.0.11
</server>
<secondary>
<server>
host 192.168.0.12
</server>
</secondary>
</match>
## match tag=myapp.** and forward and write to file
#<match myapp.**>
# @type copy
# <store>
# @type forward
# buffer_type file
# buffer_path /var/log/fluent/myapp-forward
# retry_limit 50
# flush_interval 10s
# <server>
# host 192.168.0.13
# </server>
# </store>
# <store>
# @type file
# path /var/log/fluent/myapp
# </store>
#</match>
## match fluent's internal events
#<match fluent.**>
# @type null
#</match>
## match not matched logs and write to file
#<match **>
# @type file
# path /var/log/fluent/else
# compress gz
#</match>
fluent-cat를 이용해 동작 여부를 확인한다.
$ echo '{"json":"message"}' | fluent-cat debug.test
$ fluentd -c ./fluent/fluent.conf -vv
...
2017-08-16 19:52:03.309664000 +0900 debug.test: {"json":"message"}
fluent/fluent.conf 파일을 참조해서 실제로 동작하는지 테스트해본다.
debug와 system 태그로 시작될 때에만 콘솔에 출력된다.
# HTTP input
# http://localhost:8888/<tag>?json=<json>
<source>
@type http
@id http_input
port 8888
</source>
..
## match tag=debug.** and dump to console
<match debug.**>
@type stdout
@id stdout_output
</match>
# match tag=system.** and forward to another fluent server
<match system.**>
@type forward
@id forward_output
<server>
host 192.168.0.11
</server>
<secondary>
<server>
host 192.168.0.12
</server>
</secondary>
</match>
http로 사용한다.
$ curl -v http://localhost:8888/debug.test -F 'json={"hello":"world"}'
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8888 (#0)
> POST /debug.http HTTP/1.1
> Host: localhost:8888
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Length: 156
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=------------------------5bc2e845e167b296
>
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< Content-Type: text/plain
< Connection: Keep-Alive
< Content-Length: 0
<
* Connection #0 to host localhost left intact
fluent 로그에 전달한 json이 출력된다.
2017-08-16 19:55:47.099363000 +0900 debug.test: {"hello":"world"}
그러나 매개 변수가 json 규격에 잘 맞지 않으면 로그가 발생하지 않는다. 내부적으로 파싱하고 있는 듯 하다. 쌍따옴표를 없애면 파싱 구문만 제외하고 나온다
curl -d 'json=afdafd' http://localhost:8888/debug.test
(결과 안나옴)
curl -d 'json={"hello":"world}' http://localhost:8888/debug.test
(결과 : 2017-08-16 19:59:21.074761000 +0900 debug.test: "hello")
...
curl -d 'json={"hello":"world}' http://localhost:8888/debug1.test
(결과 :
2017-08-16 20:00:39 +0900 [warn]: #0 fluent/log.rb:336:call: no patterns matched tag="debug1.test" )
fluent-logger를 설치하면 pry를 통해 fluentd에 전달할 수 있다.
$ gem i fluent-logger --no-ri --no-rdoc
Fetching: fluent-logger-0.7.1.gem (100%)
Successfully installed fluent-logger-0.7.1
1 gem installed
$ pry
[1] pry(main)> require 'fluent-logger'
=> true
[2] pry(main)> Fluent::Logger::FluentLogger.open(nil, :host => 'localhost', :port => 24224)
=> #<Fluent::Logger::FluentLogger:0x007fc0892dbad0
@buffer_overflow_handler=nil,
@con=#<TCPSocket:fd 10>,
@connect_error_history=[],
@factory=#<MessagePack::Factory:0x007fc0892db918>,
@host="localhost",
@last_error={},
@limit=8388608,
@log_reconnect_error_threshold=13,
@logged_reconnect_error=false,
@logger=
#<Logger:0x007fc0892db670
@default_formatter=#<Logger::Formatter:0x007fc0892db648 @datetime_format=nil>,
@formatter=nil,
@level=1,
@logdev=
#<Logger::LogDevice:0x007fc0892db5a8
@dev=#<IO:<STDERR>>,
@filename=nil,
@mutex=#<Logger::LogDevice::LogDeviceMutex:0x007fc0892db580 @mon_count=0, @mon_mutex=#<Mutex:0x007fc0892db530>, @mon_owner=nil>,
@shift_age=nil,
@shift_size=nil>,
@progname=nil>,
@mon=#<Monitor:0x007fc0892db760 @mon_count=0, @mon_mutex=#<Mutex:0x007fc0892db710>, @mon_owner=nil>,
@nanosecond_precision=nil,
@packer=#<MessagePack::Packer:0x007fc0892db8f0>,
@pending=nil,
@port=24224,
@socket_path=nil,
@tag_prefix=nil>
[3] pry(main)> Fluent::Logger.post("debug.test", { "id" => "1", "name" => "samuel" })
=> true
잘 나온다.
$ fluentd -c ./fluent/fluent.conf -vv
...
2017-08-16 20:03:59.000000000 +0900 debug.test: {"id":"1","name":"samuel"}
설정 파일 문법
https://docs.fluentd.org/articles/config-file#v1-format
tail log 문법
https://docs.fluentd.org/articles/in_tail
한글 착한 문서
http://blog.seulgi.kim/2014/04/fluentd-pluggable-log-collector.html
https://docs.google.com/presentation/d/1e6PXtPS_eQn7vY5hDhHlDVpCk1U6eKARoRqInoDiUkw/edit#slide=id.g271a4ef8f_018
과거에는 fluentd가 nanosecond 이슈가 있었다고 하나, 0.14부터는 nanosecond를 잘 보여주고 있다.
https://www.fluentd.org/blog/fluentd-v0.14.9-has-been-released
time_as_integer true
out_forward
Since this version, time_as_integer parameter is now false. It means v0.14.9's out_forward can't forward data to v0.12's in_forward by default. You need to set time_as_integer true explicitly. We have already mentioned this point in v0.14.0 release, so we hope this change doesn't break your v0.14 to v0.12 forwarding.
