_cat/nodes api를 통해 노드 정보를 볼 수 있다. 


$ curl -XGET localhost:9200/_cat/nodes?v


ip        heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name

127.0.0.1           45         100   6    2.03                  mdi       *      5OEGj_a



Posted by '김용환'
,


elasticsearch 5에서 elasticsearch를 실행할 때 다른 sql처럼 bootstrap 검사를 시작한다. 


관련 내용은 다음을 참고한다. 



https://www.elastic.co/guide/en/elasticsearch/reference/master/bootstrap-checks.html


https://www.elastic.co/blog/bootstrap_checks_annoying_instead_of_devastating




Posted by '김용환'
,

참조

https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-shards.html



전체 인덱스에 대한 shard 정보를 얻으려면 _cat/shards를 실행한다.


$ curl localhost:9200/_cat/shards

persons             0 p STARTED       10   4.1kb 127.0.0.1 5OEGj_a

persons             0 r UNASSIGNED

..




특정 인덱스의 샤드 정보는 다음 커맨드를 사용한다.


$ curl localhost:9200/_cat/shards/wikinews

wikinews 2 p STARTED    4214   71mb 127.0.0.1 5OEGj_a

wikinews 2 r UNASSIGNED

wikinews 1 p STARTED    4112 69.2mb 127.0.0.1 5OEGj_a

wikinews 1 r UNASSIGNED

wikinews 3 p STARTED    4310 73.6mb 127.0.0.1 5OEGj_a

wikinews 3 r UNASSIGNED

wikinews 4 p STARTED    4285 72.4mb 127.0.0.1 5OEGj_a

wikinews 4 r UNASSIGNED

wikinews 0 p STARTED    4146 69.6mb 127.0.0.1 5OEGj_a

wikinews 0 r UNASSIGNED




인덱스의 정보를 상세하게 보려면 h 매개변수에 옵션을 추가할 수 있다. 만약  unassinged라면 이유도 확인할 수 있다. 


$ curl localhost:9200/_cat/shards/wikinews?h=index,shard,prirep,state,unassigned.reason

wikinews 2 p STARTED

wikinews 2 r UNASSIGNED INDEX_CREATED

wikinews 1 p STARTED

wikinews 1 r UNASSIGNED INDEX_CREATED

wikinews 3 p STARTED

wikinews 3 r UNASSIGNED INDEX_CREATED

wikinews 4 p STARTED

wikinews 4 r UNASSIGNED INDEX_CREATED

wikinews 0 p STARTED

wikinews 0 r UNASSIGNED INDEX_CREATED




document number를 샤드단위로 볼 수 있기 때문에 라우팅 테스트를 쉽게 할 수 있다.


documents           1     p      STARTED        3   9.6kb 127.0.0.1 5OEGj_a

documents           0     p      STARTED        1   3.2kb 127.0.0.1 5OEGj_a






Posted by '김용환'
,


인덱스 통계를 보려면 _stats 또는 인덱스이름/_stats를 사용하면 된다.


https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html




curl localhost:9200/wikinews/_stats?pretty=true

{

  "_shards" : {

    "total" : 10,

    "successful" : 5,

    "failed" : 0

  },

  "_all" : {

    "primaries" : {

      "docs" : {

        "count" : 21067,

        "deleted" : 0

      },

      "store" : {

        "size_in_bytes" : 373355243,

        "throttle_time_in_millis" : 0

      },

      "indexing" : {

        "index_total" : 42134,

        "index_time_in_millis" : 110295,

        "index_current" : 0,

        "index_failed" : 0,

        "delete_total" : 0,

        "delete_time_in_millis" : 0,

        "delete_current" : 0,

        "noop_update_total" : 0,

        "is_throttled" : false,

        "throttle_time_in_millis" : 0

      },

      "get" : {

        "total" : 0,

        "time_in_millis" : 0,

        "exists_total" : 0,

        "exists_time_in_millis" : 0,

        "missing_total" : 0,

        "missing_time_in_millis" : 0,

        "current" : 0

      },

      "search" : {

        "open_contexts" : 0,

        "query_total" : 0,

        "query_time_in_millis" : 0,

        "query_current" : 0,

        "fetch_total" : 0,

        "fetch_time_in_millis" : 0,

        "fetch_current" : 0,

        "scroll_total" : 0,

        "scroll_time_in_millis" : 0,

        "scroll_current" : 0,

        "suggest_total" : 0,

        "suggest_time_in_millis" : 0,

        "suggest_current" : 0

      },

      "merges" : {

        "current" : 0,

        "current_docs" : 0,

        "current_size_in_bytes" : 0,

        "total" : 0,

        "total_time_in_millis" : 0,

        "total_docs" : 0,

        "total_size_in_bytes" : 0,

        "total_stopped_time_in_millis" : 0,

        "total_throttled_time_in_millis" : 0,

        "total_auto_throttle_in_bytes" : 104857600

      },

      "refresh" : {

        "total" : 59,

        "total_time_in_millis" : 30946,

        "listeners" : 0

      },

      "flush" : {

        "total" : 10,

        "total_time_in_millis" : 284

      },

      "warmer" : {

        "current" : 0,

        "total" : 74,

        "total_time_in_millis" : 18

      },

      "query_cache" : {

        "memory_size_in_bytes" : 0,

        "total_count" : 0,

        "hit_count" : 0,

        "miss_count" : 0,

        "cache_size" : 0,

        "cache_count" : 0,

        "evictions" : 0

      },

      "fielddata" : {

        "memory_size_in_bytes" : 0,

        "evictions" : 0

      },

      "completion" : {

        "size_in_bytes" : 0

      },

      "segments" : {

        "count" : 24,

        "memory_in_bytes" : 1619215,

        "terms_memory_in_bytes" : 1435245,

        "stored_fields_memory_in_bytes" : 42880,

        "term_vectors_memory_in_bytes" : 0,

        "norms_memory_in_bytes" : 26112,

        "points_memory_in_bytes" : 1106,

        "doc_values_memory_in_bytes" : 113872,

        "index_writer_memory_in_bytes" : 0,

        "version_map_memory_in_bytes" : 0,

        "fixed_bit_set_memory_in_bytes" : 0,

        "max_unsafe_auto_id_timestamp" : -1,

        "file_sizes" : { }

      },

      "translog" : {

        "operations" : 0,

        "size_in_bytes" : 215

      },

      "request_cache" : {

        "memory_size_in_bytes" : 0,

        "evictions" : 0,

        "hit_count" : 0,

        "miss_count" : 0

      },

      "recovery" : {

        "current_as_source" : 0,

        "current_as_target" : 0,

        "throttle_time_in_millis" : 0

      }

    },

    "total" : {

      "docs" : {

        "count" : 21067,

        "deleted" : 0

      },

      "store" : {

        "size_in_bytes" : 373355243,

        "throttle_time_in_millis" : 0

      },

      "indexing" : {

        "index_total" : 42134,

        "index_time_in_millis" : 110295,

        "index_current" : 0,

        "index_failed" : 0,

        "delete_total" : 0,

        "delete_time_in_millis" : 0,

        "delete_current" : 0,

        "noop_update_total" : 0,

        "is_throttled" : false,

        "throttle_time_in_millis" : 0

      },

      "get" : {

        "total" : 0,

        "time_in_millis" : 0,

        "exists_total" : 0,

        "exists_time_in_millis" : 0,

        "missing_total" : 0,

        "missing_time_in_millis" : 0,

        "current" : 0

      },

      "search" : {

        "open_contexts" : 0,

        "query_total" : 0,

        "query_time_in_millis" : 0,

        "query_current" : 0,

        "fetch_total" : 0,

        "fetch_time_in_millis" : 0,

        "fetch_current" : 0,

        "scroll_total" : 0,

        "scroll_time_in_millis" : 0,

        "scroll_current" : 0,

        "suggest_total" : 0,

        "suggest_time_in_millis" : 0,

        "suggest_current" : 0

      },

      "merges" : {

        "current" : 0,

        "current_docs" : 0,

        "current_size_in_bytes" : 0,

        "total" : 0,

        "total_time_in_millis" : 0,

        "total_docs" : 0,

        "total_size_in_bytes" : 0,

        "total_stopped_time_in_millis" : 0,

        "total_throttled_time_in_millis" : 0,

        "total_auto_throttle_in_bytes" : 104857600

      },

      "refresh" : {

        "total" : 59,

        "total_time_in_millis" : 30946,

        "listeners" : 0

      },

      "flush" : {

        "total" : 10,

        "total_time_in_millis" : 284

      },

      "warmer" : {

        "current" : 0,

        "total" : 74,

        "total_time_in_millis" : 18

      },

      "query_cache" : {

        "memory_size_in_bytes" : 0,

        "total_count" : 0,

        "hit_count" : 0,

        "miss_count" : 0,

        "cache_size" : 0,

        "cache_count" : 0,

        "evictions" : 0

      },

      "fielddata" : {

        "memory_size_in_bytes" : 0,

        "evictions" : 0

      },

      "completion" : {

        "size_in_bytes" : 0

      },

      "segments" : {

        "count" : 24,

        "memory_in_bytes" : 1619215,

        "terms_memory_in_bytes" : 1435245,

        "stored_fields_memory_in_bytes" : 42880,

        "term_vectors_memory_in_bytes" : 0,

        "norms_memory_in_bytes" : 26112,

        "points_memory_in_bytes" : 1106,

        "doc_values_memory_in_bytes" : 113872,

        "index_writer_memory_in_bytes" : 0,

        "version_map_memory_in_bytes" : 0,

        "fixed_bit_set_memory_in_bytes" : 0,

        "max_unsafe_auto_id_timestamp" : -1,

        "file_sizes" : { }

      },

      "translog" : {

        "operations" : 0,

        "size_in_bytes" : 215

      },

      "request_cache" : {

        "memory_size_in_bytes" : 0,

        "evictions" : 0,

        "hit_count" : 0,

        "miss_count" : 0

      },

      "recovery" : {

        "current_as_source" : 0,

        "current_as_target" : 0,

        "throttle_time_in_millis" : 0

      }

    }

  },

  "indices" : {

    "wikinews" : {

      "primaries" : {

        "docs" : {

          "count" : 21067,

          "deleted" : 0

        },

        "store" : {

          "size_in_bytes" : 373355243,

          "throttle_time_in_millis" : 0

        },

        "indexing" : {

          "index_total" : 42134,

          "index_time_in_millis" : 110295,

          "index_current" : 0,

          "index_failed" : 0,

          "delete_total" : 0,

          "delete_time_in_millis" : 0,

          "delete_current" : 0,

          "noop_update_total" : 0,

          "is_throttled" : false,

          "throttle_time_in_millis" : 0

        },

        "get" : {

          "total" : 0,

          "time_in_millis" : 0,

          "exists_total" : 0,

          "exists_time_in_millis" : 0,

          "missing_total" : 0,

          "missing_time_in_millis" : 0,

          "current" : 0

        },

        "search" : {

          "open_contexts" : 0,

          "query_total" : 0,

          "query_time_in_millis" : 0,

          "query_current" : 0,

          "fetch_total" : 0,

          "fetch_time_in_millis" : 0,

          "fetch_current" : 0,

          "scroll_total" : 0,

          "scroll_time_in_millis" : 0,

          "scroll_current" : 0,

          "suggest_total" : 0,

          "suggest_time_in_millis" : 0,

          "suggest_current" : 0

        },

        "merges" : {

          "current" : 0,

          "current_docs" : 0,

          "current_size_in_bytes" : 0,

          "total" : 0,

          "total_time_in_millis" : 0,

          "total_docs" : 0,

          "total_size_in_bytes" : 0,

          "total_stopped_time_in_millis" : 0,

          "total_throttled_time_in_millis" : 0,

          "total_auto_throttle_in_bytes" : 104857600

        },

        "refresh" : {

          "total" : 59,

          "total_time_in_millis" : 30946,

          "listeners" : 0

        },

        "flush" : {

          "total" : 10,

          "total_time_in_millis" : 284

        },

        "warmer" : {

          "current" : 0,

          "total" : 74,

          "total_time_in_millis" : 18

        },

        "query_cache" : {

          "memory_size_in_bytes" : 0,

          "total_count" : 0,

          "hit_count" : 0,

          "miss_count" : 0,

          "cache_size" : 0,

          "cache_count" : 0,

          "evictions" : 0

        },

        "fielddata" : {

          "memory_size_in_bytes" : 0,

          "evictions" : 0

        },

        "completion" : {

          "size_in_bytes" : 0

        },

        "segments" : {

          "count" : 24,

          "memory_in_bytes" : 1619215,

          "terms_memory_in_bytes" : 1435245,

          "stored_fields_memory_in_bytes" : 42880,

          "term_vectors_memory_in_bytes" : 0,

          "norms_memory_in_bytes" : 26112,

          "points_memory_in_bytes" : 1106,

          "doc_values_memory_in_bytes" : 113872,

          "index_writer_memory_in_bytes" : 0,

          "version_map_memory_in_bytes" : 0,

          "fixed_bit_set_memory_in_bytes" : 0,

          "max_unsafe_auto_id_timestamp" : -1,

          "file_sizes" : { }

        },

        "translog" : {

          "operations" : 0,

          "size_in_bytes" : 215

        },

        "request_cache" : {

          "memory_size_in_bytes" : 0,

          "evictions" : 0,

          "hit_count" : 0,

          "miss_count" : 0

        },

        "recovery" : {

          "current_as_source" : 0,

          "current_as_target" : 0,

          "throttle_time_in_millis" : 0

        }

      },

      "total" : {

        "docs" : {

          "count" : 21067,

          "deleted" : 0

        },

        "store" : {

          "size_in_bytes" : 373355243,

          "throttle_time_in_millis" : 0

        },

        "indexing" : {

          "index_total" : 42134,

          "index_time_in_millis" : 110295,

          "index_current" : 0,

          "index_failed" : 0,

          "delete_total" : 0,

          "delete_time_in_millis" : 0,

          "delete_current" : 0,

          "noop_update_total" : 0,

          "is_throttled" : false,

          "throttle_time_in_millis" : 0

        },

        "get" : {

          "total" : 0,

          "time_in_millis" : 0,

          "exists_total" : 0,

          "exists_time_in_millis" : 0,

          "missing_total" : 0,

          "missing_time_in_millis" : 0,

          "current" : 0

        },

        "search" : {

          "open_contexts" : 0,

          "query_total" : 0,

          "query_time_in_millis" : 0,

          "query_current" : 0,

          "fetch_total" : 0,

          "fetch_time_in_millis" : 0,

          "fetch_current" : 0,

          "scroll_total" : 0,

          "scroll_time_in_millis" : 0,

          "scroll_current" : 0,

          "suggest_total" : 0,

          "suggest_time_in_millis" : 0,

          "suggest_current" : 0

        },

        "merges" : {

          "current" : 0,

          "current_docs" : 0,

          "current_size_in_bytes" : 0,

          "total" : 0,

          "total_time_in_millis" : 0,

          "total_docs" : 0,

          "total_size_in_bytes" : 0,

          "total_stopped_time_in_millis" : 0,

          "total_throttled_time_in_millis" : 0,

          "total_auto_throttle_in_bytes" : 104857600

        },

        "refresh" : {

          "total" : 59,

          "total_time_in_millis" : 30946,

          "listeners" : 0

        },

        "flush" : {

          "total" : 10,

          "total_time_in_millis" : 284

        },

        "warmer" : {

          "current" : 0,

          "total" : 74,

          "total_time_in_millis" : 18

        },

        "query_cache" : {

          "memory_size_in_bytes" : 0,

          "total_count" : 0,

          "hit_count" : 0,

          "miss_count" : 0,

          "cache_size" : 0,

          "cache_count" : 0,

          "evictions" : 0

        },

        "fielddata" : {

          "memory_size_in_bytes" : 0,

          "evictions" : 0

        },

        "completion" : {

          "size_in_bytes" : 0

        },

        "segments" : {

          "count" : 24,

          "memory_in_bytes" : 1619215,

          "terms_memory_in_bytes" : 1435245,

          "stored_fields_memory_in_bytes" : 42880,

          "term_vectors_memory_in_bytes" : 0,

          "norms_memory_in_bytes" : 26112,

          "points_memory_in_bytes" : 1106,

          "doc_values_memory_in_bytes" : 113872,

          "index_writer_memory_in_bytes" : 0,

          "version_map_memory_in_bytes" : 0,

          "fixed_bit_set_memory_in_bytes" : 0,

          "max_unsafe_auto_id_timestamp" : -1,

          "file_sizes" : { }

        },

        "translog" : {

          "operations" : 0,

          "size_in_bytes" : 215

        },

        "request_cache" : {

          "memory_size_in_bytes" : 0,

          "evictions" : 0,

          "hit_count" : 0,

          "miss_count" : 0

        },

        "recovery" : {

          "current_as_source" : 0,

          "current_as_target" : 0,

          "throttle_time_in_millis" : 0

        }

      }

    }

  }

}





만약 다큐먼트 개수만 보고 싶다면 _stats/docs를 호출하면 된다.


 curl localhost:9200/wikinews/_stats/docs?pretty=true

{

  "_shards" : {

    "total" : 10,

    "successful" : 5,

    "failed" : 0

  },

  "_all" : {

    "primaries" : {

      "docs" : {

        "count" : 21067,

        "deleted" : 0

      }

    },

    "total" : {

      "docs" : {

        "count" : 21067,

        "deleted" : 0

      }

    }

  },

  "indices" : {

    "wikinews" : {

      "primaries" : {

        "docs" : {

          "count" : 21067,

          "deleted" : 0

        }

      },

      "total" : {

        "docs" : {

          "count" : 21067,

          "deleted" : 0

        }

      }

    }

  }

}

Posted by '김용환'
,


매트릭스 집계는 일래스틱서치 5.0 버전과 함께 일래스틱서치 집계 범주에 포함된다. 메트릭스 집계는 여러 필드에서 작업하고 해당 필드에서 추출한 값을 기반으로 결과 행렬을 생성하는 방법을 제공한다.

현재 매트릭스 집계 카테고리에서 지원되는 집계 타입은 매트릭스 통계(Matrix Stats) 하나뿐이다.

매트릭스 집계는 주어진 필드 집합에 대한 수치 통계를 계산한다. 


연봉과 만족도 간의 관계를 통핸 수치 통계를 매트릭트 통계로 진행할 수 있다. 

예제는 다음과 같다.


curl -XGET "http://localhost:9200/members/_search?pretty" -d'

{

  "aggs": {

    "matrixstats": {

      "matrix_stats": {

        "fields": [

          "salary",

          "self_esteem"

        ]

      }

    }

  },"size": 0

}'



이렇게 호출하면 다음 값이 나온다.  필드의 대한 자세한 내용은 아래 url에 설명되어 있다.

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-matrix-stats-aggregation.html


{

  "took" : 7,

  "timed_out" : false,

  "_shards" : {

    "total" : 1,

    "successful" : 1,

    "failed" : 0

  },

  "hits" : {

    "total" : 10,

    "max_score" : 0.0,

    "hits" : [ ]

  },

  "aggregations" : {

    "matrixstats" : {

      "fields" : [

        {

          "name" : "self_esteem",

          "count" : 10,

          "mean" : 6.109999990463257,

          "variance" : 1.1721109714508229,

          "skewness" : 0.2987179705001951,

          "kurtosis" : 2.1800689946867364,

          "covariance" : {

            "self_esteem" : 1.1721109714508229,

            "salary" : 8.953332879808212

          },

          "correlation" : {

            "self_esteem" : 1.0,

            "salary" : 0.9734162963071288

          }

        },

        {

          "name" : "salary",

          "count" : 10,

          "mean" : 159.20000000000002,

          "variance" : 72.17777777777786,

          "skewness" : 0.1035519897524805,

          "kurtosis" : 2.685063487344997,

          "covariance" : {

            "self_esteem" : 8.953332879808212,

            "salary" : 72.17777777777786

          },

          "correlation" : {

            "self_esteem" : 0.9734162963071288,

            "salary" : 1.0

          }

        }

      ]

    }

  }

}



여기서 size는 실제 데이터 정보를 포함하는 영역이다. 실제 데이터를 볼 마음이 없다면 항상 size=0을 넣어야 한다.


누락된 값에 대한 디폴트 값을 지정할 수 있다. 


curl -XGET "http://localhost:9200/members/_search?pretty" -d'

  "aggs": { 

    "matrixstats": { 

      "matrix_stats": { 

        "fields": [ 

          "salary", 

          "self_esteem" 

        ], 

    "missing": {"self_esteem" : 5} 

      } 

    } 

  },"size": 0 

}








Posted by '김용환'
,

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline.html


형제 파이프 라인 집계 

  • avg 버킷 집계
  • max 버킷 집계
  • min 버킷 집계
  • sum 버킷 집계
  • stats 버킷 집계
  • extended stats 버킷 집계
  • percentile 버킷 집계
  • moving average  집계

부모 파이프 라인 집계 

  • derivative 집계
  • cumulative 합계
  • bucket script 집계
  • bucket selector 집계
  • serial differencing 집계

파이프 라인 집계는 하위 집계를 지원하지 않지만 bucket_path 매개 변수를 사용하는 집계 체인을 지원하기 때문에 파이프 라인 집계 체인에서 최종 출력은 체인의 각 집계 출력이 포함된다. bucket_path의 구문은 다음과 같다.
AGG_SEPARATOR       =  '>' 
METRIC_SEPARATOR    =  '.' 
AGG_NAME            =  <the name of the aggregation> 
METRIC              =  <the name of the metric (in case of multi-value metrics aggregation)>
PATH                =  <AGG_NAME> [ <AGG_SEPARATOR>, <AGG_NAME> ]* [ <METRIC_SEPARATOR>, <METRIC> ]


예를 들어 "buckets_path": "AGG_NAME>METRIC"은 bucket_path는 집계와 해당 집계의 메트릭을 참조함을 알려준다.


https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggregations-matrix-stats-aggregation.html

Posted by '김용환'
,


https://www.kernel.org/doc/Documentation/sysctl/vm.txt 문서에 따르면 max_map_count의 값은 65536으로 되어 있다. 



max_map_count:

This file contains the maximum number of memory map areas a process may have. Memory map areas are used as a side-effect of calling malloc, directly by mmap and mprotect, and also when loading shared libraries.

While most applications need less than a thousand maps, certain programs, particularly malloc debuggers, may consume lots of them, e.g., up to one or two maps per allocation.

The default value is 65536.



$ cat /proc/sys/vm/max_map_count

65530


일래스틱서치 5에서는 이 값이 너무 작아서. 효율이 떨어지기 때문에 아래처럼 크게 잡아야 한다고 한다(설치할 때 자동으로 바꾼다)


sysctl -w vm.max_map_count=262144




참고 문서 : 5.3 문서 


https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html



Virtual memoryedit

Elasticsearch uses a hybrid mmapfs / niofs directory by default to store its indices. The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions.

On Linux, you can increase the limits by running the following command as root:

sysctl -w vm.max_map_count=262144

To set this value permanently, update the vm.max_map_count setting in /etc/sysctl.conf. To verify after rebooting, run sysctl vm.max_map_count.

The RPM and Debian packages will configure this setting automatically. No further configuration is required.



구 2.x 문서


https://www.elastic.co/guide/en/elasticsearch/guide/current/_file_descriptors_and_mmap.html


Elasticsearch also uses a mix of NioFS and MMapFS for the various files. Ensure that you configure the maximum map count so that there is ample virtual memory available for mmapped files. This can be set temporarily:

sysctl -w vm.max_map_count=262144

Or you can set it permanently by modifying vm.max_map_count setting in your /etc/sysctl.conf.


Posted by '김용환'
,


여러 대의 elasticsearch을 l4로 묶어 쓸 경우, 다음과 같은 NodeDisconnectedException , NoNodeAvailableException 예외가 발생할 수 있다. 






"NodeDisconnectedException: 


 org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes were available: 

 

 

 


http://asuraiv.blogspot.kr/2015/08/elasticsearch-nonodeavailableexception.html



domain이름 대신 개별 ip를 넣었더니 해결되었다. 조금 나아졌지만, 완전히 좋아지지 않는다. 

elasticsearch 상위 버전으로 가야할 듯. 






Posted by '김용환'
,



특정 로그 파일(birthday.log)을 logstash를 통해 kafka로 보내려고 한다.


이미 logstash, kafka가 설치되어 있다고 가정한다.





# logstash 설정


input {
file {
type => "birthday_log"
path => [
"/home/www/googleplus/birthday.log"
]
}
}

output {
kafka {
codec => plain { format => "%{host} %{message}" }
topic_id => "%{type}"
request_required_acks => 1
broker_list => "server-ip:9092"
}
}



# kafka에서 보기


$ /usr/local/kafka/bin/kafka-console-consumer.sh --topic birthday_log --zookeeper zk-server-ip:2181


Posted by '김용환'
,

elasticsearch 벤치마크 툴인 Rally가 4월에 출시되었다.


https://www.elastic.co/kr/blog/announcing-rally-benchmarking-for-elasticsearch


https://github.com/elastic/rally



으와.. 프로파일링과 JIT 컴파일 수까지 표현해준다.


ES 성능 테스트할 때 좋을 것 같다. 




차트와 더 자세한 내용을 보려면 여기를 클릭한다.


https://benchmarks.elastic.co/index.html



https://benchmarks-old.elastic.co/index.html

Posted by '김용환'
,