[etcd] etcd 설치와 간단 예제

Cloud 2017. 8. 23. 19:30



https://github.com/coreos/etcd/releases/ 을 접근하니 3.2.6이 최신이다. 

os 버전별로 바이너리를 다운받고 압축을 푼 디렉토리 밑에 실행하면 된다.



ETCDCTL_API의 값이 설정되야 잘 돌아간다. 

$ ETCDCTL_API=3


또는 .bash_xx에 추가한다. 

EXPORT ETCDCTL_API=3 




version을 확인한다. 


$ ./etcdctl version

etcdctl version: 3.2.6

API version: 3.2




2.x에서 사용되던 ls는 안먹힌다.. 조금씩 바꾼 형태이다. 

$ ./etcdctl ls /

Error: unknown command "ls" for "etcdctl"

Run 'etcdctl --help' for usage.

Error:  unknown command "ls" for "etcdctl"



put, get 예제


$ /etcdctl put foo bar

OK


$ ./etcdctl get foo

foo

bar



from-key 예제



$ ./etcdctl put foo bar

OK


$ ./etcdctl put foo bar

OK


$ ./etcdctl put foo1 bar1

OK


$ ./etcdctl put foo2 bar2

OK


$ ./etcdctl put foo3 bar3

OK


$ ./etcdctl get foo

foo

bar

$ ./etcdctl get --from-key foo1

foo1

bar1

foo2

bar2

foo3

bar3

$ ./etcdctl get foo1 foo3

foo1

bar1

foo2

bar2







etcd 2.x의 ls 와 비슷한 커맨드 from-key


$ ./etcdctl put x y

OK

$ ./etcdctl put y 1

OK

$ ./etcdctl get --from-key ''

x

y

y

1






delete 예제



$ ./etcdctl del foo

1

$ ./etcdctl get foo




$ ./etcdctl put key val

OK

$ ./etcdctl del --prev-kv key

1

key

val

$ ./etcdctl get key




$ ./etcdctl put a 123

OK

$ ./etcdctl put b 456

OK

$ ./etcdctl put z 789

OK

$ ./etcdctl del --from-key a

6

$ ./etcdctl get --from-key a




 ./etcdctl put zoo val

OK

$ ./etcdctl put zoo1 val1

OK

$ ./etcdctl put zoo2 val2

OK

$ ./etcdctl del --prefix zoo

3

$ ./etcdctl get zoo2





원자잭 트랙잭션 txn 예제



$ ./etcdctl txn -i

compares:

mod("key1") > "0"


success requests (get, put, delete):

put key1 "111"

put key2 "222"


failure requests (get, put, delete):


SUCCESS


OK


OK

$ ./etcdctl get key1

key1

111

$ ./etcdctl get key2

key2

222




클러스터 모드롤 실행할 때는 로컬이 아닌 3대의 장비에서 실행시키는 것이 좋다..




저작자표시

'Cloud' 카테고리의 다른 글

[오픈스택] openstack 맞보기 - virtualbox에서 호스트 전용 어댑터 추가하기/네트워크 설정/인스턴스 하나 만들기  (0) 2017.09.04
[openstack] Rally + Tempest  (0) 2017.09.02
fluent-plugin-extract_query_params  (0) 2017.08.23
[펌] fluentd 성능  (0) 2017.08.21
[td-agent] td-agent 설치 및 테스트  (0) 2017.08.18
Posted by '김용환'
,

fluent-plugin-extract_query_params

Cloud 2017. 8. 23. 16:34


fluentd에서 http query의 모든 매개변수 값을 key,value로 포함시키려면 


fluent-plugin-extract_query_params를 설치하고 @type extract_query_params를 활용한다. 




참고 

https://github.com/kentaro/fluent-plugin-extract_query_params


Imagin you have a config as below:

<filter test.**>
  @type extract_query_params

  key            url
  only           foo, baz
</match>

And you feed such a value into fluentd:

"test" => {
  "url" => "http://example.com/?foo=bar&baz=qux&hoge=fuga"
}

Then you'll get re-emmited tag/record-s below:

"extracted.test" => {
  "url" => "http://example.com/?foo=bar&baz=qux&hoge=fuga"
  "foo" => "bar",
  "baz" => "qux"
}


저작자표시

'Cloud' 카테고리의 다른 글

[openstack] Rally + Tempest  (0) 2017.09.02
[etcd] etcd 설치와 간단 예제  (0) 2017.08.23
[펌] fluentd 성능  (0) 2017.08.21
[td-agent] td-agent 설치 및 테스트  (0) 2017.08.18
fluentd 공부  (0) 2017.08.16
Posted by '김용환'
,

[펌] fluentd 성능

Cloud 2017. 8. 21. 18:46

https://github.com/fluent/fluentd-benchmark/tree/master/one_forward



CPU	Xeon E5-2670 2.60GHz x 2 (32 Cores)
Memory	24G
Disk	300G(10000rpm) x 2 [SAS-HDD]
OS CentOS release 6.2 (Final)

Result

Agent
rate of writing (lines/sec)reading (lines/sec)CPU (%)Memory (kB)Remarks
10100.229304
1001000.335812
100010001.337864
10000100006.639912
1000001000006239912
200000157148100.436280MAX
300000N/A
400000N/A
5247047N/AMAX of dummer tool


저작자표시

'Cloud' 카테고리의 다른 글

[etcd] etcd 설치와 간단 예제  (0) 2017.08.23
fluent-plugin-extract_query_params  (0) 2017.08.23
[td-agent] td-agent 설치 및 테스트  (0) 2017.08.18
fluentd 공부  (0) 2017.08.16
[펌] 한국 오픈 스택 2017 자료 올라옴  (0) 2017.08.16
Posted by '김용환'
,

[td-agent] td-agent 설치 및 테스트

Cloud 2017. 8. 18. 19:40


td-agent 설치하는 예제이다. td-agent는 fluentd의 래퍼 프로그램이다. 



rpm은 다음 링크로 설치한다. 


http://docs.fluentd.org/articles/install-by-rpm




$ sudo usermod -a -G adm td-agent

$ sudo -E /usr/sbin/td-agent-gem install fluent-plugin-record-reformer -v 0.8.2

$ sudo -E /usr/sbin/td-agent-gem install fluent-plugin-grep -v 0.3.4

$ sudo -E /usr/sbin/td-agent-gem install fluent-plugin-fields-parser -v 0.1.1




설치가 제대로 되었는지 확인한다. 


$ td-agent --version

td-agent 0.12.12





설정 파일을 생성한다.


$ sudo vi /etc/td-agent/td-agent.conf



 <source>

    type tail

    path /tmp/test.log

    pos_file /var/run/td-agent/test.log.pos

    tag public.test-service1.raw

    format /^(?<time>\S+ \S+) - \[(?<level>\S+)\] (?<msg>.+)/

    time_format %Y-%m-%d %H:%M:%S,%L

  </source>

  <match public.test-service1.raw>

    type record_reformer

    enable_ruby false

    tag public.test-service1

    <record>

      hostname ${hostname}

      phase alpha

    </record>

  </match>

  <match public.test-service1>

    type stdout

  </match>
</source>

  


문법을 확인한다.


sudo td-agent -c /etc/td-agent/td-agent.conf



데몬 실행 후 /tmp/test.log 로그를 확인한다.



2017-08-18 19:39:39,000 - [WARN] time over

2017-08-18 19:39:39 +0900 public.test-service1: {"level":"WARN","msg":"time over","hostname":"aaaaa.google.com","phase":"alpha"}



잘 동작하면 td-agent가 계속 동작하도록 한다.


sudo /sbin/chkconfig td-agent on

저작자표시

'Cloud' 카테고리의 다른 글

fluent-plugin-extract_query_params  (0) 2017.08.23
[펌] fluentd 성능  (0) 2017.08.21
fluentd 공부  (0) 2017.08.16
[펌] 한국 오픈 스택 2017 자료 올라옴  (0) 2017.08.16
[펌] fluentd 사용 사례  (0) 2017.08.15
Posted by '김용환'
,

fluentd 공부

Cloud 2017. 8. 16. 16:22


기존에 ruby 2.1.0이 설치되어 있고, fluentd 0.14를 사용해서 fluentd를 테스트해본다.



설치 방법은 다음과 같다. 


$ gem install fluentd -v "~> 0.12.0" --no-ri --no-rdoc

Fetching: string-scrub-0.0.5.gem (100%)

Building native extensions.  This could take a while...

Successfully installed string-scrub-0.0.5

Fetching: fluentd-0.12.39.gem (100%)

Successfully installed fluentd-0.14.20

2 gems installed



버전을 확인한다. 


$ fluentd --version

fluentd 0.14.20




설정 파일을 생성하고 fluentd를 실행한다. 설정 파일과 플러그인 파일을 읽는 로그를 확인할 수 있다.


$ fluentd --setup ./fluent

Installed ./fluent/fluent.conf.



$ fluentd -c ./fluent/fluent.conf -vv

2017-08-16 16:18:25 +0900 [info]: fluent/log.rb:316:call: parsing config file is succeeded path="./fluent/fluent.conf"

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered output plugin 'stdout'

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered buffer plugin 'memory'

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered formatter plugin 'stdout'

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered formatter plugin 'json'

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered output plugin 'forward'

2017-08-16 16:18:25 +0900 [info]: fluent/log.rb:316:call: adding forwarding server '192.168.0.12:24224' host="192.168.0.12" port=24224 weight=60 plugin_id="object:3ff3daa51474"

2017-08-16 16:18:25 +0900 [info]: [forward_output] adding forwarding server '192.168.0.11:24224' host="192.168.0.11" port=24224 weight=60 plugin_id="forward_output"

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered input plugin 'forward'

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered parser plugin 'in_http'

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered input plugin 'http'

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered parser plugin 'msgpack'

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered parser plugin 'json'

2017-08-16 16:18:25 +0900 [info]: [http_input] Oj is not installed, and failing back to Yajl for json parser

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered input plugin 'monitor_agent'

2017-08-16 16:18:25 +0900 [trace]: fluent/log.rb:275:call: registered input plugin 'debug_agent'

2017-08-16 16:18:25 +0900 [info]: fluent/log.rb:316:call: using configuration file: <ROOT>

  <source>

    @type forward

    @id forward_input

  </source>

  <source>

    @type http

    @id http_input

    port 8888

  </source>

  <source>

    @type monitor_agent

    @id monitor_agent_input

    port 24220

  </source>

  <source>

    @type debug_agent

    @id debug_agent_input

    bind "127.0.0.1"

    port 24230

  </source>

  <match debug.**>

    @type stdout

    @id stdout_output

  </match>

  <match system.**>

    @type forward

    @id forward_output

    <server>

      host "192.168.0.11"

    </server>

    <secondary>

      <server>

        host "192.168.0.12"

      </server>

    </secondary>

  </match>

</ROOT>

2017-08-16 16:18:25 +0900 [info]: fluent/log.rb:316:call: starting fluentd-0.14.20 pid=11375

2017-08-16 16:18:25 +0900 [info]: fluent/log.rb:316:call: spawn command to main:  cmdline=["/Users/samuel.kim/.story-spec/opt/ruby/bin/ruby", "-Eascii-8bit:ascii-8bit", "/Users/samuel.kim/.story-spec/opt/ruby/bin/fluentd", "-c", "./fluent/fluent.conf", "-vv", "--under-supervisor"]

2017-08-16 16:18:26 +0900 [info]: fluent/log.rb:316:call: gem 'fluentd' version '0.14.20'

2017-08-16 16:18:26 +0900 [info]: fluent/log.rb:316:call: gem 'fluentd' version '0.12.39'

2017-08-16 16:18:26 +0900 [info]: fluent/log.rb:316:call: adding match pattern="debug.**" type="stdout"

2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered output plugin 'stdout'

2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered buffer plugin 'memory'

2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered formatter plugin 'stdout'

2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered formatter plugin 'json'

2017-08-16 16:18:26 +0900 [info]: fluent/log.rb:316:call: adding match pattern="system.**" type="forward"

2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered output plugin 'forward'

2017-08-16 16:18:26 +0900 [info]: #0 fluent/log.rb:316:call: adding forwarding server '192.168.0.12:24224' host="192.168.0.12" port=24224 weight=60 plugin_id="object:3fe29618a3b8"

2017-08-16 16:18:26 +0900 [info]: #0 [forward_output] adding forwarding server '192.168.0.11:24224' host="192.168.0.11" port=24224 weight=60 plugin_id="forward_output"

2017-08-16 16:18:26 +0900 [info]: fluent/log.rb:316:call: adding source type="forward"

2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered input plugin 'forward'

2017-08-16 16:18:26 +0900 [info]: fluent/log.rb:316:call: adding source type="http"

2017-08-16 16:18:26 +0900 [trace]: #0 fluent/log.rb:275:call: registered parser plugin 'in_http'




로그는 fluent/fluent.conf에 있다. 로그의 내용과 동일하다. 


$ cat fluent/fluent.conf

# In v1 configuration, type and id are @ prefix parameters.

# @type and @id are recommended. type and id are still available for backward compatibility


## built-in TCP input

## $ echo <json> | fluent-cat <tag>

<source>

  @type forward

  @id forward_input

</source>


## built-in UNIX socket input

#<source>

#  @type unix

#</source>


# HTTP input

# http://localhost:8888/<tag>?json=<json>

<source>

  @type http

  @id http_input


  port 8888

</source>


## File input

## read apache logs with tag=apache.access

#<source>

#  @type tail

#  format apache

#  path /var/log/httpd-access.log

#  tag apache.access

#</source>


# Listen HTTP for monitoring

# http://localhost:24220/api/plugins

# http://localhost:24220/api/plugins?type=TYPE

# http://localhost:24220/api/plugins?tag=MYTAG

<source>

  @type monitor_agent

  @id monitor_agent_input


  port 24220

</source>


# Listen DRb for debug

<source>

  @type debug_agent

  @id debug_agent_input


  bind 127.0.0.1

  port 24230

</source>


## match tag=apache.access and write to file

#<match apache.access>

#  @type file

#  path /var/log/fluent/access

#</match>


## match tag=debug.** and dump to console

<match debug.**>

  @type stdout

  @id stdout_output

</match>


# match tag=system.** and forward to another fluent server

<match system.**>

  @type forward

  @id forward_output


  <server>

    host 192.168.0.11

  </server>

  <secondary>

    <server>

      host 192.168.0.12

    </server>

  </secondary>

</match>


## match tag=myapp.** and forward and write to file

#<match myapp.**>

#  @type copy

#  <store>

#    @type forward

#    buffer_type file

#    buffer_path /var/log/fluent/myapp-forward

#    retry_limit 50

#    flush_interval 10s

#    <server>

#      host 192.168.0.13

#    </server>

#  </store>

#  <store>

#    @type file

#    path /var/log/fluent/myapp

#  </store>

#</match>


## match fluent's internal events

#<match fluent.**>

#  @type null

#</match>


## match not matched logs and write to file

#<match **>

#  @type file

#  path /var/log/fluent/else

#  compress gz

#</match>




fluent-cat를 이용해 동작 여부를 확인한다.



$ echo '{"json":"message"}' | fluent-cat debug.test



$ fluentd -c ./fluent/fluent.conf -vv

...

2017-08-16 19:52:03.309664000 +0900 debug.test: {"json":"message"}





fluent/fluent.conf 파일을 참조해서 실제로 동작하는지 테스트해본다.


 debug와 system 태그로 시작될 때에만 콘솔에 출력된다.


# HTTP input

# http://localhost:8888/<tag>?json=<json>

<source>

  @type http

  @id http_input


  port 8888

</source>
..

## match tag=debug.** and dump to console

<match debug.**>

  @type stdout

  @id stdout_output

</match>


# match tag=system.** and forward to another fluent server

<match system.**>

  @type forward

  @id forward_output


  <server>

    host 192.168.0.11

  </server>

  <secondary>

    <server>

      host 192.168.0.12

    </server>

  </secondary>

</match>




http로 사용한다.



$ curl -v http://localhost:8888/debug.test -F 'json={"hello":"world"}'

*   Trying 127.0.0.1...

* Connected to localhost (127.0.0.1) port 8888 (#0)

> POST /debug.http HTTP/1.1

> Host: localhost:8888

> User-Agent: curl/7.43.0

> Accept: */*

> Content-Length: 156

> Expect: 100-continue

> Content-Type: multipart/form-data; boundary=------------------------5bc2e845e167b296

>

< HTTP/1.1 100 Continue

< HTTP/1.1 200 OK

< Content-Type: text/plain

< Connection: Keep-Alive

< Content-Length: 0

<

* Connection #0 to host localhost left intact





fluent 로그에 전달한 json이 출력된다.


2017-08-16 19:55:47.099363000 +0900 debug.test: {"hello":"world"}




그러나 매개 변수가 json 규격에 잘 맞지 않으면 로그가 발생하지 않는다. 내부적으로 파싱하고 있는 듯 하다. 쌍따옴표를 없애면 파싱 구문만 제외하고 나온다




curl  -d 'json=afdafd'  http://localhost:8888/debug.test

(결과 안나옴)


curl  -d 'json={"hello":"world}'  http://localhost:8888/debug.test

(결과 : 2017-08-16 19:59:21.074761000 +0900 debug.test: "hello")

...





curl  -d 'json={"hello":"world}'  http://localhost:8888/debug1.test

(결과 : 

2017-08-16 20:00:39 +0900 [warn]: #0 fluent/log.rb:336:call: no patterns matched tag="debug1.test" )




fluent-logger를 설치하면 pry를 통해 fluentd에 전달할 수 있다. 


$ gem i fluent-logger --no-ri --no-rdoc

Fetching: fluent-logger-0.7.1.gem (100%)

Successfully installed fluent-logger-0.7.1

1 gem installed

$ pry

[1] pry(main)> require 'fluent-logger'

=> true

[2] pry(main)> Fluent::Logger::FluentLogger.open(nil, :host => 'localhost', :port => 24224)

=> #<Fluent::Logger::FluentLogger:0x007fc0892dbad0

 @buffer_overflow_handler=nil,

 @con=#<TCPSocket:fd 10>,

 @connect_error_history=[],

 @factory=#<MessagePack::Factory:0x007fc0892db918>,

 @host="localhost",

 @last_error={},

 @limit=8388608,

 @log_reconnect_error_threshold=13,

 @logged_reconnect_error=false,

 @logger=

  #<Logger:0x007fc0892db670

   @default_formatter=#<Logger::Formatter:0x007fc0892db648 @datetime_format=nil>,

   @formatter=nil,

   @level=1,

   @logdev=

    #<Logger::LogDevice:0x007fc0892db5a8

     @dev=#<IO:<STDERR>>,

     @filename=nil,

     @mutex=#<Logger::LogDevice::LogDeviceMutex:0x007fc0892db580 @mon_count=0, @mon_mutex=#<Mutex:0x007fc0892db530>, @mon_owner=nil>,

     @shift_age=nil,

     @shift_size=nil>,

   @progname=nil>,

 @mon=#<Monitor:0x007fc0892db760 @mon_count=0, @mon_mutex=#<Mutex:0x007fc0892db710>, @mon_owner=nil>,

 @nanosecond_precision=nil,

 @packer=#<MessagePack::Packer:0x007fc0892db8f0>,

 @pending=nil,

 @port=24224,

 @socket_path=nil,

 @tag_prefix=nil>

[3] pry(main)> Fluent::Logger.post("debug.test", { "id" => "1", "name" => "samuel" })

=> true



잘 나온다.


$ fluentd -c ./fluent/fluent.conf -vv

...


2017-08-16 20:03:59.000000000 +0900 debug.test: {"id":"1","name":"samuel"}




설정 파일 문법

https://docs.fluentd.org/articles/config-file#v1-format



tail log 문법

https://docs.fluentd.org/articles/in_tail



한글 착한 문서


http://blog.seulgi.kim/2014/04/fluentd-pluggable-log-collector.html

https://docs.google.com/presentation/d/1e6PXtPS_eQn7vY5hDhHlDVpCk1U6eKARoRqInoDiUkw/edit#slide=id.g271a4ef8f_018




과거에는 fluentd가 nanosecond 이슈가 있었다고 하나, 0.14부터는 nanosecond를 잘 보여주고 있다. 




https://www.fluentd.org/blog/fluentd-v0.14.9-has-been-released



time_as_integer true


out_forward

Since this version, time_as_integer parameter is now false. It means v0.14.9's out_forward can't forward data to v0.12's in_forward by default. You need to set time_as_integer true explicitly. We have already mentioned this point in v0.14.0 release, so we hope this change doesn't break your v0.14 to v0.12 forwarding.



저작자표시

'Cloud' 카테고리의 다른 글

[펌] fluentd 성능  (0) 2017.08.21
[td-agent] td-agent 설치 및 테스트  (0) 2017.08.18
[펌] 한국 오픈 스택 2017 자료 올라옴  (0) 2017.08.16
[펌] fluentd 사용 사례  (0) 2017.08.15
처음 본 오픈스택 Neutron  (0) 2017.07.24
Posted by '김용환'
,

[펌] 한국 오픈 스택 2017 자료 올라옴

Cloud 2017. 8. 16. 11:23


한국 오픈 스택 2017 자료 올라옴


http://event.openstack.or.kr/2017/program.html




저작자표시

'Cloud' 카테고리의 다른 글

[td-agent] td-agent 설치 및 테스트  (0) 2017.08.18
fluentd 공부  (0) 2017.08.16
[펌] fluentd 사용 사례  (0) 2017.08.15
처음 본 오픈스택 Neutron  (0) 2017.07.24
처음 본 오픈스택 Swift  (0) 2017.07.24
Posted by '김용환'
,

[펌] fluentd 사용 사례

Cloud 2017. 8. 15. 10:37



카카오 

http://tech.kakao.com/2016/08/25/kemi/





라인에서 fludentd, hadoop를 사용한 사례


https://engineering.linecorp.com/en/blog/detail/83



https://promcon.io/2016-berlin/talks/hadoop-fluentd-cluster-monitoring-with-prometheus-and-grafana/



https://www.fluentd.org/casestudy/line





RDRC 2014 발표 자료 -fluentd

https://www.slideshare.net/tagomoris/fluentd-data-streams-in-ruby-world-rdrc2014



fluentd와 norikira 소개

https://engineering.linecorp.com/en/blog/detail/47



Log Aggregator 비교 - Scribe, Flume, Fluentd, logstash

http://blog.seulgi.kim/2014/04/log-aggregator-scribe-flume-fluentd.html



Fluentd vs. Logstash for OpenStack Log Management

https://www.slideshare.net/tagomoris/monitoring-with-fluentd



norikra - schema less fluentd sql engine 

http://tagomoris.hatenablog.com/entry/2017/07/28/123613

http://norikra.github.io/




https://www.slideshare.net/td-nttcom/fluentd-vs-logstash-for-openstack-log-management

저작자표시

'Cloud' 카테고리의 다른 글

fluentd 공부  (0) 2017.08.16
[펌] 한국 오픈 스택 2017 자료 올라옴  (0) 2017.08.16
처음 본 오픈스택 Neutron  (0) 2017.07.24
처음 본 오픈스택 Swift  (0) 2017.07.24
처음 본 오픈 스택 Glance  (0) 2017.07.21
Posted by '김용환'
,

처음 본 오픈스택 Neutron

Cloud 2017. 7. 24. 16:30

..

저작자표시

'Cloud' 카테고리의 다른 글

[펌] 한국 오픈 스택 2017 자료 올라옴  (0) 2017.08.16
[펌] fluentd 사용 사례  (0) 2017.08.15
처음 본 오픈스택 Swift  (0) 2017.07.24
처음 본 오픈 스택 Glance  (0) 2017.07.21
처음 본 오픈 스택의 Nova  (0) 2017.07.20
Posted by '김용환'
,

처음 본 오픈스택 Swift

Cloud 2017. 7. 24. 00:54



오픈 스택의 swift(https://github.com/openstack/swift)은 object stroage이다. 


동영상, 이미지, 디스크 이미지등을 저장할 수 있는 저장공간이다. 


HTTP/HTTPS REST api를 이용해 데이터를 저장할 수 있다. 





출처 : https://docs.openstack.org/security-guide/object-storage.html




 RAID가 없기 때문에 가격이 싸고 S3 API를 지원하며 스케일링을 쉽게 진행할 수 있다. 중앙 DB가 없고 TTL을 지원하는 많은 기능을 가지고 있다. https://docs.openstack.org/swift/latest/admin/objectstorage-features.html을 살펴보면 더 자세한 내용을 확인할 수 있다. 


FeaturesBenefits
Leverages commodity hardwareNo lock-in, lower price/GB.
HDD/node failure agnosticSelf-healing, reliable, data redundancy protects from failures.
Unlimited storageLarge and flat namespace, highly scalable read/write access, able to serve content directly from storage system.
Multi-dimensional scalabilityScale-out architecture: Scale vertically and horizontally-distributed storage. Backs up and archives large amounts of data with linear performance.
Account/container/object structureNo nesting, not a traditional file system: Optimized for scale, it scales to multiple petabytes and billions of objects.
Built-in replication 3✕ + data redundancy (compared with 2✕ on RAID)A configurable number of accounts, containers and object copies for high availability.
Easily add capacity (unlike RAID resize)Elastic data scaling with ease.
No central databaseHigher performance, no bottlenecks.
RAID not requiredHandle many small, random reads and writes efficiently.
Built-in management utilitiesAccount management: Create, add, verify, and delete users; Container management: Upload, download, and verify; Monitoring: Capacity, host, network, log trawling, and cluster health.
Drive auditingDetect drive failures preempting data corruption.
Expiring objectsUsers can set an expiration time or a TTL on an object to control access.
Direct object accessEnable direct browser access to content, such as for a control panel.
Realtime visibility into client requestsKnow what users are requesting.
Supports S3 APIUtilize tools that were designed for the popular S3 API.
Restrict containers per accountLimit access to control usage by user.




이해에 도움되는 swift 슬라이드 자료이다 .


Openstack Swift overview from 어형 이


저작자표시

'Cloud' 카테고리의 다른 글

[펌] fluentd 사용 사례  (0) 2017.08.15
처음 본 오픈스택 Neutron  (0) 2017.07.24
처음 본 오픈 스택 Glance  (0) 2017.07.21
처음 본 오픈 스택의 Nova  (0) 2017.07.20
[펌] 공개된 카카오 오픈 스택 관련 공개 및 강의 자료  (0) 2017.07.20
Posted by '김용환'
,

처음 본 오픈 스택 Glance

Cloud 2017. 7. 21. 20:31


오픈 스택의 Glance(https://docs.openstack.org/glance/latest/contributor/architecture.html)를 사용하면 운영체제 템플릿 이미지를 등록할 수 있다. 이미지 등록, 삭제, 변경등을 api를 통해 진행할 수 있다. 




Glance API 서버를 통해 controller를 지나 Registry Layer에서 Glance DB를 사용하는 구조이다. 



출처 : https://docs.openstack.org/glance/latest/contributor/architecture.html




이미지 포맷은 https://docs.openstack.org/image-guide/image-formats.html에서 볼 수 있듯이 다양하다.


많은 이미지와 컨테이너 타입을 지원하고 있다. 표로 설명하는 문서(https://platform9.com/support/supported-image-types-and-file-formats/)는 다음과 같다.


Format TypeDescriptionSupported By Platform9? Our Comments
raw/imgAn unstructured disk image format; if you have a file without an extension it is possibly a raw format.Yes (given .raw, .img, .dat, .bin extensions)Common extensions .raw, .img, .dat, .bin
qcow2Supported by the QEMU emulator that can expand dynamically and supports Copy-on-WriteYesMost common file format used with Linux/KVM
vmdkCommon disk format supported by many common virtual machine monitorsYes (for VMware vSphere hypervisor)VMDK format is commonly associated with VMware
vdiSupported by VirtualBox virtual machine monitor and the QEMU emulatorNot recommended(Will be recognized, but may or may not work depending on driver availability)VirtualBox image files in practice may or may not work with Linux/KVM hypervisor. You might run into compatibility issues due to missing required virtualization drivers)
isoAn archive format for the data contents of an optical disc, such as CD-ROMYes
akiAn Amazon kernel imageNoAmazon image types are not popularly used with Linux/KVM
ariAn Amazon ramdisk imageNoAmazon image types are not popularly used with Linux/KVM
amiAn Amazon machine imageNoAmazon image types are not popularly used with Linux/KVM
vhdThe VHD disk format, a common disk format used by virtual machine monitors from VMware, Xen, Microsoft, VirtualBox, and others
No
 Not a popular image format




우분투 OS이미지를 생성하는 방법(아마도 구 버전이겠지만.) 대충 이렇다.

(출처 : http://lollyrock.com/articles/glance-image-create/)


glance image-create --name 'Ubuntu 14.04 LTS' \ --container-format bare \ --disk-format qcow2 \ --is-public true \ --copy-from http://uec-images.ubuntu.com/releases/14.04/release/ubuntu-14.04-server-cloudimg-amd64-disk1.img +------------------+--------------------------------------+ | Property | Value | +------------------+--------------------------------------+ | checksum | None | | container_format | bare | | created_at | 2014-04-02T13:16:20 | | deleted | False | | deleted_at | None | | disk_format | qcow2 | | id | aa5ab6de-e461-4330-136e-fb149802bdc0 | | is_public | True | | min_disk | 0 | | min_ram | 0 | | name | Ubuntu 14.04 LTS Beta 1 | | owner | 9ab2f1a7ca521eee9c8de5a14567d377 | | protected | False | | size | 261095936 | | status | queued | | updated_at | 2014-04-02T13:16:20 | +------------------+--------------------------------------+



glance 슬라이드 자료.


OpenStack Glance from Deepti Ramakrishna


저작자표시

'Cloud' 카테고리의 다른 글

처음 본 오픈스택 Neutron  (0) 2017.07.24
처음 본 오픈스택 Swift  (0) 2017.07.24
처음 본 오픈 스택의 Nova  (0) 2017.07.20
[펌] 공개된 카카오 오픈 스택 관련 공개 및 강의 자료  (0) 2017.07.20
처음 본 오픈 스택의 Cinder  (0) 2017.07.18
Posted by '김용환'
,

티스토리툴바