apache storm은 apache zookeeper를 필요로한다. 




* apache zookeeper 설치와 실행

http://www.apache.org/dyn/closer.cgi/zookeeper/



$ ./zkServer.sh start

ZooKeeper JMX enabled by default

Using config: /usr/local/zookeeper-3.4.10/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED






* apache storm 설치 


http://storm.apache.org/downloads.html에서 설치 후 /usr/local로 옮긴다.


(archieves는 http://archive.apache.org/dist/storm/에 존재한다. 1.0.1로 테스트한다.)



설정을 변경해 local zookeeper를 바라보도록 한다. 


$ vi conf/storm.yaml


storm.zookeeper.servers:

      - localhost

nimbus.seeds: ["localhost"]





master node인 nimbus를 실행한다. 



$ ./bin/storm nimbus

Running: /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/bin/java -server -Ddaemon.name=nimbus .... org.apache.storm.daemon.nimbus




supervisor 노드를 실행한다.


$ ./bin/storm supervisor

Running: /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/bin/java -server -Ddaemon.name=supervisor... org.apache.storm.daemon.supervisor




admin ui를 위해 ui도 실행한다.


$ ./bin/storm ui

Running: /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/bin/java -server -Ddaemon.name=ui ... org.apache.storm.ui.core



웹 브라우져에서 http://localhost:8080에 접속해서 정상적으로 접근되는지 확인한다.




데몬을 확인한다. 

$ ps -ef | grep apache-storm


3개의 데몬 nimbus, ui, supervisor가 제대로 떠 있는지 확인할 수 있다. 


  


테스트를 위해 wordcount topology를 submit 한다. 


$ ./bin/storm jar ./examples/storm-starter/storm-starter-topologies-1.0.1.jar org.apache.storm.starter.WordCountTopology wordcount

Running: /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/local/apache-storm-1.0.1 -Dstorm.log.dir=/usr/local/apache-storm-1.0.1/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/local/apache-storm-1.0.1/lib/asm-5.0.3.jar:/usr/local/apache-storm-1.0.1/lib/clojure-1.7.0.jar:/usr/local/apache-storm-1.0.1/lib/disruptor-3.3.2.jar:/usr/local/apache-storm-1.0.1/lib/kryo-3.0.3.jar:/usr/local/apache-storm-1.0.1/lib/log4j-api-2.1.jar:/usr/local/apache-storm-1.0.1/lib/log4j-core-2.1.jar:/usr/local/apache-storm-1.0.1/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/apache-storm-1.0.1/lib/log4j-slf4j-impl-2.1.jar:/usr/local/apache-storm-1.0.1/lib/minlog-1.3.0.jar:/usr/local/apache-storm-1.0.1/lib/objenesis-2.1.jar:/usr/local/apache-storm-1.0.1/lib/reflectasm-1.10.1.jar:/usr/local/apache-storm-1.0.1/lib/servlet-api-2.5.jar:/usr/local/apache-storm-1.0.1/lib/slf4j-api-1.7.7.jar:/usr/local/apache-storm-1.0.1/lib/storm-core-1.0.1.jar:/usr/local/apache-storm-1.0.1/lib/storm-rename-hack-1.0.1.jar:./examples/storm-starter/storm-starter-topologies-1.0.1.jar:/usr/local/apache-storm-1.0.1/conf:/usr/local/apache-storm-1.0.1/bin -Dstorm.jar=./examples/storm-starter/storm-starter-topologies-1.0.1.jar org.apache.storm.starter.WordCountTopology wordcount

534  [main] INFO  o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -8806961472207405752:-5043320726578157353

585  [main] INFO  o.a.s.s.a.AuthUtils - Got AutoCreds []

640  [main] INFO  o.a.s.StormSubmitter - Uploading topology jar ./examples/storm-starter/storm-starter-topologies-1.0.1.jar to assigned location: /usr/local/apache-storm-1.0.1/storm-local/nimbus/inbox/stormjar-609f1ee2-82e5-4111-bba2-c36b830e0b15.jar

Start uploading file './examples/storm-starter/storm-starter-topologies-1.0.1.jar' to '/usr/local/apache-storm-1.0.1/storm-local/nimbus/inbox/stormjar-609f1ee2-82e5-4111-bba2-c36b830e0b15.jar' (62432746 bytes)

[==================================================] 62432746 / 62432746

File './examples/storm-starter/storm-starter-topologies-1.0.1.jar' uploaded to '/usr/local/apache-storm-1.0.1/storm-local/nimbus/inbox/stormjar-609f1ee2-82e5-4111-bba2-c36b830e0b15.jar' (62432746 bytes)

935  [main] INFO  o.a.s.StormSubmitter - Successfully uploaded topology jar to assigned location: /usr/local/apache-storm-1.0.1/storm-local/nimbus/inbox/stormjar-609f1ee2-82e5-4111-bba2-c36b830e0b15.jar

936  [main] INFO  o.a.s.StormSubmitter - Submitting topology wordcount in distributed mode with conf {"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-8806961472207405752:-5043320726578157353","topology.workers":3,"topology.debug":true}

1196 [main] INFO  o.a.s.StormSubmitter - Finished submitting topology: wordcount


job업로드하고 잘 실행되는지 확인할 수 있다.




실제 소스 디렉토리는 https://github.com/apache/storm/tree/1.0.x-branch/examples/storm-starter에 있다.




main 메소드는 아래 링크에 있다. 


https://github.com/apache/storm/blob/1.0.x-branch/examples/storm-starter/src/jvm/org/apache/storm/starter/WordCountTopology.java


로컬 테스트를 위해 LocalCluster를 사용하고 있고 Spout -> Bolt("split")-> Bolt("count")의 토롤리지를 구성하고 있다. 




  public static void main(String[] args) throws Exception {


    TopologyBuilder builder = new TopologyBuilder();


    builder.setSpout("spout", new RandomSentenceSpout(), 5);


    builder.setBolt("split", new SplitSentence(), 8).shuffleGrouping("spout");

    builder.setBolt("count", new WordCount(), 12).fieldsGrouping("split", new Fields("word"));


    Config conf = new Config();

    conf.setDebug(true);


    if (args != null && args.length > 0) {

      conf.setNumWorkers(3);


      StormSubmitter.submitTopologyWithProgressBar(args[0], conf, builder.createTopology());

    }

    else {

      conf.setMaxTaskParallelism(3);


      LocalCluster cluster = new LocalCluster();

      cluster.submitTopology("word-count", conf, builder.createTopology());


      Thread.sleep(10000);


      cluster.shutdown();

    }

  }




데몬을 확인하면 3개의 주요 데몬 외에.. 6개가 더 떠 있는 것을 확인할 수 있다. (주목할 점은 6700~6702!!)


$ ps -ef | grep apache-storm


  ...  org.apache.storm.daemon.worker wordcount-1-1503398931 6dd4cbd3-1f43-4f17-a90f-b470fddbccbc 6700 28fae52d-9d91-4de8-9130-fe1dff2f2487

... org.apache.storm.daemon.worker wordcount-1-1503398931 6dd4cbd3-1f43-4f17-a90f-b470fddbccbc 6701 8b53b715-0184-4639-b33f-68208cc7d2e3

...  org.apache.storm.daemon.worker wordcount-1-1503398931 6dd4cbd3-1f43-4f17-a90f-b470fddbccbc 6702 e8357b79-dd08-4d70-b4de-2bf69598eadf

비슷한 것 3개가 또 있음 

 


 


제대로 동작했는지 웹 브라우져에서 살펴볼 수 있다. 


http://localhost:8080/topology.html?id=wordcount-1-1503398931

  



http://localhost:8080/api/v1/topology/summary을 호출해 api summary로 확인할 수 있다.


$ curl http://localhost:8080/api/v1/topology/summary

{"topologies":[{"assignedTotalMem":2496.0,"owner":"samuel.kim","requestedMemOnHeap":0.0,"encodedId":"wordcount-1-1503398931","assignedMemOnHeap":2496.0,"id":"wordcount-1-1503398931","uptime":"11m 28s","schedulerInfo":null,"name":"wordcount","workersTotal":3,"status":"INACTIVE","requestedMemOffHeap":0.0,"tasksTotal":28,"requestedCpu":0.0,"replicationCount":1,"executorsTotal":28,"uptimeSeconds":688,"assignedCpu":0.0,"assignedMemOffHeap":0.0,"requestedTotalMem":0.0}],"schedulerDisplayResource":false}




목록을 확인할 수 있다. 


$ ./bin/storm list

Topology_name        Status     Num_tasks  Num_workers  Uptime_secs

-------------------------------------------------------------------

wordcount            ACTIVE     28         3            370




로그로 확인해보려면 logs 밑에 디렉토리가 있는데, 아까 봤던 ps ef로 봤던 6700~6702가 보인다. 


$ls -al logs/workers-artifacts/wordcount-1-1503398931/670

6700/ 6701/ 6702/




$ ls -al logs/workers-artifacts/wordcount-1-1503398931/6700/

gc.log.0.current    worker.log          worker.log.err      worker.log.metrics  worker.log.out      worker.pid          worker.yaml




정상적으로 작동했는지 하나 살펴본다. 


$ ls -al logs/workers-artifacts/wordcount-1-1503398931/6700/worker.log

-rw-r--r--  1 samuel.kim  staff  48832569  8 22 19:57 logs/workers-artifacts/wordcount-1-1503398931/6700/worker.log

[/usr/local/apache-storm-1.0.1] tail -n 10 logs/workers-artifacts/wordcount-1-1503398931/6700/worker.log

2017-08-22 19:57:46.245 o.a.s.d.executor [INFO] Processing received message FOR 10 TUPLE: source: split:21, stream: default, id: {}, [cow]

2017-08-22 19:57:46.245 o.a.s.d.executor [INFO] BOLT ack TASK: 13 TIME:  TUPLE: source: split:22, stream: default, id: {}, ["away"]

2017-08-22 19:57:46.245 o.a.s.d.task [INFO] Emitting: count default [cow, 4976]

2017-08-22 19:57:46.245 o.a.s.d.executor [INFO] Execute done TUPLE source: split:22, stream: default, id: {}, ["away"] TASK: 13 DELTA:

2017-08-22 19:57:46.245 o.a.s.d.executor [INFO] BOLT ack TASK: 10 TIME:  TUPLE: source: split:21, stream: default, id: {}, [cow]

2017-08-22 19:57:46.245 o.a.s.d.executor [INFO] Execute done TUPLE source: split:21, stream: default, id: {}, [cow] TASK: 10 DELTA:

2017-08-22 19:57:46.246 o.a.s.d.executor [INFO] Processing received message FOR 7 TUPLE: source: split:19, stream: default, id: {}, ["am"]

2017-08-22 19:57:46.246 o.a.s.d.task [INFO] Emitting: count default [am, 5050]

2017-08-22 19:57:46.246 o.a.s.d.executor [INFO] BOLT ack TASK: 7 TIME:  TUPLE: source: split:19, stream: default, id: {}, ["am"]

2017-08-22 19:57:46.246 o.a.s.d.executor [INFO] Execute done TUPLE source: split:19, stream: default, id: {}, ["am"] TASK: 7 DELTA:






deactivate 할 수도 있다. 


$ ./bin/storm deactivate wordcount

1650 [main] INFO  o.a.s.c.deactivate - Deactivated topology: wordcount




list로 상태를 본다.


$ ./bin/storm list

Topology_name        Status     Num_tasks  Num_workers  Uptime_secs

-------------------------------------------------------------------

wordcount            INACTIVE   28         3            617








kill해본다.

$./bin/storm kill wordcount

1696 [main] INFO  o.a.s.c.kill-topology - Killed topology: wordcount


ui에서 목록이 사라진다. 


$ curl http://localhost:8080/api/v1/topology/summary

{"topologies":[],"schedulerDisplayResource":false}



웹 브라우저에서 http://localhost:8080/index.html를 열어보면 wordcount topology가 보이지 않거나 wordcount topology가 나타나되 KILLED 상태인 것을 확인할 수도 있다. 

잠깐 wordcount topology가 보여도 결국은 사라진다..





nimbus를 중지해야 worker 데몬은 모두 종료된다. SPOF의 위험성이 높긴 하다.. 


Posted by 김용환 '김용환'


apache common의 UnmodifiableMap은 Immutable과 비슷하지만 기존 맵을 변경하지 못하도록 하지만 실제 객체의 인스턴스를 래핑하는 객체정도로 보면 좋을 것 같다. 


다음은 예시이다. hashCode를 보면 동일한 객체인지 알 수 있다. 





import org.apache.commons.collections.map.UnmodifiableMap;


Map<String, Object> map1 = Maps.newHashMap();

map1.put("key", "value");


MapUtils.debugPrint(System.out, "map1", map1);

System.out.println(map1.hashCode());

System.out.println();


Map<String, Object> map2 = UnmodifiableMap.decorate(map1);

MapUtils.debugPrint(System.out, "map2", map2);

System.out.println(map2.hashCode());



결과


map1 = 

{

    key = value java.lang.String

} java.util.HashMap

112004910


map2 = 

{

    key = value java.lang.String

} org.apache.commons.collections.map.UnmodifiableMap

112004910





UnmodifiableMap 코드 내부에서는 아래와 같이 변경하려 하면 에러를 발생시킨다. 


    public void clear() {

        throw new UnsupportedOperationException();

    }


    public Object put(Object key, Object value) {

        throw new UnsupportedOperationException();

    }


    public void putAll(Map mapToCopy) {

        throw new UnsupportedOperationException();

    }


    public Object remove(Object key) {

        throw new UnsupportedOperationException();

    }


Posted by 김용환 '김용환'


curl을 사용할 때 -F를 사용해 boundary를 사용한 예제이다. 



curl -XPOST "http://up.google.com/up/" -F "key1=value1" -F "key2=value2" -v

*   Trying ...

* Connected to up.google.com (1.1.1.1) port 80 (#0)

> POST /up/ HTTP/1.1

> Host: up.google.com

> User-Agent: curl/7.43.0

> Accept: */*

> Content-Length: 465

> Expect: 100-continue

> Content-Type: multipart/form-data; boundary=------------------------fa94669c2a0ffcbe

>

< HTTP/1.1 100 Continue

< HTTP/1.1 200 OK

< Server: openresty

< Date: Fri, 09 Jun 2017 12:25:23 GMT

< Content-Type: application/json

< Transfer-Encoding: chunked

< Connection: keep-alive

<

* Connection #0 to host up.google left intact






이를 apache http를 사용해서 boundary 를 적용한 예제이다. 


import org.apache.http.HttpEntity;

import org.apache.http.client.methods.CloseableHttpResponse;

import org.apache.http.client.methods.HttpPost;

import org.apache.http.entity.mime.HttpMultipartMode;

import org.apache.http.entity.mime.MultipartEntityBuilder;

import org.apache.http.entity.mime.content.StringBody;

import org.apache.http.impl.client.CloseableHttpClient;

import org.apache.http.util.EntityUtils;

import org.junit.Test;




@Test

public void test() throws Exception {


HttpPost httpPost = new HttpPost("http://up.google.com/up/");


String BOUNDARY = "----------fa94669c2a0ffcbe";

httpPost.setHeader("Content-Type", "multipart/form-data;boundary=" + BOUNDARY);


MultipartEntityBuilder builder = MultipartEntityBuilder.create();

builder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);

builder.setBoundary(BOUNDARY);

builder.addPart("text_1", new StringBody("304"));

builder.addPart("text_2", new StringBody(toHex("사랑합니다💕 ")));

httpPost.setEntity(builder.build());


CloseableHttpClientFactory httpClientFactory = new CloseableHttpClientFactory();

CloseableHttpClient httpClient = httpClientFactory.getObject();

CloseableHttpResponse response = httpClient.execute(httpPost);

System.out.println("response : " + response);


if (response.getStatusLine().getStatusCode() == 200 || response.getStatusLine().getStatusCode() == 201) {

HttpEntity entity = response.getEntity();

String responseString = EntityUtils.toString(entity, "UTF-8");

System.out.println("result : " + responseString);

} else {

System.out.println("error");

}


}


public String toHex(String arg) {

return String.format("%040x", new BigInteger(1, arg.getBytes()));

}



로그는 다음과 같이 나타난다.


21:49:43.806 [main] DEBUG o.a.h.c.protocol.RequestAddCookies - CookieSpec selected: default

21:49:43.814 [main] DEBUG o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context

21:49:43.816 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://up.google.com:80][total kept alive: 0; route allocated: 0 of 5; total allocated: 0 of 30]

21:49:43.824 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {}->http://up.google.com:80][total kept alive: 0; route allocated: 1 of 5; total allocated: 1 of 30]

21:49:43.825 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Opening connection {}->http://up.google.com:80

21:49:44.234 [main] DEBUG o.a.h.i.c.DefaultHttpClientConnectionOperator - Connecting to up.google.com/1.1.1.1:80

21:49:44.247 [main] DEBUG o.a.h.i.c.DefaultHttpClientConnectionOperator - Connection established 1.1.1.2:59752<->1.1.1.1:80

21:49:44.248 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Executing request POST /up/ HTTP/1.1

21:49:44.248 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Target auth state: UNCHALLENGED

21:49:44.249 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Proxy auth state: UNCHALLENGED

21:49:44.250 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> POST /up/ HTTP/1.1

21:49:44.250 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Content-Type: multipart/form-data;boundary=----------HV2ymHFg03ehbqgZCaKO6jyH

21:49:44.250 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Content-Length: 447

21:49:44.250 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Host: up.google.com

21:49:44.250 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Connection: Keep-Alive

21:49:44.250 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> User-Agent: Apache-HttpClient/4.4.1 (Java/1.8.0_101)

21:49:44.251 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "POST /up/ HTTP/1.1[\r][\n]"

21:49:44.251 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Content-Type: multipart/form-data;boundary=----------HV2ymHFg03ehbqgZCaKO6jyH[\r][\n]"

21:49:44.251 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Content-Length: 447[\r][\n]"

21:49:44.251 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Host: up.google.com[\r][\n]"

21:49:44.251 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"

21:49:44.251 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "User-Agent: Apache-HttpClient/4.4.1 (Java/1.8.0_101)[\r][\n]"

21:49:44.251 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]"

21:49:44.251 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "------------HV2ymHFg03ehbqgZCaKO6jyH[\r][\n]"

21:49:44.251 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Content-Disposition: form-data; name="text_1"[\r][\n]"

21:49:44.251 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]"

21:49:44.251 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "304"

21:49:44.251 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]"

21:49:44.251 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "------------HV2ymHFg03ehbqgZCaKO6jyH[\r][\n]"

21:49:44.252 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Content-Disposition: form-data; name="text_2"[\r][\n]"

21:49:44.252 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]"

21:49:44.252 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "ec82aceb9e91ec9db420eb8498ecb998eb8a9420ed9598eba3a8ec9e85eb8b88eb8ba42ef09f929520ebb09beb8a9420eab283eb8f8420eca28beca780eba78c20ed919ced9884ed9598eb8a9420eab283eb8f8420eca491ec9a94ed959c20eab1b020ec9584ec8b9ceca3a03ff09f988a"

21:49:44.252 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]"

21:49:44.252 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "------------HV2ymHFg03ehbqgZCaKO6jyH--[\r][\n]"

21:49:45.401 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"

21:49:45.401 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Server: openresty[\r][\n]"

21:49:45.401 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Date: Fri, 09 Jun 2017 12:49:45 GMT[\r][\n]"

21:49:45.401 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Content-Type: application/json[\r][\n]"

21:49:45.401 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]"

21:49:45.401 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Connection: keep-alive[\r][\n]"

21:49:45.401 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "[\r][\n]"

21:49:45.401 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "338[\r][\n]"

21:49:45.401 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "{.......}}[\r][\n]"

21:49:45.401 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "0[\r][\n]"

21:49:45.401 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "[\r][\n]"

21:49:45.404 [main] DEBUG org.apache.http.headers - http-outgoing-0 << HTTP/1.1 200 OK

21:49:45.405 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Server: openresty

21:49:45.405 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Date: Fri, 09 Jun 2017 12:49:45 GMT

21:49:45.405 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Content-Type: application/json

21:49:45.405 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Transfer-Encoding: chunked

21:49:45.405 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Connection: keep-alive

21:49:45.410 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Connection can be kept alive indefinitely



Posted by 김용환 '김용환'



이모티콘을 mysql에 저장할 때 자바 애플리케이션에서 다음 방식을 사용했다. 




* A 서버 

 mysql connector java : 5.1.21 환경


update set names utfmd4를 사용하지 않고 이모티콘을 잘 저장했다. 

(테이블은 utf8, 컬럼은 utfmd4이었다)



* B 서버


mysql connector java : 5.1.35 환경


update set names utfmd4를 사용해야 이모티콘을 잘 저장했다  

(테이블은 utf8, 컬럼은 utfmd4이었다)

 

  public boolean updateDNS(final DNS dns) {

return transactionTemplate.execute(new TransactionParamBuilder().add(masterJdbcTemplate).build(), new TransactionCallback<Integer>() {

@Override

public Integer doInTransaction() {

masterJdbcTemplate.update("SET NAMES utf8mb4");

String sql = "UPDATE dns SET lang=?, type=?, content=?, status=? WHERE id=?";

return masterJdbcTemplate.update(sql, dns.lang, dns.type, dns.content, 1, dns.id);

}

}) > 0;

}

 

 



아마도..다음 이슈때문은 아닌지... 예상만 한다. 


https://dev.mysql.com/doc/relnotes/connector-j/5.1/en/news-5-1-33.html


The 4-byte UTF8 (utfbmb4) character encoding could not be used with Connector/J when the server collation was set to anything other than the default value (utf8mb4_general_ci). This fix makes Connector/J detect and set the proper character mapping for any utfmb4 collation. (Bug #19479242, Bug #73663)




Posted by 김용환 '김용환'



mysql driver 버전 5에서 버전 6로 변경할 때의 이슈이다.



jdbc url에 serverTimezone도 추가한다.

jdbc driver name을 com.mysql.jdbc.Driver에서 com.mysql.cj.jdbc.Driver로 변환한다.




워닝 및 에러 에러

Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.





예)


className=com.mysql.cj.jdbc.Driver

url="jdbc:mysql://mydomain:3306/development?useUnicode=true&autoReconnect=true&useTimezone=true&serverTimezone=UTC&connectTimeout=3000&socketTimeout=3000"

...


Posted by 김용환 '김용환'


spring3는 ClassPathBeanDefinitionScanner을 제공해서 spring container에 bean을 생성할 수 있도록 도와준다.



ClassPathBeanDefinitionScanner scanner = new PlayClassPathBeanDefinitionScanner(applicationContext);

String scanBasePackage = ..

scanner.scan(scanBasePackage.split(","));



예를 들어, play1 playframework의 spring module은 ClassPathBeanDefinitionScanner을 사용한다. 

https://github.com/pepite/Play--framework-Spring-module/blob/master/src/play/modules/spring/SpringPlugin.java

그런데.. play1 playframework의 spinrg module을 spring4로 변경하면 동작이 안된다. 

spring3과 spring4의 차이를 살펴본다.


문제가 되는 부분을 설명한다. 내부에 ConfigurationClassParser 클래스의 asSourceClass 메소드가 변경되었다.
spring3 코드와 달리 spring4에서는 외부에서 읽힐 때 현재 classload에서 읽지 목한다면 ClassNotFoundException이 생긴다.(안전함을 더 중요하게 여기고 수정되었다)

<spring 3>


/**
* Factory method to obtain a {@link SourceClass} from a {@link Class}.
*/
public SourceClass asSourceClass(Class<?> classType) throws IOException {
try {
// Sanity test that we can read annotations, if not fall back to ASM
classType.getAnnotations();
return new SourceClass(classType);
}
catch (Throwable ex) {
// Enforce ASM via class name resolution
return asSourceClass(classType.getName());
}
}




<spring 4>


https://github.com/spring-projects/spring-framework/blob/5b98a54c9b9f8c2f4332734ee23cd483b7df0d22/spring-context/src/main/java/org/springframework/context/annotation/ConfigurationClassParser.java#L659


/**

 * Factory method to obtain a {@link SourceClass} from a class name.
 */
public SourceClass asSourceClass(String className) throws IOException {
if (className.startsWith("java")) {
// Never use ASM for core java types
try {
return new SourceClass(this.resourceLoader.getClassLoader().loadClass(className));
}
catch (ClassNotFoundException ex) {
throw new NestedIOException("Failed to load class [" + className + "]", ex);
}
}
return new SourceClass(this.metadataReaderFactory.getMetadataReader(className));
}


참고

https://jira.spring.io/browse/SPR-15245



Posted by 김용환 '김용환'

spring 3.2.2부터 Scheduled에 String 문자열로 간단한 문자열 형태를 받을 수 있어 크론 잡을 실행할 수 있었다. 



https://github.com/spring-projects/spring-framework/blob/master/spring-context/src/main/java/org/springframework/scheduling/annotation/Scheduled.java


/**

* Execute the annotated method with a fixed period in milliseconds between the

* end of the last invocation and the start of the next.

* @return the delay in milliseconds as a String value, e.g. a placeholder

* @since 3.2.2

*/

String fixedDelayString() default "";



http://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/scheduling/annotation/Scheduled.html#fixedDelayString--



spring 3.2.2부터 fixedDelayString, fixedRateString, initialDelayString 을 사용할 수 있다. 



https://github.com/spring-projects/spring-framework/blob/3.2.x/spring-context/src/main/java/org/springframework/scheduling/annotation/ScheduledAnnotationBeanPostProcessor.java



즉, 아래와 같은 코드를 사용할 수 있다. 

@Scheduled(fixedDelayString="10")


@Scheduled(fixedRate = 10)



annotation 레벨에서 fixedDelayString을 사용하려면 다음 테스트 코드를 사용해 테스트해 볼 수 있다.




private AnnotationConfigApplicationContext ctx;

@Test

public void ScheduleTest() throws InterruptedException {

  ctx = new AnnotationConfigApplicationContext(TestScheduleClass.class);


  Thread.sleep(10);

  assertThat(ctx.getBean(AtomicInteger.class).get(), greaterThanOrEqualTo(1));

}


@Configuration

@EnableScheduling

private static class TestScheduleClass {

  public TestScheduleClass() {}


    @Bean

  public AtomicInteger counter() {

    return new AtomicInteger();

  }


  @Scheduled(fixedDelayString="10")

  public void run() {

    new Integer(10);

    int a = counter().incrementAndGet();

    System.out.println("xxxx : " + new Date() + "," + a);

  }

}




간단한 property 또는 아주 간단한 spel(예, properties 파일에서 읽은 설정) 등이 될 것이다.


private static final String delay = "new Integer(10)";

@Scheduled(fixedDelayString=delay)



@Scheduled(fixedDelayString="${fix.Delay}")


그러나, @Scheduled에 fixedDelayString에 SPEL을 사용하려면 에러가 발생한다.


@Scheduled(fixedDelayString="#{new Integer(10)}")



예외는 다음과 같다.


Caused by: java.lang.IllegalStateException: Encountered invalid @Scheduled method 'run': Invalid fixedDelayString value "#{new Integer(10)}" - cannot parse into integer

at org.springframework.scheduling.annotation.ScheduledAnnotationBeanPostProcessor$1.doWith(ScheduledAnnotationBeanPostProcessor.java:259)

at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:495)

at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:502)

at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:473)

at 






spring 3.x에서는 @Scheduled에 spel을 사용할 수 없다.




따라서 @Schedule에 spel을 사용하려면 Spring 4.3.0을 써야 한다. (SPEL 처리하는 아래 Resolver가 추가되었다..)


https://github.com/spring-projects/spring-framework/blob/master/spring-beans/src/main/java/org/springframework/beans/factory/config/EmbeddedValueResolver.java






@Scheduled(fixedRate = 6000, initialDelayString = #{ T(java.lang.Math).random() * 10 } )




Posted by 김용환 '김용환'




타입에 대한 byte[]를 쉽게 만드는 방법이이다. 



ByteBuffer.allocate(4).putInt(data).array();




또는 apache commons의 SerializationUtils.serialize 메소드를 사용한다.



import org.apache.commons.lang.SerializationUtils;


SerializationUtils.serialize(data);

    public static byte[] serialize(Serializable obj) {

        ByteArrayOutputStream baos = new ByteArrayOutputStream(512);

        serialize(obj, baos);

        return baos.toByteArray();

    }



Posted by 김용환 '김용환'


spring에는 cron 표현식을 사용하지만, 너무 똑같은 시간(0초)에 동작되지 않게 할 수 있다. 



spring 3.x에는 어노테이션에서 spel을 지원하지 않아 XML로 설정해야 한다. 



<beans xmlns="http://www.springframework.org/schema/beans"

       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

       xmlns:task="http://www.springframework.org/schema/task"

       xsi:schemaLocation="

http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd

http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task.xsd">


    <task:executor id="googleExecutor" pool-size="5-30" queue-capacity="10000"/>

    <task:annotation-driven scheduler="googleScheduler" executor="googleExecutor"/>


    <task:scheduler id="googleScheduler" pool-size="30"/>



    <task:scheduled-tasks scheduler="googleScheduler">

        <task:scheduled ref="googleFeedService" method="reloadCache" fixed-delay="#{new Double(T(java.lang.Math).random()*3000).intValue() + 600000}"/>

    </task:scheduled-tasks>


</beans>





반면, spring 4.3에서 어노테이션을 사용하면 다음과 같이 쉽게 사용할 수 있다.


@Scheduled(fixedRate = 600000, initialDelayString = #{ T(java.lang.Math).random() * 3000} )




Posted by 김용환 '김용환'



logback에서 마음에 드는 것 중 하나는 log에 항상 나오는 패턴을 지정할 수 있다는 점(encoder)이다.




<appender name="birthday" class="ch.qos.logback.core.rolling.RollingFileAppender">

<file>${app.home}/logs/birthday.log</file>

<encoder>

<pattern>%d{yyyyMMdd}\t%d{HHmmssSSS}\t%m%n</pattern>

</encoder>

<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">

<fileNamePattern>${app.home}/logs/birthday.log.%d{yyyyMMdd}</fileNamePattern>

<maxHistory>21</maxHistory>

</rollingPolicy>

</appender>

<logger name="birthday_logger" level="INFO" additivity="false">

<appender-ref ref="birthday"/>

</logger>



birthday logger를 사용해서 로그를 저장할 때 탭 단위로 날짜, 시간,  메시지를 저장할 수 있다는 점이 매력적이다. 

Posted by 김용환 '김용환'