[hadoop] Exit Code 154

hadoop 2019. 6. 4. 17:20

 

아래와 같은 클라이언트에서 hadoop에러가 발생했다.

org.apache.http.conn.HttpHostConnectException: Connection to http://google-hadoop-dn4.dakao.io:40060 refused

 

hadoop ui에 다음과 같이 154 에러가 발생했다. 

Application application_1544089295328_663490 failed 1 times due to AM Container for appattempt_1544089295328_663490_000001 exited with exitCode: 154

 

이 이유는 Job 실행 중간에 노드가 내려가면서 발생한 것이었다.

 

 

컨테이너 (프로세스 트리)가 아직 실행 중일 때 RecoveredContainerLaunch(https://hadoop.apache.org/docs/r2.9.2/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/apidocs/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/RecoveredContainerLaunch.html)이 주기적으로 폴링하여 프로세스가 종료되었는지 확인하고 생성된 종료 코드 파일에서 종료 코드를 얻는다.

 

 

컨테이너에서 종료 코드가 없어서 다시 가져 오기를 하려고 하고. 

다시 실패하면 컨테이너는 LOST (Exit 154)로 알린다.

 

 

 

154번에 대한 에러 코드는 다음과 같다. 

 

https://github.com/apache/hadoop/blob/master/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java#L288

  public enum ExitCode {
    FORCE_KILLED(137),
    TERMINATED(143),
    LOST(154);
    private final int code;

    private ExitCode(int exitCode) {
      this.code = exitCode;
    }

    public int getExitCode() {
      return code;
    }

    @Override
    public String toString() {
      return String.valueOf(code);
    }
  }

 

RecoveryContainerLaunch 소스

 

https://zgrepcode.com/hadoop/apache/3.2.0-rc1/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/recoveredcontainerlaunch.java

Posted by '김용환'
,


https://spark.apache.org/releases/spark-release-2-4-3.html
2.4.3이 얼마전에 출시되었는데. 달랑 지라 처리 하나인줄 알았더니. 이제 스칼라 2.12 버전으로 사용할 수 있는 버전이다.

문서 보면 2.11, 2.12 사용 가능하다고 한다. (하도 많이 바뀌어서 아예 지원 안할 줄 알았는데.)

그외 추가된 내용은 kafka key/value 에 null 처리를 반영했다.
https://issues.apache.org/jira/browse/SPARK-27494
https://github.com/apache/spark/pull/24441/files

Posted by '김용환'
,

일래스틱서치에 6.7, 7.x에 CCR 이라는 기능이 생겼다.

클러스터 복제 기능이다. 

https://www.elastic.co/guide/en/elasticsearch/reference/current/ccr-apis.html

 

Cross-cluster replication APIs | Elasticsearch Reference [7.1] | Elastic

 

www.elastic.co

https://www.elastic.co/kr/blog/benchmarking-elasticsearch-cross-cluster-replication

 

Benchmarking Elasticsearch cross-cluster replication

Wondering about the performance of Elasticsearch cross-cluster replication? Here's how we've been benchmarking and stress testing it using Rally.

www.elastic.co

 

클러스간 복제를 진행하려면 그동안 snapshot, restore를 진행했었고 너무 느리고 잘 안되는 부분들이 많았었는데.. 

이제 6.7, 7.0 부터는 쓸만해 질 것 같다.

 

Posted by '김용환'
,

death by amazon 지수와 amazon survivors 라는 지수가 있을 정도로 영향도 높다.

 

https://www.bespokepremium.com/chart-of-the-day/chart-of-the-day-introducing-the-amazon-survivors-index/

 

Chart of the Day: Introducing The "Amazon Survivors" Index | Bespoke Investment Group

Back in February of 2014, we introduced the Bespoke “Death By Amazon” Index (DBA), a portfolio of companies culled from retailer indices that we judged as being extremely exposed to Amazon’s assault on the traditional retailing business model.  The “Death

www.bespokepremium.com

https://www.bespokepremium.com/?s=Death+by+amazon

 

Death by amazon | Bespoke Investment Group

Apr 23, 2019 Our “Death By Amazon” index was created many years ago to provide investors with a list of retailers we view as vulnerable to competition from e-commerce.  In 2016, we also created our “Amazon Survivors” index which is made up of companies tha

www.bespokepremium.com

 

 

회사 목록은 다음과 같다.

 

출처: 

https://books.google.co.kr/books?id=IBOSDwAAQBAJ&pg=PT9&lpg=PT9&dq=%EB%8D%B0%EC%8A%A4+%EB%B0%94%EC%9D%B4+%EC%95%84%EB%A7%88%EC%A1%B4+%EC%BD%94%EC%8A%A4%ED%8A%B8%EC%BD%94+%ED%8E%98%EB%8B%88&source=bl&ots=_FOLkeEvTz&sig=ACfU3U2ql8DWpsWyti266ftkw9cLJd6N8A&hl=ko&sa=X&ved=2ahUKEwipy6mr4s7iAhWHF4gKHTbVBrEQ6AEwB3oECAkQAQ#v=onepage&q=%EB%8D%B0%EC%8A%A4%20%EB%B0%94%EC%9D%B4%20%EC%95%84%EB%A7%88%EC%A1%B4%20%EC%BD%94%EC%8A%A4%ED%8A%B8%EC%BD%94%20%ED%8E%98%EB%8B%88&f=false

 

 

 

출처

https://www.cnbc.com/2017/08/23/these-10-stocks-can-survive-amazons-retail-apocalypse.html

 

Posted by '김용환'
,