'2018/03/29'에 해당되는 글 3건

  1. 2018.03.29 [spark] 로컬 모드 아키텍처 [펌]
  2. 2018.03.29 [spark] 스파크 잡 튜닝 및 아키텍처 URL 펌질
  3. 2018.03.29 git log 예제

[spark] 로컬 모드 아키텍처 [펌]

scala 2018. 3. 29. 11:56


가상 클러스터 모드 (로컬 모드)의 아키텍처



https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-local.html




로컬 모드를 때때로 가상 클러스터(pseudocluster) 실행 모드라고 한다. 또한 해당 모드는 분산되지 않고 단일 JVM 기반 배포 모드로서 스파크는 드라이버 프로그램, 익스큐터, LocalSchedulerBackend, 마스터와 같은 모든 실행 컴포넌트를 단일 JVM에 배포한다. 로컬 모드는 드라이버 자체가 익스큐터로 사용되는 유일한 모드이다. 다음 그림은 스파크 잡 제출과 관련된 로컬 모드의 하이 레벨 아키텍처를 보여준다.








저작자표시

'scala' 카테고리의 다른 글

[spark] spark-shell - Failed to start database 'metastore_db' with class loader 처리하기  (0) 2018.05.28
[spark] EOFException:Cannot seek after EOF/java.lang.NumberFormatException: For input string: "" 해결하기  (0) 2018.05.12
[spark] 스파크 잡 튜닝 및 아키텍처 URL 펌질  (0) 2018.03.29
[spark] 스파크 애플리케이션 튜닝 방법 - 펌  (0) 2018.03.26
[spark] log4j 직렬화하기 - org.apache.spark.SparkException: Task not serializable 해결하기  (0) 2018.03.25
Posted by '김용환'
,

[spark] 스파크 잡 튜닝 및 아키텍처 URL 펌질

scala 2018. 3. 29. 11:20


spark job을 제출할 때 가장 중요한 요소는 메모리와 core 수이다.




관련 설정은 다음과 같은데. 아래 블로그에 잘 설명되어 있다.





http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/



이를 잘 이해하고 설명한 한글 내용은 다음 블로그에 있다.  



http://kysepark.blogspot.kr/2016/04/how-to-tune-your-apache-spark-jobs-part.html





그리고 스파크 내부 아키텍처에 관련 설명은 다음 url을 살펴본다.


https://0x0fff.com/spark-architecture/

https://0x0fff.com/spark-architecture-shuffle/






저작자표시

'scala' 카테고리의 다른 글

[spark] EOFException:Cannot seek after EOF/java.lang.NumberFormatException: For input string: "" 해결하기  (0) 2018.05.12
[spark] 로컬 모드 아키텍처 [펌]  (0) 2018.03.29
[spark] 스파크 애플리케이션 튜닝 방법 - 펌  (0) 2018.03.26
[spark] log4j 직렬화하기 - org.apache.spark.SparkException: Task not serializable 해결하기  (0) 2018.03.25
[spark] 스파크 MLlib으로 비정상 데이터를 찾기에 좋은 참조 자료  (0) 2018.03.14
Posted by '김용환'
,

git log 예제

svn 2018. 3. 29. 11:01


https://github.com/adamfisk/LittleProxy를 clone해서 git log 예를 살펴본다.




git log


$ git log

commit 70edbe1c502d79449fb5ceb9d00cca6a0759d56f

Merge: ac8a466 1792fac

Author: adamfisk <afisk@bravenewsoftware.org>

Date:   Mon Sep 25 06:43:29 2017 -0700


    Merge pull request #388 from shs96c/guava-23


    Bump guava to version 23


commit 1792facf4799347b9ae841bd577917f4b0296fd7

Author: Simon Stewart <simon.m.stewart@gmail.com>

Date:   Thu Aug 17 22:21:02 2017 +0100


    Bump guava to version 23


commit ac8a466f6ca9db64efecbc72972502cfe6ec19d5

Merge: 47bee7b 9630e71

Author: Jason Hoetger <jekh@outlook.com>

Date:   Sun Mar 5 14:34:20 2017 -0800


    Merge pull request #355 from jekh/add-mitm-links-to-readme


    1.1.2 doc update & MITM links


commit 9630e71af4e5708ce87deb3d8b63a6a7b4a9019c

Author: Jason Hoetger <jekh@outlook.com>

Date:   Sun Mar 5 11:31:09 2017 -0800


    Added MITM links to readme


commit 47bee7b5e44b9ec514cd55be919aa3512abb74dc

Author: Jason Hoetger <jekh@outlook.com>

Date:   Sun Mar 5 11:07:19 2017 -0800


    [maven-release-plugin] prepare for next development iteration


commit f1ce0b6428bc86f1137a82ea6f9dae93a393f3c9

Author: Jason Hoetger <jekh@outlook.com>

Date:   Sun Mar 5 11:06:01 2017 -0800


    [maven-release-plugin] prepare release littleproxy-1.1.2


commit 27de86026edf4e1e86c9327a933237aae5a3a0bb

Merge: 070e6c1 31c19ff

Author: Jason Hoetger <jekh@outlook.com>

Date:   Sun Mar 5 10:51:47 2017 -0800


    Merge pull request #352 from jekh/dependency-updates


    Updated guava, dnsjava, slf4j, commons-lang, and test dependencies


commit 070e6c11574ebd1b0c2edb5f87421664d8b8db18

Merge: d1dffa8 08a36bc

Author: Jason Hoetger <jekh@outlook.com>

Date:   Sun Mar 5 10:47:37 2017 -0800


    Merge pull request #351 from jekh/fix-memory-leak-on-initial-request







너무 길어 한줄로..


$ git log --oneline
70edbe1 Merge pull request #388 from shs96c/guava-23
1792fac Bump guava to version 23
ac8a466 Merge pull request #355 from jekh/add-mitm-links-to-readme
9630e71 Added MITM links to readme
47bee7b [maven-release-plugin] prepare for next development iteration
f1ce0b6 [maven-release-plugin] prepare release littleproxy-1.1.2
27de860 Merge pull request #352 from jekh/dependency-updates
070e6c1 Merge pull request #351 from jekh/fix-memory-leak-on-initial-request
d1dffa8 Merge pull request #350 from luleyl/Keep-Alive_fixes
31c19ff Updated guava, dnsjava, slf4j, commons-lang, and test dependencies
08a36bc Fixing memory leak on initial request when HttpObjectAggregator is enabled
f816f3e Fix the repeated TimeOut response.
55fd04d Demonstrate that the TimeOut response is repeated.
43ee962 Merge pull request #349 from jekh/update-netty
a01674b Removed Transfer-Encoding portion of test due to change in netty behavior from netty PR#6270
f4fa693 Updated netty version to avoid false positive leak detection
2371cd7 Merge pull request #346 from jekh/allow-requests-to-origin-server
4ab9829 Added allowRequestsToOriginServer option to Bootstrap to allow handling origin-form requests
dfdb375 Merge pull request #343 from danielkyu/bug-format-accept-encoding-remove-sdch
6e0d253 Support for removing sdch encoding regardless of whether it is stored as a single/multi value
b92ed54 Merge pull request #334 from jekh/fix-invalid-client-request
faba98f Fixed test failure issue
7062538 Fixed bug where the 'Accept-Encoding' header could be improperly formatted after removing the 'sdch' encoding
b0792ae Responding with 400 and closing connection when client sends an invalid request
27dc4f4 Merge pull request #333 from jekh/update-docs
86e75d7 Update readme for 1.1.1 release
904aea3 [maven-release-plugin] prepare for next development iteration
098e16a [maven-release-plugin] prepare release littleproxy-1.1.1
7bf4483 Merge pull request #331 from jekh/remove-ossrh-parent-pom
ee7c934 Merge pull request #328 from jekh/remove-commons-dependencies
0d16280 Merge pull request #330 from jekh/fix-npe-from-invalid-request
3382c6e Replaced deprecated ossrh parent pom with updated nexus-staging-maven-plugin config. Updated plugin versions.
f1a86d4 Returning a Bad Gateway to clients when receiving an indecipherable response from upstream servers
e8af6a5 Added test for invalid server responses
7f5a850 Update tracking of current HTTP request in ProxyToServerConnection to avoid NPEs
2f13621 Removed redundant isHead method
034f80b Made commons-cli an optional dependency, since it is only used with the standalone Launcher
f65a0a7 Replaced commons-io and commons-code dependencies with guava and jdk equivalents
8ce850c Merge pull request #326 from jekh/update-dependencies-and-cleanup
da03f5d Merge pull request #327 from jekh/javadoc-error-fixes
a3d9123 Replaced self-closing <p/> tags with <p> to fix javadoc errors
c2ddbc1 Made remaining immutable variables in DefaultHttpProxyServer final
7acb652 Updated netty dependencies
96f09e4 Merge pull request #299 from jekh/remove-java-6
91bb503 Bumped selenium version to 2.53.1





브랜치 보이도록 decoreate 옵션을 사용한다

$ git log --oneline --decorate
70edbe1 (HEAD -> master, origin/master, origin/HEAD) Merge pull request #388 from shs96c/guava-23
1792fac Bump guava to version 23
ac8a466 Merge pull request #355 from jekh/add-mitm-links-to-readme
9630e71 Added MITM links to readme
47bee7b [maven-release-plugin] prepare for next development iteration
f1ce0b6 (tag: littleproxy-1.1.2) [maven-release-plugin] prepare release littleproxy-1.1.2
27de860 Merge pull request #352 from jekh/dependency-updates
070e6c1 Merge pull request #351 from jekh/fix-memory-leak-on-initial-request
d1dffa8 Merge pull request #350 from luleyl/Keep-Alive_fixes
31c19ff Updated guava, dnsjava, slf4j, commons-lang, and test dependencies
08a36bc Fixing memory leak on initial request when HttpObjectAggregator is enabled
f816f3e Fix the repeated TimeOut response.
55fd04d Demonstrate that the TimeOut response is repeated.
43ee962 Merge pull request #349 from jekh/update-netty
a01674b Removed Transfer-Encoding portion of test due to change in netty behavior from netty PR#6270
f4fa693 Updated netty version to avoid false positive leak detection
2371cd7 Merge pull request #346 from jekh/allow-requests-to-origin-server
4ab9829 Added allowRequestsToOriginServer option to Bootstrap to allow handling origin-form requests
dfdb375 Merge pull request #343 from danielkyu/bug-format-accept-encoding-remove-sdch
6e0d253 Support for removing sdch encoding regardless of whether it is stored as a single/multi value
b92ed54 Merge pull request #334 from jekh/fix-invalid-client-request
faba98f Fixed test failure issue
7062538 Fixed bug where the 'Accept-Encoding' header could be improperly formatted after removing the 'sdch' encoding
b0792ae Responding with 400 and closing connection when client sends an invalid request
27dc4f4 Merge pull request #333 from jekh/update-docs
86e75d7 Update readme for 1.1.1 release
904aea3 [maven-release-plugin] prepare for next development iteration
098e16a (tag: littleproxy-1.1.1) [maven-release-plugin] prepare release littleproxy-1.1.1
7bf4483 Merge pull request #331 from jekh/remove-ossrh-parent-pom





그래프로 보고 싶다면 graph 옵션을 사용한다.

$ git log --oneline --decorate  --graph
*   70edbe1 (HEAD -> master, origin/master, origin/HEAD) Merge pull request #388 from shs96c/guava-23
|\
| * 1792fac Bump guava to version 23
|/
*   ac8a466 Merge pull request #355 from jekh/add-mitm-links-to-readme
|\
| * 9630e71 Added MITM links to readme
|/
* 47bee7b [maven-release-plugin] prepare for next development iteration
* f1ce0b6 (tag: littleproxy-1.1.2) [maven-release-plugin] prepare release littleproxy-1.1.2
*   27de860 Merge pull request #352 from jekh/dependency-updates
|\
| * 31c19ff Updated guava, dnsjava, slf4j, commons-lang, and test dependencies
* |   070e6c1 Merge pull request #351 from jekh/fix-memory-leak-on-initial-request
|\ \
| * | 08a36bc Fixing memory leak on initial request when HttpObjectAggregator is enabled
| |/
* |   d1dffa8 Merge pull request #350 from luleyl/Keep-Alive_fixes
|\ \
| |/
|/|
| * f816f3e Fix the repeated TimeOut response.
| * 55fd04d Demonstrate that the TimeOut response is repeated.
|/
*   43ee962 Merge pull request #349 from jekh/update-netty
|\
| * a01674b Removed Transfer-Encoding portion of test due to change in netty behavior from netty PR#6270
| * f4fa693 Updated netty version to avoid false positive leak detection
|/
*   2371cd7 Merge pull request #346 from jekh/allow-requests-to-origin-server
|\
| * 4ab9829 Added allowRequestsToOriginServer option to Bootstrap to allow handling origin-form requests
|/
*   dfdb375 Merge pull request #343 from danielkyu/bug-format-accept-encoding-remove-sdch
|\
| * 6e0d253 Support for removing sdch encoding regardless of whether it is stored as a single/multi value
| * faba98f Fixed test failure issue
| * 7062538 Fixed bug where the 'Accept-Encoding' header could be improperly formatted after removing the 'sdch' encoding
* |   b92ed54 Merge pull request #334 from jekh/fix-invalid-client-request
|\ \
| |/
|/|
| * b0792ae Responding with 400 and closing connection when client sends an invalid request
|/
*   27dc4f4 Merge pull request #333 from jekh/update-docs
|\
| * 86e75d7 Update readme for 1.1.1 release
|/
* 904aea3 [maven-release-plugin] prepare for next development iteration
* 098e16a (tag: littleproxy-1.1.1) [maven-release-plugin] prepare release littleproxy-1.1.1
*   7bf4483 Merge pull request #331 from jekh/remove-ossrh-parent-pom
|\
| * 3382c6e Replaced deprecated ossrh parent pom with updated nexus-staging-maven-plugin config. Updated plugin versions.
* |   ee7c934 Merge pull request #328 from jekh/remove-commons-dependencies
|\ \
| * | 034f80b Made commons-cli an optional dependency, since it is only used with the standalone Launcher
| * | f65a0a7 Replaced commons-io and commons-code dependencies with guava and jdk equivalents




특정 기간부터 보고 싶다면. since와 until를 사용한다. (before, after도 사용할 수 있다)

$ git log --oneline --decorate --graph --since=10.weeks



$ git log --oneline --decorate --graph --until=1.days --since=5.years




특정 파일보고 싶다면. --를 사용한다.


$ git log -- COPYRIGHT.txt

commit 1627266fc58b4a1c2c3ba1d027389e55f9906a30

Author: afisk <a@littleshoot.org>

Date:   Sat Oct 24 22:45:04 2009 +0000


    Initial implementation of LittleShoot proxy based on Netty core





특정 커미터의 로그를 보고 싶다면 committer를 사용한다.


$ git log --oneline --decorate --graph --committer adamfisk

* 58765c5 Merge pull request #279 from jekh/upgrade-dependencies

* 075ec1d Merge pull request #278 from jekh/update-surefire-settings

* c3a4cec Merge pull request #211 from jekh/fix-ipv6-parsing

* 0d0c35b Merge pull request #191 from jekh/travis-ci-update

* f5c8ea7 Merge pull request #183 from jekh/add-abort-shutdown

* 7409b0b Merge pull request #187 from compcruz/master

* 2bf48bc Merge pull request #170 from jekh/request-sent-callback-after-content-sent

* bf3fd76 Merge pull request #172 from schoeffm/master

* 4c60fa8 Merge pull request #179 from jekh/remove-shutdown-hook-on-stop

* e3acc48 Merge pull request #176 from jekh/update-maven-plugin-versions

* 91db6fb Merge pull request #163 from jekh/allow-port-zer





stat를 사용하면 커밋에 대한 diff 정보를 간략히 볼 수 있다. 


 git log --stat

commit 70edbe1c502d79449fb5ceb9d00cca6a0759d56f

Merge: ac8a466 1792fac

Author: adamfisk <afisk@bravenewsoftware.org>

Date:   Mon Sep 25 06:43:29 2017 -0700


    Merge pull request #388 from shs96c/guava-23


    Bump guava to version 23


commit 1792facf4799347b9ae841bd577917f4b0296fd7

Author: Simon Stewart <simon.m.stewart@gmail.com>

Date:   Thu Aug 17 22:21:02 2017 +0100


    Bump guava to version 23


 pom.xml                                                               | 2 +-

 src/main/java/org/littleshoot/proxy/impl/ProxyToServerConnection.java | 4 ++--

 2 files changed, 3 insertions(+), 3 deletions(-)


commit ac8a466f6ca9db64efecbc72972502cfe6ec19d5

Merge: 47bee7b 9630e71

Author: Jason Hoetger <jekh@outlook.com>

Date:   Sun Mar 5 14:34:20 2017 -0800


    Merge pull request #355 from jekh/add-mitm-links-to-readme


    1.1.2 doc update & MITM links


commit 9630e71af4e5708ce87deb3d8b63a6a7b4a9019c

Author: Jason Hoetger <jekh@outlook.com>

Date:   Sun Mar 5 11:31:09 2017 -0800


    Added MITM links to readme


 README.md | 8 +++++++-

 1 file changed, 7 insertions(+), 1 deletion(-)


commit 47bee7b5e44b9ec514cd55be919aa3512abb74dc

Author: Jason Hoetger <jekh@outlook.com>

Date:   Sun Mar 5 11:07:19 2017 -0800


    [maven-release-plugin] prepare for next development iteration


 pom.xml | 4 ++--

 1 file changed, 2 insertions(+), 2 deletions(-)


commit f1ce0b6428bc86f1137a82ea6f9dae93a393f3c9




 

최신 로그 해시만 보려면 다음과 같이 reflog를 사용한다.


$ git reflog

70edbe1 HEAD@{0}: clone: from https://github.com/adamfisk/LittleProxy.git

$ git log

commit 70edbe1c502d79449fb5ceb9d00cca6a0759d56f

Merge: ac8a466 1792fac

Author: adamfisk <afisk@bravenewsoftware.org>

Date:   Mon Sep 25 06:43:29 2017 -0700


    Merge pull request #388 from shs96c/guava-23


    Bump guava to version 23



저작자표시

'svn' 카테고리의 다른 글

[대용량 파일 다운받기] git annex in gitlab  (0) 2015.02.24
PC에 svn 서버 설치 도움되는 Site  (0) 2010.09.06
Setting property in svn repository  (0) 2009.04.17
php 로깅 관련  (0) 2008.01.18
cvs에서 cvs checkout: failed to create lock directory 에러 문제  (0) 2007.12.06
Posted by '김용환'
,

티스토리툴바