[spark] spark-shell에서 특정 라이브러리의 의존성 라이브러리 다운받기(spark cassandra connector 라이브러리 다운받기)
scala 2017. 3. 25. 08:47spark-shell에서 특정 라이브러리를 사용하려면 의존성 라이브러리도 같이 사용해야 한다.
매번 메이븐 저장소에서 찾는 것이 귀찮은데, spark-shell에 packages 플래그를 사용하면 편하게 라이브러리를 다운받을 수 있다.
./spark-shell --packages datastax:spark-cassandra-connector:1.6.0-s_2.11
Ivy Default Cache set to: /Users/samuel.kim/.ivy2/cache
The jars for the packages stored in: /Users/samuel.kim/.ivy2/jars
:: loading settings :: url = jar:file:/usr/local/spark-2.1.0-bin-hadoop2.7/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
datastax#spark-cassandra-connector added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found datastax#spark-cassandra-connector;1.6.0-s_2.11 in spark-packages
found org.apache.cassandra#cassandra-clientutil;3.0.2 in central
found com.datastax.cassandra#cassandra-driver-core;3.0.0 in central
found io.netty#netty-handler;4.0.33.Final in local-m2-cache
found io.netty#netty-buffer;4.0.33.Final in local-m2-cache
found io.netty#netty-common;4.0.33.Final in local-m2-cache
found io.netty#netty-transport;4.0.33.Final in local-m2-cache
found io.netty#netty-codec;4.0.33.Final in local-m2-cache
found io.dropwizard.metrics#metrics-core;3.1.2 in list
found org.slf4j#slf4j-api;1.7.7 in list
found org.apache.commons#commons-lang3;3.3.2 in list
found com.google.guava#guava;16.0.1 in list
found org.joda#joda-convert;1.2 in list
found joda-time#joda-time;2.3 in central
found com.twitter#jsr166e;1.1.0 in central
found org.scala-lang#scala-reflect;2.11.7 in list
[2.11.7] org.scala-lang#scala-reflect;2.11.7
downloading http://dl.bintray.com/spark-packages/maven/datastax/spark-cassandra-connector/1.6.0-s_2.11/spark-cassandra-connector-1.6.0-s_2.11.jar ...
[SUCCESSFUL ] datastax#spark-cassandra-connector;1.6.0-s_2.11!spark-cassandra-connector.jar (3339ms)
:: resolution report :: resolve 4700ms :: artifacts dl 3348ms
:: modules in use:
com.datastax.cassandra#cassandra-driver-core;3.0.0 from central in [default]
com.google.guava#guava;16.0.1 from list in [default]
com.twitter#jsr166e;1.1.0 from central in [default]
datastax#spark-cassandra-connector;1.6.0-s_2.11 from spark-packages in [default]
io.dropwizard.metrics#metrics-core;3.1.2 from list in [default]
io.netty#netty-buffer;4.0.33.Final from local-m2-cache in [default]
io.netty#netty-codec;4.0.33.Final from local-m2-cache in [default]
io.netty#netty-common;4.0.33.Final from local-m2-cache in [default]
io.netty#netty-handler;4.0.33.Final from local-m2-cache in [default]
io.netty#netty-transport;4.0.33.Final from local-m2-cache in [default]
joda-time#joda-time;2.3 from central in [default]
org.apache.cassandra#cassandra-clientutil;3.0.2 from central in [default]
org.apache.commons#commons-lang3;3.3.2 from list in [default]
org.joda#joda-convert;1.2 from list in [default]
org.scala-lang#scala-reflect;2.11.7 from list in [default]
org.slf4j#slf4j-api;1.7.7 from list in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 16 | 1 | 1 | 0 || 16 | 1 |
---------------------------------------------------------------------
만약 proxy 같은 환경 설정을 받는다면 다음과 같이 사용한다
./spark-shell --conf "spark.driver.extraJavaOptions=-Dhttp.proxyHost=<proxyHost> -Dhttp.proxyPort=<proxyPort> -Dhttps.proxyHost=<proxyHost> -Dhttps.proxyPort=<proxyPort>" --packages datastax:spark-cassandra-connector:1.6.0-s_2.11
'scala' 카테고리의 다른 글
[spark] RDD join 예제 (0) | 2017.03.29 |
---|---|
[spark2.0] dataframe의 filter,where,isin,select,contains,col,between,withColumn, 예제 (0) | 2017.03.28 |
[spark] Only one SparkContext may be running in this JVM - StreamingContext (0) | 2017.03.24 |
[spark] rdd의 stats 함수 (0) | 2017.03.24 |
[spark 1.6] hive 접근하기 (0) | 2017.03.22 |