SparkSession을 종료하지 않으면 아래와 같은 에러가 발생할 수 있다. 




Driver stacktrace:

19/01/17 19:42:45 INFO DAGScheduler: Job 1 failed: count at FirstSparkSample.scala:21, took 0.194101 s

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 22, localhost, executor driver): java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V




SparkSession을 stop하는 코드를  추가한다.


sparksession.stop()

Posted by '김용환'
,