[spark] Unable to find encoder for type XXX 해결하기

rawdata를 case class로 정의해서 spark Dataset으로 encoding할 때 아래 에러가 발생할 수 있다.

Error:(14, 31) Unable to find encoder for type Flight. An implicit Encoder[Flight] is needed to store Flight instances in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._  Support for serializing other types will be added in future releases.

    val flights = flightsDF.as[Flight]

Error:(14, 31) not enough arguments for method as: (implicit evidence$2: org.apache.spark.sql.Encoder[Flight])org.apache.spark.sql.Dataset[Flight].

Unspecified value parameter evidence$2.

    val flights = flightsDF.as[Flight]

먼저 case class를 main 앞에 위치시키고 import spark.implicits._ 를 main 안에 위치한다.

case class Flight(DEST_COUNTRY_NAME: String, 
           ORIGIN_COUNTRY_NAME: String, count: BigInt)

object FlightMain extends SparkHelper {
  def main(args: Array[String]): Unit = {
    
    // 3.2
    import sparkSession.implicits._

    val flightsDF = sparkSession.read.parquet("origin-source/data/flight-data/parquet/2010-summary.parquet")
    val flights = flightsDF.as[Flight]