rawdata를 case class로 정의해서 spark Dataset으로 encoding할 때 아래 에러가 발생할 수 있다.



Error:(14, 31) Unable to find encoder for type Flight. An implicit Encoder[Flight] is needed to store Flight instances in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._  Support for serializing other types will be added in future releases.

    val flights = flightsDF.as[Flight]

Error:(14, 31) not enough arguments for method as: (implicit evidence$2: org.apache.spark.sql.Encoder[Flight])org.apache.spark.sql.Dataset[Flight].

Unspecified value parameter evidence$2.

    val flights = flightsDF.as[Flight]



먼저 case class를 main 앞에 위치시키고 import spark.implicits._ 를 main 안에 위치한다.



case class Flight(DEST_COUNTRY_NAME: String, 
           ORIGIN_COUNTRY_NAME: String, count: BigInt)

object FlightMain extends SparkHelper {
def main(args: Array[String]): Unit = {

// 3.2
import sparkSession.implicits._

val flightsDF = sparkSession.read.parquet("origin-source/data/flight-data/parquet/2010-summary.parquet")
val flights = flightsDF.as[Flight]


Posted by '김용환'
,