Sparkcandandra 커넥터

Question

나는 Spark 1.6과 Cassandra connector 1.4.3을 사용하여 spark에서 cassandra에 데이터를 쓰고 있습니다.

이제, 우리는 Spark 2.1.0로 업그레이드하고 난 cassandra connector to 2.0.0-M3를 업그레이드하려고하지만 아래와 같은 오류를 반환 : -

java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror; at com.datastax.spark.connector.util.JavaApiHelper$.mirror(JavaApiHelper.scala:25) at com.datastax.spark.connector.util.JavaApiHelper$.getTypeTag(JavaApiHelper.scala:29) at com.datastax.spark.connector.util.JavaApiHelper.getTypeTag(JavaApiHelper.scala) at com.datastax.spark.connector.japi.CassandraJavaUtil.typeTag(CassandraJavaUtil.java:159) at com.datastax.spark.connector.japi.CassandraJavaUtil.mapToRow(CassandraJavaUtil.java:1172) at com.datastax.spark.connector.japi.CassandraJavaUtil.mapToRow(CassandraJavaUtil.java:1205) at com.datastax.spark.connector.japi.CassandraJavaUtil.mapToRow(CassandraJavaUtil.java:1222) at com.cisco.cmcdp.EBdoUtils.upsertToJobAuditTable(EBdoUtils.java:370) at com.cisco.cmcdp.EBdoContactPublisher.main(EBdoContactPublisher.java:245) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:637)

의 pom.xml :

이

<!-- Spark and Spark SQL, Hive Dependencies --> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-core_2.11</artifactId> 
     <version>2.1.0-mapr-1703</version> 
        <exclusions> 
      <exclusion> 
       <groupId>com.google.guava</groupId> 
       <artifactId>guava</artifactId> 
      </exclusion> 
     </exclusions> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-sql_2.11</artifactId> 
     <version>2.1.0-mapr-1703</version> 
        <exclusions> 
      <exclusion> 
       <groupId>com.google.guava</groupId> 
       <artifactId>guava</artifactId> 
      </exclusion> 
     </exclusions> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-hive_2.11</artifactId> 
     <version>2.1.0-mapr-1703</version> 
        <exclusions> 
      <exclusion> 
       <groupId>com.google.guava</groupId> 
       <artifactId>guava</artifactId> 
      </exclusion> 
     </exclusions> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-streaming_2.11</artifactId> 
     <version>2.1.0-mapr-1703</version> 
        <exclusions> 
      <exclusion> 
       <groupId>com.google.guava</groupId> 
       <artifactId>guava</artifactId> 
      </exclusion> 
     </exclusions> 
    </dependency> 


    <!--Spark Cassandra Connector --> 
    <dependency> 
     <groupId>com.datastax.cassandra</groupId> 
     <artifactId>cassandra-driver-core</artifactId> 
     <version>3.0.0</version> 
     <exclusions> 
      <exclusion> 
       <groupId>org.xerial.snappy</groupId> 
       <artifactId>snappy-java</artifactId> 
      </exclusion> 
      <exclusion> 
       <groupId>com.google.guava</groupId> 
       <artifactId>guava</artifactId> 
      </exclusion> 
     </exclusions> 
    </dependency> 

    <dependency> 
     <groupId>com.datastax.cassandra</groupId> 
     <artifactId>cassandra-driver-mapping</artifactId> 
     <version>3.0.0</version> 
     <exclusions> 
      <exclusion> 
       <groupId>com.google.guava</groupId> 
       <artifactId>guava</artifactId> 
      </exclusion> 
     </exclusions> 
    </dependency> 

    <dependency> 
     <groupId>org.apache.cassandra</groupId> 
     <artifactId>cassandra-clientutil</artifactId> 
     <version>2.0.0</version> 

     <exclusions> 
      <exclusion> 
       <groupId>org.xerial.snappy</groupId> 
       <artifactId>snappy-java</artifactId> 
      </exclusion> 
      <exclusion> 
       <groupId>com.google.guava</groupId> 
       <artifactId>guava</artifactId> 
      </exclusion> 
     </exclusions> 
    </dependency> 

    <dependency> 
     <groupId>com.datastax.spark</groupId> 
     <artifactId>spark-cassandra-connector_2.10</artifactId> 
     <version>2.0.0-M3</version> 
      <exclusions> 
      <exclusion> 
       <groupId>com.google.guava</groupId> 
       <artifactId>guava</artifactId> 
      </exclusion> 
     </exclusions> 
    </dependency> 

    <dependency> 
     <groupId>com.datastax.spark</groupId> 
     <artifactId>spark-cassandra-connector_2.10</artifactId> 
     <version>2.0.0-M3</version> 
     <exclusions> 
      <exclusion> 
       <groupId>com.google.guava</groupId> 
       <artifactId>guava</artifactId> 
      </exclusion> 
     </exclusions> 
    </dependency> 

적절한 제안하십시오 Cassandra 커넥터의 방식과 버전.

감사합니다, Rishabh 인스

Answer 1

이 문제는 대부분 부적절 스칼라 버전의 사용에 의한 것입니다, 당신은 스칼라 2.10 및 스칼라 클래스 경로에서 2.11 라이브러리의 조합이있을 수 있습니다. Spark 2.x, Spark Cassandra Connector 2.x 및 Scala 2.11을 사용하고 있는지 확인하십시오 (Scala 2.10이 지원되지만 Scala 2.11을 사용하는 것이 더 좋습니다). 다음 작업 구성을 살펴보십시오. -

<dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-core_2.11</artifactId> 
     <version>2.1.0</version> 
     <exclusions> 
      <exclusion> 
       <groupId>commons-codec</groupId> 
       <artifactId>commons-codec</artifactId> 
      </exclusion> 
     </exclusions> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-sql_2.11</artifactId> 
     <version>2.1.1</version> 
     <exclusions> 
      <exclusion> 
       <groupId>commons-codec</groupId> 
       <artifactId>commons-codec</artifactId> 
      </exclusion> 
     </exclusions> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-streaming_2.11</artifactId> 
     <version>2.1.1</version> 
    </dependency> 
    <dependency> 
     <groupId>com.datastax.spark</groupId> 
     <artifactId>spark-cassandra-connector_2.11</artifactId> 
     <version>2.0.1</version> 
    </dependency> 
    <dependency> 
     <groupId>commons-codec</groupId> 
     <artifactId>commons-codec</artifactId> 
     <version>1.10</version> 
    </dependency> 

도움이되기를 바랍니다.