scala spark 버전이 일치하지 않습니다.

Question

스칼라에서 단어 개수 프로그램을 실행하는 중에 예외가 발생합니다.

Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps; 
    at org.apache.spark.util.Utils$.getCallSite(Utils.scala:1406) 

해결을위한 Google 검색 스파크와 스칼라간에 불일치가있을 때 이것이 발생하는 것을 이해할 수 있습니다. 내 팸 dependeny 스칼라-SDK-2.11.8 무엇이 잘못되었는지

확실하지가 이리와 다른 버전으로 시도 받는다는 저장소에 시간의 상당한 금액을 지출 한

<?xml version="1.0" encoding="UTF-8"?> 
<project xmlns="http://maven.apache.org/POM/4.0.0" 
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 
    <modelVersion>4.0.0</modelVersion> 

    <groupId>Test</groupId> 
    <artifactId>Test</artifactId> 
    <version>1.0-SNAPSHOT</version> 

    <dependencies> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-core_2.11</artifactId> 
      <version>2.1.0</version> 
     </dependency> 

    </dependencies> 

</project> 

및 프로젝트 설정에서 스칼라 버전입니다 조합. 내 로컬 스파크 설치에서

나는 내가 project.but 행운에 대해 같은 스칼라 SDK를 선택한 명령 scala.util.Properties.versionString에게

을 실행하여 바로 스칼라 버전을 알아 냈어.

귀하의 도움에 크게 감사드립니다.

Answer 1

다음은 spark 응용 프로그램 용 uber jar 파일을 만드는 작업 pom.xml입니다. 그룹 및 이슈 ID를 대체하여 사용하십시오.

mvn clean package을 실행하면 uber (단일) jar 파일이 빌드됩니다.

<?xml version="1.0" encoding="UTF-8"?> 
<project xmlns="http://maven.apache.org/POM/4.0.0" 
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 
    <modelVersion>4.0.0</modelVersion> 

    <groupId>com.aravind.spark</groupId> 
    <artifactId>wordcount</artifactId> 
    <version>1.0.0-SNAPSHOT</version> 
    <name>${project.artifactId} ${project.version}</name> 

    <properties> 
     <maven.compiler.source>1.8</maven.compiler.source> 
     <maven.compiler.target>1.8</maven.compiler.target> 
     <encoding>UTF-8</encoding> 
     <spark.core.version>2.0.2</spark.core.version> 
     <spark.sql.version>2.0.2</spark.sql.version> 
     <scala.tools.version>2.11</scala.tools.version> 
     <scala.version>2.11.8</scala.version> 
    </properties> 

    <dependencies> 
     <dependency> 
      <groupId>org.scala-lang</groupId> 
      <artifactId>scala-library</artifactId> 
      <version>${scala.version}</version> 
      <scope>provided</scope> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-core_2.11</artifactId> 
      <version>${spark.core.version}</version> 
      <scope>provided</scope> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-sql_2.11</artifactId> 
      <version>${spark.sql.version}</version> 
      <scope>provided</scope> 
     </dependency> 
    </dependencies> 

    <build> 
     <sourceDirectory>src/main/scala</sourceDirectory> 
     <testSourceDirectory>src/test/scala</testSourceDirectory> 
     <pluginManagement> 
      <plugins> 
       <plugin> 
        <!-- see http://davidb.github.com/scala-maven-plugin --> 
        <groupId>net.alchim31.maven</groupId> 
        <artifactId>scala-maven-plugin</artifactId> 
        <version>3.2.1</version> 
        <executions> 
         <execution> 
          <goals> 
           <goal>compile</goal> 
           <goal>testCompile</goal> 
          </goals> 
          <configuration> 
           <args> 
            <arg>-dependencyfile</arg> 
            <arg>${project.build.directory}/.scala_dependencies</arg> 
           </args> 
          </configuration> 
         </execution> 
        </executions> 
       </plugin> 
       <plugin> 
        <groupId>org.apache.maven.plugins</groupId> 
        <artifactId>maven-surefire-plugin</artifactId> 
        <version>2.19.1</version> 
        <configuration> 
         <useFile>false</useFile> 
         <disableXmlReport>true</disableXmlReport> 
         <!-- If you have classpath issue like NoDefClassError,... --> 
         <!-- useManifestOnlyJar>false</useManifestOnlyJar --> 
         <includes> 
          <include>**/*Test.*</include> 
          <include>**/*Suite.*</include> 
         </includes> 
        </configuration> 
       </plugin> 
       <plugin> 
        <artifactId>maven-assembly-plugin</artifactId> 
        <version>2.4.1</version> 
        <configuration> 
         <descriptorRefs> 
          <descriptorRef>jar-with-dependencies</descriptorRef> 
         </descriptorRefs> 
        </configuration> 
        <executions> 
         <execution> 
          <id>make-assembly</id> 
          <phase>package</phase> 
          <goals> 
           <goal>single</goal> 
          </goals> 
         </execution> 
        </executions> 
       </plugin> 
       <plugin> 
        <artifactId>maven-compiler-plugin</artifactId> 
        <configuration> 
         <source>${maven.compiler.source}</source> 
         <target>${maven.compiler.target}</target> 
        </configuration> 
       </plugin> 
      </plugins> 
     </pluginManagement> 

     <plugins> 
      <plugin> 
       <groupId>org.apache.maven.plugins</groupId> 
       <artifactId>maven-surefire-plugin</artifactId> 
      </plugin> 
      <plugin> 
       <groupId>net.alchim31.maven</groupId> 
       <artifactId>scala-maven-plugin</artifactId> 
      </plugin> 
      <plugin> 
       <groupId>org.apache.maven.plugins</groupId> 
       <artifactId>maven-shade-plugin</artifactId> 
       <version>2.4.3</version> 
       <executions> 
        <execution> 
         <phase>package</phase> 
         <goals> 
          <goal>shade</goal> 
         </goals> 
         <configuration> 
          <transformers> 
           <!--transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> 
            <mainClass>your main job pipeline class</mainClass> 
           </transformer--> 
          </transformers> 
          <createDependencyReducedPom>false</createDependencyReducedPom> 
          <finalName>${project.artifactId}-${project.version}</finalName> 
          <artifactSet> 
           <excludes> 
            <exclude>org.scala-lang:scala-library</exclude> 
            <exclude>org.apache.spark:spark-core_2.10</exclude> 
            <exclude>log4j:log4j</exclude> 
            <exclude>org.slf4j:slf4j-api</exclude> 
            <exclude>ml-apis:xml-apis</exclude> 
            <exclude>org.apache.httpcomponents:httpclient</exclude> 
            <exclude>org.apache.httpcomponents:httpcore</exclude> 
            <exclude>commons-codec:commons-codec</exclude> 
            <exclude>org.apache.ant:ant</exclude> 
            <exclude>org.apache.ant:ant-junit</exclude> 
            <exclude>org.codehaus.jettison</exclude> 
            <exclude>stax:stax-api</exclude> 
            <exclude>commons-configuration:commons-configuration</exclude> 
            <exclude>commons-digester:commons-digester</exclude> 
            <exclude>commons-beanutils:*</exclude> 
            <exclude>com.google.code.findbugs:jsr305</exclude> 
           </excludes> 
          </artifactSet> 
          <filters> 
           <filter> 
            <artifact>*:*</artifact> 
            <excludes> 
             <exclude>META-INF/*.SF</exclude> 
             <exclude>META-INF/*.DSA</exclude> 
             <exclude>META-INF/*.RSA</exclude> 
            </excludes> 
           </filter> 
          </filters> 
         </configuration> 
        </execution> 
       </executions> 
      </plugin> 
     </plugins> 
    </build> 

</project> 

Answer 2

내 경험은 모두 Microsoft의 것입니다. 방금 Java 기술을 탐구하기 시작했습니다. 질문에 잘못된 점을 묻는다면 질문을 표시 한 사람이 저와 커뮤니티에 도움을 주었을 것입니다. 이유가 설명되어있는 이유는 무엇입니까?

나는 아래의 참조를 추가하여 프로젝트를 만들었습니다. 프로젝트 구조> 모듈> 종속성

screen shot

이 저를 도와 시도 problem.Thanks을 모두 해결했다.