EMR Spark에서 JDBC로드가 처음 실패한 다음 작동합니다.

Question

AWS Elastic Map Reduce 5.3.1에서 Spark 2.1.0과 함께 spark-shell을 사용하여 Postgres 데이터베이스에서 데이터를로드합니다. loader.load은 항상 실패한 다음 성공합니다. 왜 이런 일이 일어 났을까요?

[hadoop@[SNIP] ~]$ SPARK_PRINT_LAUNCH_COMMAND=1 spark-shell --driver-class-path ~/postgresql-42.0.0.jar 
Spark Command: /etc/alternatives/jre/bin/java -cp /home/hadoop/postgresql-42.0.0.jar:/usr/lib/spark/conf/:/usr/lib/spark/jars/*:/etc/hadoop/conf/ -Dscala.usejavacp=true -Xmx640M -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError=kill -9 %p org.apache.spark.deploy.SparkSubmit --conf spark.driver.extraClassPath=/home/hadoop/postgresql-42.0.0.jar --class org.apache.spark.repl.Main --name Spark shell spark-shell 
======================================== 
Setting default log level to "WARN". 
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 
17/02/28 17:17:52 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. 
17/02/28 17:18:56 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException 
Spark context Web UI available at http://[SNIP] 
Spark context available as 'sc' (master = yarn, app id = application_1487878172787_0014). 
Spark session available as 'spark'. 
Welcome to 
     ____    __ 
    /__/__ ___ _____/ /__ 
    _\ \/ _ \/ _ `/ __/ '_/ 
    /___/ .__/\_,_/_/ /_/\_\ version 2.1.0 
     /_/ 

Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_121) 
Type in expressions to have them evaluated. 
Type :help for more information. 

scala> val loader = spark.read.format("jdbc") // connection options removed 
loader: org.apache.spark.sql.DataFrameReader = org.apache.spark.sql.DataFrameReader@46067a74 

scala> loader.load 
java.sql.SQLException: No suitable driver 
    at java.sql.DriverManager.getDriver(DriverManager.java:315) 
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions$$anonfun$7.apply(JDBCOptions.scala:84) 
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions$$anonfun$7.apply(JDBCOptions.scala:84) 
    at scala.Option.getOrElse(Option.scala:121) 
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:83) 
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:34) 
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:32) 
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:330) 
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:152) 
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:125) 
    ... 48 elided 

scala> loader.load 
res1: org.apache.spark.sql.DataFrame = [id: int, fsid: string ... 4 more fields] 

Answer 1

동일한 문제가 발생합니다. JDBC를 사용하여 Spark를 통해 vertica에 연결하려고합니다. 내가 사용하고 있습니다 : 스파크 쉘에게 스파크 버전입니다 2.2.0 자바 버전을 1.8

외부 항아리를 연결 : Vertica의-8.1.1_spark2.1_scala2.11-20170623.jar Vertica의-JDBC - 8.1. 1-0.jar

코드 연결 : 난 같은 명령을 두 번 실행하면

import java.sql.DriverManager 
import com.vertica.jdbc.Driver 


val jdbcUsername = "<username>" 
val jdbcPassword = "<password>" 
val jdbcHostname = "<vertica server>" 
val jdbcPort = <vertica port> 
val jdbcDatabase ="<vertica DB>" 
val jdbcUrl = s"jdbc:vertica://${jdbcHostname}:${jdbcPort}/${jdbcDatabase}?user=${jdbcUsername}&password=${jdbcPassword}" 

val connectionProperties = new Properties() 
connectionProperties.put("user", jdbcUsername) 
connectionProperties.put("password", jdbcPassword) 

val connection = DriverManager.getConnection(jdbcUrl, connectionProperties) 
java.sql.SQLException: No suitable driver found for jdbc:vertica://${jdbcHostname}:${jdbcPort}/${jdbcDatabase}?user=${jdbcUsername}&password=${jdbcPassword} 

    at java.sql.DriverManager.getConnection(Unknown Source) 
    at java.sql.DriverManager.getConnection(Unknown Source) 
    ... 56 elided 

이, 나는 다음과 같은 출력과 연결이 설정 얻을

scala> val connection = DriverManager.getConnection(jdbcUrl, connectionProperties) 
connection: java.sql.Connection = com.vertica.jdbc.VerticaJdbc4ConnectionImpl@7d994c 

Answer 2

오늘 PySpark와 sqlserver jdbc 드라이버로이 문제가 발생했습니다. 처음에는 간단한 해결 방법을 만들었습니다 - Py4JJavaException을 잡아 두 번 다시 시도 할 수있는 곳에서 다시 시도하십시오.

트릭은 DataStreamReader.jdbc 메소드에서 드라이버 클래스를 지정하는 것입니다.

사용 pyspark :

spark.read.jdbc(..., properties={'driver': 'com.microsoft.sqlserver.jdbc.SQLServerDriver'}) 

그런 다음 필요한 모든는 스칼라와 Raje의 예 @를 사용

spark-submit --jars s3://somebucket/sqljdbc42.jar script.py 

입니다