Spark Cassandra 통합 Calliope 라이브러리 사용 - 레코드가 표시되지 않음

Question

tuplejump Calliope-sql을 사용하여 Cassandra를 spark-shell을 통해 연결하려고 시도합니다.

Sprak 버전 1.1.0 :

연결 :

./spark-shell --master spark://PCSS-HDOP04:7077 --jars calliope-sql-assembly-1.1.0-CTP-U2.jar,calliope-sql_2.10-1.1.0-CTP-U2.jar,spark-cassandra-assembly-1.0.0-SNAPSHOT-jar-with-dependencies.jar,stargate-core-0.9.9.jar,calliope-core-assembly-1.1.0-CTP-U2.jar --conf "spark.cassandra.connection.host=10.234.31.231" 

명령 실행 된 :

import com.datastax.spark.connector._ 
import org.apache.spark.SparkContext 
import org.apache.spark.SparkContext._ 
import org.apache.spark.SparkConf 
val conf = new SparkConf(true).set("spark.cassandra.connection.host", "10.234.31.231") 
val sc = new SparkContext("spark://PCSS-HDOP04:7077", "test", conf) 
val sqlContext = new org.apache.spark.sql.CassandraAwareSQLContext(sc) 
import sqlContext.createSchemaRDD 
sqlContext.sql("select * from roadtrips.roadtrip") 

출력 :

,691 363,210

scala> val res = sqlContext.sql("select * from roadtrips.roadtrip") 
15/01/15 14:55:41 INFO CassandraAwareSQLContext$$anon$1: LOOKING UP DB [None] for CF [roadtrips.roadtrip] 
15/01/15 14:55:41 INFO CassandraAwareSQLContext$$anon$1: INTERPRETED AS DB [Some(roadtrips)] for CF [roadtrip] 
ArrayBuffer(id#21, destination_city_name#22, destination_state_abr#23, distance#24, elapsed_time#25, origin_city_name#26, origin_state_abr#27) 
res: org.apache.spark.sql.SchemaRDD = 
SchemaRDD[6] at RDD at SchemaRDD.scala:103 
== Query Plan == 
== Physical Plan == 
CassandraTableScan [id#21,destination_city_name#22,destination_state_abr#23,distance#24,elapsed_time#25,origin_city_name#26,origin_state_abr#27], (CassandraRelation 10.234.31.231, 9042, 9160, roadtrips, roadtrip, org.apache.spark.sql.CassandraAwareSQLContext@54bebc7b, None, None, false, Some(Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml)), [] 

scala> 

카산드라 테이블 : 열 이름이 아닌 기록 만 검색

id | destination_city_name | destination_state_abr | distance | elapsed_time | origin_city_name | origin_state_abr 
----+-----------------------+-----------------------+----------+--------------+------------------+------------------ 
23 |   Los Angeles |     CA |  2475 |   1700 |   New York |    NY 
33 |   Los Angeles |     CA |  2475 |   1444 |   New York |    NY 

명령.

Answer 1

쿼리에서 반환하는 레코드 수가 잠재적으로 많기 때문에 기본적으로 결과가 표시되지 않습니다. RDD의 검색된 레코드 일부를 보려면 first 또는 take 메서드를 사용할 수 있습니다.

val res = sqlContext.sql("select * from roadtrips.roadtrip") 
res.first() 
res.take(3)