spark sql 能够通过thriftserver 访问hive数据,默认spark编译的版本是不支持访问hive,因为hive依赖比较多,因此打的包中不包含hive和thriftserver,因此需要自己下载源码进行编译,将hive,thriftserver打包进去才能够访问,详细配置步骤如下:
1、下载源码
2、下载Maven,并配置
此配置简单,就略过
3、使用maven进行打包:
打包命令:
mvn -Pyarn -Dhadoop.version=2.3.0-cdh5.0.0 -Phive -Phive-thriftserver -DskipTests clean package
上面的hadoop.version可以根据自己的需要设置相应的版本
注:在windows环境下编译时,由于编译需要的内存很大,因此要设置一下maven使用的内存数据,
进入MAVEN_HOME/bin/ 目录,修改mvn.bat文件,定位到下面一行位置,此行已被注释
@REM set MAVEN_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000
然后在下一行增加下面内容,并保存
set MAVEN_OPTS= -Xms1024m -Xmx1024m -XX:MaxPermSize=512m
编译时可能会失败,这是由于网络中断的原因,多试几次就能成功
4、将SPARK_SRC_HOME\assembly\target\scala-2.10\spark-assembly-1.4.0-hadoop2.3.0-cdh5.0.0.jar 文件替换掉SPARK_HOME/lib 目录下的 spark-assembly*.jar文件
SPARK_SRC_HOME:为spark源码路径
SPARK_HOME:spark安装路径
5、cp HIVE_HOME/conf/hive-site.xml SPARK_HOME/conf/
将hive安装目录下的hive-site.xml文件复制到spark的conf目录下
6、进入SPARK_HOME/bin/ 运行 ./spark-sql --master spark://masterIp:7077
7、查询hive表数据
> select * from test2;
15/07/16 14:07:13 INFO ParseDriver: Parsing command: select * from test2
15/07/16 14:07:13 INFO ParseDriver: Parse Completed
15/07/16 14:07:20 INFO BlockManagerInfo: Removed broadcast_0_piece0 on 192.168.209.140:51413 in memory (size: 1671.0 B, free: 267.3 MB)
15/07/16 14:07:20 INFO BlockManagerInfo: Removed broadcast_0_piece0 on 192.168.209.141:35827 in memory (size: 1671.0 B, free: 267.3 MB)
15/07/16 14:07:21 WARN HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
15/07/16 14:07:21 INFO deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
15/07/16 14:07:22 INFO MemoryStore: ensureFreeSpace(377216) called with curMem=0, maxMem=280248975
15/07/16 14:07:22 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 368.4 KB, free 266.9 MB)
15/07/16 14:07:22 INFO MemoryStore: ensureFreeSpace(32203) called with curMem=377216, maxMem=280248975
15/07/16 14:07:22 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 31.4 KB, free 266.9 MB)
15/07/16 14:07:22 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.209.141:35827 (size: 31.4 KB, free: 267.2 MB)
15/07/16 14:07:22 INFO SparkContext: Created broadcast 1 from processCmd at CliDriver.java:423
15/07/16 14:07:23 INFO FileInputFormat: Total input paths to process : 1
15/07/16 14:07:23 INFO SparkContext: Starting job: processCmd at CliDriver.java:423
15/07/16 14:07:23 INFO DAGScheduler: Got job 1 (processCmd at CliDriver.java:423) with 2 output partitions (allowLocal=false)
15/07/16 14:07:23 INFO DAGScheduler: Final stage: ResultStage 1(processCmd at CliDriver.java:423)
15/07/16 14:07:23 INFO DAGScheduler: Parents of final stage: List()
15/07/16 14:07:23 INFO DAGScheduler: Missing parents: List()
15/07/16 14:07:23 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[5] at processCmd at CliDriver.java:423), which has no missing parents
15/07/16 14:07:23 INFO MemoryStore: ensureFreeSpace(7752) called with curMem=409419, maxMem=280248975
15/07/16 14:07:23 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 7.6 KB, free 266.9 MB)
15/07/16 14:07:23 INFO MemoryStore: ensureFreeSpace(4197) called with curMem=417171, maxMem=280248975
15/07/16 14:07:23 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 4.1 KB, free 266.9 MB)
15/07/16 14:07:23 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.209.141:35827 (size: 4.1 KB, free: 267.2 MB)
15/07/16 14:07:23 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:874
15/07/16 14:07:23 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 1 (MapPartitionsRDD[5] at processCmd at CliDriver.java:423)
15/07/16 14:07:23 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks
15/07/16 14:07:24 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, 192.168.209.139, ANY, 1435 bytes)
15/07/16 14:07:24 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 2, 192.168.209.140, ANY, 1435 bytes)
15/07/16 14:07:24 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.209.140:51413 (size: 4.1 KB, free: 267.3 MB)
15/07/16 14:07:24 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.209.140:51413 (size: 31.4 KB, free: 267.2 MB)
15/07/16 14:07:25 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.209.139:44770 (size: 4.1 KB, free: 267.3 MB)
15/07/16 14:07:30 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.209.139:44770 (size: 31.4 KB, free: 267.2 MB)
15/07/16 14:07:34 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 2) in 10767 ms on 192.168.209.140 (1/2)
15/07/16 14:07:39 INFO DAGScheduler: ResultStage 1 (processCmd at CliDriver.java:423) finished in 15.284 s
15/07/16 14:07:39 INFO StatsReportListener: Finished stage: org.apache.spark.scheduler.StageInfo@9aa2a1b
15/07/16 14:07:39 INFO DAGScheduler: Job 1 finished: processCmd at CliDriver.java:423, took 15.414215 s
15/07/16 14:07:39 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 15287 ms on 192.168.209.139 (2/2)
15/07/16 14:07:39 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
15/07/16 14:07:39 INFO StatsReportListener: task runtime:(count: 2, mean: 13027.000000, stdev: 2260.000000, max: 15287.000000, min: 10767.000000)
15/07/16 14:07:39 INFO StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100%
15/07/16 14:07:39 INFO StatsReportListener: 10.8 s 10.8 s 10.8 s 10.8 s 15.3 s 15.3 s 15.3 s 15.3 s 15.3 s
15/07/16 14:07:39 INFO StatsReportListener: task result size:(count: 2, mean: 15663.000000, stdev: 15.000000, max: 15678.000000, min: 15648.000000)
15/07/16 14:07:39 INFO StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100%
15/07/16 14:07:39 INFO StatsReportListener: 15.3 KB 15.3 KB 15.3 KB 15.3 KB 15.3 KB 15.3 KB 15.3 KB 15.3 KB 15.3 KB
15/07/16 14:07:39 INFO StatsReportListener: executor (non-fetch) time pct: (count: 2, mean: 72.195332, stdev: 18.424154, max: 90.619485, min: 53.771178)
15/07/16 14:07:39 INFO StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100%
15/07/16 14:07:39 INFO StatsReportListener: 54 % 54 % 54 % 54 % 91 % 91 % 91 % 91 % 91 %
15/07/16 14:07:39 INFO StatsReportListener: other time pct: (count: 2, mean: 27.804668, stdev: 18.424154, max: 46.228822, min: 9.380515)
15/07/16 14:07:39 INFO StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100%
15/07/16 14:07:39 INFO StatsReportListener: 9 % 9 % 9 % 9 % 46 % 46 % 46 % 46 % 46 %
111
222
333
444
555
666
777
888
999111
222
333
444
555
666
777
888
999111
............
Time taken: 26.523 seconds, Fetched 889 row(s)
15/07/16 14:07:39 INFO CliDriver: Time taken: 26.523 seconds, Fetched 889 row(s)
spark-sql>
由于我是自己笔记本上的虚拟机上运行,同时只有两个work节点,所以运行会慢一点,但不影响功能
看到有的同学会要求先启动thriftserver,经测试可以不需要手动启动,在运行./spark-sql时会自动启动
同时hive-site.xml也不需要修改任何配置
相关推荐
通用load/write方法 手动指定选项 Spark SQL的DataFrame接口支持多种数据源的操作。...修改配置项spark.sql.sources.default,可修改默认数据源格式。 scala> val df = spark.read.load(hdfs://hadoop001:9000/nam
spark下安装hive标准配置文档。Ubuntu安装hive,并配置mysql作为元数据库时候需要的标准hive-site.xml配置文件,可以根据这个文件联系我的博文内容就行修改,避免入坑。实现快捷启动hive。
项目实战:Java一站式解决Hive内用Spark取数,新建ES索引,灌入数据,并且采用ES别名机制,实现ES数据更新的无缝更新,底层采用Spark计算框架,数据较快。
Hive中SQL详解
spark-hive_2.11-2.3.0 spark-hive-thriftserver_2.11-2.3.0.jar log4j-2.15.0.jar slf4j-api-1.7.7.jar slf4j-log4j12-1.7.25.jar curator-client-2.4.0.jar curator-framework-2.4.0.jar curator-recipes-2.4.0....
《Hive数据仓库案例教程》教学课件 第5章 Hive数据操作.pdf《Hive数据仓库案例教程》教学课件 第5章 Hive数据操作.pdf《Hive数据仓库案例教程》教学课件 第5章 Hive数据操作.pdf《Hive数据仓库案例教程》教学课件 第...
CDH6针对hive on spark的调优文档,这个是生产的实战经验
hive-on-spark客户端
使用hive3.1.2和spark3.0.0配置hive on spark的时候,发现官方下载的hive3.1.2和spark3.0.0不兼容,hive3.1.2对应的版本是spark2.3.0,而spark3.0.0对应的hadoop版本是hadoop2.6或hadoop2.7。 所以,如果想要使用高...
这是每个学习spark必备的jar包,是根据我的个人试验后所得,官网正版,在spark官网下载。 资源包里不仅有需要的jar包,并且给不会再官网上下载的新手官方网址,可以自由下载资源
Scala代码积累之spark streaming kafka 数据存入到hive源码实例,Scala代码积累之spark streaming kafka 数据存入到hive源码实例。
Hive.sql
建立Hive和Hbase的映射关系,通过Spark将Hive表中数据导入ClickHouse对应的jar包
补丁文件包,依赖包,hive3.1.2-spark3.0.0和hive3.1.3-spark3.1.3二进制包已经全部放进该压缩
Hive数据查询详解,基础篇
// 设置元数据服务 Delegate.getDelegate().setMetaColumnService(new IMetaColumnService() { @Override public List<String> queryMetaColumn(String dbName, String tableName) { return Collections.empty...
大数据笔记,包含Hadoop、Spark、Flink、Hive、Kafka、Flume、ZK...... 大数据笔记,包含Hadoop、Spark、Flink、Hive、Kafka、Flume、ZK...... 大数据笔记,包含Hadoop、Spark、Flink、Hive、Kafka、Flume、ZK.......
IDEA中使用Spark SQL 连接Hive 折腾了两个小时,好久没有弄了,记录一下 一、复制hive-site.xml 复制hive-site.xml到idea工程中的resource目录下 二、在windows中安装hadoop并配置环境变量 1、解压hadoop 2、修改...
非常全面的hive参数配置,总共有600多项,中文注释是用软件翻译的,勉强能看,引用请注明出处。
基于Spark的大数据分析工具Hive的研究(毕业论文)