add jars to hbase server side

[chenbodeng719]

Add jars to hbase server side according to https://github.com/LucaCanali/Miscellaneous/blob/master/Spark_Notes/Spark_HBase_Connector.md. But it won't work for me. I get error as below. Please help me.

java.lang.NoSuchMethodError: org.apache.hadoop.hbase.spark.protobuf.generated.SparkFilterProtos$SQLPredicatePushDownFilter$Builder.addValueFromQueryArray(Lorg/apache/hbase/thirdparty/com/google/protobuf/ByteString;)Lorg/apache/hadoop/hbase/spark/protobuf/generated/SparkFilterProtos$SQLPredicatePushDownFilter$Builder;
	at org.apache.hadoop.hbase.spark.SparkSQLPushDownFilter.toByteArray(SparkSQLPushDownFilter.java:257)
	at org.apache.hadoop.hbase.spark.datasources.SerializedFilter$.$anonfun$toSerializedTypedFilter$1(HBaseTableScanRDD.scala:273)
	at scala.Option.map(Option.scala:230)
	at org.apache.hadoop.hbase.spark.datasources.SerializedFilter$.toSerializedTypedFilter(HBaseTableScanRDD.scala:273)
	at org.apache.hadoop.hbase.spark.datasources.HBaseTableScanRDD.$anonfun$getPartitions$2(HBaseTableScanRDD.scala:85)
	at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
	at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
	at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
	at org.apache.hadoop.hbase.spark.datasources.HBaseTableScanRDD.getPartitions(HBaseTableScanRDD.scala:77)

[LucaCanali]

A few hints for troubleshooting:

• Which version of HBase are you using?
• Did you add the 3 jars listed in the note to $HBASE_HOME/lib ?
• Did you restart the region servers after adding the jars?

[chenbodeng719]

@LucaCanali

• Which version of HBase are you using?
  2.4.8
• Did you add the 2 jars listed in the note to $HBASE_HOME/lib ?
  I'using emr hbase. $HBASE_HOME/lib is /usr/lib/hbase/lib. I put them here.
• Did you restart the region servers after adding the jars?
  I create a new emr hbase. And add them to bootstrap part, it means I put them to lib path before hbase start.

Did I miss something?

[LucaCanali]

On the client side:

• which version of Spark do you use?
• do you run it with --jars $JAR1,$JAR2 --packages org.apache.hbase:hbase-shaded-mapreduce:2.4.9 ?

[chenbodeng719]

On the client side:

• which version of Spark do you use?
• do you run it with --jars $JAR1,$JAR2 --packages org.apache.hbase:hbase-shaded-mapreduce:2.4.9 ?

• which version of Spark do you use?
  3.1.2
• do you run it with --jars $JAR1,$JAR2 --packages org.apache.hbase:hbase-shaded-mapreduce:2.4.9 ?
  I use jupyter + Livy + spark. I put jar to spark class path.

def get_data_from_hbase():
    data_source_format = 'org.apache.hadoop.hbase.spark'
    tname = "candidate"
    tmap = "uid STRING :key, oridata STRING f1:data"
    df = sqlc.read.format(data_source_format) \
        .option('hbase.table',tname) \
        .option('hbase.columns.mapping', tmap) \
        .option('hbase.spark.use.hbasecontext', False) \
        .option("hbase.spark.pushdown.columnfilter", False) \
        .load()
    # print(df.count())
    tlist = ["tiq_ed4f3ab4-d2d4-4108-83c8-408ff3cf5f2b"]
    df = df \
    .filter(df.uid.isin(tlist)) \
    .withColumn("position_title",get_json_object(col("oridata"), "$.basic.current_position.position_title") ) \
    .select(
        "uid",
        "oridata",
    )
                  
    df.show()

If I set hbase.spark.pushdown.columnfilter false, it's ok. If ture, not ok.

[LucaCanali]

Does it work from the spark-shell?

[chenbodeng719]

Does it work from the spark-shell?

Same error

[chenbodeng719]

My error is different with error in md "java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/spark/datasources/JavaBytesEncoder". Maybe it's a server config error?

[LucaCanali]

Can you try using HBase 2.3.x ?

[chenbodeng719]

Can you try using HBase 2.3.x ?

There is no 2.3.x hbase in aws emr release.

[LucaCanali]

Unfortunately I cannot against HBase (server ) 2.4 yet to test this.
I have just compiled the connector jars using Spark 3.3.1 and HBase 2.4.15 and linked the URLs at https://github.com/LucaCanali/Miscellaneous/blob/master/Spark_Notes/Spark_HBase_Connector.md , not sure if it helps though.

[chenbodeng719]

Considering that our query is more complex than range query and the query may not trigger the push down filter, so now we ignore it. Thanks for your patient. We'll test later. @LucaCanali

[amilosevic-grid]

hi @LucaCanali, have you managed to test against HBase 2.4?
I am having this issue pop up:

def catalog = s"""{
|"table":{"namespace":"dev", "name":"amilosevic"},
|"rowkey":"key",
|"columns":{
|"col0":{"cf":"rowkey", "col":"key", "type":"binary"},
|"col1":{"cf":"a", "col":"col1", "type":"string"}
|}
|}""".stripMargin

scala> spark.sqlContext.read.option("catalog",catalog).format("org.apache.hadoop.hbase.spark").load()
java.lang.NullPointerException
at org.apache.hadoop.hbase.spark.HBaseRelation.(DefaultSource.scala:138)
at org.apache.hadoop.hbase.spark.DefaultSource.createRelation(DefaultSource.scala:69)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:354)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:326)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:308)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:308)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:226)
... 47 elided

on a local spark 3.3.0 + hbase 2.4.14 using your connector jars for spark 3.3.1(placed in hbase/lib) and in spark-submit

[amilosevic-grid]

after some inspection, this class seems to be failing the predicate pushdown
https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java

Caused by: java.lang.ClassCastException: com.google.protobuf.LiteralByteString cannot be cast to org.apache.hbase.thirdparty.com.google.protobuf.ByteString
at org.apache.hadoop.hbase.spark.SparkSQLPushDownFilter.parseFrom(SparkSQLPushDownFilter.java:208)
... 12 more