[hotfix][connector-v2-hbase]fix and optimize hbase source problem

[zhangshenghang]

#7149

Purpose of this pull request

• Add Hbase Source parameter to support large data volume reading and optimize performance
  • Support caching parameter settings
  • Support batch parameter settings
  • Support cache_blocks parameter settings
• Delete unused code

How was this patch tested?

Running locally and on servers

Check list

• If any new Jar binary package adding in your PR, please add License Notice according
  New License Guide
• If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
• If you are contributing the connector code, please check that the following files are updated:
  1. Update change log that in connector document. For more details you can refer to connector-v2
  2. Update plugin-mapping.properties and add new connector information in it
  3. Update the pom file of seatunnel-dist
• Update the release-note.

[Hisoka-X]

Why remove static?

[zhangshenghang]

Why remove static?

The previous methods using ROW_KEY were all non-static methods. Using this.ROW_KEY to call private final static String ROW_KEY = "rowkey"; is not a good choice.

Here is how to use ROW_KEY in historical code：

[Hisoka-X]

how about remove this when call ROW_KEY?

[zhangshenghang]

how about remove this when call ROW_KEY?

Thank you for your reminder, You are right, this will be better, I will optimize it

[Hisoka-X]

Why remove this feature?

[zhangshenghang]

Why remove this feature?

This feature was not implemented in previous versions, and query_columns was an invalid configuration. After this modification, the query columns use the schema configuration. It is redundant to set both schema and query_columns at the same time.

The columns configured by the schema are also the columns for querying

schema = {
            columns = [
                  {
                     name = rowkey
                     type = string
                  },
                  {
                     name = "info1:age"
                     type = int
                  },
                  {
                      name = "info1:name"
                      type = string
                  }
]
}

[Hisoka-X]

cc @EricJoy2048 @TyrantLucifer @TaoZex @hailin0 from #6348

[Hisoka-X]

+1 from my side.

[zhilinli123]

hi, can you add the hbase sink document together

[zhangshenghang]

hi, can you add the hbase sink document together

Hi, the HBase sink documentation is available now. What do you want to do specifically?

[zhilinli123]

hbase source & sink The document is changed to v2， Add Chinese document

reference :
https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/sink/Mysql.md
https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/source/Mysql.md

[TaoZex]

Cache is often used for frequently querying hot data. In data synchronization read scenarios, data is usually queried only once. Is it effective to configure cache-related settings in this case? Looking forward to your reply.

[zhangshenghang]

Cache is often used for frequently querying hot data. In data synchronization read scenarios, data is usually queried only once. Is it effective to configure cache-related settings in this case? Looking forward to your reply.

You are right. In the case of synchronous data, it only needs to be queried once, so I set the default value of cache_blocks to false. The default cache_blocks value of Hbase Scan is True. At the same time, I am not sure whether any user will use cache_blocks = true, so I set the cache_blocks parameter to an optional parameter for special cases.

public static final Option<Boolean> HBASE_CACHE_BLOCKS_CONFIG =
Options.key("cache_blocks")
.booleanType()
.defaultValue(false)
.withDescription(
"When it is false, data blocks are not cached. When it is true, data blocks are cached. This value should be set to false when scanning a large amount of data to reduce memory consumption. The default value is false");

[TaoZex]

Cache is often used for frequently querying hot data. In data synchronization read scenarios, data is usually queried only once. Is it effective to configure cache-related settings in this case? Looking forward to your reply.

You are right. In the case of synchronous data, it only needs to be queried once, so I set the default value of cache_blocks to false. The default cache_blocks value of Hbase Scan is True. At the same time, I am not sure whether any user will use cache_blocks = true, so I set the cache_blocks parameter to an optional parameter for special cases.

public static final Option<Boolean> HBASE_CACHE_BLOCKS_CONFIG =
Options.key("cache_blocks")
.booleanType()
.defaultValue(false)
.withDescription(
"When it is false, data blocks are not cached. When it is true, data blocks are cached. This value should be set to false when scanning a large amount of data to reduce memory consumption. The default value is false");

Thank you for your reply. I believe setting the batch size is necessary. Should we keep the cache enabled? I would like to hear your suggestions. @Hisoka-X @hailin0

[Hisoka-X]

Cache is often used for frequently querying hot data. In data synchronization read scenarios, data is usually queried only once. Is it effective to configure cache-related settings in this case? Looking forward to your reply.

You are right. In the case of synchronous data, it only needs to be queried once, so I set the default value of cache_blocks to false. The default cache_blocks value of Hbase Scan is True. At the same time, I am not sure whether any user will use cache_blocks = true, so I set the cache_blocks parameter to an optional parameter for special cases.

public static final Option<Boolean> HBASE_CACHE_BLOCKS_CONFIG =
Options.key("cache_blocks")
.booleanType()
.defaultValue(false)
.withDescription(
"When it is false, data blocks are not cached. When it is true, data blocks are cached. This value should be set to false when scanning a large amount of data to reduce memory consumption. The default value is false");

Thank you for your reply. I believe setting the batch size is necessary. Should we keep the cache enabled? I would like to hear your suggestions. @Hisoka-X @hailin0

OK for me. Give the choice to users.

[TaoZex]

+1

[Hisoka-X]

Please add test case in

seatunnel/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-hbase-e2e/src/test/java/org/apache/seatunnel/e2e/connector/hbase/HbaseIT.java

Line 56 in 2058f5d

public class HbaseIT extends TestSuiteBase implements TestResource {

[zhangshenghang]

Please add test case in

seatunnel/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-hbase-e2e/src/test/java/org/apache/seatunnel/e2e/connector/hbase/HbaseIT.java

Line 56 in 2058f5d

public class HbaseIT extends TestSuiteBase implements TestResource {

Okay, I will add it

[zhangshenghang]

Please add test case in

seatunnel/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-hbase-e2e/src/test/java/org/apache/seatunnel/e2e/connector/hbase/HbaseIT.java

Line 56 in 2058f5d

public class HbaseIT extends TestSuiteBase implements TestResource {

@Hisoka-X add completed , Please confirm if there is any problem

[Hisoka-X]

LGTM. Thanks @zhangshenghang