Yige

Yige

Build
HomeArchives作品集

HBase Series - HBase Read and Write Process

#HBase44
AI Translation
This post is translated from Chinese into English through AI.View Original
AI-generated summary
The document outlines the read and write processes in HBase, detailing both client-side and server-side operations. ### Write Process 1. **Client-Side**: - When a user submits a put request, HBase checks the `autoflush` setting. If `false`, requests are buffered locally until a threshold is reached to improve performance, though this risks data loss if the client crashes. - HBase locates the appropriate region server using the row key and groups batch requests for efficiency. - A remote RPC request is constructed and executed, concluding the client-side operation. 2. **Server-Side**: - The region server deserializes the request and performs checks (e.g., read-only status, memstore size). - It acquires row and region locks to ensure atomic updates. - A write transaction begins, utilizing MVCC for non-locking reads. - Data is written to a memstore (cache) before being logged in HLog for reliability. - After releasing locks, the HLog is synced to HDFS, and if successful, the transaction is completed, making updates visible to other requests. - When the memstore reaches a certain size, it flushes data to disk, creating storefiles that may later undergo compaction. ### WAL Persistence Levels - Various levels of WAL persistence are discussed, ranging from `SKIP_WAL` (no logging) to `FSYNC_WAL` (strict logging with forced disk write). ### Read Process - The client retrieves data by first accessing the META table to find the region server for the desired row key. - Subsequent reads utilize cached information unless the region moves, requiring a refresh from the META table. ### Query Methods - Full table scans, single row queries, and range scans are supported, with methods to set cache and batch sizes for server responses.

HBase Series - HBase Read and Write Process#

Content organized from:

Data Writing Process#

Client-Side Writing Process#

  1. After the user submits a put request, the HBase client will determine whether to directly submit it to the server for processing based on the setting autoflush=true/false (default is true). If it is false, the request will be added to the local buffer, and will only be submitted after exceeding a certain threshold (default is 2M, configurable via configuration files). This can improve write performance, but there is a risk of request loss due to client crashes.

  2. Before submission, HBase will find the corresponding region server based on the rowkey in the meta metadata table. This locating process is obtained through the HConnection's locateRegion method. If it is a batch request, these rowkeys will also be grouped according to HRegionLocation, with each group corresponding to one RPC request.

  3. HBase will construct a remote RPC request MultiServerCallable<Row> for each HRegionLocation, and then execute the call through rpcCallerFactory.<MultiResponse> newCaller(), ignoring failures and error handling for resubmission. The client's submission operation ends here.

Server-Side Writing Process#

After the server-side Region Server receives the client's write request, it will first deserialize it into a Put object, then check operations such as whether the region is read-only, whether the memstore size exceeds blockingMemstoreSize, and then perform the following core operations:
image.png

  1. Acquire row lock, Region update shared lock: HBase uses row locks to ensure that updates to the same row of data are mutually exclusive operations, ensuring the atomicity of updates—either the update succeeds or it fails.

  2. Start write transaction: Obtain write number for implementing MVCC, enabling non-locking reads of data while improving read performance under the premise of ensuring read-write consistency.

  3. Write to memstore cache: Each column family in HBase corresponds to a store for storing that column's data. Each store has a write cache memstore for caching written data. HBase does not directly write data to disk but first writes it to the cache, and only writes to disk when the cache reaches a certain size.

  4. Append HLog: HBase uses the WAL mechanism to ensure data reliability, meaning it first writes to the log and then to the cache. Even in the event of a crash, the original data can be restored by recovering the HLog. This step constructs the data into a WALEdit object and sequentially writes it into the HLog, without needing to execute a sync operation. Version 0.98 adopted a new write thread model to implement HLog logging, greatly enhancing overall data update performance.

  5. Release row lock and shared lock

  6. Sync HLog: HLog is truly synced to HDFS. Executing the sync operation after releasing the row lock is to minimize lock holding time and improve write performance. If the sync fails, a rollback operation will remove the data already written in memstore.

  7. End write transaction: At this point, the update operation of this thread will be visible to other read requests, and the update will take effect.

  8. Flush memstore: When the write cache reaches 64M, a flush thread will be activated to refresh the data to the hard disk. When the memstore data in HBase is flushed to disk, a storefile is created. When the number of storefiles reaches a certain level, a compaction operation is needed on the storefile files. The purpose of Compact: merge files, clear expired and redundant version data, improve read and write efficiency.

WAL Persistence Levels#

  • SKIP_WAL: Only write to cache, not to HLog. This provides good performance but risks data loss; not recommended.
  • ASYNC_WAL: Asynchronously write data to HLog.
  • SYNC_WAL: Synchronously write data to the log file. It should be noted that data is only written to the file system and not actually persisted to disk.
  • FSYNC_WAL: Synchronously write data to the log file and force it to disk. This is the strictest log writing level, ensuring data is not lost, but performance is relatively poor.
  • USER_DEFAULT: By default, if the user does not specify a persistence level, HBase uses the SYNC_WAL level to persist data.

Users can set the WAL persistence level through the client, code: put.setDurability(Durability.SYNC_WAL);

Reading Process#

The process for the client to read and write data on HBase for the first time:

  1. The client obtains the Region Server where the META table is located from Zookeeper;
  2. The client accesses the Region Server where the META table is located, queries the Region Server for the access row key from the META table, and then caches this information along with the location of the META table;
  3. The client retrieves data from the Region Server where the row key is located.

If reading again, the client will obtain the Region Server for the row key from the cache. This way, the client does not need to query the META table again unless the region moves, causing the cache to become invalid. In that case, it will re-query and update the cache.

Note: The META table is a special table in HBase that stores the location information of all regions, while the location information of the META table itself is stored in ZooKeeper.

-w816

For a more detailed reading data process, refer to:

HBase Principles - Data Reading Process Analysis

HBase Principles - Late 'Data Reading Process Partial Details

HBase Query Methods#

  • Full table query: scan tableName

  • Single row query based on rowkey: get tableName, '1'

  • Range scan based on rowkey: scan tableName, {STARTROW=>'1', STOPROW=>'2'}

setCache() and setBatch() methods
Cache sets the number of rows returned by the server at once, while Batch sets the number of columns returned by the server at once.

Loading...
Ownership of this post data is guaranteed by blockchain and smart contracts to the creator alone.