Yige

Yige

Build
HomeArchives作品集

HBase Series - Basic Concepts of HBase

#HBase29
AI Translation
This post is translated from Chinese into English through AI.View Original
AI-generated summary
The document provides an overview of HBase, focusing on its basic concepts, data structure, and architecture. ### Key Concepts: - **Row Key**: The primary key used to retrieve records, sorted lexicographically. It can be any string, but integer strings should be zero-padded to maintain natural order. - **Column Family**: Groups of columns defined during table creation, with all columns prefixed by the family name. - **Column Qualifier**: Specific column names within a column family, which can be dynamically created during data insertion. - **Cell**: A storage unit identified by a row key and column, containing a value and timestamp. - **Timestamp**: Each cell can store multiple versions of data, indexed by a 64-bit timestamp. ### Data Storage: - Data is organized into Regions, which are horizontal splits of tables based on row key ranges. Regions are distributed across Region Servers for load balancing. - Each Region Server manages multiple Regions and includes components like Write Ahead Log (WAL), BlockCache, and MemStore for efficient data handling. ### System Architecture: - HBase follows a Master/Slave architecture, with components including Zookeeper (for cluster coordination), Master (for managing Regions), and Region Servers (for handling data operations). - Zookeeper ensures that only one Master is active at a time and monitors the status of Region Servers. ### Design Principles: - Row keys should be designed to avoid hotspots and maintain efficiency. - Row key length should be controlled to optimize storage and performance. Overall, HBase is structured to handle large datasets efficiently with a focus on scalability and performance through its distributed architecture.

HBase Series - Basic Concepts of HBase#

Content organized from:

Basic Concepts#

The logical arrangement of data in HBase:

Row-KeyValue（CF、Qualifier、Version）
1
info{'last name': 'Zhang'，'first name':'San'}
pwd{'password': '111'}
2
Info{'last name': 'Li'，'first name':'Si'}
pwd{'password': '222'}

The physical arrangement of data in HBase:

Row-KeyCFTimestampCell Value
1info123456789Zhang
1info123456789San
2info123456789Li
2info123456789Si

Row Key#

  • Row Key is the primary key used to retrieve records. Accessing data in HBase can only be done through Rowkey, Rowkey range, or full table scan; full table scans have poor performance.

  • Row Key can be any string and is sorted in dictionary order when stored. It is important to note:

    1. Because the dictionary order for Int sorting results in 1,10,100,11,18,19,2,20,21,…,9,91,...,99. If you use integer strings as row keys, to maintain the natural order of integers, row keys must be left-padded with 0.
    2. A read and write operation on a row is atomic (regardless of how many columns are read or written at once).

Column Family#

Each column in an HBase table belongs to a specific column family. Column families are part of the table's schema, so they need to be defined when creating the table. All columns in a column family are prefixed with the column family name, for example, info:ln, info:fn both belong to the info column family.

Column Qualifier#

Can be understood as the specific column name, for example, info:ln, info:fn both belong to the info column family, with their column qualifiers being ln and fn, respectively. It is important to note that column qualifiers are not part of the table schema; you can dynamically create columns during data insertion.

Column#

Columns in HBase consist of a column family and a column qualifier, separated by : (colon), meaning a complete column name should be expressed as column family name : column qualifier.

Cell#

A storage unit determined by row key and column in HBase is called a Cell, which is a combination of row, column family, and column qualifier, and contains a value and a timestamp.

Timestamp#

Each Cell stores multiple versions of the same data. Versions are indexed by timestamps, which are of type 64-bit integer. Timestamps can be automatically assigned by HBase when data is written or explicitly specified by the client. In each Cell, different versions of data are arranged in descending order by timestamp, meaning the latest data is at the front.

Rowkey Design Principles#

Reference: Things to Note When Designing HBase RowKey

Dispersing Rowkey
Rows in HBase are sorted in dictionary order by RowKey. This is very friendly for Scan operations, as rows with similar RowKeys are always stored in close proximity, making sequential reads more efficient than random reads. However, if a large number of read and write operations are always concentrated in a certain RowKey range, it can cause Region hotspots, degrading the performance of the RegionServer. Therefore, it is necessary to appropriately disperse RowKeys.

Controlling RowKey Length

  • In HBase's underlying storage HFile, RowKey is a field in the KeyValue structure. Assuming a RowKey length of 100B, then among 10 million records, just the RowKey would occupy nearly 1G of space, affecting the storage efficiency of HFile.
  • HBase is designed with MemStore and BlockCache, corresponding to write caches at the column family/store level and read caches at the RegionServer level. If RowKey is too long, the density of stored data in the cache will decrease, affecting data persistence or query efficiency.

Ensuring RowKey Uniqueness

Storage Structure#

Regions#

image.png

  • All rows in an HBase Table are arranged in dictionary order by Row Key. HBase Tables are horizontally split into multiple Regions based on the range of row keys. A Region contains all rows between the start key and end key.

  • Each table initially has only one Region. As data continues to increase, the Region will grow larger, and when it reaches a threshold, the Region will split into two new Regions. As the number of rows in the Table continues to increase, there will be more and more Regions.

  • A Region is the smallest unit of distributed storage and load balancing in HBase. This means different Regions can be distributed across different Region Servers. However, a single Region will not be split across multiple servers.

Region Server#

image.png

Region Server runs on the DataNode of HDFS. It has the following components:

  • WAL (Write Ahead Log): Used to store data records that have not yet been persisted, allowing for recovery in case of failure.
  • BlockCache: Read cache. It stores frequently read data in memory, and if storage is insufficient, it will evict excess data based on the Least Recently Used principle.
  • MemStore: Write cache. It stores new data that has not yet been written to disk and sorts it before writing to disk. Each column family on each Region has a MemStore.
  • StoreFile: The underlying StoreFile is HFile, which is Hadoop's binary format file, storing row data in the form of Key\Values in the file system.

When a Region Server accesses a sub-table, it creates a Region object and then creates a Store instance for each column family of the table. Each Store may have 0 or more corresponding StoreFiles, and each StoreFile corresponds to one HFile, which is the actual file stored on HDFS.

image.png

System Architecture#

image.png
The HBase system follows a Master/Slave architecture, consisting of three different types of components:

Zookeeper

  1. Ensures that there is only one Master in the cluster at any time;
  2. Stores the addressing entry for all Regions;
  3. Monitors the status of Region Servers in real-time and notifies the Master of Region Server's online and offline information;
  4. Stores HBase's Schema, including which Tables exist and what Column Families each Table has.

Master

  1. Allocates Regions to Region Servers;
  2. Responsible for load balancing of Region Servers;
  3. Detects failed Region Servers and reallocates their Regions;
  4. Garbage collection of junk files on GFS;
  5. Handles Schema update requests.

Region Server

  1. Region Server is responsible for maintaining the Regions assigned to it by the Master and processing IO requests sent to the Regions;
  2. Region Server is responsible for splitting Regions that become too large during operation.

Component Collaboration#

image.png

HBase uses ZooKeeper as a distributed coordination service to maintain the server status in the cluster. Zookeeper is responsible for maintaining the list of available services and providing services such as failure notifications:

  • Each Region Server creates a temporary node on ZooKeeper, and the Master monitors the node through ZooKeeper's Watcher mechanism, allowing it to detect newly added Region Servers or failed Region Servers;
  • All Masters will competitively create the same temporary node on ZooKeeper. Since ZooKeeper can only have one node with the same name, only one Master can successfully create it, making that Master the primary Master, which will periodically send heartbeats to ZooKeeper. Backup Masters will monitor the node where the primary HMaster is located through the Watcher mechanism;
  • If the primary Master fails to send heartbeats on time, its ZooKeeper session will expire, and the corresponding temporary node will be deleted, triggering the Watcher event defined on that node, notifying the backup Master Servers. Upon receiving the notification, all backup Master Servers will compete to create temporary nodes again, completing the election of the primary Master.
Loading...
Ownership of this post data is guaranteed by blockchain and smart contracts to the creator alone.