MongoDB Sharding

 

For starters, MongoDB is a document-oriented NoSQL database used for high volume data storage. In the traditional relational databases, we use tables and rows. In contrast, MongoDB makes use of collections and documents. Documents consist of key-value pairs which are the basic unit of data in MongoDB.

Without any further delay, we jump into what is Sharding, its requirement, sharding cluster architecture in MongoDB, and a practical example with Docker.

 

Sharding

Sharding is a method for distributing data across multiple machines. MongoDB uses sharding to support deployments with very large data sets and high throughput operations.

Why Sharding?

Database systems with large data sets or high throughput applications can challenge the capacity of a single server. For example, high query rates can exhaust the CPU capacity of the server. Working set sizes larger than the system’s RAM stress the I/O capacity of disk drives.

For addressing system growth, we have 2 methods :

1. Vertical Scaling.
2. Horizontal Scaling.

Vertical Scaling

Vertical scaling is increasing the capacity of a single server by using a more powerful CPU, adding more RAM, and increasing the amount of storage space. But there are practical limits to vertical scaling like the hardware limits and the load a single server can handle.

Horizontal Scaling

Horizontal Scaling involves dividing the system dataset and load over multiple servers, adding additional servers to increase capacity as required. While the overall speed or capacity of a single machine may not be high, each machine handles a subset of the overall workload, potentially providing better efficiency than a single high-speed high-capacity server.

MongoDB supports horizontal scaling through sharding.

Sharding cluster

Before going further, we need to understand the components in the sharding cluster.

1. Shard: Each shard contains a subset of the sharded data. Each shard can be deployed as a replica set to provide redundancy and high availability. Together, the cluster’s shards hold the entire data set for the cluster.
2. Mongos: The mongos acts as a query router, providing an interface between client applications and the sharded cluster.
3. Config Servers: Config servers store metadata and configuration settings for the cluster. They are also deployed as a replica set.

Sharded cluster

Shard Keys

MongoDB uses the shard key to distribute the collection’s documents across shards. The shard key consists of a field or fields that exist in every document in the target collection.

You choose the shard key when sharding a collection. The choice of shard key cannot be changed after sharding. A sharded collection can have only one shard key.

To shard a non-empty collection, the collection must have an index that starts with the shard key. For empty collections, MongoDB creates the index if the collection does not already have an appropriate index for the specified shard key. See Shard Key Indexes.

Note: The choice of shard key affects the performance, efficiency, and scalability of a sharded cluster. A cluster with the best possible hardware and infrastructure can be bottlenecked by the choice of the shard key.

Chunks

MongoDB partitions sharded data into chunks. Each chunk has an inclusive lower and exclusive upper range based on the shard key. MongoDB migrates chunks across the shards in the sharded cluster using the sharded cluster balancer. The balancer attempts to achieve an even balance of chunks across all shards in the cluster.

Balancer and Even Chunk Distribution

In an attempt to achieve an even distribution of chunks across all shards in the cluster, a balancer runs in the background to migrate chunks across the shards

Advantages of Sharding

1. Reads/Writes: MongoDB distributes the read and write workload across the shards in the sharded cluster, allowing each shard to process a subset of cluster operations. Both read and write workloads can be scaled horizontally across the cluster by adding more shards.
2. Storage Capacity: Sharding distributes data across the shards in the cluster, allowing each shard to contain a subset of the total cluster data. As the data set grows, additional shards increase the storage capacity of the cluster.
3. High Availability: A sharded cluster can continue to perform partial read/write operations even if one or more shards are unavailable. While the subset of data on the unavailable shards cannot be accessed during the downtime, reads or writes directed at the available shards can still succeed.

In production environments, individual shards should be deployed as replica sets, providing increased redundancy and availability.

Sharded and Non-Sharded Collections

A database can have a mixture of sharded and unsharded collections. Sharded collections are partitioned and distributed across the shards in the cluster. Unsharded collections are stored on a primary shard. Each database has its own primary shard.

Sharded and Unsharded collections

Connecting to a Sharded Cluster

You must connect to a mongos router to interact with any collection in the sharded cluster. This includes sharded and unsharded collections. Clients should never connect to a single shard in order to perform read or write operations.

You can connect to a mongos the same way you connect to a mongod, such as via the mongo shell or a MongoDB driver.

You can find the docker-compose.yaml and same set up without docker here.

Github link: MongoDB sharding

 

References and additional resources:

1. https://docs.mongodb.com/manual/core/sharding-shard-key/
2. https://www.mongodb.com/lp/white-paper/usage/ops-best-practices
3. https://www.mongodb.com/presentations/webinar-sharding-methods-mongodb?jmp=docs