The capacity-resilience problem

Computing performance is derived from standard CPU, RAM and storage modules – HDDs, SSDs. If more processing power is needed for a computing task then more CPUs are added. If greater storage is required then more storage devices are attached. The main question arises – how are these CPU, RAM and storage modules put together in order to maximize performance and reliability.

Vertical Scaling

The expansion can be done by adding these resources to one system – a process called vertical scaling. In this case all components are tightly interconnected which is good for performance because it enables fast data paths between internal components with relatively simple protocols. However, it is not good for reliability as tight interconnections imply significant mutual reliance between components. Failure of one internal device can easily bring the entire system down. For this reason the bigger the system the more reliable components are required. Besides increased cost, there is the problem that the level of reliability has an upper limit induced by physical limitations and the closer you get to this limit, the higher the price per component gets. In the end this architecture reaches an unacceptable price/performance ratio. The shared components, like system bus and bridges, become bottlenecks for performance. Still, the poor price/performance ratio is compensated for by the simplicity of management – after all, this is just one computer and all business applications are straightforward to deploy. Another benefit is the lack of partitioning between resources. This means that all components are available for use by all other components which allows for maximum resource utilization, whatever the business applications require. Further growth, however, brings the need to remove the bottleneck of common system bus between devices. The answer lies in the breaking of the architecture symmetry by partitioning the system components. An example is the NUMA architecture where each CPU has a preferred memory device which is ‘closer’ to it while the rest of the memory is slower to access. The non-symmetrical approach, however, leaves the resilience problem untouched.

Horizontal Scaling

In order to grow the overall power beyond the limit of a single system while keeping high resilience, partitioning of computing resources to smaller units is required. This allows each of them to fail without bringing down any other. This is achieved by replacing the tight interconnections of system bus and bridges with loose connections that can handle failure of connected devices.

Scaling via One Type of Nodes

The simplest way is to use small computers – nodes, interconnected with a LAN/WAN connection. The benefit is that conventional hardware can be used to increase the total bandwidth of buses between CPU, RAM, IO and storage devices at the cost of resource partitioning. The overall computing task must be partitioned in order to utilize the CPU devices of all nodes. To enable high availability of computing tasks, they must be moved between computers in case of failure, however most business applications are not designed to be moved between computers. The solution to this problem is virtualization of computer hardware on which the business application is installed and moving the entire virtual machine between nodes. However, this approach creates further problems by introducing additional partitioning of computing tasks on virtual machine boundaries. The virtualization itself imposes overhead to both storage and computing devices because the system software is installed and run multiple times on one physical machine. The increased amount of VMs and their overhead demands the appropriate increase in storage. Because the storage devices are split between different computers the processed data needs to be partitioned as well within the internal storage of each computer. This data partitioning has these major problems:

1. The growth of data needs to happen gradually on all nodes’ chunks so that it does not overflow onto one before the others
2. The CPU/storage ratio is determined by all computers and to change it one needs to upgrade all nodes
3. The failure of a node would lose the data on its storage devices, potentially rendering the overall system nonoperational. To avoid that, every piece of data needs to be placed on at least two nodes, changing the CPU/storage device ratio.
4. As such a system is non-standard it requires a complicated proprietary data management system that would be incompatible with most of the existing software – it would be a closed system. This would not allow easy deployment of new software if the need arises.

Obviously, the above problems mean that power and resilience come at the very high cost of developing proprietary data management software and doubling or tripling the amount of hardware in order to avoid data loss in case of any single or double failure of nodes.

Scaling via Two Types of Nodes

Instead of designing and implementing a very complex closed system that solves the above problems, a simple solution is to use two types of systems – computing nodes and storage nodes (disk arrays). All storage devices (HDD, SSD) would be attached to storage nodes only. In this way the ratio between overall CPU and storage devices in the computing cluster can be precisely controlled by adding the appropriate type of nodes as demand arises. Because the storage nodes are freed from computation their internal architecture can be optimized for the specific storage requirements – for example, using battery-backed RAM for write-back caching of RAID 1, 5, 6 and 10 dramatically boosts performance in some cases. A storage node can be designed for a certain amount of storage devices and their respective throughput of data through RAM to link can be guaranteed. Provided that any computing node can access any storage node, the reliability requirements to computing nodes become more relaxed as their failure does not make data inaccessible. In turn they can be optimized for the business applications they would run. A link that is optimized for mass storage can be used to connect every storage node to every computing node and present it as local SCSI device (SAN). The storage nodes can be configured as one large shared, fault-tolerant volume so failure of a storage node does not lose data. In this way the storage is aggregated again via the storage network. And the most important benefit is that any application can run on the shared storage just as it would on local storage. Data availability is guaranteed if at least one computing node and a sufficient amount of storage nodes (depending on RAID configuration) are available. The only missing link is a shared volume manager that can create shared volumes out of shared storage and a shared file system that can mount safely shared volumes. Melio provides both services. Its architecture is designed so that it does not create artificial partitioning or unavailability. The main principle is that if the hardware allows operation the software should provide it. For example, if two computing nodes A and B are accessing a storage node S, the failure of B should not cause A to fail operations. Melio FS fully guarantees this principle.

Scaling Method Comparison

Storage nodes aggregate HDD and SSD drives so they allow for much higher transfer rates than when storage is scattered among many nodes. The latter case has the benefit of being able to replicate data with high demand to many nodes to increase read rates.

Application-level Clustering

Despite the fact that storage is not partitioned in SAN the computing resources still are. The straightforward approach is again virtualization of hardware on which to install conventional applications but this brings the problems described above. Melio addresses the problems of clustering in a more efficient way than virtualization. It provides a framework (AppCluster) that controls conventional business applications so that they do not move between servers but attach and detach data files from the shared storage, making sure their respective services are always available. In this way the overhead of VMs is spared and the freed computing and storage power can be utilized by business tasks.

SAN vs. NAS

NAS has been used traditionally to aggregate and share storage. Its major benefit is that it is simple to manage the storage devices because they are all concentrated in one big computer. All RAID configurations are possible, all devices can be aggregated in one big volume to avoid partitioning, backup and security are centralized. However the NAS approach to storage is in essence vertical scaling applied to storage only and as such it suffers from all the problems described before. Its margin of scaling is limited and the NAS server presents a single point of failure. SAN on the other hand is horizontal scaling applied to storage and computing nodes. Data partitioning is removed with the use of shared volume manager and shared file system. Every FS request in NAS is performed by two file systems – at the NAS client and at the NAS server. This overhead of using two file systems instead of one is significant and requires the NAS server machine to be sufficiently powerful and dedicated to file serving only. The same applies for its fail-over partner(s), wasting at least two systems that could perform business tasks otherwise. Melio FS is designed so that each server processes its own FS requests and interacts only with the previous owners of the locks it acquires. This approach guarantees that one node is not consuming resources of another, potentially creating bottleneck in it. In this way the computing nodes can grow horizontally without the need to grow the NAS server vertically at the same time.

Metadata controller vs. Fully Symmetrical FS

The fastest way to implement shared FS is to start with a NAS server that would perform all metadata operations and would grant leased locks to the other nodes that allow them direct access to user data. In this way it can keep all the lock state consistent and perform complex operations. This approach, despite being simple, leaves the vertical scaling for metadata with all its problems and adds to NAS the horizontal scaling for user data. To avoid a single point of failure the metadata role can be moved to backup server. However this process loses the state of all open files, because the NAS connection operates on the file level and the server knows the state of every file opened by clients. This surely leads to interruption of all clients operation just as it would in the case of a plain NAS server, no matter how fast the fail-over process is. All operations would fail and all data cached at the clients would be lost by the unexpected closure of files. Melio FS uses distributed locking protocol between servers that guarantees that the owner of a specific lock is its server as well. This means that whichever machines fail, the remaining machines are unaffected because there is no interruption of their ownership of shared resources. Furthermore they would experience just a delay in their operation only if they request locks that were owned by a failed machine.

Persisting lock state NAS vs. Fully Symmetrical FS

Using NAS server cluster on top of a SAN with metadata controller allows hiding the high availability deficiencies of metadata controller approach within the NAS implementation. To avoid NAS client interruption during metadata node fail-over the following is required:

1. The state of all open files within clients must be persisted on shared storage in order to be able to recover it on the new metadata controller.
2. No user data write caching is allowed on server in order to avoid data loss when it fails.
3. To support caching of metadata on server, clients must replay metadata operations to the new metadata controller node on fail over.

Point one requires that every file open or lock operation be written to the shared storage prior completing the client’s request, inserting the time for at least one storage write operation in their latency. Point three requires that clients consume memory and CPU power to keep the changes that the metadata server has not flushed to storage yet. Therefore all three points have negative performance impact. Points one and three slow the failover process by requiring that the new metadata server restores the state of all clients’ open files and held locks from the shared storage and then replaying all metadata operations from all clients. This means that on failover the new metadata controller has to perform all write work, cached by its predecessor. This creates two contradictions:

1. Between cache size and fail-over time: the more metadata changes are cached the better write performance and poorer fail-over speed.
2. Between number of clients and fail-over times: the more clients are served – the greater amount of handles, locks and work on fail over and hence greater delay in restoring operation for all of them.

In a fully symmetrical FS the process of server failure is most natural. No owned locks are lost to recover and no handles are closed, no operations are repeated. User data and Meta data can be cached as much as RAM quantity allows for each individual server. The FS is much simpler because the main difference from non-shared FS is the shared lock protocol, all other operations being essentially the same. It supports lock caching and lock-ahead behaviors to minimize communication between nodes. The fail-over process in Melio requires nothing. The failed node is awaited for a defined time and then is declared dead. Even the recovery of its locks does not happen immediately but is postponed until they are actually required. The simplicity of the shared file system approach is far greater than a mix of technologies that are not designed from ground up for the particular purpose but are patched with one another along the way.

Soft SAN

To deploy SAN architecture onto existing set of hardware, Melio provides the ability to represent existing NAS servers as computing and storage nodes merged together. This enables the creation of a shared volume out of the internal storage already present in existing servers, while using them as computing nodes. As the need arises for more storage nodes, computing nodes or storage-and-computing nodes can be added, allowing risk-free gradual  scaling of performance.

Conclusion

Today there are many solutions for performance scalability and each of them has made different compromises based on different priorities. Common one is the desire to reuse existing technologies in order to keep development costs low. The main principle behind development of Melio, however, was to fully utilize the SAN potential in the simplest and most efficient way of aggregating its storage into one shared volume and presenting it to servers via shared file system. By designing and implementing it from the ground up we were able to make it as simple as possible without sacrificing capabilities of hardware.