• sysstat
• munin
• strace
• memcached
• nginx
• keepalived
• HAProxy
• etc.

The document is divided in three distinct sections. The first two chapters explain how to configure monitoring and profiling tools. The next section presents six different server architectures and explains their advantages and shortcomings. Finally, the third section provides hands on examples on how to install the required software for all server architectures given in the previous section. How to configure the software and manipulate your Linux system through scripts and commands is also explained.

All the examples are written for the Debian Linux operating system (Lenny v5.0 version). If you use a different Linux OS, you will need to adapt the commands and the scripts accordingly. Therefore, intermediate skills in system administration are required if you intend to use a different OS than Debian to implement the recipes given in this eBook. If you are using Debian, beginner skills are sufficient.

Product Details

• Author: Alex Williams
• PDF: 51 pages
• Publisher: Scaling Experts; First Edition, v1.1 (July 2011)
• Language: English
• ISBN: 978-2-924017-00-5

• local storage (DAS)
• storage on a standard network (NAS)
• storage on a specialized network (SAN)

Each technology has strengths and weaknesses. While the local storage is the most widespread and has a high level of performance, network storage offers the possibility to share data between different servers. This last feature is increasingly sought either to synchronize data between different servers, to consolidate disk space or to enable advanced virtualization architectures.

Hard disks

In recent years, the size of hard drives has increased rapidly. Storage capacity of several terabytes is no longer expensive especially if storage space is created with SATA drives. However, the use of SATA drives to provide storage space may result in poor I/O performance. The use of SAS drives with a rotational speed of 15000 rpm instead of SATA drives at 7200 rpm provides up to two times higher I/O performance. SSD drives can be used to reach even higher I/O performance.

The table below shows the basic performance of different types of hard drives in terms of MB/s and IOPS (Input/Output Operation per Second). These data, collected from tom’s hardware website, may be handy to evaluate the performance level of a data storage solution :

For a database server, high I/O performance is desirable while required disk space is usually low. Small-sized SAS or SSD drives are more useful than SATA drives in this case. For file servers, high throughput is recommended especially if large-sized files are served (music, videos, etc.).

Furthermore, the discs can be arranged in a RAID configuration to achieve a performance level higher than that offered by a single disk.

DAS (Direct Attached Storage)

Local storage matches the disks installed in a server. This is obviously the most widespread type of storage since most servers, although they may be connected to a network storage system, access files from their operating system through local disks. Servers booting directly from a SAN are an exception to this rule since they can operate without local disks.

DAS is very efficient because disks are connected to the server processor via the built-in motherboard controller or an hardware RAID card. It is therefore a broadband and exclusive connection as it is not shared with other servers.

NAS (Network Attached Storage)

A storage system on a standard network share files via an IP network. These file sharing protocols are usually offered:

• CIFS : Common Internet File System (natively on Microsoft Windows operating system)
• NFS : Netowork File System (natively on Linux operating system)
• AFP : Apple File Protocol (natively on Apple Mac OS X operating system)
• WebDAV : Web-based Distributed Authoring and Versioning (HTTP protocol extension)

A simple file server may be converted into a NAS storage system. A portion of its local storage is then made ​​available to the other networked servers. Commercial solutions are also available and can provide additional functionality:

• redundancy for the core system components (controller, network connectivity, electrical power)
• optimization of file sharing protocols
• caching of frequently accessed files
• web-based management console

The performance of a NAS system is affected by several factors:

• Type of disk drives (SATA, SAS or SSD)
• RAID configuration of disk drives
• Quantity of disk drives
• Network connectivity
• Type of shared files

The level of performance for a NAS solution depends on the limiting factor. For example, a set of sixteen (16) SATA drives in RAID 10 can offer a data throughput above 1 Gb/s. By cons, if these discs are placed in a NAS system with a 1 Gb/s network connectivity, the system performance can not exceed 1 Gb/s because of insufficient network connectivity. The limiting factor is then network connectivity. By using a 10 Gb/s network connectivity, system performance will be increased. The improvement of a factor other than network connectivity will have virtually no effect on system performance.

SAN (Storage Area Network)

A storage solution on a specialized network (SAN) uses iSCSI or Fibre Channel protocols to share storage space. In the case of a NAS, the storage resource is directly connected to the IP network while in the case of a SAN, storage space is available at the block level to the server file system. Each server sees the disk space on a SAN as its own hard drive. Traffic on a SAN follows similar conventions to those in force during communication with a local storage system. A SAN volume connected to a server can be formatted just like a local disk can be. This is obviously not possible in the case of a NAS volume.

In recent years, the iSCSI protocol has become increasingly attractive. The Fibre Channel protocol still exists, but tends to disappear in favor of the iSCSI protocol. Various reasons explains this phenomenon:

• compatibility with standard IP network equipment
• availability of 1 Gb/s and 10 Gb/s connectivity
• low cost of IP network equipment (cables, switches, routers)
• similarity of skills needed by network administrators

A SAN can meet some special needs:

• Creating a Microsoft Windows cluster (ActiveDirectory, SQL, Exchange, etc.) to obtain an high level of redundancy
• Allowing seamless migration of machines between servers in a virtualization solution (VMware, Xen)
• Replicating a database between to distant locations
• Consolidating storage space for a large quantity of servers
• etc.

Apart from these specific needs, a NAS solution can usually reach a similar level of performance at a lower cost.

Conclusion

The conventional DAS storage solution (local hard drives) offers a high level of performance. The cost of this solution is also lower than that of NAS and SAN network storage solutions. For your network storage needs, the NAS allows sharing of files between different servers while a SAN solution makes available storage space at the block level.

Used to ease the management of your server, the control panel will simplify the following tasks:

• Mail account creation
• User account creation
• Web application deployment
• FTP configuration
• Database configuration
• etc.

A control panel simplifies server management by offering a graphical user interface (GUI) suitable for both end-users and system administrators. It turns complex server administration tasks into user-friendly processes. An application programming interface (API) should also be available for advanced scripting and automation.

If you intend on using your web hosting solution to resell hosting package, you should consider a control panel allowing your customers and other users to maintain their own hosting environments. This could drastically reduce the number of assistance requests from your customer base. cPanel and Plesk are two control panels making this possible.

The control panel overrides the normal behaviour of the operating system installed on your server. Since applications and services are configured through the control panel, using the default tools of your operating system to modify configurations or install software may cause unexpected results. When using a control panel on your server, you should rely exclusively on the control panel tools to manage your server. For example, a server with the CentOS operating system and cPanel should not be considered as a CentOS server but rather as a cPanel server. This is especially true when you use Google or discussion forums to find a solution for a problem on your hosting server.

Another drawback is that building an highly available web server cluster with control panel powered servers is usually not supported. The tools required for server synchronization lie at the operating system level while the web service is managed at the control panel level. This is the main reason why control panels are not suitable for high-end web hosting solutions.

cPanel

cPanel is the most popular control panel on the market. It provides an intuitive GUI to help website owners manage their sites. A specialized component of cPanel called WebHost Manager (WHM) automates server management tasks for server administrators. cPanel intuitive interface includes video tutorials and on-screen help.

cPanel is only supported on Linux. Most cPanel installations are realized on the CentOS operating system. Both 32 and 64bit architectures are supported.

Product website: http://www.cpanel.com/products/cpanelwhm/

Enkompass

The Enkompass control panel has been released recently. Made by the same company that produced cPanel, Enkompass is only be supported on Microsoft Windows 2008 and above.

The Enkompass interface mirrors that of cPanel/WHM for Linux. While many features of Enkompass are the same as cPanel, features incompatible with Windows have been removed, and some Windows-only features have been added.

Product website: http://www.cpanel.com/windows/overview.html

Plesk

Plesk is a control panel supporting both Linux and Windows operating systems with a unified GUI. It provides an easy to learn interface with a simple look and feel.
Features such as the billing manager for customer invoicing and the site builder for website editing and management are convenient.

Under Linux, most Plesk installations are realized on the CentOS operating system. Both 32 and 64bit architectures of this Linux distribution are supported. Under Microsoft Windows, Plesk installation is supported on Windows Server 2003, Windows Server 2008 and Windows Server 2008R2 under both 32 and 64bit architectures.

Product website: http://www.parallels.com/products/plesk/

Snapshots

cPanel

Enkompass

Plesk for Linux

Plesk for Windows

References

Webhostingfan.com, Top Three Control Panels for Reseller Hosting

• BigBrother
• munin
• Nagios
• OpenNMS
• pingdom

BigBrother

BigBrother is a solution for IT infrastructure monitoring and diagnostics. It uses a client/server model to provide real-time monitoring for various operating systems (Windows, UNIX, Linux). A BigBrother installation is formed of the following elements:

• A web server: this server processes the status information and creates a dashboard to display your web solution status. The dashboard can be accessed through any web browser.
• A notification server: this server processes alerts and dispatches them to the correct people. Notifications can be sent out by e-mail, SMS text, alpha/numeric pages and/or SNMP traps.
• A network monitor: common network protocols are supported (http/ftp/pop3). Simple text-based protocols can be easily added to the configuration and monitored for availability.
• Local system monitors: a client must be installed on each monitored server to collect system information and send it to the web and/or notification server.

This commercial software is well-suited for mid-sized to large web solutions.

Website: http://bb4.com/

munin

Munin is a resource monitoring tool that can help analyze resource trends and diagnose performance issues on your servers. It uses a master/node architecture where the master connects to all the nodes at regular interval to collect data. Nodes can be computers, networks, SANs, etc. A piece of software needs to be installed on the nodes to allow communication with the master.

The collected information is stored in RRD files which are transformed into graphs and presented to the end-user through a web interface. Munin uses the OpenSource software RRDTool and its framework is written in Perl, while plugins may be written in any language.The emphasis of munin is on its plug and play capabilities and the ease of creating new plugins (graphs). Munin will help you determine what is different when a performance problem is detected compared to when things were working correctly. It will also help you see how your servers are in term of capacity for various resources (disk capacity, CPU, RAM, network).

This OpenSource software is well-suited for all web solutions.

Website: http://munin-monitoring.org/

Nagios

Nagios is a monitoring system to help you ensure systems, applications, services and business processes are functioning properly. It uses a client/server model to provide real-time monitoring for various operating systems (Windows, Linux, Netware) and network equipment (routers, switches). Therefore, a piece of software needs to be installed on your servers to allow Nagios to collect information about its resources. No software is required on your monitored servers if you only need to monitor publicly available services (HTTP, FTP, SSH, etc.).

In the event of a failure, Nagios can send alerts to allow you to begin remediation processes before outages affect business processes, end-users, or customers. Designed with scalability and flexibility in mind, Nagios will help you reduce the occurrence of unexpected outages.

Nagios is a popular OpenSource software well-suited for mid-sized to large web solutions.

Website: http://www.nagios.org/

OpenNMS

OpenNMS is a network management application platform with four main features:

• Automated and Directed Discovery
• Event and Notification Management
• Service Assurance
• Performance Measurement

By simply pointing OpenNMS at your network, it will discover services being provided and start monitoring them. Like Nagios, no software is required on your monitored servers if you only need to monitor publicly available services (HTTP, FTP, SSH, etc.). Devices can also be added manually to OpenNMS if you need to have fine-grained control over what is being managed.

OpenNMS can generate its own events or receive events from outside sources, such as SNMP Traps. OpenNMS which is able to handle burst of thousands of events per second, can serve as the central repository for your network event stream. Once important events have been identified, they can generate notices (email, SMS) or trouble tickets through an API allowing bidirectional communication with many trouble ticketing systems (Atlassian JIRA, Best Practical Solutions RT, Concursive ConcourseSuite – formerly Centric CRM, etc.).

The OpenNMS application allow the tracking of Service Level Agreements (SLAs) using synthetic transactions ranging from a simple ICMP request or port check, up through complex website monitoring and round trip e-mail testing. Detailed reports can also be generated on the availability of the services.

To collect performance data, OpenNMS includes support for the SNMP, JMX and HTTP protocols. The HTTP protocol can be used to retrieve data on a web page and use regular expressions to extract values in order to extend an application to integrate with this monitoring tool. Stored data can be converted into graphs and are available through OpenNMS web interface.

This OpenSource software is well-suited for mid-sized to large web solutions.

Website: http://www.opennms.org/

pingdom

Pingdom provides an uptime monitoring service. This service monitors the uptime and responsiveness of your web solution from different worldwide locations. It provides information about the availability of your web solution from the Internet perspective. No software is required on your servers to enable this service.

The service provided by Pingdom will discover any problem with your website within minutes. Different monitoring probes can be configured: web checks (HTTP and HTTPS), network checks (ping, specific TCP/UDP port, DNS) and email checks (SMTP, POP3, IMAP). Alerts can be sent through email or SMS. Detailed statistics of your website performance are also available giving you a historical record of all incidents.

This commercial service is well-suited for small to mid-sized web solutions.

Website: http://www.pingdom.com/

Conclusion

Various monitoring software and services are available to help you maintain your web solution availability. Using them will help maintain your web presence and give historical data about the performance of your web solution.

• MySQL Cluster
• MySQL replication
• Shared storage
• Block-level replication

This book is mainly focused on the MySQL Cluster approach with four (4) entire chapters dedicated to this method. The three (3) remaining approaches are covered in dedicated single chapters while the last chapter covers performance tuning of MySQL.

The configuration examples presented in the book are rich and well explained. These examples could be used to quickly set up an highly available MySQL database from scratch. However, the inner workings of MySQL in a highly available configuration could be more detailed. The graphical representations used to illustrate the theory behind the examples are poorly designed.

Another weakness of this book is the proposed method of installation for the Multi Master Replication Manager (MMM). The author recommends the installation of source packages. While these tools seem to be a valuable resource for a multi-master replication configuration, the use of manually installed software on a production environment should be avoided. The author should provide a way to install these tools via a set of packages specifically built for the CentOS distribution.

Nonetheless, this book is a valuable resource for a novice to intermediate level MySQL database administrator in his journey toward making its database highly available.

Product Details

• Author: Alex Davies
• Paperback: 276 pages
• Publisher: Packt Publishing; 1st New edition edition (April 2010)
• Language: English
• ISBN-10: 1847199941
• ISBN-13: 978-1847199942

Goals and limitations

The backup of your website data should be considered as a process. Your goals in term of data recovery combined with the limitations of the backup process must be taken into account in order to implement an adequate backup solution.

Recovery Point Objective

The frequency at which the backup operation is done will influence the length of the time-gap in case of data recovery. For example, if you scheduled a weekly backup for your website database, recovering your data from the backup will roll you back in time for up to a week. Setting up your Recovery Point Objective (RPO) will help you schedule your backup operations correctly and size your backup storage accordingly.

Recovery Time Objective

On the other hand, the time required to restore your backup data in case of data loss is called the Recovery Time Objective (RTO). The choice of storage media, its location and the backup strategy will influence the recovery time. Security concerns should also be addressed by choosing an appropriate location for your storage media and using access restriction tools like encryption.

Constraints

Several limitations must be considered during the design of the backup process: performance impact, bandwidth requirements and costs are three important factors. The performance impact is the additional payload imposed on your website due to the backup operation. To minimize such performance impact, the backup operation can be scheduled when your website is seeing the least usage. The backup process will then have the least amount of interference with normal operations. You will also need to evaluate the bandwidth required during backup and restoration operations if the backup media is accessed through the network. In this common scenario, the recovery time objective (RTO) is directly related to the size of the bandwidth between your server and the backup media.

Cost is also a limitation and will guide your choice of a backup solution. All types of storage media have a finite capacity with a real cost. Matching the correct amount of storage capacity with your needs is an important part of the backup scheme design. On the other hand, using a simple scheme will help you avoid excessive labour costs. Finally, while choosing a commercial backup software may simplify the backup scheme, the cost of a proprietary solution can be important.

Storage media

Magnetic tape

Magnetic tape has been used for backup data storage for a long time. Tape is a proven storage medium, which is its main advantage along with its portability. A magnetic tape is a purely sequential access medium. While access time for random files is poor, performance for continuous writing or reading can be high. Magnetic tape features the lowest price/capacity ratio. However, the complexity of a tape backup system and the specialized hardware required for reading and writing to this medium are major drawbacks.

Hard disk

Hard disk is a common medium for storage solutions. Relying on commodity hardware, this storage technology features low access time, high capacity and a low price/capacity ratio. The main disadvantage of hard disk backups is that their portability is limited. They can be easily damaged during transport.

Solid state device

Flash memory, USB keys and USB flash drives are solid state devices that can be used for storage solutions. Their main disadvantage is their high price/capacity ratio. However, this medium offers portability, the lowest access time and ease of use.

Backup strategy

Full backup

A full backup is the first step in a rotation scheme. All the relevant data is copied on the backup media. Some backup software will allow you to restore a whole server after a full system backup is completed. This feature is called bare-metal restoration. Without this feature, the restoration of the backup data to the server will have to be executed after the restoration of the original operating system on the server.

Differential backup

A differential backup copies files and folders that have been created or modified since the last full backup. When performing a combination of normal and differential backups, the restoration will most likely require the last full backup along with the latest differential backup.

Incremental backup

An incremental backup copies files and folders that have been created or modified since the last full or incremental backup. When performing a combination of normal and incremental backups, the restoration will require the last full backup along with all the incremental backups taken in the meantime.

Rotation scheme

A complete backup strategy should include a rotation scheme to allow the conservation of historical data. Such a method is also useful for managing multiple media in the backup process. The rotation scheme determines how and when each piece of storage is used for a backup job and how long it is retained once it has backup data stored on it. A balance between the cost of extra storage media and the need for data retention must be found.

The most used rotation strategies are the Grand-Father/Father/Son (GFS) and the Towers of Hanoi. Both of these strategies use full and differential/incremental backups.

Location of the backup data

On-site

Storing the backup data on the same site as the backup target is convenient since there is no need to move backup media physically or through the network. Handling or bandwidth issues can be avoided with such a solution.

However, on-site storage is insecure. Since both original and backup data are on the same location, a major disaster (fire, flood, theft, etc.) could be catastrophic.

Off-site

Storing the backup data on a secondary location increases the reliability of the backup process at the expense of convenience. To reduce the required effort, data can be moved between both sites using the network instead of relying on the transportation of physical media.

Reference

• Wikipedia, backup

Available solutions

Different web hosting solutions are available. We divide them into the following categories:

• Shared web servers
• Virtual private servers (VPS)
• Dedicated web servers
• Homemade web hosting
• Colocation

Division of responsibilities

The first table below illustrates the division of responsibilities for all five web hosting solutions. The end-user layer represents the interaction between your web solution and your visitors. In all cases, you are directly responsible toward your web solution users. The application layer encompass the operating system and the application stack. Unless you are using an homemade or a colocation solution, the hosting provider will share some responsibilities with you in this aspect. The device and network layers represent the physical server and networking hardware respectively. Finally, the data center layer encompass the electricity, cooling, access control and fire protection of the hardware equipment.

Strengths and weaknesses

This second table is a summary of the strengths and weaknesses of the different web hosting solutions.

A shared web server is generally a slice of a physical server along with a control panel to simplify the management of your web solution. It is available at low cost but will offer low performance since resource allocation is minimal. Uptime is also below average and scalability is not available.

A virtual private server (VPS) is similar to a shared web server since it is sitting on an actual physical server. However, most of the allocated resources of the VPS are determined at purchase time and will determine its cost. Uptime and scalability are generally good.

A dedicated web server gives you the complete resource ownership of a physical server. The cost of a dedicated server is high especially for low-end servers compared to VPS offerings. Uptime is generally good and scalability is excellent with such a solution.

Building your web solution with your own equipment and network connectivity could save you a lot of money on the long term. While uptime may be lacking since your infrastructure is likely less sophisticated than the one offered by web hosting companies, resource allocation and scalability are excellent.

The colocation option is the intermediate solution between a dedicated web server and an homemade solution. The cost is high but resource allocation, uptime and scalability are excellent.

Reference: Charles Hotchkin (RackSpace), Managed Colocation

• increase data reliability
• increase Input/Output (I/O) performance

Theory

Concepts

A RAID array is composed of two or more physical hard disks combined into a single logical storage unit. To give RAID array additional features compared to JBOD (Just a Bunch Of Disk), three main concepts are used:

• Mirroring
• Striping
• Error correction

Mirroring is the writing of identical data to more than one disk. The basic example of mirroring is a RAID 1 array formed by two disks. Both disks have the same content at any time. If the first drive fails, read and write operation can be done directly on the second disk. Read operations on mirrored arrays is faster compared to a single disk since the system can fetch data from multiple disks at the same time. However, write operations are slower since the data must be written to all disks instead of only one. The reconstruction of a failed mirror array is quite simple: data must be copied from the healthy disk to the new one. During reconstruction, the read performance boost of the mirror array is reduced since only the healthy disk is fully usable.

Striping is the splitting of data across multiple disks. For example, a RAID 0 array formed by two disks strips data to both disks. Striping does not provide fault tolerance, only a performance boost. Read and write operations on a striped array are faster compared to a single disks as both operation are split between the available disks.

Error correction stores parity data on disk to allow the detection and possibly the correction of problems. RAID 5 is a good example of the error correction mechanism. For example, a RAID 5 array composed of three drive strips data on the first two disks and stores parity on the third disk to provide fault tolerance. The error correction mechanism will slow down performance especially for write operation since both data and parity information needs to be written instead of data only. Moreover, the reconstruction of a failed array using parity information incurs severe performance degradation as data needs to be fetched from all drives in the array to rebuild the information for the new disk.

The design of any RAID scheme is a compromise between data protection and performance. The comprehension of your server requirements in terms of storage is crucial to select the appropriate RAID configuration.

Hardware vs Software RAID

There are two types of RAID managers:

• hardware
• software

Hardware solutions are specialized hardware components connected to the server motherboard. Most of the time, these components will provide a post-BIOS configuration interface that can be run before booting your server operating system. Each configured RAID array will present himself to the operating system as a single storage drive. The RAID array can be partitioned into various RAID volumes at the operating system level.

On the other hand, software solutions are implemented at the operating system level and directly create RAID volumes from entire physical disks or partitions. Each RAID volume is seen as a standard storage space for the applications running within the operating system. Both approaches have advantages and disadvantages compared to each other.

Depending on the manufacturer, an hardware RAID card supporting up to 8 drives is usually sold between 400$ and 1200$ while a software RAID solution is usually included free of charge with the operating system of your server. Under Linux, the md RAID subsystem is able to support most RAID configurations. Under Microsoft Windows, Software RAID is provided through the use of dynamic disks in the disk management console.

The required processing power for RAID 0, RAID 1 and RAID 10 is relatively low. Parity-based arrays like RAID 5, RAID 6, RAID 50 and RAID 60 require more complex data processing during write or integrity check operations. However, this processing time is minimal on modern CPU units as the increase in speed of commodity CPUs has been consistently greater than the increase in speed of hard disk drive throughput over history. Thus, the percentage of server CPU time required to saturate an hard disk RAID array throughput has been dropping and will probably continue to do so in the future.

A more serious issue with software RAID array is how the operating system deals with the boot process. Since the RAID information is kept at the operating system level, booting a faulty RAID array is problematic. At boot time, the operating system is not available to coordinate the failover to another drive if the usual boot drive fails. Such systems may require manual intervention to make them bootable again after a failure. A hardware RAID controller is initialized before the boot process starts looking for information on the disk drives. Therefore, hardware RAID controller will increase the robustness of your server compared to software RAID.

An hardware RAID controller may also support hot swappable hard drives. With such a feature, hard disks can be changed in a server without having to turn off the power and open up server case. Removing a failed hard drive and replacing it with a new one is a simple task with a hardware RAID controller supporting hot swappable disks. Without this feature, the server needs to be powered off before replacing the failed drive. This will lead to downtime unless your web solution is properly clustered.

Finally, only hardware RAID controllers can carry a Battery Backup Unit (BBU) to preserve the cache memory of the controller if the server is shut down abruptly. Without such a protection, write-back cache should be disabled on the RAID array to prevent data corruption. Turning off write-back cache comes with a performance penalty for write operations on the RAID array. The use of a BBU on your RAID controller is a solution to safely enable write-back caching and improve write performance.

A RAID array is not a backup solution

Most RAID arrays provide protection in case of a disk failure. While such a protection is important to protect yourself from data loss due to hardware failure, it does not provide historical data. A RAID array does not allow to recover a deleted or corrupted file due to a bug in your application. A backup solution will allow you to go back in time to recover deleted or corrupted files.

Implementation

Note: images were adapted from those available on Wikipedia.

RAID 0

RAID 0 is a pure implementation of striping. A minimum of two (2) disks is required for RAID 0. No parity information is stored for redundancy. It is important to note that RAID 0 was not one of the original RAID levels and provides no data redundancy. RAID 0 is normally used to increase performance. RAID 0 is useful for setups where redundancy is irrelevant.

A RAID 0 array can be created with disks of differing sizes, but the total available storage space in the array is limited by the size of the smallest disk. For example, if a 450GB disk is striped together with a 300GB disk, the usable size of the array will be 2 x min(450GB, 300GB) = 600GB.

For reads and writes operations dealing with small data blocks such as database access, the data will be fetch independently on each disk of the RAID 1 array. If the data sectors accessed are spread evenly between the two disks, the apparent seek time of the array will be half that of a single disk. The transfer speed of the array will be the transfer speed of all the disks added together, limited only by the speed of the RAID controller. For reads and writes operations dealing with large data blocks such as copying files or video playback, the data will most likely be fetch on a single disk reducing the performance gain of the RAID 0 array.

RAID 1

RAID 1 is a pure implementation of mirroring. A minimum of two (2) disks is required for RAID 1. This is useful when read performance or reliability are more important than data storage capacity. A classic RAID 1 mirrored pair contains two disks (see diagram), which increases reliability over a single disk. Since each member contains a complete copy of the data, and can be addressed independently, ordinary wear-and-tear reliability is raised.

A RAID 1 array can be created with disks of differing sizes, but the total available storage space in the array is equal to the size of the smallest disk. For example, if a 450GB disk is mirrored with a 300GB disk, the usable size of the array will be min(450GB, 300GB) = 300GB.

The read performance of a RAID 1 array can go up roughly as a linear multiple of the number of copies. That is, a RAID 1 array of two disks can query two different places at the same time so the read performance should be two times higher than the performance of a single disk. RAID 1 is a good starting point for applications such as email and web servers as well as for any other use requiring above average read I/O performance and hardware failure protection.

RAID 5

RAID 5 array uses block-level striping with distributed parity blocks across all member disks. The disk used for the parity block is staggered from one stripe to the next, hence the term distributed parity blocks. A minimum of three (3) disks is required for RAID 5. This RAID configuration is mainly used to maximize disk space while providing a protection for your data in case of a disk failure.

Given the diagram of the RAID 5 array, where each column is a disk, let assume A1=00000101 and A2=00000011. The parity block Ap is generated by applying the XOR operator on A1 and A2:
Ap = A1 XOR A2 = 00000110

If the first disk fails, A1 will no longer be accessible, but can be reconstructed:
A1 = A2 XOR Ap = 00000101

A RAID 5 array can be created with disks of differing sizes, but the total available storage space in the array is limited by the size of the smallest disk. The parity data consumes a complete disk, leaving N-1 disks for usable storage space in an array composed of N disks. For example, on an array formed of three 450GB disks and one 300GB disk, the usable size of the array will be (4-1) x min(450GB, 300GB) = 900GB.

RAID 5 writes are expensive in terms of disk operations and traffic between the disks and the RAID controller since both data and parity information need to be written to disk. The parity blocks are not read on data reads, since this would add unnecessary overhead and would diminish performance. However, the parity blocks are read when a defective disk sector is present in the required data blocks. Likewise, should a disk fail in the array, the parity blocks and the data blocks from the surviving disks are combined mathematically to reconstruct data from the failed drive in real-time. This situation leads to severe performance degradation on the array for read and write operations.

RAID 6

RAID 6 extends RAID 5 by adding an additional parity block. Block-level striping is combined with two parity blocks distributed across all member disks. A minimum of four (4) disks is required for RAID 6. This RAID configuration is mainly used to maximize disk space while providing a protection for up to two disk failures.

Both parity blocks Ap and Aq are generated from the data blocks A1, A2 and A3. Ap is generated by applying the XOR operator on A1, A2 and A3. Aq is generated using a more complex variant of the Ap formulae. If the first disk fails, A1 will no longer be accessible, but can be reconstructed using A2 and A3 plus the Ap parity block. If both the first and the second disk fail, A1 and A2 will no longer be accessible, but can be reconstructed using A3 plus both Ap and Aq parity blocks. The computation of Aq is CPU intensive, in contrast to the simplicity of Ap. Thus, a software RAID 6 implementation may have a significant effect on system performance especially during the reconstruction of a failed disk.

A RAID 6 array can be created with disks of differing sizes, but the total available storage space in the array is limited by the size of the smallest disk. The parity data consumes two complete disks, leaving N-2 disks for usable storage space in an array composed of N disks. For example, on an array formed of four 450GB disks and one 300GB disk, the usable size of the array will be (5-2) x min(450GB, 300GB) = 900GB.

RAID 6 writes are even more expensive than RAID 5 writes in terms of disk operations and traffic between the disks and the RAID controller since both data and parity information need to be written to disk. The parity blocks are not read on data reads, since this would add unnecessary overhead and would diminish performance. However, the parity blocks are read when a defective disk sector is present in the required data blocks. Likewise, should a disk fail in the array, the parity blocks and the data blocks from the surviving disks are combined mathematically to reconstruct data from the failed drive in real-time. This situation leads to severe performance degradation on the array for read and write operations.

RAID 10

RAID 10 is a combination of RAID 1 (mirroring) and RAID 0 (striping) where 4N mirrored disks are striped together. A minimum of four (4) disks are required for RAID 10. One disk in each RAID 1 mirror can fail without damaging the data contained in the entire array.

A RAID 10 array can be created with disks of differing sizes, but the total available storage space in the array is limited by the size of the smallest disk. The mirroring consumes half of disk space, leaving 2N disks for usable storage space in an array composed of 4N disks. For example, on an array formed of seven 450GB disks and one 300GB disk, the usable size of the array will be (7+1)/2 x min(450GB, 300GB) = 1200GB.

RAID 10 provides better performance than all other redundant RAID levels. It is the preferable RAID level for I/O intensive applications such as database servers as well as for any other use requiring high disk performance.

RAID 50

RAID 50 is a combination of RAID 5 (striping and error correction) and RAID 0 (striping) where RAID 5 sub-arrays are striped together. A minimum of six (6) disks are required for RAID 50. One disk in each RAID 5 sub-array can fail without damaging the data contained in the entire array.

A RAID 50 array can be created with disks of differing sizes, but the total available storage space in the array is limited by the size of the smallest disk. The parity data consumes a complete disk in each RAID 5 sub-array, leaving N-2 disks for usable storage space in an array composed of N disks. For example, on an array formed of seven 450GB disks and one 300GB disk, the usable size of the array will be (8-2) x min(450GB, 300GB) = 1800GB.

RAID 50 provides better performance than RAID 5 but requires more disks. The performance gain is particularly observed for write operations. This level is recommended for applications that require high fault tolerance along with high capacity.

Hot spare disks

Both hardware and software redundant RAID arrays may support the use of hot spare disks. Such disks are physically installed in the array and are inactive until an active disk fails. The RAID controller automatically replaces the failed drive with the spare and starts the rebuilding process for the affected array. This reduces the vulnerability window of the array by providing a healthy disk to the array as soon as a problematic disk is identified.

For example, a RAID 5 array with a single hot spare disk uses the same number of disks as a RAID 6 array while providing a similar level of protection. The use of hot spare disks is particularly important for RAID arrays formed by multiple disks. For example, a RAID 10 array formed of 12 disks will most likely have a higher disk failure rate than a RAID 10 array of 4 disks. Putting aside one or two disks as hot spare for your large RAID array will provide additional protection in case of disk failure.

Conclusion

RAID arrays allow a higher level of reliability and performance for your server storage. While RAID 1 is a good starting point for applications such as email and web servers, RAID 10 is recommended for database applications. RAID 5 or RAID 50 can be used for backup appliances where high fault tolerance along with high capacity are needed.

References

• Wikipedia, RAID
• Art S. Kagel, RAID 5 vs RAID 10

MySQL DRBD

The MySQL database service can be transformed into an highly available service with DRBD. This software technology, only available under Linux, mirrors data of a service from the active node to the standby node of a highly available cluster. DRBD works on top of block devices (e.g. hard disk partitions, LVM’s logical volumes). It mirrors each data block that it is written to disk to the standby node.

Mirroring can be done in a tightly coupled fashion called protocol C (synchronous mirroring). In this mode, the file system on the active node is notified that the writing of the block was finished only when the block is written to both disks of the cluster. It is the good choice for highly available clusters when losing a single transaction in case of a complete crash of the active node must be avoided. The other available option is asynchronous mirroring. In this mode, the file system on the active node is notified that the writing of the block was finished as soon as the data is written to the local disk. It is the good choice when building mirrors over long distances.

The diagram below presents a solution formed by a web server cluster and a MySQL DRBD cluster connected on a local network. Both MySQL server nodes are inter-connected by a DRBD network link. This network interface is used to synchronize data block between the servers and avoid private network congestion.

A consequence of mirroring data at the block device level is that you can only access your data on the active node. This is caused by the nature of most Linux file systems (ext3, ext4, XFS, JFS, etc). These file systems do not support multiple servers accessing the same shared disk for read and write operations. Their design only allow one server accessing each disk. As a consequence, the performance of a MySQL DRBD cluster is roughly equal to the performance of a single MySQL server. DRBD provides high availability but no performance enhancement for a MySQL database service.

MySQL replication

A cluster composed of two or more MySQL servers can also be created using a redundant load balancing solution and MySQL database replication feature.

MySQL master/master replication

A cluster with MySQL master/master replication consists of two MySQL servers in an active/active mode. The load balancing appliance must define two separate IP addresses:

• Write operation IP
• Read operation IP

Write operations to the database must be performed on a single MySQL server to prevent data corruption. A server is defined as the primary server in the load balancing appliance configuration. The second server is configured as a backup. In case the primary server fails, write operations will be directed automatically to the second server. The data contained in each server database is synchronized between them using replication. Read operations can be performed on both MySQL servers.

In a master/master replication setup, both servers are on an equal footing. If one server fails, the second can take the relay without having to change the configuration of the web servers. For write operations, such a configuration provides performance equal to that of a single server. For read operations, the theoretical performance of such a cluster is two times higher than that of a single server. However, to achieve higher performance during read operations, the web application must be coded to make a distinction between SQL write and read requests.

MySQL master/slave replication

A cluster with MySQL master/slave replication consists of two MySQL servers in an active/active mode. The load balancing appliance must define two separate IP addresses:

• Write operation IP
• Read operation IP

Write operations to the database must be performed on the master server exclusively. The data contained in the slave server is synchronized with the master server. Read operations can be performed on both MySQL servers.

In a master/slave replication setup, only the master server is able to properly manage the write operations. In case of failure of the master server, the web application will stop working properly. For write operations, such a configuration provides performance equal to that of a single server. For read operations, the theoretical performance of such a cluster is two times higher than that of a single server. However, to achieve higher performance during read operations, the web application must be coded to make a distinction between SQL write and read requests.

Comparing master/master and master/slave replications shows there is no apparent advantage to use a master/slave configuration. However, while a master/master replication consists of two servers, the master/slave replication can be formed of several servers with one master server and several slave server. Therefore, a master/slave configuration is interesting when the number of slave servers is greater than one.

MSSQL high availability cluster

A Microsoft Windows cluster consists of two servers in an active/passive configuration, at least one domain controller (DC) and a data storage solution. The diagram below presents a MSSQL cluster formed by two database servers, two redundant domain controller servers and an iSCSI SAN storage solution. The iSCSI SAN is connected to the database servers with multipath connections to provide redundant connectivity links.

This Microsoft Windows based setup provides high availability to your web solution. If the active server stops working, the passive server will take the relay. The performance of the database cluster depends mainly on the following elements:

• database server CPU
• SAN disks and RAID configuration
• SAN data caching mechanism
• communication link between the servers and the SAN

MSSQL replication

A cluster composed of two or more MSSQL servers can also be created using a redundant load balancing solution and MSSQL database replication feature.

MSSQL master/master replication

A cluster with MSSQL master/master replication consists of two MSSQL servers in an active/active mode. The load balancing appliance must define two separate IP addresses:

• Write operation IP
• Read operation IP

Write operations to the database must be performed on a single MSSQL server to prevent data corruption. A server is defined as the primary server in the load balancing appliance configuration. The second server is configured as a backup. In case the primary server fails, write operations will be directed automatically to the second server. The data contained in each server database is synchronized between them using the transactional publication with updatable subscriptions mechanism. Read operations can be performed on both MSSQL servers.

The transactional publication with updatable subscriptions mechanism adds a column named msrepl_tran_version on each published table to keep track of transactions. All SQL INSERT commands of your application must be adapted to accommodate this change.

In a master/master replication setup, both servers are on an equal footing. If one server fails, the second can take the relay without having to change the configuration of the web servers. For write operations, such a configuration provides performance equal to that of a single server. For read operations, the theoretical performance of such a cluster is two times higher than that of a single server. However, to achieve higher performance during read operations, the web application must be coded to make a distinction between SQL write and read requests.

MSSQL master/slave replication

A cluster with MSSQL master/slave replication consists of two MSSQL servers in an active/active mode. The load balancing appliance must define two separate IP addresses:

• Write operation IP
• Read operation IP

Write operations to the database must be performed on the master server exclusively. The data contained in the slave server is synchronized with the master server using the transactional publication mechanism. Read operations can be performed on both MSSQL servers.

In a master/slave replication setup, only the master server is able to properly manage the write operations. In case of failure of the master server, the web application will stop working properly. For write operations, such a configuration provides performance equal to that of a single server. For read operations, the theoretical performance of such a cluster is two times higher than that of a single server. However, to achieve higher performance during read operations, the web application must be coded to make a distinction between SQL write and read requests.

Comparing master/master and master/slave replications shows there is no apparent advantage to use a master/slave configuration except the requirement to use modified SQL INSERT statements in master/master mode. However, while a master/master replication consists of two servers, the master/slave replication can be formed of several servers with one master server and several slave server. Therefore, a master/slave configuration is interesting when the number of slave servers is greater than one.

• Increase the availability
• Facilitate the rise in system load
• Allow for load distribution
• Allow the use of low-cost servers

Scaling up from a single server solution

The first recommended step when scaling up your web solution is to place your web and database services on two separate servers. The web server will handle requests from your clients and communicate with the database server to obtain dynamic data. Such a configuration is compatible with most control panel software available on the market. The control panel will be installed on the web server and will connect with the remote database server. Your web pages configured via the control panel will then have database functionality.

The partition of the web and database services on two different servers has several advantages.

First, both services have different requirements in terms of security and access control. Putting them on two different servers allows the creation of a more secure design for your whole web solution. While the web server will accept all incoming connections on the HTTP and maybe the HTTPS ports, the database server should only accept connections on the SQL port from the web server. The use of a private network (not shown on the diagram above) provides a much more secure communication channel.

Second, the resource allocation requirements for the web and database servers are different. The web server must have enough space to store your text, image, music and video files. A single SATA drive or a RAID 1 configuration should provide enough space and IO performance in most cases. Also, a multi-core CPU with a low clock speed should provide enough computational power for your web service (apache, nginx, IIS, etc). On the other hand, your database may have higher IO and CPU requirements. The use of SAS or SSD drives in a RAID 1 or RAID10 configuration will provide an higher IO throughput. A single or dual multi-core CPU with an higher clock speed will provide higher CPU performance for your database service (MySQL, PostgreSQL, MSSQL, Oracle, etc).

Third, a split between the web and databases services will allow you to move seamlessly toward high availability with web and database clustering in the future. A different approach is required to put both of these services into a highly available cluster. Separating them at this stage of the evolution of your web solution will simplify a future migration toward an high availability configuration.

Web server clustering

To achieve high availability and increase the performance of your web service, you will need a redundant load balancing solution and at least a pair of back-end servers. There are two major ways to load balance network connections:

• Reverse proxy with layer 7 (application layer) packet filtering
• Direct routing with layer 4 (transport layer) content switching

In the diagram shown below, the load balancing appliance is formed by an active/passive pair of reverse proxy servers. The network traffic from the Internet is dispatched to the back-end web servers via the load balancing appliance. The IP address associated with the domain name of your web service is bonded to the load balancing appliance. All incoming connections to this IP address are treated by the active load balancer server and forwarded according to its configuration to the web servers through Network Address Translation (NAT). Web server responses are forwarded back to the clients via the active load balancer.

With such a configuration, the load balancing appliance may also work as a firewall device to protect your web and database servers from hostile intrusion. Moreover, your database server is now in a much more secure location as it can respond to SQL requests from the private network and safely ignore requests from the public network.

Distribution algorithm

Various scheduling algorithms are used to distribute connection requests to the back-end servers. A simple algorithm could consist of a random choice or the round robin technique where connection requests are assigned to each back-end server in equal portions and in circular order. Some sophisticated methods taking into account additional factors, such as the back-end server loads and response time are also available depending on your choice of load balancing appliance.

Connection persistence

Handling information that must be conserved between multiple client requests is an important issue when operating a load balancing appliance. When this information is kept locally on each server, subsequent requests must be sent to the appropriate server. Storing session data on the client browser instead of the web server is an elegant solution as it does not require connection persistence. The load balancing appliance can then use a simple round robin scheduling algorithm to distribute client requests. However, this method of handling session data is not always suitable or desirable.

A simple solution to the connection persistence problem is to send all requests from a user session consistently to the same back-end server to provide network connection persistence. However, if a server goes down, its local information is lost and any session depending on it must be restarted. The persistence can be established using the client IP address or by inserting a browser cookie into the web session. This cookie containing information about the back-end server is analyzed by the load balancing appliance before the transmission of the request to the appropriate back-end web server.

Content synchronization

Configuration and data files need to be synchronized between all cluster node servers. Under Linux where the configuration is a set of files, the web server configuration can be copied from the first web server toward the other ones on a regular basis using a scheduled task. Under Microsoft Windows, the configuration of IIS is not stored directly on a file and the configuration needs to be replicated manually on every web server.

Text, image, music and video files can be copied from the first web server toward the other ones on a regular basis using a scheduled task under Linux and Microsoft Windows. However, if you need to maintain a continuous synchronization of data content between the web servers, you will most likely need a Network Attached Storage (NAS) solution. To illustrate this requirement, we could imagine two clients surfing your web site. While the first client is connected to web server A, the second one is connected to web server B. The first client uploads a picture of himself on your website. Without continuous synchronization, the picture will not be visible to the second client until the synchronization script is executed between server A and server B. While this situation is acceptable on most situations, you may have a requirement for continuous synchronization for your web solution.

Control panel compatibility

An important drawback of a web cluster architecture is its incompatibility with the majority of control panel software. The required synchronization of the web server configuration and data files is the main obstacle to the use of a control panel on a web cluster solution.

Performance

Theoretically, since multiple back-end web servers are able to respond to client requests, the performance of such a web solution measured in terms of data throughput and responsiveness will be superior compared to an equivalent solution with only one web server. However, the performance may not scale up linearly for different reasons:

• the database may have insufficient resources to answer web server SQL requests quickly enough;
• the bandwidth of the load balancing appliance with the Internet may be saturated;
• etc.

Until now, we only discussed about clustering the web servers. However, to achieve true high availability, the database service should also be transformed into a cluster. We discuss how to create a database cluster in this article.