• sysstat
• munin
• strace
• memcached
• nginx
• keepalived
• HAProxy
• etc.

The document is divided in three distinct sections. The first two chapters explain how to configure monitoring and profiling tools. The next section presents six different server architectures and explains their advantages and shortcomings. Finally, the third section provides hands on examples on how to install the required software for all server architectures given in the previous section. How to configure the software and manipulate your Linux system through scripts and commands is also explained.

All the examples are written for the Debian Linux operating system (Lenny v5.0 version). If you use a different Linux OS, you will need to adapt the commands and the scripts accordingly. Therefore, intermediate skills in system administration are required if you intend to use a different OS than Debian to implement the recipes given in this eBook. If you are using Debian, beginner skills are sufficient.

Product Details

• Author: Alex Williams
• PDF: 51 pages
• Publisher: Scaling Experts; First Edition, v1.1 (July 2011)
• Language: English
• ISBN: 978-2-924017-00-5

• local storage (DAS)
• storage on a standard network (NAS)
• storage on a specialized network (SAN)

Each technology has strengths and weaknesses. While the local storage is the most widespread and has a high level of performance, network storage offers the possibility to share data between different servers. This last feature is increasingly sought either to synchronize data between different servers, to consolidate disk space or to enable advanced virtualization architectures.

Hard disks

In recent years, the size of hard drives has increased rapidly. Storage capacity of several terabytes is no longer expensive especially if storage space is created with SATA drives. However, the use of SATA drives to provide storage space may result in poor I/O performance. The use of SAS drives with a rotational speed of 15000 rpm instead of SATA drives at 7200 rpm provides up to two times higher I/O performance. SSD drives can be used to reach even higher I/O performance.

The table below shows the basic performance of different types of hard drives in terms of MB/s and IOPS (Input/Output Operation per Second). These data, collected from tom’s hardware website, may be handy to evaluate the performance level of a data storage solution :

For a database server, high I/O performance is desirable while required disk space is usually low. Small-sized SAS or SSD drives are more useful than SATA drives in this case. For file servers, high throughput is recommended especially if large-sized files are served (music, videos, etc.).

Furthermore, the discs can be arranged in a RAID configuration to achieve a performance level higher than that offered by a single disk.

DAS (Direct Attached Storage)

Local storage matches the disks installed in a server. This is obviously the most widespread type of storage since most servers, although they may be connected to a network storage system, access files from their operating system through local disks. Servers booting directly from a SAN are an exception to this rule since they can operate without local disks.

DAS is very efficient because disks are connected to the server processor via the built-in motherboard controller or an hardware RAID card. It is therefore a broadband and exclusive connection as it is not shared with other servers.

NAS (Network Attached Storage)

A storage system on a standard network share files via an IP network. These file sharing protocols are usually offered:

• CIFS : Common Internet File System (natively on Microsoft Windows operating system)
• NFS : Netowork File System (natively on Linux operating system)
• AFP : Apple File Protocol (natively on Apple Mac OS X operating system)
• WebDAV : Web-based Distributed Authoring and Versioning (HTTP protocol extension)

A simple file server may be converted into a NAS storage system. A portion of its local storage is then made ​​available to the other networked servers. Commercial solutions are also available and can provide additional functionality:

• redundancy for the core system components (controller, network connectivity, electrical power)
• optimization of file sharing protocols
• caching of frequently accessed files
• web-based management console

The performance of a NAS system is affected by several factors:

• Type of disk drives (SATA, SAS or SSD)
• RAID configuration of disk drives
• Quantity of disk drives
• Network connectivity
• Type of shared files

The level of performance for a NAS solution depends on the limiting factor. For example, a set of sixteen (16) SATA drives in RAID 10 can offer a data throughput above 1 Gb/s. By cons, if these discs are placed in a NAS system with a 1 Gb/s network connectivity, the system performance can not exceed 1 Gb/s because of insufficient network connectivity. The limiting factor is then network connectivity. By using a 10 Gb/s network connectivity, system performance will be increased. The improvement of a factor other than network connectivity will have virtually no effect on system performance.

SAN (Storage Area Network)

A storage solution on a specialized network (SAN) uses iSCSI or Fibre Channel protocols to share storage space. In the case of a NAS, the storage resource is directly connected to the IP network while in the case of a SAN, storage space is available at the block level to the server file system. Each server sees the disk space on a SAN as its own hard drive. Traffic on a SAN follows similar conventions to those in force during communication with a local storage system. A SAN volume connected to a server can be formatted just like a local disk can be. This is obviously not possible in the case of a NAS volume.

In recent years, the iSCSI protocol has become increasingly attractive. The Fibre Channel protocol still exists, but tends to disappear in favor of the iSCSI protocol. Various reasons explains this phenomenon:

• compatibility with standard IP network equipment
• availability of 1 Gb/s and 10 Gb/s connectivity
• low cost of IP network equipment (cables, switches, routers)
• similarity of skills needed by network administrators

A SAN can meet some special needs:

• Creating a Microsoft Windows cluster (ActiveDirectory, SQL, Exchange, etc.) to obtain an high level of redundancy
• Allowing seamless migration of machines between servers in a virtualization solution (VMware, Xen)
• Replicating a database between to distant locations
• Consolidating storage space for a large quantity of servers
• etc.

Apart from these specific needs, a NAS solution can usually reach a similar level of performance at a lower cost.

Conclusion

The conventional DAS storage solution (local hard drives) offers a high level of performance. The cost of this solution is also lower than that of NAS and SAN network storage solutions. For your network storage needs, the NAS allows sharing of files between different servers while a SAN solution makes available storage space at the block level.

Work experience

CBC/Radio-Canada (dec 2010 – …)
Project engineer

• Implantation of systems for production and broadcasting of quality projects for Radio, Television and Internet
• Management of various design projects, construction of studios and master control room related to the production and broadcast of radio, television, web and other content
• Work coordination of engineers, associates, other employees and subcontractors engaged in complex projects

iWeb Technologies (jan 2009 – dec 2010)
Sales engineer

• Technical lead during the planing and the delivery of our larger customer projects
• Definition of the logical (WAN, DMZ, LAN, firewall, VPN) and physical architecture for the web solutions offered to our clients in order to meet their business requirements (high availability, performance, cost)
• Team coordination during the delivery of complex web solutions
• Documentation of our customer solutions using a collaborative tool (secured extranet wiki)
• Planing, cost evaluation and testing of the possible new technologies for our customers (NAS/DAS/SAN storage solutions, SSD drives, MSSQL cluster, DRBD cluster, Content Delivery Network, etc.)

Outbox Technology (jan 2005 – nov 2008)
IT director

• Design and installation of the system architecture for Cirque du Soleil et Bell Center ticketing system (firewall, NAT, DHCP, Bind, Squid, Dovecot, Postfix + Amavis+ ClamAV + SpamAssassin, etc)
• Installation of a monitoring system (OpenNMS) for the IT infrastructure
• Integration and configuration of the various elements of the ticketing system (Apache and IIS front-end, Onyx CRM system, MySQL and MSSQL databases, credit card processor, CUPS printing system, reporting system, etc.)
• Segmentation of the different environments (development, QA, production) and implementation of a version control system
• Documentation of the IT processes using a collaborative tool (intranet wiki)
• Evolution of the IT architecture with the growth of the business (local disks -> SAN; physical servers -> virtual servers; etc.)
• Coordination and supervision of our team of IT technicians

OpenSource community involvment

CentOS LiveCD project

• First live distribution based on Red Hat Enterprise Linux 5
• Backport of the livecd-tools package from Fedora to CentOS 5
• Providing support, bug fixes and new features for the LiveCD toolset

Spacewalk documentation for CentOS

• Spacewalk provides a management interface for software content across registered servers and desktops

Skills

• Evaluation of datacenter infrastructures (generators, transfer switch, UPS, PDU, electrical circuits, cooling system, etc.)
• Conception of highly available and scalable web solutions (DRBD, heartbeat, HAProxy, LVS, Varnish, CDN, Microsoft cluster and MySQL replication)
• Documentation of processes and system architectures using collaborative tools
• Virtualization of Windows and Linux servers with VMware, Xen and HyperV technologies
• Configuration and use of Compellent and Dell EqualLogic SAN products
• Administration of computer networks (network architecture, VPN, DHCP, DNS, IMAP/POP3, SMTP, etc.)
• Administration of Windows and Linux servers
• Management of software versions

Contact

You can contact me via email at info@patriceguay.com

Used to ease the management of your server, the control panel will simplify the following tasks:

• Mail account creation
• User account creation
• Web application deployment
• FTP configuration
• Database configuration
• etc.

A control panel simplifies server management by offering a graphical user interface (GUI) suitable for both end-users and system administrators. It turns complex server administration tasks into user-friendly processes. An application programming interface (API) should also be available for advanced scripting and automation.

If you intend on using your web hosting solution to resell hosting package, you should consider a control panel allowing your customers and other users to maintain their own hosting environments. This could drastically reduce the number of assistance requests from your customer base. cPanel and Plesk are two control panels making this possible.

The control panel overrides the normal behaviour of the operating system installed on your server. Since applications and services are configured through the control panel, using the default tools of your operating system to modify configurations or install software may cause unexpected results. When using a control panel on your server, you should rely exclusively on the control panel tools to manage your server. For example, a server with the CentOS operating system and cPanel should not be considered as a CentOS server but rather as a cPanel server. This is especially true when you use Google or discussion forums to find a solution for a problem on your hosting server.

Another drawback is that building an highly available web server cluster with control panel powered servers is usually not supported. The tools required for server synchronization lie at the operating system level while the web service is managed at the control panel level. This is the main reason why control panels are not suitable for high-end web hosting solutions.

cPanel

cPanel is the most popular control panel on the market. It provides an intuitive GUI to help website owners manage their sites. A specialized component of cPanel called WebHost Manager (WHM) automates server management tasks for server administrators. cPanel intuitive interface includes video tutorials and on-screen help.

cPanel is only supported on Linux. Most cPanel installations are realized on the CentOS operating system. Both 32 and 64bit architectures are supported.

Product website: http://www.cpanel.com/products/cpanelwhm/

Enkompass

The Enkompass control panel has been released recently. Made by the same company that produced cPanel, Enkompass is only be supported on Microsoft Windows 2008 and above.

The Enkompass interface mirrors that of cPanel/WHM for Linux. While many features of Enkompass are the same as cPanel, features incompatible with Windows have been removed, and some Windows-only features have been added.

Product website: http://www.cpanel.com/windows/overview.html

Plesk

Plesk is a control panel supporting both Linux and Windows operating systems with a unified GUI. It provides an easy to learn interface with a simple look and feel.
Features such as the billing manager for customer invoicing and the site builder for website editing and management are convenient.

Under Linux, most Plesk installations are realized on the CentOS operating system. Both 32 and 64bit architectures of this Linux distribution are supported. Under Microsoft Windows, Plesk installation is supported on Windows Server 2003, Windows Server 2008 and Windows Server 2008R2 under both 32 and 64bit architectures.

Product website: http://www.parallels.com/products/plesk/

Snapshots

cPanel

Enkompass

Plesk for Linux

Plesk for Windows

References

Webhostingfan.com, Top Three Control Panels for Reseller Hosting

• BigBrother
• munin
• Nagios
• OpenNMS
• pingdom

BigBrother

BigBrother is a solution for IT infrastructure monitoring and diagnostics. It uses a client/server model to provide real-time monitoring for various operating systems (Windows, UNIX, Linux). A BigBrother installation is formed of the following elements:

• A web server: this server processes the status information and creates a dashboard to display your web solution status. The dashboard can be accessed through any web browser.
• A notification server: this server processes alerts and dispatches them to the correct people. Notifications can be sent out by e-mail, SMS text, alpha/numeric pages and/or SNMP traps.
• A network monitor: common network protocols are supported (http/ftp/pop3). Simple text-based protocols can be easily added to the configuration and monitored for availability.
• Local system monitors: a client must be installed on each monitored server to collect system information and send it to the web and/or notification server.

This commercial software is well-suited for mid-sized to large web solutions.

Website: http://bb4.com/

munin

Munin is a resource monitoring tool that can help analyze resource trends and diagnose performance issues on your servers. It uses a master/node architecture where the master connects to all the nodes at regular interval to collect data. Nodes can be computers, networks, SANs, etc. A piece of software needs to be installed on the nodes to allow communication with the master.

The collected information is stored in RRD files which are transformed into graphs and presented to the end-user through a web interface. Munin uses the OpenSource software RRDTool and its framework is written in Perl, while plugins may be written in any language.The emphasis of munin is on its plug and play capabilities and the ease of creating new plugins (graphs). Munin will help you determine what is different when a performance problem is detected compared to when things were working correctly. It will also help you see how your servers are in term of capacity for various resources (disk capacity, CPU, RAM, network).

This OpenSource software is well-suited for all web solutions.

Website: http://munin-monitoring.org/

Nagios

Nagios is a monitoring system to help you ensure systems, applications, services and business processes are functioning properly. It uses a client/server model to provide real-time monitoring for various operating systems (Windows, Linux, Netware) and network equipment (routers, switches). Therefore, a piece of software needs to be installed on your servers to allow Nagios to collect information about its resources. No software is required on your monitored servers if you only need to monitor publicly available services (HTTP, FTP, SSH, etc.).

In the event of a failure, Nagios can send alerts to allow you to begin remediation processes before outages affect business processes, end-users, or customers. Designed with scalability and flexibility in mind, Nagios will help you reduce the occurrence of unexpected outages.

Nagios is a popular OpenSource software well-suited for mid-sized to large web solutions.

Website: http://www.nagios.org/

OpenNMS

OpenNMS is a network management application platform with four main features:

• Automated and Directed Discovery
• Event and Notification Management
• Service Assurance
• Performance Measurement

By simply pointing OpenNMS at your network, it will discover services being provided and start monitoring them. Like Nagios, no software is required on your monitored servers if you only need to monitor publicly available services (HTTP, FTP, SSH, etc.). Devices can also be added manually to OpenNMS if you need to have fine-grained control over what is being managed.

OpenNMS can generate its own events or receive events from outside sources, such as SNMP Traps. OpenNMS which is able to handle burst of thousands of events per second, can serve as the central repository for your network event stream. Once important events have been identified, they can generate notices (email, SMS) or trouble tickets through an API allowing bidirectional communication with many trouble ticketing systems (Atlassian JIRA, Best Practical Solutions RT, Concursive ConcourseSuite – formerly Centric CRM, etc.).

The OpenNMS application allow the tracking of Service Level Agreements (SLAs) using synthetic transactions ranging from a simple ICMP request or port check, up through complex website monitoring and round trip e-mail testing. Detailed reports can also be generated on the availability of the services.

To collect performance data, OpenNMS includes support for the SNMP, JMX and HTTP protocols. The HTTP protocol can be used to retrieve data on a web page and use regular expressions to extract values in order to extend an application to integrate with this monitoring tool. Stored data can be converted into graphs and are available through OpenNMS web interface.

This OpenSource software is well-suited for mid-sized to large web solutions.

Website: http://www.opennms.org/

pingdom

Pingdom provides an uptime monitoring service. This service monitors the uptime and responsiveness of your web solution from different worldwide locations. It provides information about the availability of your web solution from the Internet perspective. No software is required on your servers to enable this service.

The service provided by Pingdom will discover any problem with your website within minutes. Different monitoring probes can be configured: web checks (HTTP and HTTPS), network checks (ping, specific TCP/UDP port, DNS) and email checks (SMTP, POP3, IMAP). Alerts can be sent through email or SMS. Detailed statistics of your website performance are also available giving you a historical record of all incidents.

This commercial service is well-suited for small to mid-sized web solutions.

Website: http://www.pingdom.com/

Conclusion

Various monitoring software and services are available to help you maintain your web solution availability. Using them will help maintain your web presence and give historical data about the performance of your web solution.

Screenshots

Here are two pictures taken during the boot process of this LiveCD. Syslinux is still used to mimic a GRUB interface at boot time. Several options are given including a graphical installation through anaconda. The default boot option will take you to the Gnome Display Manager where the an automatic login has been configured.

The Gnome desktop interface is shown in the pictures below. Gnome has been updated to version 2.28 and Firefox 3.6 is now included. Red Hat has kept a similar menu item organization compared to the previous distribution release while the newer version of Gnome provides updated tools and functionalities.

A graphical installation program is available directly from the LiveCD desktop. The different steps involved in the installation process are shown below. This installation mechanism is similar to the one available from the LiveCD boot menu. The main difference between this installer and the standard one available from the regular installation media is the lack of a package selection option. The installation mechanism of the LiveCD will copy the content of the LiveCD directly to the selected storage device. The resulting installation will therefore only include the software already available from the LiveCD.

Conclusion

The LiveCD creation process for the upcoming CentOS 6 distribution is working great. Additional features like the long awaited installer should be available.

• MySQL Cluster
• MySQL replication
• Shared storage
• Block-level replication

This book is mainly focused on the MySQL Cluster approach with four (4) entire chapters dedicated to this method. The three (3) remaining approaches are covered in dedicated single chapters while the last chapter covers performance tuning of MySQL.

The configuration examples presented in the book are rich and well explained. These examples could be used to quickly set up an highly available MySQL database from scratch. However, the inner workings of MySQL in a highly available configuration could be more detailed. The graphical representations used to illustrate the theory behind the examples are poorly designed.

Another weakness of this book is the proposed method of installation for the Multi Master Replication Manager (MMM). The author recommends the installation of source packages. While these tools seem to be a valuable resource for a multi-master replication configuration, the use of manually installed software on a production environment should be avoided. The author should provide a way to install these tools via a set of packages specifically built for the CentOS distribution.

Nonetheless, this book is a valuable resource for a novice to intermediate level MySQL database administrator in his journey toward making its database highly available.

Product Details

• Author: Alex Davies
• Paperback: 276 pages
• Publisher: Packt Publishing; 1st New edition edition (April 2010)
• Language: English
• ISBN-10: 1847199941
• ISBN-13: 978-1847199942

Goals and limitations

The backup of your website data should be considered as a process. Your goals in term of data recovery combined with the limitations of the backup process must be taken into account in order to implement an adequate backup solution.

Recovery Point Objective

The frequency at which the backup operation is done will influence the length of the time-gap in case of data recovery. For example, if you scheduled a weekly backup for your website database, recovering your data from the backup will roll you back in time for up to a week. Setting up your Recovery Point Objective (RPO) will help you schedule your backup operations correctly and size your backup storage accordingly.

Recovery Time Objective

On the other hand, the time required to restore your backup data in case of data loss is called the Recovery Time Objective (RTO). The choice of storage media, its location and the backup strategy will influence the recovery time. Security concerns should also be addressed by choosing an appropriate location for your storage media and using access restriction tools like encryption.

Constraints

Several limitations must be considered during the design of the backup process: performance impact, bandwidth requirements and costs are three important factors. The performance impact is the additional payload imposed on your website due to the backup operation. To minimize such performance impact, the backup operation can be scheduled when your website is seeing the least usage. The backup process will then have the least amount of interference with normal operations. You will also need to evaluate the bandwidth required during backup and restoration operations if the backup media is accessed through the network. In this common scenario, the recovery time objective (RTO) is directly related to the size of the bandwidth between your server and the backup media.

Cost is also a limitation and will guide your choice of a backup solution. All types of storage media have a finite capacity with a real cost. Matching the correct amount of storage capacity with your needs is an important part of the backup scheme design. On the other hand, using a simple scheme will help you avoid excessive labour costs. Finally, while choosing a commercial backup software may simplify the backup scheme, the cost of a proprietary solution can be important.

Storage media

Magnetic tape

Magnetic tape has been used for backup data storage for a long time. Tape is a proven storage medium, which is its main advantage along with its portability. A magnetic tape is a purely sequential access medium. While access time for random files is poor, performance for continuous writing or reading can be high. Magnetic tape features the lowest price/capacity ratio. However, the complexity of a tape backup system and the specialized hardware required for reading and writing to this medium are major drawbacks.

Hard disk

Hard disk is a common medium for storage solutions. Relying on commodity hardware, this storage technology features low access time, high capacity and a low price/capacity ratio. The main disadvantage of hard disk backups is that their portability is limited. They can be easily damaged during transport.

Solid state device

Flash memory, USB keys and USB flash drives are solid state devices that can be used for storage solutions. Their main disadvantage is their high price/capacity ratio. However, this medium offers portability, the lowest access time and ease of use.

Backup strategy

Full backup

A full backup is the first step in a rotation scheme. All the relevant data is copied on the backup media. Some backup software will allow you to restore a whole server after a full system backup is completed. This feature is called bare-metal restoration. Without this feature, the restoration of the backup data to the server will have to be executed after the restoration of the original operating system on the server.

Differential backup

A differential backup copies files and folders that have been created or modified since the last full backup. When performing a combination of normal and differential backups, the restoration will most likely require the last full backup along with the latest differential backup.

Incremental backup

An incremental backup copies files and folders that have been created or modified since the last full or incremental backup. When performing a combination of normal and incremental backups, the restoration will require the last full backup along with all the incremental backups taken in the meantime.

Rotation scheme

A complete backup strategy should include a rotation scheme to allow the conservation of historical data. Such a method is also useful for managing multiple media in the backup process. The rotation scheme determines how and when each piece of storage is used for a backup job and how long it is retained once it has backup data stored on it. A balance between the cost of extra storage media and the need for data retention must be found.

The most used rotation strategies are the Grand-Father/Father/Son (GFS) and the Towers of Hanoi. Both of these strategies use full and differential/incremental backups.

Location of the backup data

On-site

Storing the backup data on the same site as the backup target is convenient since there is no need to move backup media physically or through the network. Handling or bandwidth issues can be avoided with such a solution.

However, on-site storage is insecure. Since both original and backup data are on the same location, a major disaster (fire, flood, theft, etc.) could be catastrophic.

Off-site

Storing the backup data on a secondary location increases the reliability of the backup process at the expense of convenience. To reduce the required effort, data can be moved between both sites using the network instead of relying on the transportation of physical media.

Reference

• Wikipedia, backup

• Persistence feature: with livecd-tools 014-7 and later, it is possible to create a LiveUSB media featuring a persistent overlay
• OpenOffice.org 3.1.1: support for files created with Microsoft Office 2007 or Microsoft Office 2008 for Mac OS X (.docx, .xlsx, .pptx, etc)
• Improved wireless network adapter drivers: better support for Intel Wireless devices
• ext4 support: ext4 partitions of your system should now be detected automatically by the LiveCD

A size constraint of 700MB has been respected for both i386 and x86_64 iso images. Users who want to burn the LiveCD iso images to regular CD medias will be able to do so.

However, transferring the LiveCD iso image to a USB key is the preferred method. Boot time is significantly reduced on a USB key. Moreover, the persistence feature is only available for LiveUSB medias.

Customization

As usual, the CentOS LiveCD 5.5 can be customized in order to support additional features:

• NTFS support
• Additional packages
• Additional services (httpd, mysql, bind, etc.)
• Custom configurations
• Localization
• etc.

Contact

Patrice Guay, IT systems engineer
info@patriceguay.com

Available solutions

Different web hosting solutions are available. We divide them into the following categories:

• Shared web servers
• Virtual private servers (VPS)
• Dedicated web servers
• Homemade web hosting
• Colocation

Division of responsibilities

The first table below illustrates the division of responsibilities for all five web hosting solutions. The end-user layer represents the interaction between your web solution and your visitors. In all cases, you are directly responsible toward your web solution users. The application layer encompass the operating system and the application stack. Unless you are using an homemade or a colocation solution, the hosting provider will share some responsibilities with you in this aspect. The device and network layers represent the physical server and networking hardware respectively. Finally, the data center layer encompass the electricity, cooling, access control and fire protection of the hardware equipment.

Strengths and weaknesses

This second table is a summary of the strengths and weaknesses of the different web hosting solutions.

A shared web server is generally a slice of a physical server along with a control panel to simplify the management of your web solution. It is available at low cost but will offer low performance since resource allocation is minimal. Uptime is also below average and scalability is not available.

A virtual private server (VPS) is similar to a shared web server since it is sitting on an actual physical server. However, most of the allocated resources of the VPS are determined at purchase time and will determine its cost. Uptime and scalability are generally good.

A dedicated web server gives you the complete resource ownership of a physical server. The cost of a dedicated server is high especially for low-end servers compared to VPS offerings. Uptime is generally good and scalability is excellent with such a solution.

Building your web solution with your own equipment and network connectivity could save you a lot of money on the long term. While uptime may be lacking since your infrastructure is likely less sophisticated than the one offered by web hosting companies, resource allocation and scalability are excellent.

The colocation option is the intermediate solution between a dedicated web server and an homemade solution. The cost is high but resource allocation, uptime and scalability are excellent.

Reference: Charles Hotchkin (RackSpace), Managed Colocation