This page is part of my virtualization context i.e. from my point of view talking/doing virtualization includes

• the OS (Operating System) part e.g. OpenVZ, Xen, Linux-VServer, VMware, KVM, etc. and
• the storage part e.g. LVM (Logical Volume Manager), a world-class solution for doing storage virtualization.

Ohhhhh my Goooooood ... it is virtual ... does it breathe?

A Glance on Xen

This section should provide the reader with brief information in order to get a notion about Xen. In addition, one might also skim over the Wikipedia page on Xen — I consider this a good idea in general ...

Lean and Secure

Xen is exceptionally lean — less than 50,000 lines of code. That translates to extremely low overhead and near-native performance for guests. Xen re-uses existing device drivers (both closed and open source) from Linux, making device management easy. Moreover Xen is robust to device driver failure and protects both guests and the hypervisor from faulty or malicious drivers.

One of the Highest Performing Virtualization Software

Enterprises are embracing Xen for server virtualization because it enables them to increase server utilization, consolidate servers, and dramatically reduce complexity and overall TCO (Total Cost of Ownership). Xen is one of the fastest and most secure virtualization software solutions available today, enabling every server to support multiple virtual servers each with resource guarantees to ensure that its application layer SLA (Service Level Agreement) is met.

Xen's paravirtualization technology is widely acknowledged as one of the fastest and most secure virtualization software approaches to virtualization in the industry. Xen offers near-native performance for virtual servers with up to 10 times less overhead than proprietary offerings, and benchmarked overhead of well under 5% in most cases compared to 35% or higher overhead rates for other virtualization technologies.

With regards to performance, one might also take a look at the Linux-VServer project — it has native performance but is overall seen more as a isolation/virtualization approach to virtualization.

Server Virtualization with the Xen Hypervisor

With Xen virtualization, a thin software layer known as the Xen hypervisor is inserted between the server's hardware and the operating system. This provides an abstraction layer that allows each physical server to run one or more virtual servers, effectively decoupling the operating system and its applications from the underlying physical server.

Resource Independence — Run Applications on Any Server

Once a virtual server image has been created it can run on any server, at any time, and multiple virtual servers can simultaneously share a single physical server, increasing its utilization while receiving a resource guarantee that ensures that application layer performance criteria are met.

Virtual servers are hardware agnostic. One key feature of a virtual server is the independence from the actual hardware it is running on. Most hardware issues are irrelevant for a virtual server installation.

The host environment (the Xen hypervisor, its services, file-system etc.) acts as a host and takes care of all the hardware related details. The virtual server is just a guest and can ignore all the hardware details. As such, the guest can be migrated to another physical server running a Xen host environment with very few manipulations -– for example, imagine, migrating a whole web presence (web server plus data, database server plus data, eventually a mail system with its data etc.) from one hardware to another becomes almost as easy as moving a file from one physical server to another.

To migrate the virtual server with all its services and data inside from one physical machine to another, it is sufficient to do the following:

• Shutdown the running virtual server that should be migrated
• Copy it over to the new machine already running a Xen host environment
• Copy the configuration
• Start the virtual server on the new machine

Note: It is also possible to perform a so called Live Migration if certain requirements are meet which means no shutdown is necessary thus the whole process becomes fully transparent to the outside world e.g. a visitor on a website will not notice that, as he is surfing the site, it is migrated to another physical server.

No adjustments to user setup, password database or hardware configuration are required, as long as both machines are binary compatible (e.g. both are x86-64). Thus, once one has found that a project is using more resources than expected, he can easily migrate it to another, yet more powerful hardware without tinkering around in hardware configuration files.

Xen is Fee Software

The Xen hypervisor is a unique open source technology, developed collaboratively by a community of volunteers around the globe and software engineers at over 20 of the most innovative data center solution vendors, including Intel, AMD, Cisco, Dell, Egenera, HP, IBM, Mellanox, Network Appliance, Novell, Red Hat, SGI, Sun, Unisys, Veritas, Voltaire, and of course, XenSource. Xen is licensed under the GNU General Public License (GPLv3).

Xen is available free for download in source code, binary packages or even better via the package management system of numerous Linux distributions e.g. Debian offers it via its world-class package management system called APT (Advanced Packaging Tool). XenSource also provides simple tools for the community to download, install, test and develop using Xen.

Greater Flexibility and Lower TCO

Xen enables IT (Information Technology) staff to plan in long-term endeavours to increase utilization of server resources, achieve server consolidation, scale their test & development environments, and achieve greater business continuity through dynamic provisioning. Thus increasing flexibility and reducing TCO (Total Cost of Ownership).

From my personal point of view, the fact that I can steer Xen with IBM Director and IBM Virtualization Manager on IBM's BladeCenter makes this even more attractive.

Fastest Live Relocation

Xen's live relocation capability, combined with its extraordinary performance, have led analysts to pronounce it the first virtualization technology that is capable of being deployed in the heart of the data-center, bringing the benefits of server consolidation and increased utilization to the vast majority of servers in the enterprise.

Utility Based Computing

Xen is a key enabler in enterprise's aspirations towards a utility-based computing model, in which any server can run any operating system and any application, at any time, with dynamic load balancing of resources both within each server, between the guests, and across servers.

Just imagine one has a single application running within a single domU, next to several other domUs on a physical box. Then, as the application increases in performance demands, he either moves this domU to a dedicated box or makes it the only domU running on a dedicated box. If, after all that is not enough or the performance demand for this application is steadily increasing (e.g. a website with daily increasing visitor counts), one can simply stretch this domU running the application inside it across more then one physical box ... in essence, this is known as load balancing.

Per VM Resource Guarantees

Xen provides resource partitioning/limiting capabilities, for CPU, memory, storage and network I/O resources. This resource protection model leads to improved security because guests and drivers are DoS-proof. Xen is fully open to scrutiny by the security community and its security is continuously tested. Xen is also the foundation for a Multi-Level Secure system architecture being developed by XenSource, IBM and Intel.

An Emerging Open Industry Standard

Xen enjoys extraordinary community support. It is a de-facto, industry endorsed open source virtualization standard backed by the industry's leading enterprise solution vendors. I already mentioned IBM's BladeCenter above.

History

Xen was originally developed by the Systems Research Group at the University of Cambridge Computer Laboratory as part of the XenoServers project, funded by the UK-EPSRC.

XenoServers aim to provide a public infrastructure for global distributed computing. Xen plays a key part in that, allowing one to efficiently partition a single machine to enable multiple independent clients to run their operating systems and applications in an environment. This environment provides protection, resource isolation and accounting. The project web page contains further information along with pointers to papers and technical reports: http://www.cl.cam.ac.uk/xeno

Xen has grown into a fully-fledged project in its own right, enabling us to investigate interesting research issues regarding the best techniques for virtualizing resources such as the CPU, memory, disk and network. Project contributors now include XenSource, Intel, IBM, HP, AMD, Novell, Red Hat.

Xen was first described in a paper presented at SOSP in 2003¹, and the first public release (1.0) was made that October. Since then, Xen has significantly matured and is now used in production scenarios on many sites.

Structure of a Xen-Based System

Xen is a hypervisor that runs directly on the system hardware. Xen inserts a virtualization layer between the system hardware and the virtual machines, turning the system hardware into a pool of logical computing resources that Xen can dynamically allocate to any guest operating system. The operating systems running in virtual machines interact with the virtual resources as if they were physical resources. A Xen system has multiple layers, the lowest and most privileged of which is Xen itself.

Xen may host multiple guest operating systems, each of which is executed within a secure virtual machine. In Xen terminology, a domain. Domains are scheduled by Xen to make effective use of the available physical CPUs. Each guest OS manages its own applications. This management includes the responsibility of scheduling each application within the time allotted to the virtual machine by Xen.

The above figure shows a system with Xen running virtual machines. Xen is running three virtual machines. Each virtual machine is running a guest operating system (Linux in this case but others are possible as well) and applications independent of the other virtual machines, while sharing the same physical resources.

When we are talking about virtualization, a domain is one of the virtual machines that run on the system. Domain0 is the first domain started by the Xen hypervisor at boot, and will be running a Linux OS (Operating System). This domain is privileged — it may access the hardware and can run the XenControlTools that manage other domains. These other domains are referred to as domUs, the U standing for user. They are unprivileged, and could be running any operating system that has been ported to Xen respectively if the hostsystem's CPU (Central Processing Unit) has support for AMD's Pacifica or Intel's Vanderpool technology, then unmodified guest OSs can be run on top dom0 inside a domU.

dom 0

XenLinux is the Linux kernel with patches applied so that it will run on the virtual architecture presented by the Xen hypervisor rather than on real hardware. This means it can serve as the kernel of a domain, but it will no longer run natively on a machine. Although other operating systems have been ported to run on Xen, Linux is the only one that has been given the extra functionality needed to run in dom0.

The first domain (also known as domain 0 or dom0 for short), is the first domain started by the Xen hypervisor on boot. It has special privileges, like being able to cause new domains to start, and being able to access the hardware directly. It also performs administrative tasks such as suspending, resuming and migrating other virtual machines (domUs). Domain 0 builds other domains and manages their virtual devices. Unless DriverDomains are being used, it is responsible for running all of the device drivers for the hardware. For hardware that is made available to other domains, like network interfaces and disks, it will run the BackendDriver, which multiplexes and forwards to the hardware requests from the FrontendDriver in each domU.

Although any operating system can be ported to run on Xen as a domU, only Linux has been given the tools and kernel patches necessary to run in dom0. To compile a Linux kernel for dom0, one must configure in CONFIG_XEN_PRIVILEGED_GUEST as well as the drivers for all of his hardware. A kernel that has been compiled with these options can run in either a dom0 or a domU. A kernel without will be smaller but can only run in a domU.

domU

domU is the counterpart to dom0. It is an unprivileged domain with (by default) no access to the hardware. It must run a FrontendDriver for multiplexed hardware it wishes to share with other domains. A domU is started by xend in dom0, which the user accesses with the xm command-line tool. The kernel for a domU comes from dom0s filesystem, not from the filesystem exported to the domU

DriverDomain

Hardware drivers are the most failure-prone part of an operating system. It would be good for safety if we could isolate a driver from the rest of the system so that, when it fails, it could just be restarted without affecting the rest of the machine. Driver domains allow this.

A driver domain is a dom0 with the backend driver part plus actual hardware driver. It has has been given responsibility for a particular piece of hardware. It runs a minimal kernel with only that hardware driver and the BackendDriver for that device class.

Thus, if the hardware driver fails, the other domains (including dom0) will survive and, when the driver domain is restarted, will be able to use the hardware again.

BackendDriver

To allow unprivileged domUs to share hardware, dom0 must give them an interface by which to make requests for access to the hardware. This is accomplished by using a BackendDriver. The BackendDriver runs in dom0 or a DriverDomain and communicates with FrontendDrivers via XenBus, XenStore, and shared memory pages. It queues requests from domUs and relays them to the real hardware driver.

xend

Within domain 0, a process called xend runs to manage the system. Xend is responsible for managing virtual machines and providing access to their consoles. Commands are issued to xend over an HTTP interface, via a command-line tool. Xend is written in the programming language python.

XenContolTools

There is a dedicated section to them.

Xen offloads most of the hardware support issues to the guest OS running in the domain 0 management virtual machine. Xen itself contains only the code required to detect and start secondary processors, set up interrupt routing, and perform PCI (Peripheral Component Interconnect) bus enumeration. Device drivers run within a privileged guest OS rather than within Xen itself. This approach provides compatibility with the majority of device hardware supported by Linux.

Random Knowledge surrounding Xen

Hypervisor and Kernel

TLS Libraries

Users of the XenLinux 2.6 kernel should disable Thread Local Storage (TLS) (e.g. by doing a mv /lib/tls /lib/tls.disabled) before attempting to boot a XenLinux kernel^2.4. One can always reenable TLS by restoring the directory to its original location (i.e. mv /lib/tls.disabled /lib/tls).

The reason for this is that the current TLS implementation uses segmentation in a way that is not permissible under Xen. If TLS is not disabled, an emulation mode is used within Xen which reduces performance substantially. To ensure full performance, one should install a `Xen-friendly' (nosegneg) version of the library.

Xen Tools

Install and Configure Xen

Networking

Routing vs. Switching vs. Bridging

Types of Networking with Xen

Xen provides for three basic approaches to do networking

• Via routing or
• Bridging or
• NAT (Network Address Translation)

However, intermixed setups are possible.

Firewall / DMZ

Storage

Resource Limits

RAM (Random Access Memory)

CPU Scheduling