VHACS-VM

The Linux SCSI Target Wiki

Jump to: navigation, search
VHACS
Logo
Storage Cloud Controller
Original author(s) Jerome Martin
Developer(s) Datera, Inc.
Initial release June 26, 2008 (2008-06-26)
Preview release 0.8.15 / August 25, 2008;
5 years ago
 (2008-08-25)
Development status Deprecated
Written in Python
Operating system Linux
Type Cloud controller
License GNU GPLv2
Website datera.io
VHACS-VM across two physical x86_64 nodes, one on Linux x86_64 and one on Windows32 with 12 active 2GB storage clouds.
VHACS-VM v0.8.15 running eight 1 GB Storage Clouds on x86_64 on a single physical Linux x86_64 Machine.
VHACS-VM Alpha x86_64 Demo in action with eight active VHACS storage clouds on a Debian virtualized host.

VHACS-VM is an Alpha release of the VHACS high availability storage cloud stack running in a virtualized x86_64 environment. VHACS is a highly available storage cloud based on at least eight open source technologies running on Linux v2.6.25.

The VHACS-VM images are available, and oriented towards kernel developers, storage and network systems researchers, C programmers, Python and Shell programmers, and anyone who wants to run and see a real life open storage cloud on their virtualized x86 PC. Using VHACS-VM is the fastest and easiest method for users to get involved with the project. This is a developer-oriented release, and a complete development environment is included in the images.

Currently, VHACS can manage 4-8 storage clouds as long as the network bridge setup is working as expected (see below).

Contents

Current setup

There are two (vhacs64-west and vhacs64-east) x86_64 VMware v6 compatible images, and there is one build .vmdk containing the source for the VHACS Alpha. The build disk image is distributed seperately in order to reduce the size for obtaining a two node VHACS-VM cloud of your own.

# ls -al
-rw-r--r-- 1 root root 830805854 Jul 16 19:57 VHACS64-BUILD-0.8.15.zip
-rw-r--r-- 1 root root 314996216 Jul 16 19:56 VHACS64-East-0.8.15-2.zip
-rw-r--r-- 1 root root 299372440 Jul 16 19:56 VHACS64-West-0.8.15-2.zip

# cat md5sum.txt 
e1923fbea84b2e91739ca1199e4695f4  VHACS64-BUILD-0.8.15.zip
32bc1e16c1961c3fb3e66ea52976060a  VHACS64-East-0.8.15-2.zip
160e3cdee8344faedf148c5eb845ed2c  VHACS64-West-0.8.15-2.zip

Each image has a Debian Etch v4.0 r3 x86_64 installation. The Linux kernel is v2.6.25.9 (see kernel.org) with KDB patches. The VHACS-VM kernel is also configured with ParaVirt-Ops for the least amount of virtualized execution overhead on modern PC hardware.

The .vmx images are configured with:

Requirements

A x86_64 CPU with hardware x86 virtualization is required, i.e. the CPU has a virtualized memory management unit (MMU) capable of handling multiple page tables from multiple different unmodified operating systems.

Hardware x86 virtualization is the easiest way to get a high-performance open-source storage cloud running on recent x86_64 CPUs quickly (for the purpose of the Alpha preview) - VHACS itself doesn't require x86 hardware virtualization, of course.

Setup

/etc/hosts

Based on your DHCP configured IP address (obtained with ifconfig eth0), set the vhacs64-west.linux-iscsi.org and vhacs64-east.linux-iscsi.org hostnames to your locally allocated IP addresses on both vhacs64-west and vhacs64-east VM images.

/etc/vhacs.conf

Configure the storage network and mask:

STORAGE_NETWORK = "192.168.0.0"
STORAGE_NETMASK = "24"

Configure the range of IP addresses that are assigned to VHACS cloud from the resource pool.

WARNING: Make sure this range is either explictly not in use by your DHCP server, or won't have an affect on your active network.

FIRST_STORAGE_IP = "192.168.0.50"
LAST_STORAGE_IP = "192.168.0.99"

NIC configuration

VHACS-VM can be configured to use different STORAGE_IFNAME and HEARTBEAT_IFNAME interfaces. For instance, this consists of:

The current setup using two network bridges looks something like:

HOST0: 192.168.0.*/ethX via DHCP bridged into vhacs64-east VM
HOST1: 192.168.0.*/ethX via DHCP bridged into vhacs64-west VM
vhacs64-east: 192.168.0.*/eth0 via DHCP
vhacs64-west: 192.168.0.*/eth0 via DHCP
HOST0: 10.10.0.5/ethY bridged into vhacs64-east VM 
HOST1: 10.10.0.10/ethY bridged into vhacs64-west VM
vhacs64-east: 10.10.0.15/eth1 via static IP
vhacs64-west: 10.10.0.20/eth1 via static IP

Also, you need to change your /etc/ais/openais.conf to the 10.10.0.0 network for broadcast packets (example).

Results

A correctly running VHACS setup with working cluster communication should produce the following crm_mon -1 output after booting:

vhacs64-west:~# crm_mon -1

============
Last updated: Thu Jul  3 00:38:29 2008
Current DC: vhacs64-east (vhacs64-east)
2 Nodes configured.
0 Resources configured.
============

Node: vhacs64-west (vhacs64-west): online
Node: vhacs64-east (vhacs64-east): online

Problems

Failure to communicate

If cluster communication somehow cannot be established between nodes, try calling the VHACS cluster initialization logic to restart services on both vhacs64-west and vhacs64-east images:

vhacs cluster -I vhacs64-west,vhacd64-east

Again, double check that the hostnames vhacs64-west and vhacs64-east in both VHACS-VM images /etc/hosts files match each VHACS-VM images' individual ifconfig output.

Failure to join

If crm_mon -1 only shows the local node on each VHACS-VM image, it is an indication that the UDP broadcast packets for cluster communciation didn't come up cleanly or got wedged.

For reference purposes, the VHACS-VM prototype was a 2.2 Ghz Core 2 Duo with 1 GB of DDR2-800 running Debian Etch i386. The local bridged Ethernet interface is an on-board Realtek RTL8111/8168B PCI-Express 1 Gb/sec MAC.

Note that not all ethernet interfaces support bridging sufficently to run VHACS-VM. We still need to determine how to easily find the networking hardware that can be used with the various virtualization hypervisors with VHACS-VM images.

Launching the storage cloud

Now that active cluster communication between the two VHACS nodes (vhacs64-west and vhacs64-east) is established, the storage cloud can be launched as follows.

Assigning roles

Roles can be assigned to the nodes in the VHACS cluster as follows:

vhacs64-west:~# vhacs node -s ALL vhacs64-west,vhacs64-east
role vhost now set on node vhacs64-west
role storage now set on node vhacs64-west
role vhost now set on node vhacs64-east
role storage now set on node vhacs64-east

Allocating the storage cloud

The VHACS storage cloud can then be allocated as follows:

vhacs64-west:~# vhacs storage -c vmcloud0 -s 200M
storage vmcloud0 has been assigned IP 192.168.0.50
LVM volume vmcloud0 created on vhacs64-west
LVM volume vmcloud0 created on vhacs64-east
Promoting node vhacs64-west
Waiting until sync is finished (be patient) ...
drbd devices configured and formated as ext3 on vhacs64-west and vhacs64-east
master/slave drbd resource created and constrained for storage vmcloud0
iSCSI target iqn.2003-01.org.linux-iscsi:192.168.0.50 created and constrained for storage vmcloud0
iSCSI filesystem for iqn.2003-01.org.linux-iscsi:192.168.0.50 created and constrained for storage vmcloud0

Monitoring

The storage cloud can be monitord by opening up another ssh connection to one of the VHACS-VM nodes, and running the VHACS monitor. The following reflects the simple configuration descibed here:

vhacs64-east:~# vhacs cluster -m
[00:56:00] [vhacs64-east] [VHACS cluster storage resources status]
 ____________________________________________________________________________________________________________________________________________________________________
|                                |                                |                                |                                |                                |
| NODE                           | HA STATUS                      | FREE STORAGE                   | STORAGE ROLE                   | VHOST ROLE                     |
|________________________________|________________________________|________________________________|________________________________|________________________________|
|                                |                                |                                |                                |                                |
| (A)vhacs64-west                | online                         | 39.80G/39.99G                  | 1 exported                     | 0 mounted                      |
| (A)vhacs64-east                | online                         | 39.80G/39.99G                  | 0 exported                     | 1 mounted                      |
|________________________________|________________________________|________________________________|________________________________|________________________________|
 ____________________________________________________________________________________________________________________________________________________________________
|                                |                                |                                |                                |                                |
| STORAGE                        | PRI/SEC DRBD                   | DRBD TARGET                    | ISCSI MOUNT                    | FREE SPACE                     |
|________________________________|________________________________|________________________________|________________________________|________________________________|
|                                |                                |                                |                                |                                |
| (A)vmcloud0                    | (P)vhacs64-west/(S)vhacs64-east| (S)vhacs64-west                | (S)vhacs64-east                | 179M/194M (96%)                |
|________________________________|________________________________|________________________________|________________________________|________________________________|

A more complex VHACS storage cloud with four volumes could look as follows:

vhacs64-east:~# vhacs cluster -M
 _________________________________________________________________________________________________________________________________________________________________________
|                                 |                                 |                                 |                                 |                                 |
| NODE                            | HA STATUS                       | FREE STORAGE                    | STORAGE ROLE                    | VHOST ROLE                      |
|_________________________________|_________________________________|_________________________________|_________________________________|_________________________________|
|                                 |                                 |                                 |                                 |                                 |
|                                 |                                 |                                 |                                 |                                 |
| (A)vhacs64-west                 | online                          | 39.21G/39.99G                   | 3 exported                      | 2 mounted                       |
| (A)vhacs64-east                 | online                          | 39.21G/39.99G                   | 1 exported                      | 2 mounted                       |
|_________________________________|_________________________________|_________________________________|_________________________________|_________________________________|
 _________________________________________________________________________________________________________________________________________________________________________
|                                 |                                 |                                 |                                 |                                 |
| NODE                            | WHEN                            | OPERATION TYPE                  | RESOURCE                        | RETURN CODE                     |
|_________________________________|_________________________________|_________________________________|_________________________________|_________________________________|
|                                 |                                 |                                 |                                 |                                 |
| vhacs64-west                    | 2h30m28s ago                    | initial start                   | ms_drbd_vmcloud1_CLONE:0        | ERROR                           |
|_________________________________|_________________________________|_________________________________|_________________________________|_________________________________|
 _________________________________________________________________________________________________________________________________________________________________________
|                                 |                                 |                                 |                                 |                                 |
| STORAGE                         | PRI/SEC DRBD                    | DRBD TARGET                     | ISCSI MOUNT                     | FREE SPACE                      |
|_________________________________|_________________________________|_________________________________|_________________________________|_________________________________|
|                                 |                                 |                                 |                                 |                                 |
| (A)vmcloud0                     | (P)vhacs64-west/(S)vhacs64-east | (S)vhacs64-west                 | (S)vhacs64-east                 | 179M/194M (96%)                 |
| (A)vmcloud1                     | stopped/(P)vhacs64-east         | (S)vhacs64-east                 | (S)vhacs64-west                 | 179M/194M (96%)                 |
| (A)vmcloud2                     | (P)vhacs64-west/(S)vhacs64-east | (S)vhacs64-west                 | (S)vhacs64-west                 | 179M/194M (96%)                 |
| (A)vmcloud3                     | (S)vhacs64-east/(P)vhacs64-west | (S)vhacs64-west                 | (S)vhacs64-east                 | 179M/194M (96%)                 |
|_________________________________|_________________________________|_________________________________|_________________________________|_________________________________|

Trouble shooting

As this is a very developer oriented Alpha preview, so you can help us find bugs! Here is the 10,000 ft view:

VHACS kernel/systems developer

The included v2.6.25.9 x86_64 kernel is patched with SGI Kernel Debugger. At any time, you can break into the kernel debugger with the good old' PC key 'Pause/Break'. Note that not all laptops have this key, so you may have to use a USB keyboard to access KDB.

Cluster infrastructure developer

The cluster stack (see VHACS for the stack components) is enabled with verbose debug. You can view the complete logs with tail -f /var/log/syslog.

VHACS systems administator

vhacs cluster -m is the primary status monitor (see above). Exceptions related to cluster RAs will appear as they occur.

See also

External links

Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox
Google AdSense