Part I of this posting, walked through the steps to create a iSCSI target and LUN on a QNAP server. Part II now considers provisioning guests using iSCSI storage from virt-manager.
Storage pool management
After launching virt-manager and connecting to the desired hypervisor, QEMU/KVM in my case, the first step is to open up the storage management pane. From the main virt-manager window, this can be found by selecting the menu “Edit -> Host Details” and then navigating to the “Storage” tab. The host shown here already has two storage pools configured, both pointing at local filesystem directories. The “default” storage pool will usually be visible on all libvirt hosts managed by virt-manager and lives in /var/lib/libvirt/images. This isn’t much use if you plan to migrate guests between machines, because some form of shared storage is required between the hosts. This is where iSCSI comes into the equation. To add a iSCSI storage pool, click the “+” button in the bottom-left of the window
Storage pool list
Adding a storage pool
The first stage of setting up a storage pool is to provide a short name and select the type of storage to be accessed. For sake of consistency this example gives the storage pool the same name as the iSCSI target previously configured on the QNAP.
Storage pool creation
Entering iSCSI parameters
The “Target path” determines how libvirt will expose device paths for the pool. Paths like /dev/sda, /dev/sdb, etc are not a good choice because they are not stable across reboots, or across machines in a cluster, the names are assigned on a first come, first served basis by the kernel. Thus virt-manager helpfully suggests that you use paths under “/dev/disk/by-path”. This results in a naming scheme that will be stable across all machines. The “Host Name” is simply the fully qualified DNS name of the iSCSI server. Finally the “Source Path” is that adorable IQN seen earlier when creating the iSCSI target (“iqn.2004-04.com.qnap:ts-439proii:iscsi.kvmguests.bf6d84“)
iSCSI storage pool parameters
Browsing iSCSI LUNs
If all went to plan, libvirt connected to the iSCSI server and imported the LUNs associated with the iSCSI target specified in the wizard. Selecting the new storage pool, it should be possible to see the LUNs and their sizes. These steps can be repeated on other hosts if the intention is to migrate guests between machines. Obviously care should be taken to not run the same VM on two machines at once though. Ideally use clustering software to protect against this scenario.
iSCSI LUN browsing
Selecting storage for a new VM
With the iSCSI storage pool configured in virt-manager, it is now possible to create a guest. After breezing through the first couple of steps in the “New VM wizard”, it is time to specify what storage to use for the new guest. By default virt-manager will allocate storage from the local filesystem. This isn’t what we want now, so go for the “Select managed or other existing storage” option and hit the “Browse” button
New guest storage selection
Browsing iSCSI LUNs for the new VM
The dialog that has appeared shows all the storage pools this libvirt connection knows about. This should match the pools seen a short while ago when configuring the iSCSI storage pool. It should be fairly obvious what todo at this point, select the iSCSI storage pool and the desired LUN (volume) within it and press “Choose Volume”
iSCSI LUN selection
Continuing the new VM wizard
After choosing an iSCSI LUN, its path should now be displayed. It’ll be a rather long and scary looking path under /dev/disk/by-path, don’t stare at it for too long. Just continue with the rest of the new VM wizard in the normal manner and the guest will shortly be up & running using iSCSI storage for its virtual disk.
New guest storage path
Final words
Hopefully this quick walkthrough has shown that provisioning KVM guests on Fedora 12 using iSCSI is as easy as 1..2..3.. The hard bit is probably going to be on your iSCSI server if you don’t have a NAS with a nice administrative interface like the QNAP’s. The libvirt storage pool management architecture includes support for actually constructing storage pools & allocating LUNs. For a LVM storage pool libvirt would do this using vgcreate and lvcreate respectively. The iSCSI protocol standard does not support these kinds of operations but many vendors provide ways todo this with custom APIs, for example, Dell EqualLogic iSCSI arrays have a SSH command shell that can be used for LUN creation/deletion. It would be desirable to add support for these vendor specific APIs for LUN creation/deletion in libvirt. It could even be possible to support iSCSI target creation from libvirt if suitable APIs were available. This would dramatically simplify the steps required to provision new guests on iSCSI by enabling everything to be done from virt-manager, without the need to touch the NAS admin interfaces.
I recently got a rather nice QNAP NAS to replace my crufty, cobbled together home fileserver. Unpack the NAS, insert 4 SATA disks and turn it on. The first time it starts up, it prompts to set up RAID5 across the disks by default (overridable to any other RAID config or no RAID) and after a few minutes formatting the disks, it is up and running ready todo work. It exposes storage shares via all commonly used protocols such as SAMBA, NFS, FTP, SSH/SCP, WebDAV, Web UI, and most interestingly to this post, iSCSI. The QNAP marketing material is strongly pushing iSCSI as a solution for use in combination with VMWare but here in Fedora & the open source virt world it is KVM we’re interested in. libvirt has a fairly generic set of APIs for managing storage which allows impls for many different types of storage, including iSCSI. The virt-manager app provides a UI for nearly all of the libvirt storage capabilities, again including iSCSI. This blog posting will graphically illustrate how to deploy a new guest on Fedora 12 using virt-manager and the QNAP iSCSI service.
iSCSI Service Enablement
In the QNAP administration UI, navigate to the “Disk Management -> iSCSI” panel. Ensure the iSCSI Target Service is enabled and running on the default TCP port 3260. libvirt currently has no need for iSNS, so that can be left disabled unless you have other reasons for requiring its use.
iSCSI Service Enablement
iSCSI Targets
An iSCSI target is simply a means to group a set of LUNs (storage volumes). There are many ways to organize your targets + LUNs, but to keep things simple I’m going to keep all my KVM guest storage volumes in a single target and enable this target on all my virtualization hosts. Some people like the other extreme of one target per guest, and only enabling the guest’s target on the host currently running the guest. In this screengrab I’ve already got one iSCSI target for other experimentation and intend to use the ‘Quick Configuration Wizard’ to add a new one for my KVM guests.
iSCSI target list
iSCSI Target + LUN creation
Since this is my first guest, I need to create both he iSCSI target and a LUN. For future guests I can create the iSCSI LUN only and associate it with the previously created target.
iSCSI target/LUN creation wizard
Setting the iSCSI target name
Every iSCSI target needs a unique identifier, known as the IQN. For reasons I don’t want to understand, the typical IQN format is a rather unpleasant looking string, but fortuitously the QNAP admin UI only expects you to enter a short name, which it then uses to form the full IQN. In this example I’m giving my new iSCSI target the name “kvmguests” which results in the adorable IQN “iqn.2004-04.com.qnap:ts-439proii:iscsi.kvmguests.bf6d84“. Remember this IQN, you’ll need it later in virt-manager.
iSCSI Target naming
iSCSI target authentication
iSCSI includes support for an authentication scheme known as CHAP. Unfortunately libvirt does not yet support configuration of iSCSI storage using CHAP, so this has to be left disabled. This sucks if you’re using a shared/untrusted local network between your virt hosts & iSCSI server, but if you’ve got a separate network or VLAN dedicated to storage traffic this isn’t so much of a problem. I also don’t care for home usage.
iSCSI target authentication
iSCSI LUN allocation
Now the iSCSI target is configured, it is time to allocate storage for the LUN. This is what will provide the guest’s virtual disk. Each LUN configured via the QNAP admin UI ends up being backed by a plain file on the NAS’ (ext4) filesystem. If you choose the “Thin Provisioning” option at this stage, the LUN’s backing file will be a sparse file with no storage allocated upfront. It will grow on demand as data is written to it. This allows you to over-commit allocation of storage on the NAS, with the expectation that should you actually reach the limit on the NAS some time later, you can simply swap in larger disks to the RAID array or attach some extra external SATA devices. “Instant allocation” meanwhile, fully allocates the LUN at time of creation so there is never a problem of running out of disk space at runtime. The LUN name is just an aid-memoir for the admin. I name my LUNs to match the KVM guest name, so this one is called “rhel6x86_64”. The size of 10 GB will be more than enough for the basic RHEL6 install I plan.
iSCSI LUN allocation
iSCSI setup confirmation
Before actually creating the iSCSI target and LUN, the QNAP wizard allows a chance to review the configuration choices just made.
iSCSI target + LUN creation summary
iSCSI target list (updated)
After completing the wizard, the browser returns to the list of iSCSI targets. The newly created target is visible, with one LUN beneath it. The status of ‘Processing’ just means that storage for the LUN is still being allocated & goes away pretty much immediately for LUNs using ‘Thin provisioning’ since there’s no data to allocate upfront. In theory full allocation should be pretty much instantaneous on ext4 too, if QNAP is using posix_fallocate, but I’ve not had time to check that yet.
iSCSI target list (updated)
With the iSCSI target and LUN created, it is now time to provision a new KVM guest using this storage. This is the topic of Part II