Open up the I/O turnstiles by setting the queue depth.
When you're facing performance bottlenecks, you'll often find the I/O layer is the culprit. If you're using a Storage Area Network (SAN), it's natural to look at the SAN configuration. You might focus on RAID levels, the layout of data on the disks, and the speed of the disks themselves; however, even with a fast SAN, you can still experience slow I/O because of bottlenecks at the operating system level.
There are a few parameters on AIX which are worth looking at to ease I/O congestion, but the one I've found gives the most quick wins is the queue depth parameter. I once saw a company brought to its knees by slow I/O response times, especially during busy periods such as month end. Hundreds of users were affected. Although there were application and database tweaks that might have helped, the big fix was upping the queue depth. It literally fixed the I/O problem overnight.
No Time to Wait
On AIX, disks—whether they are physical or virtual—are called Physical Volumes (PVs) and they can be added to a Volume Group (VG). Before adding new PVs to a VG, it's important to set the queue depth. The queue depth allows you to submit more than one I/O to a disk at a time. If your queue depth is too small, I/O requests will queue up.
The queue depth is set for each PV, which is why a large LUN with a small queue depth can sometimes be slower than several smaller LUNs, even if the SAN subsystem configuration is in other ways identical.
Determining the queue depth can seem a bit of a black art, but there are some helpful guidelines in the IBM Technote “AIX disk queue depth tuning for performance.” There's a link to that document in the Resources section at the end of this article.
You can view the queue depth for a PV with the command:
Setting the queue depth is easy if the disk is not in an active volume group. In this example, hdisk7 will have its queue depth set to 20:
If you get an error because the disk is busy, you can add the -P flag at the end of the chdev command to make the change permanent. The new setting will become active the next time the volume group is deactivated and activated again, which will probably end up being the next time you reboot.
Remember the VIOS
It's very common these days to assign SAN disk to a Virtual I/O Server (VIOS), and then use the VIOS mkvdev command to create a virtual SCSI disk for the VIO client. If that's how your configuration looks, then the first step after you assign the SAN LUN to the VIOS should be to set the queue depth on the VIOS disk itself. You should do this before you create the virtual SCSI disk for the client—for example, an AIX logical partition—to use.
Here's how to set the queue depth from the VIOS restricted shell (that's when you're logged in as padmin):
Once again, if the disk is in use—perhaps because it's already been mapped to a Virtual SCSI adapter—you might have to make the change permanent. On the VIOS restricted shell, you can do that using the -perm flag.
However, this change will only take effect next time the VIOS is rebooted.
If you're mapping a SAN LUN to two VIO servers, you'll need to set the reserve policy as well. This allows you to present the LUN to the VIO client through two different paths (one from each VIO server).
Once these settings are changed on the VIOS' hdisk, you can map the disk to the Virtual SCSI adapter on each VIOS. Once again, this is done on the VIOS restricted shell. Here I'll map it to vhost3, which corresponds to the Virtual SCSI adapter for the client logical partition on my system.
The -dev parameter at the end is optional, but I prefer to use it to help identify the disk by the name of the client it's assigned to. That way, the VIOS lsmap command will be more readable.
The article “The VIO cheat sheet” gives a quick rundown of some commonly used VIOS commands (also see the Find Out More section below).
And Finally, the VIO Client
On the VIO client, which in this case is an AIX logical partition, the queue depth should also be set. It should match the queue depth setting for the disk on the VIO server (or servers).
Setting the queue depth is a little like opening up some turnstiles at the grandstand. A higher queue depth can let more I/O requests to the SAN, and as long as the SAN infrastructure is able to manage those requests, the queue depth tweak can be a quick and simple way of relieving I/O contention.
Find Out More
The Power Systems Virtualisation from IBM wiki (This wiki has some very helpful tips on tuning virtualized environments. The Session 5: Virtualization Best Practices webinar includes some important tuning parameters for adapters and hdisks.)