QuantaStor Troubleshooting Guide
Contents
Common Issues
Don't use the Ubuntu 'apt-get upgrade' command
With Quantastor you do not want to use the command "apt-get upgrade" or the dist-upgrade. This is because QuantaStor uses a custom kernel based off of a specific version of Ubuntu. Calling apt-get upgrade will result in Ubuntu trying to update to a newer version of the standard kernel which does not include the custom SCSI target drivers.
To keep our software up to date only apt-get update and apt-get install <quantastor packages> need to be called. The four quantastor packages are qstormanager, qstorservice, qstortomcat, and qstortarget. The safest way to do this is through the web interface using the Upgrade Manager but you can also do: apt-get update apt-get install qstormanager qstorservice qstortarget qstortomcat
What to do when you upgrade on accident
The easiest way is to reinstall quantastor. After the install is complete you can go into the Recovery Manager in the web interface. This will recover the metadata and the network configuration, and the pools will auto recover.
Resetting the admin password
If you forget the admin password you can reset it by logging into the system via the console or via SSH and then run these commands:
sudo -i cd /opt/osnexus/quantastor/bin service quantastor stop ./qs_service --reset-password=newpass service quantastor start
In the above example the new password for the system is set to 'newpass' but you can change that to anything of your choice.
Storage pool creation fails at 16%
Many motherboards include onboard RAID support which in some cases can conflict with the software raid mechanism QuantaStor utilizes. There's an easy fix for this, simply remove the driver using these two commands after logging in via the console as 'qadmin':
sudo apt-get remove dmraid sudo update-initramfs -u
Here are a couple of articles that go into the problem in more detail here and here.
The two commands noted above removes the dmraid driver that linux utilizes to communicates with the RAID chipset in your BIOS. Once removed the devices will no longer be locked down so the software RAID mechanism we utilize (mdadm) is then unable to use the disks.
XenServer Issues
Verify Network Configuration
Often times XenServer issues can be traced to network issues so anytime you're having trouble accessing your storage it's best to start by doing a 'ping' test from each of your XenServer hosts to each of your QuantaStor systems. To do this just bring up the console window on each of your Dom0 XenServer hosts and then type 'ping <ipaddress of quantastor box>' for example, if you have a QuantaStor system at 192.168.10.10 you would type 'ping 192.168.10.10'. If it says 'destination host unreachable' then you have a network configuration issue. The issue may be with the VLAN configuration in your switch or it may be that the QuantaStor system and XenServer host are on separate networks. Be sure to review your subnet mask (eg 255.255.255.0) and the IP addresses for both. Once corrected try the ping test again. Once you can successfully ping you're ready for the next level of checking.
Verify Storage Volume Assignment
Another common mistake is to forget to assign all the storage volumes to all the XenServer hosts. If you're having trouble connecting a specific host that should be the first thing to check. To do this, go to the Hosts section within the QuantaStor manager UI, select the host that should have the volume and then make sure the volume is in the list off the host. If you don't see it there, right-click on the host and assign the missing volume to the host.
Verify XenServer iSCSI Service Configuration
If you're still not able to successfully 'Repair Storage Repository..' in XenServer the next step is to try some low level iSCSI commands from the XenServer host that's not able to connect. The most basic of these is to do an iSCSI discover:
iscsiadm -m discovery -t st -p <quantastor-ipaddress>
For example you 'iscsiadm -m discovery -t st -p 192.168.10.10'. If you don't see the target list come back from the storage system and the list is blank then you've got a storage assignment issue. It could be that you have the incorrect iqn assigned to the host or some typo in it. Re-verify the IQN for the XenServer host and make sure that your QuantaStor system has the volume(s) assigned to the correct IQN. If you get an error back like "iscsiadm: Could not scan /sys/class/iscsi_transport" or the iSCSI service is not started then you'll want to try restarting the iSCSI service on the XenServer host.
/etc/init.d/open-iscsi stop /etc/init.d/open-iscsi start
You'll also want to look at the file '/etc/iscsi/initiatorname.iscsi' If that file is missing, that's a problem. You'll need to create that file to have contents that look something like this:
InitiatorName=iqn.2012-01.com.example:ee46dcfc InitiatorAlias=osn-prod1
Note you will want to change the part 'ee46dcfc' to have different letters and numbers so that you have a unique IQN for the host. The host may already have one assigned which you can find in the 'General' tab within XenCenter. Take that IQN and replace the one above with it and the InitiatorAlias should be the name of your XenServer host. Once you have that file in place, try stopping and starting the open-iscsi service as noted above. Then try the iscsiadm discovery command again as noted above.
At this point you should be able to see a list of iSCSI targets from your QuantaStor system. If you're working on repairing an existing SR, try repairing it again.
