Upgrading Membase 1.7.2 to Couchbase 1.8.1, provisioned IOPS for EBS

I just completed an upgrade of our prod Membase cluster to go from 1.7.2 to Couchbase 1.8.1 community edition, which was made available recently.  Since I am planning to do the upgrade by taking all the nodes down and update all the server software which will incur downtime, I figured I will also try out the newly available “EBS with provisioned IOPS”.  Things went well for the most part, however there is one key thing when doing the upgrade that was not covered in Couchbase’s documentation.

If you are deploying your Couchbase server in a cloud service like EC2, you likely have changed your server setting so it uses a DNS name rather than a self-reported IP address (see: http://www.couchbase.com/docs/couchbase-manual-1.8/couchbase-bestpractice-cloud-ip.html).  And if that is part of your setup, you have to also do the same to the database upgrade script that will convert your data to 1.8.1 format.  Here are the steps I used to do the upgrade on my boxes running Amazon Linux AMI:

1. Of course, backup the server.  You never know what’s going to happen in one of these major upgrades.

2. rpm -e membase-server

3. INSTALL_DONT_START_SERVER=1 INSTALL_DONT_AUTO_UPGRADE=1 \
   INSTALL_UPGRADE_CONFIG_DIR=/opt/membase/var/lib/membase/config \
   rpm -i couchbase-server-community_x86_64_1.8.1.rpm

4. vim /opt/couchbase/bin/cbupgrade

5. Find “127.0.0.1” in the file, and replace it with your custom DNS

6. vim /opt/couchbase/bin/couchbase-server

7. Find the following and add in your server name:

   ...
   -ns_server config_path "\"/opt/couchbase/etc/couchbase/static_config\"" \
   -name ns_1@YOUR_DNS_HERE \
   -ns_server pidfile "\"$PIDFILE\"" \
   ...

8. Do a dry-run of the upgrade:

   /opt/couchbase/bin/cbupgrade -c /opt/membase/var/lib/membase/config -n

9. If it all works, then run it without the -n option and it should complete very quickly (took no more than a few seconds for me)
10. I found myself needing to reboot the box after the upgrade.  Simply starting Couchbase right after the upgrade didn’t bring the server up properly.

After that, it’s back to waiting for the server to finish warming up.  I am getting much better warm-up time with the new EBS with provisioned IOPS.  No surprise there.  Knowing how often disk ends up being the bottleneck (like just about any database under the sun), I wouldn’t setup any future Couchbase nodes without this puppy.

1. vim /opt/couchbase/bin/cbupgrade