Happy holidays everyone! \u00a0I realize that from working with Membase in our production over the last year, I’ve collected a few key commands in my .bashrc for quickly checking vital stats on my Membase servers, many of them came from the good folks at Couchbase. \u00a0 Their wiki has improved over the year as well, and you can find a lot of good information there. \u00a0Here I will list the most common commands I run for monitoring and troubleshooting, along with related links to the Membase wiki:<\/p>\n
<\/p>\n
mbstats<\/strong> is the most useful utility that will print out all the important stats in the system. \u00a0Like I said, many of these stats are accessible via the web console. \u00a0But ultimately, if you want a script to periodically capture the data and alert if something goes wrong, you still want to keep this command handy:<\/p>\n When first growing out your database, it’s good to observe how quickly\u00a0mem_used <\/strong>approaches high_wat<\/strong>, which is when Membase will have to start ejecting data from memory, and subsequent read of ejected data will result in a disk read. \u00a0For animal party, we’ve been near high_wat for many months, but our working dataset is fairly small part of the entire data, so we don’t have a lot of disk reads. \u00a0But if you see your cluster is approaching high_wat quickly, start to keep a watch on how often server is fetching data from disk. \u00a0If you see the cache miss ratio<\/strong>\u00a0from the web console start to climb, then you should look into adding more memory to the cluster.<\/p>\n For more detailed monitoring on the effect of cache miss, ep_bg_fetched<\/strong>\u00a0will tell the numbers disk fetches that has been performed since server was started. \u00a0You can pull the number twice with a time lapse in between to estimate disk fetches per second or minute to give you a stat to record and later compare. \u00a0Adding up\u00a0ep_bg_load_avg <\/strong>and ep_bg_wait_avg <\/strong>should tell you how long on average it takes an item to get loaded off the disk in microsecond, and if it starts to get too high you know you need to get more memory into the cluster to reduce ejecting item and subsequently reduce disk fetches.<\/p>\n eq_queue_size<\/strong>\u00a0and ep_flusher_todo<\/strong>\u00a0combine to tell you how many items are waiting to be written to the disk. \u00a0When an item needs to be written, it first gets added into a queue that goes towards the ep_queue_size count. \u00a0A dedicated disk writing process or processes (ala flusher) will on a periodic basis\u00a0take items out of this queue and write them to the disk. \u00a0 The items that are off the queue but have not made it onto the disk yet will be accounted for in ep_flusher_todo. \u00a0Obviously, you always want the disk queue to drain more quickly than it’s being filled, or server will run out of memory to store these pending items.<\/p>\n In general, you don’t want the queue to be too large anyways, because that means you have a lot of data that are not persisted to disk. \u00a0Granted, they will be replicated in another node in the cluster, but if both the main copy and the replica copy are both stuck waiting in the disk queue and servers are rebooted, you would lose those data. \u00a0During a rebalance, you will see the disk queue shoot up dramatically, since data needs to be relocated to the appropriate node. \u00a0This is why I choose to schedule a downtime for the application during rebalance even though theoretically it can still serve data normally – all the new writes that are happening during a rebalance may take much longer than normal before they are persisted to disk, increasing the risk of losing those data.<\/p>\n The Membase wiki has a good explanation on what stats to monitor and why: As a reference, here’s a link for complete list of all the stats and what they mean: <\/p>\n <\/p>\n dispatcher<\/strong>\u00a0is another interesting command that gives you visibility into what Membase is actually doing at this moment. \u00a0You can run it with the following command:<\/p>\n\/opt\/membase\/bin\/mbstats \u00a0localhost:11210 all<\/pre>\n
\nhttp:\/\/www.couchbase.org\/wiki\/display\/membase\/Ongoing+Monitoring+and+Maintenance<\/a><\/p>\n
\nhttps:\/\/github.com\/membase\/ep-engine\/blob\/master\/docs\/stats.org<\/a><\/p>\n\/opt\/membase\/bin\/mbstats localhost:11210 dispatcher logs<\/pre>\n