Key stats to monitor for your Membase cluster

By Shawn | Published: December 23, 2011

Happy holidays everyone!  I realize that from working with Membase in our production over the last year, I’ve collected a few key commands in my .bashrc for quickly checking vital stats on my Membase servers, many of them came from the good folks at Couchbase.   Their wiki has improved over the year as well, and you can find a lot of good information there.  Here I will list the most common commands I run for monitoring and troubleshooting, along with related links to the Membase wiki:

Read More »

Posted in Development | Tagged | Leave a comment

Migrating Membase Cluster – Part 2

By Shawn | Published: December 11, 2011

After up all night babysitting the rebalance process, I am happy to report that it was a rather uneventful night of maintenance.  The rebalance itself took 8-9 hours to complete, and then took another hour for all the replicas to get saved to the disk also.  Theoretically, I didn’t need to take the site down while the rebalance was happening, but I took the game down just to be safe and not compromise the game experience.

The disk access was definitely the bottleneck through out the rebalance process once again.  One the the reason we went for more # of smaller nodes rather than smaller # of bigger nodes is to spread out our disk activities over more # of EBS drives during a rebalance, conceptually similar to a RAID 0.  We do increase the higher risk of hardware failure simply by having more nodes in the cluster, but the disk performance gain is definitely worth it.

Some folks are doing a RAID 0 setup using multiple EBS as described here on alestic, but I haven’t tried it personally.  If anyone has attempted that setup, especially in a production environment, please share your experience in the comments!

Posted in Development | Leave a comment

Migrating Membase Cluster – Part 1

By Shawn | Published: December 11, 2011

I got an email from EC2 a few days ago that a number of our instances that serves Animal Party have pending reboots coming up.   It turns out that ALL of the 8 nodes in our membase cluster are scheduled for a reboot.   Sure, a reboot on any database server is expected to cause some interruption, but I know from my past experience that warming up a membase node after reboot can take a really long time, especially if the data on disk is highly fragmented.  Since I’m waiting for our scheduled downtime to kick in, I figured it’s a good time for me to share my learning with membase – especially how and why we are doing the cluster migration.

Read More »

Posted in Development | Leave a comment

Trouble handling Icon\r file with Git

By Shawn | Published: December 9, 2011

We installed a new process for managing content couple weeks ago, where we had one of the game designer check in his CSV data file into git so we can pull his changes and run the data through our tests automatically. He was on a Mac, so he used GitHub for Mac (very user friendly, by the way) to get his files in.  I tried to pull his changes, and I got an error about some “Icon\r” file. What on earth?!

It turns out that “Icon\r” is a file that exists in all directories that have a custom icon in the Mac OSX Finder.   Normally hidden, but GitHub for Mac sees the file and committed it without any issue.  However, when I was trying to pull that file through git on my Ubuntu box, I get an error telling me that it’s unable to create “Icon\r”.   I tried to use GitHub for Mac to delete the file and commit that change, but GitHub for Mac is failing when trying to commit that delete change with a generic error message.  It looked like they are all having problem with the “\r”, the carriage return character.

I am able to login to GitHub and see the file “Icon ” (\r is not rendered in the browser), but unfortunately there was no way to delete the file there.  After an hour of futile attempt, I finally was able to get rid of it by running “git gui” in my cygwin, and use the interface to select the file and commit that delete.  I’m not sure why or how “git gui” was able to get it done, but there you go – I hope this post may help someone out there.

By the way, you probably want to add that pesky “Icon\r” to the .gitignore file as well so you can prevent them from getting into the codebase in the first place.   For that, you should check out this post on dealing with the “\r” character.

Posted in Development | 1 Comment

Setup Namespace URL for Flex project in FlashDevelop

By Shawn | Published: November 4, 2011

Our game client at Tribal Crossing is all written in Flex, and naturally we used Flash Builder to handle our project.  However, I’ve always been partial to FlashDevelop, a much leaner IDE that has great editor features, faster compile time, and fantastic community support.  After being fed up with another Eclipse (the IDE that Flash Builder is built on top of) crash, I decided to give FD a try for our project.

We had multiple Flex Library projects, and since there’s no direct support for that in FD, I simply added the src folder from each of the library as additional project code path – and that worked.  There was a problem with the namespace URL used in MXML and CSS, however.

There is no UI in FD to specify namespace URL.  But you can point your project to the manifest.xml files that contains the namespace URL info by adding this to your advanced compiler options (under  Project settings->Compiler Options->Advanced Compiler Options):

-namespace library://project.com/comp c:\code\src\manifest.xml

Hope this helps!

Posted in Development | Tagged , , | Leave a comment

Issue with “npm install socket.io” on CentOS 5

By Shawn | Published: October 31, 2011

Hey guys,

I just ran into some issue installing socket.io using npm on my personal CentOS 5 box.  All the other node modules installed fine – but not socket.io.  After some searching around, I found that it appears to be an issue with the version of tar that came with CentOS 5, which was 1.15.  After upgrading to 1.26, the issue went away.

You can find the original fix here https://github.com/LearnBoost/socket.io/issues/456 from korch towards the bottom of the page.

Hope this helps!

Posted in Development | Tagged | Leave a comment

Video Games Live Tour – Damn it was good.

By Shawn | Published: October 30, 2011

I went to the Video Games Live tour with Rick in LA earlier this weekend.  We heard tracks from their past shows on Spotify, and we just decided to grab some tickets on a whim a couple months ago.   We know they play music from many classic games which are also our favorites, and the tour seems well received from our scant research prior to the purchase.

We got there, sat down… and it just blew me away.

On a personal level, it reminded me how video games have been such a tremendous part of my life, and how I am grateful to be part of this community.  I sat through the entire concert with goosebumps running through every square-inch of my skin, and I nearly teared up upon hearing the Halo and Chrono Trigger/Cross pieces.

I want to write more about it, but since the concert I’ve been reinvigorated to get some tedious game programming done.  But I certainly hope that you have a chance to check them out live!

P.S. Laura “Flute Link” Intravia performed at our show.  I have never seen her Zelda flute performance on YouTube, but now I understand why there is a major buzz surrounding her.  She is one impressively multi-talented video game geek who could be mistaken for a model.

Posted in Uncategorized | Leave a comment

Sending batch app-to-user request on Facebook

By Shawn | Published: September 1, 2011

With the release of the “live game feed” on Facebook, game requests and notifications now gets their own real estate on the upper right corner of your screen.   This immediately reminded me of the good old days of “notification war” on Facebook circa 2009.   Applications had free reign over your notifications and many spammy apps went viral by abusing that channel.  At Tribal Crossing, we created a populate polling app that reached 3 million MAU that was in part due to this powerful viral feature.   Oh yes, everyone succumbs to the mystical power of the Red Bubble that screams CLICK ME.

Anyways, Facebook took the app-to-user notifications away, but now it has returned in a different implementation – a much better one IMO.   So we plugged back into app-to-user requests so we can send user individual updates besides going through the application wall.  Each notification requires a Graph API call, and it can take a long time to send millions of these requests.  There is a batch API available, but there wasn’t any good documentation so it took me a bit of time to get it to work.   Here’s the detail on formatting your Graph API calls to send batch app-to-user requests:

Read More »

Posted in Development | 4 Comments

Animal Party gets coverage in Inside Social Game

By Shawn | Published: July 18, 2011

Animal Party hit the big 100k DAU and 800k MAU mark over the weekend!   Inside Social Game interviewed Tommy Wu, my co-founder/CEO, for a few quotes.  You can see the article here.

We’re now doing various optimization and content tweak for Animal Party while continuing ahead on our next title.

Posted in Gaming Bizdev | Leave a comment

Animal Party hitting 500k MAU; Facebook Credits as currency

By Shawn | Published: July 13, 2011

Animal Party joined the 6waves publishing network several weeks ago and we’ve been getting more traffic.  We also transitioned our monetization strategy to use Facebook Credits as a in-game currency, which made us eligible for various Facebook promotion that brought in additional traffic.  As a result, we broke 500k MAU for the first time since the game was launched!

Facebook Credits as a currency is definitely has both pros and cons.  While Facebook provides a lot of additional user through their promotion channels, it still remains to be seen if the overall effect on revenue per user will be a positive one.  Many smaller studio like Digital Chocolate have adopted that strategy fully, while bigger studios like Zynga and Playdom are still sticking to their own in-game currency.  This makes sense, because bigger studios don’t really need the Facebook promotion since they already have a very large audience playing their existing games, and they can funnel traffic between their own games through cross promotion.

Posted in Gaming Bizdev | Leave a comment