http://www.kickstarter.com/projects/66710809/double-fine-adventure

On the KS page, they pointed out that “even something as ‘simple’ as an Xbox LIVE Arcade title can cost upwards of two or three million dollars.  For disc-based games, it can be over ten times that amount.”  This is something I mentioned in my rant about in my other post about the rise of game clones in the social/freemium space.  Traditional games are expensive to make, and developers have to finish all the content at the time of release because players don’t continue to download updates to content and game mechanics each time they play.

Yes, downloadable content things like Steam update are starting to change that situation slowly, but it is not the same as freemium games for one major reason.  If I paid $20 up front, I need to know that I will have $20 worth of content ready for me.  But if I started a game for free, I wouldn’t mind if it only has three weeks worth of content and I just have to see how the game evolves as I continue to play.  So instead of having no revenue stream until the entire game is finished, freemium games can start to receive revenue at a much earlier stage.

Crowd-sourcing however, is giving game developers another viable way to fund-raise through the dev cycle.  There have been a few indie games that were funded and eventually released through KS (e.g. No Time To Explain), but Double Fine just proved (this morning!) that crowd-sourcing can do a lot more.  Their original pledge goal of $400k is rather small for any studio quality game, but they already hit $700k in just over 9 hours.  Obviously, having Tim Schafer as a lead makes a night-and-day difference (to the point where they didn’t even need to reveal any info or screenshot on the game being made), but this reinforces two of my existing believes:

1. Story driven, click-adventure games are viable today

The recent success of Machinarium and Sword & Sworcery EP and the wild funding success of Double Fine show that there is a demand for adventure games.  Their audience is somewhat different from the popular FPS, RTS, or MMORPG players, but developers are finding new ways to reach those players.  We’re also seeing less adventure games focused on challenging puzzles and more focus on making sure puzzles do not impede players from progressing through the game plots.

2. Studios are finding new paths to funding and revenue outside of the old developer-publisher relationship

Whether it’s freemium, crowd-sourcing or episodic releases, developers are finding new ways to get it done without relying on a publisher.  I think this bodes well for everybody, because this will allow more courageous and out-of-the-box ideas to see the light of day.

Can’t wait to see how much momentum Tim Schafer and Double Fine will generate from this KS project.

https://github.com/esnme/ultramemcache

It’s written and maintained by the good folks at ESN.me.  They built Battlelog, the social network for Battlefield 3, using gevent and this memcache client.  They also released a gevent-compatible MySQL driver and a few other interesting python projects.

I’ll be doing load/stress testing with various different clients and setups for flask/gevent in the coming weeks, and I’ll post my findings here.

http://kotaku.com/5879046/zynga-totally-rips-off-tiny-tower
http://venturebeat.com/2012/01/29/buffalo-studios-blasts-zynga-for-copying-bingo-blitz-social-game/

And here is a more recent and more serious accusation that is actually turning into a lawsuit, against our own publisher 6waves/LOLAPPS:

http://www.edery.org/2012/01/standing-up-for-ourselves/

Having been in contact with several folks involved in the accusation and lawsuit, it’s been interesting to hear people’s take on the issue.  There is a really fine line between inspiration and copying, and this problem has been seen in any creative field for a long time.  It has come up before in the game industry in the past, but it has become more of a focus in the current social/freemium game landscape.  Why?  Because the efforts involved in cloning a game have been reduced while the financial reward has gone up.

For a long time, shipping any computer game was not for the faint of heart.  Development cycles often lasted a year or more, and studios lived and died by the whims of publisher because games generate no revenue until it finally hits the shelf.  Game developers worked long and hard hours to deliver milestones to just keep the publishers happy, and it was commonplace to see game projects canned for financial reasons.  Even when a game was completed, it still needed the publisher’s buy-in to get the shelve placement and marketing effort required to reach financial success.

However, the tough landscape did bring the best out of the game industry.  The game economy filtered out anyone that didn’t have the passion for it.  It had never been a good paying gig outside of a few rock stars, but folks stuck around because of a sense of pride and ownership in what they created.  A lot of industry visionaries made their marks, and they all brought something unique to the industry – in the form of game design or technology.

Nowadays, with the advancement of dev tools, distribution channel, and new revenue models, online freemium games are achieving financial success while having shorter production cycles.  I am still trying to wrap my brain around this every day.  Some of my friends in the traditional gaming space are making almost half of what my other friends are making at some social game studios.  Hey, game developers are finally able to pay their bills and put food on the table!

The new landscape brought in more money, but also fundamentally changed how games were designed.  In traditional games, we have seen the occasional bad games that sold well due to licensing and buzz (Enter the Matrix, 62/100 on metacritic, 5m copies sold), critically acclaimed games flopping commercially (Grim Fandango, Okami, etc), and bad games sold poorly despite big marketing effort (Haze, <500k copies sold).  It’s a hits-driven business, and a fun game or strong marketing alone does not guarantee commercial success. Combining with the fact that traditional game productions are rather expensive and lengthy, it makes cloning games a rather risky business.

In freemium games, it becomes a very different numbers game.  There are many fun games that simply don’t monetize (unfortunately, I’ve worked on a couple), and not-so-fun games that monetizes well.  In fact, just a few key numbers determine the entire fate of nearly any freemium games – conversion rate, retention rate, and revenue-per-active-user.  If a game converts free players at a high rate and monetizes them well, the developer/publisher can continue to ramp up marketing and buy installs as long as each player brings in more revenue than the acquisition cost.  Conversely, if you have a lot of people playing your game but few are paying, the game is going to eventually shut down, regardless how fun it is.  The financial success of games can now be projected early and accurately from the sea of metrics, and this is something that is very hard to do in the traditional game space.  And this gives rise to a new kind of visionaries – the ones who master metric driven design, A/B testing, and monetization techniques. With the advanced understanding of metrics, it has become easier and less risky to replicate the financial success of an existing freemium game.  Not to mention the time to market on these games are also shorter than the traditional games, making the decision to clone a game an even more profitable one. While I believe it’s a good thing that more dollars are flowing into the market and developers, I also believe that the recent focus on monetization has relegated game design innovation to the back seat.

One other key difference is that these cloned games are serving the new “freemium casual game audience”.  These players are not the ones who read reviews in gaming magazine.  They have not been playing games long enough to build a “refined” taste that traditional gamers have.  They are not the ones who will tell you that Angry Bird is a copy of “Crush the Castle”, which is inspired by “Worms”, which is inspired by “Scorched Earth” (they call it The Mother of All Games for good reason).  In a landscape where audience does not yet fully recognize and reward innovation, it is natural that the market supplies only what the consumers demand.

What used to be a creative process of “designing games for the sake of fun” becomes a financially driven process of “quickly expanding revenue generation portfolio”.

Let me go on a tangent for a second and look at another sector flooded with copycat ideas – the current day Silicon Valley.  With regards to the start-up world, the industry seems to have relatively little problem with cloning ideas.  Just look at all the daily deal companies (Groupon, LivingSocial, Bloomspot, etc), FourSquare/Gowalla, and of course, Friendster/Facebook/Myspace.

Why hasn’t it been an issue?  I believe it’s because the ultimate gauge of success for start-ups is financial success.  At the end of the day, whoever is able to get a bigger market share, generate more revenue, and offer more value to the shareholders is the winner.  The better product is the one that generates more revenue, not necessarily the one that is functionally and aesthetically superior.  Folks are constantly bashing Microsoft products, but we all recognize that the company is enormously successful.  Zynga has a terrible reputation among game developers for their design practice, but they are one of the mos most valuable gaming company in the US.  They are innovative in proving out the freemium business model, but not so in game design.  The game industry looks down on them for delivering copycat products, cranking out uninspired ideas that are based metrics and not fun factors.  But as long as they are able to reward their shareholders and offer their developers twice as much money as traditional developers, you bet they will keep their formula going.

When a good, innovative game idea comes about, it will attract designers who are inspired by the game mechanics, as well as players who has grown familiar and become attached to those same game mechanics.  As future “inspired” games sprouts out from the first innovation, they form a game genre over time.  The success of Dune 2 and Warcraft draws in many subsequent developers and designers who create the RTS genre together.  In a way, Zynga has also paved way for many of the freemium games today with their innovation in monetization strategy.  The forming of a new game genre tends to happen organically and is sprinkled with small bits of innovation in each new iteration along the way.

However, there is definitely a line between adhering to a genre and copying a game.  It’s a very subjective matter, and I don’t think there’s an easy way to define how the line is drawn.  But when a company crosses it, the public tends to see it quickly.  But in our current space, the developers are the ones who recognize and speak up, but not the audience – yet.  Companies looking at these short term clones are making the business decision between short term gain versus a long term reputation and growth.  Will the consumers be knowledgeable enough with a discerning taste to reward the ones who come up with cool ideas and good execution versus the ones with recycled ideas and clones?  Only time will tell.  But I believe that all businesses should to stay true to their core value.  And for us at FableLabs, it’s about creating games with original content that offer a unique experience to players.  We want to be the next group of visionaries that marry the learnings from the freemium model with the good ol’ spirit of game design innovation.

I do think that this new freemium/social/casual game audience is quickly maturing.  And this will only in turn foster a new crop of games that will build on the existing mechanics that they are familiar with, while introducing new ones that provides many of the things old school gamers have general come to recognize as being a good, original games:

• A story line that create emotional attachment
• New game mechanics that departs from or combines existing ones
• Emergent game play stemming from a good set of game rules

FWIW, here’s Zynga’s internal response to the accusations:

http://kotaku.com/5880941/zyngas-formerly+secret-defense-against-copycat-accusations

mbstats is the most useful utility that will print out all the important stats in the system.  Like I said, many of these stats are accessible via the web console.  But ultimately, if you want a script to periodically capture the data and alert if something goes wrong, you still want to keep this command handy:

/opt/membase/bin/mbstats  localhost:11210 all

When first growing out your database, it’s good to observe how quickly mem_used approaches high_wat, which is when Membase will have to start ejecting data from memory, and subsequent read of ejected data will result in a disk read.  For animal party, we’ve been near high_wat for many months, but our working dataset is fairly small part of the entire data, so we don’t have a lot of disk reads.  But if you see your cluster is approaching high_wat quickly, start to keep a watch on how often server is fetching data from disk.  If you see the cache miss ratio from the web console start to climb, then you should look into adding more memory to the cluster.

For more detailed monitoring on the effect of cache miss, ep_bg_fetched will tell the numbers disk fetches that has been performed since server was started.  You can pull the number twice with a time lapse in between to estimate disk fetches per second or minute to give you a stat to record and later compare.  Adding up ep_bg_load_avg and ep_bg_wait_avg should tell you how long on average it takes an item to get loaded off the disk in microsecond, and if it starts to get too high you know you need to get more memory into the cluster to reduce ejecting item and subsequently reduce disk fetches.

eq_queue_size and ep_flusher_todo combine to tell you how many items are waiting to be written to the disk.  When an item needs to be written, it first gets added into a queue that goes towards the ep_queue_size count.  A dedicated disk writing process or processes (ala flusher) will on a periodic basis take items out of this queue and write them to the disk.   The items that are off the queue but have not made it onto the disk yet will be accounted for in ep_flusher_todo.  Obviously, you always want the disk queue to drain more quickly than it’s being filled, or server will run out of memory to store these pending items.

In general, you don’t want the queue to be too large anyways, because that means you have a lot of data that are not persisted to disk.  Granted, they will be replicated in another node in the cluster, but if both the main copy and the replica copy are both stuck waiting in the disk queue and servers are rebooted, you would lose those data.  During a rebalance, you will see the disk queue shoot up dramatically, since data needs to be relocated to the appropriate node.  This is why I choose to schedule a downtime for the application during rebalance even though theoretically it can still serve data normally – all the new writes that are happening during a rebalance may take much longer than normal before they are persisted to disk, increasing the risk of losing those data.

The Membase wiki has a good explanation on what stats to monitor and why:

http://www.couchbase.org/wiki/display/membase/Ongoing+Monitoring+and+Maintenance

As a reference, here’s a link for complete list of all the stats and what they mean:
https://github.com/membase/ep-engine/blob/master/docs/stats.org

dispatcher is another interesting command that gives you visibility into what Membase is actually doing at this moment.  You can run it with the following command:

/opt/membase/bin/mbstats localhost:11210 dispatcher logs

There are a few dispatchers, and to be frank, I don’t really know any details about them (you can read http://www.couchbase.org/wiki/display/membase/DGM+Implementation+Details if you are curious).  But for monitoring purpose, I look at the entries in the “Slow jobs” section.   For example, this is one of the jobs that is returned in the dispatcher log:

        runtime:   2s
        starttime: 43439
        task:      Fetching item from disk:  StoryQuest40000001045816

So here I see that item “StoryQuest40000001045816″ took 2 seconds to come off the disk, and the job was started at a time that is 43439 seconds after the server booted up.  If you didn’t see any recent slow jobs in the log, then you know things are doing fine.  If a task is taking a long time to finish, you may get an idea what is going on by looking at the description, but chances are you may need to get some support help from the Membase folks to decipher the issue.

Last useful command is mbcollect_info, which is what folks at Membase always ask me to run whenever I report an issue.  Since I need to run this on all the nodes in the cluster and aggregate the output, I ended up adding this to a larger script so this can be automated.

/opt/membase/bin/mbcollect_info FILENAME.zip

The disk access was definitely the bottleneck through out the rebalance process once again.  One the the reason we went for more # of smaller nodes rather than smaller # of bigger nodes is to spread out our disk activities over more # of EBS drives during a rebalance, conceptually similar to a RAID 0.  We do increase the higher risk of hardware failure simply by having more nodes in the cluster, but the disk performance gain is definitely worth it.

Some folks are doing a RAID 0 setup using multiple EBS as described here on alestic, but I haven’t tried it personally.  If anyone has attempted that setup, especially in a production environment, please share your experience in the comments!

Our cluster currently holds about 240m key-value pairs over a total of about 160GB of data.   We have only about half of those data in memory, but it’s enough to cover our working set and the cluster doesn’t read from our disks much at all.  However, when a member node is rebooted, it must warm-up all the items that the node is “master” of.

What do i mean by “master”?  We have a replication count of 1 in our cluster, so while any item (a key-value pair) will exist on two nodes at any given time, all the writes to that piece of data will only take place on a single node to ensure consistency.  When a membase server reboots, it needs to “warm-up” and finds “all the data that it is master of” from the disk.  Since we got ~32m items per node, it can take a while.

Another thing that causes this warm-up to take a long time is the fact that membase uses sqlite3 engine for persisting data to the disk.  Sqlite3 uses btree to store its data, and when items are deleted, the underlying btree pages are merely marked as “free”.  Later on when new items are stored, their content can be spread over different pages, causing fragmentation.  So if the membase cluster is seeing a lot of delete or expiration, which ours does, the warm-up time will slowly increase overtime.  This fragmentation issue will be addressed in the next major release Couchbase 2.0, since it will be replacing sqlite3 with CouchDB.  But in the mean time, this is a real problem that we will need to deal with in production.

While it’s possible to vacuum the sqlite database, it’s not really a viable option for us because data persistence needs to be suspended during the process, leaving the system vulnerable to data loss.  And we know EBS has pretty awful disk performance, so vacuum is surely going to take a really long time with the amount of data we have.

After talking to Perry Krug from Couchbase (who has always been extremely helpful) about the situation, he suggested doing a complete cluster migration by rebalancing in 8 new nodes while removing the 8 old nodes.  This will take care of the reboot issue since we’ll be moving to new instances, but we will also get a free vacuuming of our data.  As a side bonus, Perry said we can also use this opportunity to upgrade our cluster from version 1.7.1.1 to 1.7.2 too.  I don’t know if this “rebalace your entire cluster” upgrade path works for older versions too, since you will need to mix two versions of membase server in your cluster during the rebalance –  but I am glad it works for 1.7.2

One caveat Perry mentioned was that the webconsole UI does not allow you to remove all the nodes, but this can be circumvented with the command line tools.  After all the new nodes are added to the existing cluster and show up as “pending for rebalance” in the webconsole, this command will remove all the old nodes and rebalance them into the new ones:

/opt/membase/bin/membase rebalance -c 10.0.0.10 -u <username> -p <password> –server-remove=10.0.0.1 –server-remove=10.0.0.2 …

(assuming 10.0.0.10 is one of the new nodes)

I’ll write up a postmortem after the actual migration, and I hope I will have nothing but good news to report…

It turns out that “Icon\r” is a file that exists in all directories that have a custom icon in the Mac OSX Finder.   Normally hidden, but GitHub for Mac sees the file and committed it without any issue.  However, when I was trying to pull that file through git on my Ubuntu box, I get an error telling me that it’s unable to create “Icon\r”.   I tried to use GitHub for Mac to delete the file and commit that change, but GitHub for Mac is failing when trying to commit that delete change with a generic error message.  It looked like they are all having problem with the “\r”, the carriage return character.

I am able to login to GitHub and see the file “Icon ” (\r is not rendered in the browser), but unfortunately there was no way to delete the file there.  After an hour of futile attempt, I finally was able to get rid of it by running “git gui” in my cygwin, and use the interface to select the file and commit that delete.  I’m not sure why or how “git gui” was able to get it done, but there you go – I hope this post may help someone out there.

By the way, you probably want to add that pesky “Icon\r” to the .gitignore file as well so you can prevent them from getting into the codebase in the first place.   For that, you should check out this post on dealing with the “\r” character.

We had multiple Flex Library projects, and since there’s no direct support for that in FD, I simply added the src folder from each of the library as additional project code path – and that worked.  There was a problem with the namespace URL used in MXML and CSS, however.

There is no UI in FD to specify namespace URL.  But you can point your project to the manifest.xml files that contains the namespace URL info by adding this to your advanced compiler options (under  Project settings->Compiler Options->Advanced Compiler Options):

-namespace library://project.com/comp c:\code\src\manifest.xml

Hope this helps!

I just ran into some issue installing socket.io using npm on my personal CentOS 5 box.  All the other node modules installed fine – but not socket.io.  After some searching around, I found that it appears to be an issue with the version of tar that came with CentOS 5, which was 1.15.  After upgrading to 1.26, the issue went away.

You can find the original fix here https://github.com/LearnBoost/socket.io/issues/456 from korch towards the bottom of the page.

Hope this helps!

We got there, sat down… and it just blew me away.

On a personal level, it reminded me how video games have been such a tremendous part of my life, and how I am grateful to be part of this community.  I sat through the entire concert with goosebumps running through every square-inch of my skin, and I nearly teared up upon hearing the Halo and Chrono Trigger/Cross pieces.

I want to write more about it, but since the concert I’ve been reinvigorated to get some tedious game programming done.  But I certainly hope that you have a chance to check them out live!

P.S. Laura “Flute Link” Intravia performed at our show.  I have never seen her Zelda flute performance on YouTube, but now I understand why there is a major buzz surrounding her.  She is one impressively multi-talented video game geek who could be mistaken for a model.