Asherons Call: So, Why No Hotfix for the Money Exploit?

[i]Let's start by looking at the side effects of this bug. Players can make money -- lots and lots of money. That has three real effects in the game: players can very easily buy things from vendors, players can gamble lots, and players have lots of cash or cash equivalents to use in player trades.

Well, most players don't have severe difficulty in buying what they need from vendors, so the effect on that facet of the game is not huge. Money is not a limited resource in AC -- there is an unending influx of loot entering the system, and that loot can always be sold to a vendor for cash. In the past, in the early days of AC, the supply of cash was more limited than it is today, but in any game with our type of treasure and vendor system, you'll eventually end up where we are today -- with readily available money.

And actually, I want to branch off here a second and mention the quote that's floating around. A couple of people have brought up an old "Pages from the Producer" article that deals with the infamous rollover bug of January 2001. This bug, if you don't remembered, involved "An exploit ... that allowed the creation of infinite money via vendor manipulation." The particular quote I'm referring to is this one: "A stupendous glut of money, however, would have tainted the game for a very long time." (Quotes from http://zone.msn.com/asheronscall/news/pagesJan.asp.)

Several people have asked why, if a stupendous glut of money would have tainted the game for a very long time *then*, does the same not hold true *now*? And the answer, of course, is that the two situations are quite different. First, that bug really did allow for the creation of infinite money, whereas this one merely allows the creation of lots and lots of money. I don't mean to sound facile here -- but there really are several orders of magnitude difference. And secondly, back to the point I was getting at above, the meaning of money has changed tremendously since that time. The game was much younger then; loot was different, hunting was different, and money was different. Because of changes in the game -- a natural evolution of game systems and content -- money is much more available today than it was back then. Taking those points together, a stupendous glut then would have tainted the game for a very long time; a large influx today will not.

So we felt that the side effects as far as players buying things from vendors were not sufficient to warrant a hotfix.

Now, the second side effect of players with lots and lots of money is that they can gamble. And since gambling can get you mnemosynes, and mnemosynes can get you experience, you can use your lots and lots of money to get lots and lots of experience, right? Well, not exactly. You can use money to get experience this way, but in the long run it just doesn't compare to the experience you can get by going out hunting. Even if we increase the experience rewards at the casinos, the best you can really say is that casino gambling is an alternate but not particularly efficient game path. In fact, leaving aside the question of this bug in particular, we quite like the notion that a character can use the casinos as an alternate advancement path. AC doesn't have many of those, you know.

So we did not feel that the side effects as far as players gambling were sufficient to warrant a hotfix.

The third side effect I mention above is the doozy -- the player economy. And I want to say right off that we care a great deal about the player economy. We care about it because you care about it; we know that it adds a valuable element to the game, an intriguing social grace note that wouldn't be there otherwise; and we like that -- we do what we can to support it. And we know that this bug will have effects on the player economy; that a large influx of cash and cash equivalents will have repurcussions on player trades. It's not like we're rubbing our hands together gleefully, here, wanting to screw traders -- we worry about these things.

BUT! But we also know from experience how resilient the economy is. We've seen what's happened in the past; we've seen the player economy recover from shocks worse than this one. We've seen the effect of the older money macros -- and while they weren't as intense as this bug, they were also around for a lot longer. And we know too that today's economy is more diversified than it's ever been before -- even young characters can gather valuable salvage to trade, and there's more desirable trophy and quest items. We'd prefer not to affect the player economy in this way, but these factors at least reassure us that if we do, it won't by any means collapse.

So the side effects on the player economy are not what we would have chosen, but they are not an absolute catastrophe. Are these side effects sufficient to justify a hotfix? Well, let's look at the side effects of a hotfix.

First, any time we do a hotfix there is a risk of introducing a more serious problem. Hotfixes are by their very nature 'hot' -- they are tested over a shorter period of time and less thoroughly. I'm not saying that we don't test hotfixes -- I'm saying that in a complex game like AC, sometimes specific point testing doesn't do it. Sometimes problems can develop between two behaviors that you'd swear were not connected, and the more playtesting you do in normal game conditions, the more likely you are to find them. But hotfixing doesn't allow for playtesting 'normal game conditions' -- there isn't time -- and so hotfixes will always be riskier than a normal update. This particular bug seems simple enough, but I can't hammer it home enough that there are risks here.

Secondly, any hotfix involves bringing the servers down. The design of AC does not allow for patching the data while the servers are running. So right up front we're talking about several hours when no one can play. Players are used to the monthly updates, but unscheduled downtimes, even to hotfix serious bugs, and always very unpopular. And if there is some doubt about how serious the bug is -- if, for instance, it doesn't affect you personally ... well, things can get ugly. Add that to the ever-present risk, outlined above, that something will go wrong and we'll have to hotfix the hotfix, or that the servers will be down much longer than usual, and you can see why we prefer to spare our customers from hotfixes if we can.

In this particular case, we also need to be careful about the harware upgrades that are in the works. The Zone and the Zone Operations team has been putting in an incredible amount of time and effort, working with us to test the new hardware configuration on Shadowclaim so that we can minimize the risk of destabilizing the live worlds with new hardware. We've been leaning hard on those guys for this, because upgrading our server hardware is very important to us. But they are also the team that does most of the grunt work in a hotfix -- bringing down the servers, distributing the new data and executables, coordinating the process. So for this particular hotfix, we also have to weigh the importance of an immediate fix for this bug against the importance of getting the new server hardware installed. If we had to, we could do both -- but it would stress the ops team and could lead to mistakes. We'd rather avoid that if we can.

And, of course, there is also the live team itself to consider. Even a relatively simple bug takes time to investigate, fix, and test, and that's time that could be spent working on new content for the next update. Now there are some criticisms that the April update was thin -- and you know what, you're right. April was not a heavy content month. There are reasons for that that I won't go into right now, but suffice it to say that May is shaping up to be pretty damned solid. I'm very excited about May, and I think you will be too. All of the designers are working flat out to get their content in in time, and if we pulled one of them off that for a hotfix, we'd neccesarily lose something. Maybe not something major -- what's a little less lore, a little less storyline, a little less decoration? Ah, but there's another point -- testing. Hotfixes are tested intensely, and hotfix testing means that we would lose some testing time for May content. Decorations don't get tested (well, they do, but they don't take much time), so if we're losing testing time we're talking about losing the time to test quest logic, or a new weapon. We know how much you want solid story; we know how much you crave new dungeons and quests and trinkets. So what do we sacrifice for this hotfix?

So let's sum up: This bug has some side effects, the largest of which is an effect on the player economy. It's not a good effect, but we know the economy can recover. On the other hand, if we hotfix we will have guarenteed downtime with a risk of more serious problems and extended downtime, we very well may jeopardize our server upgrade schedule, and we'll have to punt some of the amazingly cool May content.

These are the factors we considered when we considered hotfixing this bug. Looking at all of these factors, we decided that the effects of the bug do not justify the risk of the hotfix.

Egad, it's 4 am! Well, I did warn you that I might ramble. I hope that this helps explain our reasoning somewhat. I know that my original post on this issue was rather short and to the point, and I know that that led some of you to believe that it was a glib dismissal, and therefore that we don't care about the effects of this bug or about the continued health of AC. But nothing could be further from the truth. If we didn't care, I could have gome to bed three hours ago when I got home from work. *grin* Seriously, we're not any happier with this bug than you are, but the realities of the situation have convinced us that this is the best course of action for everyone involved at this point.

[/i]