EVE: The Anti-Lag Arsenal

The latest EVE Online developer log deals with the issue of lag and how developers do their level-best to prevent it. Check it out below:

Wow, this has turned into quite a long blog, but it's chock full of much info-goodness which hopefully addresses many of the questions and concerns people have had about public testing and how we approach large-scale performance and feature testing for EVE.

What's in our anti-lag arsenal?

As EVE grows, more features are added, more players are online at once, and the whole system becomes more complex we find that our ability to weed out sources of lag becomes increasingly difficult. Throughout EVE's development we've used various methods to identify and fix sources of lag; server performance monitoring, special task-forces to 'seek and destroy' lag, monitoring forums and petitions, public test servers, and intensive regression tests of all new code, for example. Though these are great tools, we don't feel that it's enough. Over the years, while creating all of our expansions and the new features and changes that come with them, it became apparent that we had a problem: We had no good way to simulate high-load situations such as fleet fights, factional warfare, Jita, etc. This is a particularly difficult situation for us. In any high-load situation there are always a very wide range of actions being performed simultaneously, with many different hardware configurations on the client machines, and many small factors quickly add up. But these factors are all relatively unpredictable and almost random in practice.

Consider Jita. Think of how many people are there; now think of all the different market transactions, combat, chatting, industry, and other activities that are going on simultaneously in that system - you can see that testing this scenario is not as simple as "load up 1000 clients and let them run." We needed a way to get as many different transactions and machine configurations as possible in order to confidently say that we've gotten a valid test for high-load performance.

Over a year ago, in order to address this issue, we began exploring several different options. We looked at everything, from specially built load-testing clients (bots), to hiring many more testers, to community-supported testing, to developing specialized test suites and scripts, and everything in between. After some prototyping, tests, and wracking our brains, we finally came up with a multi-prong approach that includes community-supported testing (i.e. Mass-Testing - more on this shortly), massive test-server upgrades and changes, load-test clients (still in the early stages of development), special debugging code, and more data logging than you can shake a stick at.

There's a boatload more to read so head to the link above.

Permalink