Darkfall: Exclusive Interview with Erik Johansen

A small preface regarding the technology - Darkfall is expected to use 40-50 load balanced servers inter-connected to form one massive server cluster. This cluster will work in unison to form one huge "server" that will be presented to the players as a single game-world.

On to the interview!

----------------------------------------------------------------

Warcry - It was stated at one point that databases would be able to be updated in real-time without bringing down the servers, does this mean that the game-worlds could be updated with patches without having to come offline?

Erik - Almost any server component can theoretically be updated without downtime for the world simulation. We have however chosen to not use these features on production servers, to improve the QA of the update process. Patching an online simulation will inevitably lead to a higher risk of errors, and the gain is simply not proportional to this increased risk.

Warcry - When you were originally talking about the server-cluster design, the theoretical limit was 60,000 players. After factoring in internet bandwidth etc you revised this to approximately 15,000-20,000. Have the estimates for number of supportable players changed or should we still expect to see those kind of numbers?

Erik - Theoretically there's no limit to the number of simultaneous players in a single world. In practice, the individual server load will exponentially increase as the number of server computers in a world simulation increases, and at some point we will not get additional gain from further horizontal scaling (adding more computers). At what point we reach this limit is hard to say before open beta and real measurements, but currently my requirement is at least 10000 players, and my hopes are for double that.

Warcry - It would make sense that all MMOG's would want to take advantage of this kind of technology, is there a reason why you think we haven't seen it used before now?

Erik - I think the technology we use is very hard to add into a game after development has started. In order to successfully create a truly distributed simulation, every single component must be written with parallelism in mind from day one. We have designed and implemented everything we use in-house, and we have always been aware of the need to run this in a distributed environment. We use a combination of several well known and tested technologies and a few inventions of our own to make it possible. There's nothing magical about it, it's simply a matter of thorough design with distribution as one of the main design goals.

Warcry - You mentioned in the past that the servers will balance load between them. Will this also be able to handle loads that have crashed other MMOG servers like several hundred (or even several thousand given what Darkfall is estimated to do) people in a single place all battling at once?

Erik - The servers will handle it, but the clients may not. There will be a limit to the number of players we can have in a small area, simply because there's a limit to how much information we can pass on to each player in that area. We're using a few tricks to get this number as high as possible, but it most definitely will be limited.

Warcry - Kind of a technical question - but is it possible to give a little more detail about how exactly the load will be transferred to another server in the cluster if one server fails during a major battle for example?

Erik - We are using a variation of a relatively new technology for persistent data storage to achieve this. I can't go into the technical details of this, as it's a key aspect of how we intend to achieve the high number of simultaneous players, but I'll mention that it includes real-time state data mirroring on multiple computers, and uses our load-balancing technology to distribute responsibilities among server machines in case of a server failure.

Warcry - Some gamers have indicated that they don't believe you can have that number of simultaneous players without a huge internet connection. Can you tell us anything about the datacenter that will house the game servers and what kind of bandwidth you will have access to at that facility? Could you give us an idea where the servers will be physically located?

Erik - We will need a very huge internet connection in order to sustain 10000 players. assuming an average bandwidth usage per player of 5k/sec, 10000 players will require 50 MB/sec, which approximately corresponds to a 400 MBit internet connection. We've been working with several providers with that kind of capacity.

Warcry - Some current MMOG's have a problem with server instability and going offline. Could you comment on why you think such instability occurs and what Razorwax is doing to prevent and or reduce it?

Erik - There are many reasons for servers to go offline. You could have Denial of Service attacks, which are almost impossible to control. You could have ISP network problems, which are outside our control. And then you can of course have software or hardware problems on the servers itself. This part we can do something about, and we work hard to make our servers as fault tolerant as possible. We are fully aware that there will be hardware and software faults from time to time, and that we need to reduce the effects of these. We will have no single point of failure in a server cluster, and our load balancing mechanisms are fully able to cope with a limited number of failing servers. We can't guarantee 100% stability, but we can guarantee that we'll do all we can to make such problems as invisible as possible to the players.

That concludes our questions and this interview. We want to thank Erik Johansen and Tasos Flambouras for taking the time to make this possible.

-Malevolent