Asherons Call: Worlds Down (Updated)

This was the global message seen:
In preparing for the May Release tomorrow, it has become important that we move up the release to today. We will be bringing down the worlds in 30 minutes to apply the May Release. The worlds will be down for approximated 6 hours for the prop. We apologize that this happened during prime time.

AC Live Program Manager Ken Karl has just made a post over at the Official AC message boards here.
So some behind the scene stuff -- Preparation for a release starts usually 24 hours before the scheduled downtime. Things such as, putting out the client to all the servers that end up sending it to you, staging the new files so that once we turn off the worlds, we can make the prop downtime be as minimal as possible, getting out articles, etc.

In doing all of this, there was a failure in one of the processes that handles moving the new files out to some of the servers. The process decided that rather then staging it into a "holding pattern", that it would just go ahead and prop the file immediately. This put us in a situation where on some of the servers, we actually had May files on some servers. Now, this would not expose content, but it would put the game in a situation where if it was left unchecked or we did not go forward with the process that was already set in motion, over time, it could result in some bad data being stored to our databases. Unfortunately, it is also not as easy as just replacing files with April files either once this process is put in motion.

So while there is a slight chance that this would have been fine for 12 hours until the scheduled prop window tomorrow morning, we did not want to take any chances. We decided to take the worlds down immediately and go forward with the May release. The biggest downside of this is that it will result in downtime during primetime. Something we would never do intentionally.

How long will this take? Our normal scheduled window for a prop is 6 hours. Now yes, we do schedule in buffer for unknowns. So it is possible that this will take shorter. As you know, most of our props do infact take much less time. But it is always possible that it will go to the full 6 hours. We will do everything we can to make the downtime as minimal as possible. As I get updates, I will come post them. I expect to get them hourly or so.

Sorry for the inconvenience.

Update:
UPDATE - The update is humming along. There is a chance that we may still have to re-push out all the files again due to the initial error. This looks unlikely right now, but I wanted to let you know that this has not been completely ruled out. This is probably the one outstanding thing that would definately cause us to eat up the entire prop window.

Next update in a hour or so or as I get more news on the above item

Yes, Patch day is TONIGHT instead of the scheduled tomorrow date. Stay tuned to WCoD as more info comes in.