Dangerous World, Mistrust, Slaughter
The open PvP is small for a reason as well. There are a number of reasons that such a system turns off a vast majority of players. Generally speaking, open PvP tends to create a much less casual-player-friendly game, with a lack of trust between other players, and the general dislike of griefers.
Since anyone can attack anyone at any time, for any reason, players that find themselves 'behind the curve' in experience in the cookie-cutter MMO design will find themselves lagging far behind other players. Effectively, they become easy prey with little that they can do to beat the higher level players. That said, to effectively create an open PvP system to appeal to people who do not have the hours to invest into their characters, a developer would effectively have to tone down the meaning of character advancement - something that can be a bit dangerous to do. If nothing else, lower experienced players need to have a role that they can fill. The latter is a major reason why EVE Online has seen success in its system, as players in small ships have certain benefits in larger battles - they are hard to hit and can prevent other, larger ships, from leaving a fight.
The lack of trust between players is a major problem as far as community goes. It can be hard for a new player to find an organization and a group of people that they can rely on, that they can feel comfortable that they won't get stabbed in the back by them. Who wants to group with some random people, when you never know if they'll just turn around at the end of the night, kill you, and ensure you have no share of the rewards of the evening?
It can be very difficult to meet people that you do not have a prior relationship with, and that can be very dangerous as far as community goes. That said, if you are able to get in with a group, you will likely form close bonds with them. However, that is an 'if' for many people. Without a solid guild, a player may well find themselves playing in a very lonely world.
Of course, for all the player roles that griefers can form, they're also vastly irritating for the vast majority of players. People just flat out don't like being killed when they're about to complete some goal that they've been investing a fair chunk of their time in, or even just to be randomly killed at all. This system doesn't bode well for people who aren't always in the mood for PvP.
Success Not Guaranteed
In the end, it can probably be summed up as this: EVE Online is a successful game. Shadowbane is not.
Both of these games used open PvP systems, but Shadowbane did not manage to take off like EVE Online did - and EVE had a very hard start as well. There are a limited number of people who are going to be willing to play in an open PvP game. Even in EVE Online, if it did not design its systems to help out the casual player, and if the (relatively) safe areas of space protected by NPC police were removed, it would likely suffer a pretty sharp drop in population.
A game built around open PvP can be successful... but it must be designed with care.
