The problem with dialogue system components in RPGs tends to be that they’re components in RPGs.
I’ve been playing Fallout: New Vegas: This game, like the earlier Fallout games, is premised on making important dialogue choices which can have a big impact on how the rest of the game plays out – the name Fallout, itself, being a double entendre for the aftermath of a nuclear war and the aftermath of a difficult choice.
As with all Fallout games, you have stats which govern how skilled a speaker and negotiator your character is, and these govern what you can and can’t say in dialogue trees. Unlike earlier iterations, you can see in the dialogue menu which skill is being tested by choosing a particular dialogue option, along with whether or not it will succeed ahead of time. Together with the fact that you always get an experience bonus for passing a skill check in dialogue, this has a couple of unfortunate effects:
First, the dialogue option for a failed skill check will almost never be triggered by any players. Even if the dialogue choice for a failed skill check seems reasonable to the player, and even if the consequences of it are more interesting, no one will ever select that choice if they can possibly avoid it. This kills an entire alternate dialogue branch for each skill check, and allows the player to game the system in strange ways, like spotting an option that’s too demanding, leaving, killing a few monsters to level up, and coming back with an improved speech stat.
Second, the player will always choose a choice that uses a skill check. Not only are these usually guaranteed to solve the problem at hand, they also give a small experience bonus for passing the skill check. This discourages the player from choosing the option that seems best to them in favor of choosing the option the game tells them is best, removing an opportunity to make an interesting choice by essentially making the choice for them.
Now, the problem isn’t that certain dialogue choices are rewarded and others aren’t: The idea of choosing to do something because it will create an advantage for the player, potentially at the cost of screwing over another character, is integral to the idea of Fallout. The problem is that some of these rewards are part of the narrative universe of the game and others are not: I feel like a heel extorting extra money out of someone who I’ve done work for, but I feel like an idiot for passing up the free exp I can get by passing a check of my barter skill.
A similar problem came up in the Mass Effect series. In the first game, you can increase your ‘paragon’ and ‘renegade’ skills – essentially good cop and bad cop dialogue choices – the same way you’d level any other skill in the game. This means that, if the player wants to, they can max both of them out, opening up all possible dialogue options and allowing the player full rein to choose whichever they prefer. However, in later games you only get paragon/renegade points by choosing the associated dialogue options, which creates a good cop/bad cop feedback loop. Thus, if you want to be able to choose the goodest good cop actions or the baddest bad cop actions, one or the other of which will be necessary in order to achieve optimal outcomes in later scenarios, you have to ignore your own moral compass or role-playing desires and always choose to be the badass or the softy.
These systems were all made with the best of intentions, and there were other good reasons, in terms of streamlined design and player feedback, to make these decisions – however, when carelessly implemented, they end up undermining the greater design goal of naturalistic and compelling player choice.