(with thanks to Elizabeth Linington, whose title I have borrowed for this post; and to Lexi Revellian, for suggesting the topic)
The subject of the "bad guys" in mystery fiction has come up. I think we all spend a lot of time thinking about what makes a good protagonist, as well as a good supporting cast. But what makes for a good "villain"?
This is a bit of a tricky question in that I usually think of the "villain" of a mystery as the murderer. And in a most mysteries, we don't know who the killer is until quite late in the book. (When we know early, I put the book into the "suspense" or "thriller" category.) The villains in the thrillers I've read lately have all been of a type or two: power-mad egomaniacs or garden-variety psychopaths.
So maybe we are better off thinking about the villain in a mystery as being an antagonist to the protagonist. Such characters need not be the murderer, but they do point to one of those frequently-unsaid truisms that good character development is based on conflict. Conflict comes in all forms, from basic family issues (such as sibling rivalry) to highly abstract types of conflict based on competing belief systems or deeply held values. In a good mystery, conflict drives not only the characters but also the plot.
That said, I do think that some types of antagonists are better than others. I always find intra-family conflict to be quite realistic, because I don't think I know of any family that isn't rife with it. Conflict on the job is good too, and raises all sorts of opportunities to explore relationships in the workplace (indeed, this has become a staple of many of the modern police procedurals). Antagonism based on personality conflict is always great fun--put a modest wall-flower in the same room with a narcissistic drama queen, and the resulting fireworks can be great fun to read. (Better yet, let's watch two narcissistic drama queens battle it out!) A mega-corporation or the government as the antagonist seems to me to be more the stuff of thrillers, as it's tough to make the avatars of this antagonism anything beyond a two-dimensional stereotype (money-mad CEO, hard-as-nails G-man).
In a way, we can think of the "villains" of the typical mystery as anyone who stands in the sleuth's way, or who erects obstacles for whatever reason. It can be quite interesting when an antagonist blocks the protag for psychologically rich reasons, such as wanting to protect a child or a secret; it can be equally enjoyable to watch the psychologically acute sleuth jump over these barriers or find a way to circumvent them.
If all is done well, when the identity of the murderer is revealed, we should have a good sense of what makes him or her tick. We should understand WHY the murder was committed, even if we don't condone it. The motive really does have to make sense; I've read some books where I've thought, "Huh? X had absolutely no reason to kill Y over that lost game of bridge [or shufleboard]." This may be the hardest thing for the novelist to pull off--to make the murder believable or inevitable based on the conflicts that have been set up and explored, but not fully realized until the denouement.