Really, it all boils down to character and Humphrey Bogart's performance as Sam Spade.
Sam Spade isn't a nice guy. He's greedy, misogynistic, homophobic, unconcerned with the death of innocents, and more interested in beating people up than justice. What do you say about a man who is carrying on an affair with his partner's wife only to try and brush her off after said partner's murder? That's just one scene that illustrates Sam Spade isn't the sort of hero you'd expect in cinema.
Sam doesn't have any real stake in the Maltese Falcon affair. His partner has been murdered but he's largely ambivalent about the man in death as well as life. Sam doesn't express any real loathing of the villains for their actions either, he's the kind of guy who notes there's murders every week so why should be concerned about these ones? The cynicism of the character drips through every scene and there's no eleventh hour redemption for him. He's as crusty and bad-tempered at the end as he was in the beginning.
Really, the meat of the story is Sam playing all the various parties involved against one another. Unlike your typical knock-off of Sam Spade, he rarely draws his gun and his fists don't do much more than annoy his enemies. Instead, Sam achieves most of his goals in the movie through the use of his wits. As Batman would say, criminals are a cowardly and superstitious lot, so Sam's strategy is to consistently appeal to their greed. They think he's on their side, so they let information slip and attempt to recruit him to their cause.
The thing is, I'm pretty sure Sam WOULD join with one of them if he didn't think they'd shoot him in the back the moment he turned around. Sam's greed isn't just an informed flaw, he's genuinely capable of making deals with the bad guys for money. It's just he's dealing with a group of treacherous weasels. They don't trust each other and Sam knows well they can't be trusted in turn. They don't trust Sam either but are all convinced he can be persuaded to their cause.
Contrasting Sam Spade is Mary Astor's Brigid O'Shaughnessy. She plays a weepy, sweet, innocent girl who appears to be the polar opposite of a femme fatale. Except, well, Sam sees through her disguise within minutes. It's an interesting character that lies to the hero's face then and when called on it, lies some more. It's one of the first realistic depictions of a sociopath in cinema. Right up until the very end, she's attempting to manipulate Sam by playing the helpless damsel-in-distress. The fact she's a dangerous monster is something she never let's the mask slip on.
Sydney Greenstreet's Gutman is my favorite character after Sam Spade, however. He's one of the original affably evil villains in cinema. You really get the sense he admires Sam Spade and wishes he could recruit him as an ally. When Mister Gutman turns on Sam, and it happens a few times, you get the sense it's strictly business. Plus, you have to admire a villain so cold he can mutter the line, "I couldn't be fonder of you if you were my own son. But, well, if you lose a son, it's possible to get another. There's only one Maltese Falcon. "
Peter Lorre's character Joel Cairo is the only character modern audiences may have a problem with. A flamboyant Dandy, the movie was written with the idea audiences would be repulsed by him. Repeated allusions are made to both Cairo's homosexuality and the implications are we're supposed to think of him as a deviant for it. Due to changing values, Joel Cairo comes off as the sanest one in the group. He's still a ruthless criminal but the intended effect is lost.
Good riddance, I say. He's still an interesting character.
Overall, I like The Maltese Falcon for its ability to tell a story drained of idealism. Sam Spade wants cash and possibly sex from his client. He's also willing to betray her, possibly, for a bigger payday. The villains are not caricatures, all of them want an object because it's valuable as opposed to any irrational motives. The tight plotting means there's possibly only a dozen characters overall and it works extremely well due to all of them being distinct. The Maltese Falcon may well be the first Noir movie and is certainly one of the best examples of the style.