Secondary characters done right are the ones who aren't just there as part of a story, aren't just there to "support" the protagonist . . . and aren't obviously appearing to fulfill a function for some purpose ordained by a writer in another universe. These characters breathe. They feel. They have independent emotions and they don't behave as if they're less of a person just because they have less time on stage. They feel like they started living when they were born, not when they walked into the protagonist's life.
One of my favorite secondary characters is Butler from the Artemis Fowl series by Eoin Colfer. Butler supports his underage charge, Artemis, through thick and thin, and is probably the best bodyguard in existence (for a person who really needs one). He's actually come very close to death more than a dozen times while protecting Artemis, and he doesn't just protect him physically—he supports him in all his ridiculous evil genius schemes, and his actions are inspired by love for his charge just as much as they are inspired by his sense of duty.
But besides just being a great bodyguard and a loyal protector, Butler has depth. His family has protected Artemis's family for generations. (His sister, Juliet Butler, is similarly trained, and later protects Artemis's little brothers, among others.) Artemis didn't know Butler's first name until he really almost died because of a promise he made. He's also fiercely protective of his little sister even though she can take care of herself. He's a layered dude and an inspiring (if imposing) character. I love what his protection allows Artemis to do in the stories, and I love how he develops certain relationships with the other characters that also braid loyalty and compassion together with competence and badassery. (And the jokes about him being too big to fit in certain cars, chairs, and rooms are delightfully visual and fun for the younger kids who read the books.)
I love Butler. I wish I had one.
Another secondary character I love is Bailey from The Sisterhood of the Traveling Pants by Ann Brashares. Okay, so Bailey's "job"—in a book where Tibby is the rightful main character of her storyline—is to inspire Tibby to care about the right things and do some tearjerking. She's supposed to do this by being a sad girl with cancer whom everyone figures will die at the end and exists primarily to teach everyone about the preciousness of life. Too bad Bailey had other plans.
Bailey is inspirational partly because she isn't trying to be. She isn't what you'd expect. She and Tibby pretty much detest each other and develop what could only be described as a grudging respect for one another. It was so amazing to see that Bailey's gruffness and unpleasantness continued to be part of her personality even after she warmed to Tibby and tried to emulate her—because after all, her life is about her, and she doesn't want anyone's pity friendship. In stories containing an inspirational kid with cancer, usually they're angry at the world because of their disease and experience a personality overhaul when they realize their time is limited. Bailey isn't like that. She's her. She's not just a kid with cancer.
I loved seeing her wear those magical pants.
When I write, I try my best not to stick supporting characters into the mix just to do their thing and leave. I don't want to make them unique by tacking on a catch phrase or a quirky behavior. I want readers to understand them as complex people, with evidence that they have opinions and preferences and life stories that aren't part of the book. I want them to be as fully formed as any main character—and I want them to be fleshed out enough that if the story happened to be about them, there'd be enough material there to make it interesting. Stories in which the "secondary character" is the protagonist, like Confessions of an Ugly Stepsister by Gregory Maguire or Book of a Thousand Days by Shannon Hale, are especially wonderful in this way. They show us how the supporting character can be the whole show in the end.