(CNN) -- — Admit it: You've gone to the grocery store in your PJs.
Maybe you just needed half and half for your morning coffee or went out after a day of marathoning whole seasons of "Divine Design." Maybe you really like how your pajamas feel and aspire to wear them Hugh Hefner-style, which is to say, as often as possible.
But deep inside your soul -- in the place that determines the difference between right and wrong -- Clinton Kelly believes you know it's taboo.
"It's a universal fashion affront tied into social appropriateness," the "What Not To Wear" style maven said of loose, knit ensembles.
He's witnessed nothing less than a paradigm shift in Americans' concept of appropriate clothing over the past 10 years on "What Not to Wear," as he helped 600 fashion victims learn to put their best face (and well-dressed-self) forward.
But the allure of pajamas, yoga pants and track suits is strong.
"We are continuing, as a culture, on this downward spiral of style," Kelly said, "I think that there will always be a group of people, a strong percentage of the population, who will care about their appearance. But now we have permission not to care."
What changed? He said it's that people no longer face the same social pressures they would have centuries ago, or even a couple decades ago.
That social pressure, the unwritten rules about how someone should or should not dress, has always existed, said Susan Kaiser, a University of California, Davis, professor in textiles and clothing, and women and gender studies.
"Clothing has been described as a broadcast signal, the way it communicates," Kaiser said. "We know (an appropriate outfit) when we see it. When something doesn't look quite right, then we notice that."
That knowledge is called a social norm, she said, and the norms surrounding fashion incorporate the function of clothing as well as the cultural attitudes of the day.
"We're a pretty Puritanic kind of country," Kaiser said. "That has influenced our clothing a lot."
As a result, American social norms have always boiled down to modesty, she said. Colonists hardly showed any skin or even hair in public, an aspect of American dress that didn't change much until the 1900s.
The notion of privacy has been intertwined with ideas of modesty since Colonial days as well, said Karen Herbaugh, the curator of the American Textile History Museum in Lowell, Massachusetts.
While performing chores around the house or going to the market, "(colonists) definitely would not have been wearing clothing that would have been considered private," Herbaugh said. "They would have been wearing all the proper layers of clothing."
In Colonial times, that meant women would wear a chemise, a corset, a corset cover, a bodice, multiple layers of petticoats and then a dress, Herbaugh explained.
"And you would have been wearing that regardless of whether you saw someone else or not," she said. "It just wouldn't have been appropriate otherwise."
The line between public and private clothing started to blur with the advent of the "indoor wrapper" during Victorian times, Herbaugh said.
"These were considered appropriate for a woman to wear to receive a woman friend at her house. They still were proper, they were just a little less restrictive," she said. Barely more formal than a robe, this precursor to the housecoat allowed women the opportunity to be in a semipublic situation without having to wear all the bustles and petticoats that public dress usually required. Men had their own version of wrappers as well, she said.
Still, it's a far cry from the pajama-clad parents Herbaugh sees dropping children off at school.
That boundary of public and private dress has moved from the house into the neighborhood, Kelly said. We no longer seek clues about appropriate dress from our community. These days, we look to celebrities, he said.
"I think that one photograph of one celebrity wearing something (inappropriate) can validate someone," he said. "And they will leave the house wearing the same thing."
Before Cameron Diaz was photographed wearing Ugg boots, no one would have considered wearing them in public, he said.
The cult of comfort is not only determined by stylish starlets, said Kaiser, the textiles professor. Clothing experienced a revolution in the 1960s, thanks to textiles such as polyester, rayon and spandex, as well as mass production technology.
"Rudi Gernreich, the famous designer in the '60s known for designing the one-piece topless bathing suit, was also known for doing a lot of knits," she said. "He predicted that we would all be wearing, like, knit dancewear by now. And I think he's kind of right.
"We've become accustomed to a kind of comfort level in knits."
But the devil is in that forgiving detail of stretch, Kelly said.
"It's very easy in this country to do two things," he said. "One: overeat. Two: wear really comfortable clothes. The problem is, when you combine both of those things you get caught in a spiral."
Large portions of cheap, overly processed foods keep us wearing stretchy, knit clothes -- that makes it easier to eat more large, overprocessed portions.
"I've heard this from women time after time: 'Before you know it, you've gained 30 pounds.' And then you get to a point where it's hopeless -- 'I can't lose 30 pounds.' "
"You say, 'Uh, I give up,'" he said. "'I'll keep wearing my big comfy pants.'"
Would you wear pajamas or other ultracasual clothes outside the house? Tell us why or why not in the comments, on Twitter @CNNLiving or on CNN Living's Facebook page.