Even thoughPrince Charlesis next in line for the British crown, after his marriage toPrincess Dianahe was no longer the main event — and he knew it.

According to a new memoir from retired Supreme Court JusticeJohn Paul Stevens, the heir to the throne once joked that he was a “second-class” citizen when he attended parties with his more dazzling wife.

Stevens met the royal couple in 1985 while at a formal dinner at the National Gallery of Art in Washington, D.C., and recalls Charles’ comment inThe Making of a Justice: Reflections on My First 94 Years, published Tuesday.

“I have no memory of any of the works of art displayed on that occasion, but I vividly recall meeting Princess Diana and her then husband,Prince Charles,” Stevens writes. “Diana, an unforgettably beautiful and gracious woman, was the center of attention for one group of admirers after another.”

Corbis via Getty

Princess Diana, Prince Charles

As Diana’s profile rose, making headlines and drawing interest around the globe, Charles was eclipsed.

“[Diana] sort of metamorphosed into a very, very elegant blonde beauty,” Lady Elizabeth Anson, a party planner and a cousin to the Queen, told PEOPLE last year. “They did a visit to Wales which was the beginning of, forPrince Charles, realizing, ‘They’ve come out to see my wife, they haven’t come out to see me.’ ”

“She could make any person, whether they were the grandest or the most humble, totally at ease,” Diana’s brother Charles, 9th Earl Spencer, told PEOPLE in 2017. “It’s an incredible gift.”

From left: Princess Diana and Prince Charles in 1987.Georges De Keerle/Getty

Princess Diana, Prince Charles

Prince Charles and Princess Diana.David Levenson/Getty

Princess Diana, Prince Charles

Princess Diana, Prince Charles, Prince Harry and Prince William.Tim Graham/Getty

Princess Diana, Prince Charles, Harry, William

Married at the age of 20, Diana had two sons with Charles:Prince William, now 36, and Prince Harry, now 34.

While Diana loved motherhood, she also grappled with personal and marital problems, includingdepression, bulimia and her husband’s infidelity. (After splitting from Diana, Charles went on to marry his longtime lover,Camilla Parker Bowles, now Duchess of Cornwall.)

And she didn’t keep quiet about her struggles: Diana secretly spoke to author Andrew Mortonfor a revealing 1992 biography, and in 1995 she gave an infamous interview to BBC’sPanoramasaying, “There were three of us [in the marriage].”

Diana “wanted to tell the world how ghastly things had been and how appalling,” her friend Dr. James Colthurst told PEOPLE in 2017. “She was worried she would be blamed and then sidelined. And then would lose her position as mum.”

“I was thrown into the deep end,” Diana once said of royal life, according to the Morton tapes. “Nobody ever helped me at all.”

source: people.com