Holly Madison talks about the fear she felt when she left her relationship with Hugh Hefner: 'I was kinda synonymous with the Playboy brand'

Holly Madison can look back now and see that her relationship with late Playboy publisher Hugh Hefner, whom she dated from 2001 to 2008, wasn't good for her. But, when their breakup was happening, not so much.

"For me, I think," Madison said Wednesday on Nick Viall's podcast The Viall Files, "the scariest thing about leaving the relationship was — because I'd been in this hit TV show and been presented a certain way and everybody kinda thought they knew me, and I was kinda synonymous with the Playboy brand, and I didn't really know if I’d be able to go out and, like, lead a normal life or date like a normal person, so that was a big fear for me."

Madison co-starred on reality series The Girls Next Door, which aired from 2005 to 2010 and followed her and other Hefner girlfriends, Bridget Marquardt and Kendra Wilkinson, in their lives at the Playboy Mansion.

Looking back, Madison said there wasn't one incident that prompted her exit from Hefner's estate.

"It was a combination of things that finally made me ready to leave. It was like, you know, getting into my late 20s and realizing that I wanted to have kids someday and also, I'd started working more, whether it was through the show and also like working at the Playboy Studio," Madison said. "I started to get a lot more confidence, because I was working. And that helped."

And there was something else.

"But also, like, a strange thing kind of happened toward the end of the relationship where the other girls were leaving and it was just me, and he started to get really verbally abusive toward me..." Madison said. "He liked to, you know, encourage drama between the other girls, and he didn’t have that anymore. So it was just me realizing that he was the problematic one. Like, I could no longer say, 'Oh, I'm miserable here because of the other girls. Oh, if the other girls weren't so mean.' Not that Bridget or Kendra were mean. There was a long history of, like, other women that lived there before that, so I don't want anybody to think I am talking about either of them. But, for a long time I made excuses, saying, 'Oh, it’s just the other girls. If they weren’t here, you know, this could be a fun situation. But then when it was just me and him, I realized… this is what I'm dealing with, I can't do it anymore."

Madison looked back on her time with Hefner in her 2015 memoir, Down the Rabbit Hole, and earlier this year on A&E's Secrets of Playboy docuseries, calling the vibe at the Playboy Mansion "cult-like" and explaining that she felt she had been pushed into having sex with the media mogul on their first date. She also referred to Hefner, who was in his 70s when they began dating, as "manipulative."

She told Viall why she'd gone public with her true feelings.

"For me, when I shared my story, it was just important for me to share everything about it, like the bad things as well as the good things, because on Girls Next Door, our lifestyle was packaged to look just very fun and happy and almost like family friendly in a way," Madison said. "And after I left the show I was kinda like just going out living my life and I'd run into people every day who would come up to me and just have assumptions about my life or think they knew how things were, and it really started to bother me after a while because I felt like, when I would just respond to these people and say what I thought was the nice thing to say, like, 'Oh, yeah, everybody's great or whatever.' I started to feel like I was living a lie, and it just really started to weigh on me.

"And I felt like I had a lot to share as far as, like you said, power dynamics in relationships and relationships that are abusive in different kinds of ways, whether they’re, like, emotionally abusive or, you know, financially abusive."

Madison said she's been able to connect with others through her story.

"I mean, obviously the situation I was in was very unusual, and I think because it was so unusual, I was able to make excuses for it for a long time because, obviously, you get into it knowing it's not like a typical relationship," Madison said. "So I made a lot of excuses for it but I found that, you know, my story connects with a lot of people, and they see echoes of it sometimes in their own relationships that they then got out of."