Back in 2007, it emerged that phone-in competitions had been rigged on their ITV shows Gameshow Marathon and Saturday Night Takeaway.
ITV was later fined over £5 million by TV regulator Ofcom in relation to the misconduct.
Declan Donnelly told BBC Radio 4's Desert Island Discs: "We thought that was probably it. I think the viewers would have been justified in thinking their trust had been breached and if that had been the end I don't think we could have had many complaints.
"It was, without doubt, our worst moment, our worst point. Totally. We couldn't quite comprehend how it was allowed to happen."
Ant McPartlin added: "I think our audience trust us and with that case we worried that the trust had gone.
"It did affect us... it was probably the worst part of our career.