Cardinal Keith O'Brien resigned last month after claims of inappropriate behavior towards several priests came to light.
His four accusers are dismissing reports that quote church sources saying O'Brien was forced to quit by gay priests angry at his hypocrisy about same-sex marriages, the UK Guardian reports.
"This isn't about people being gay. It's about abuse of power," said "Father Peter", who revealed in the Observer report which broke the story that he had been involved in an inappropriate relationship with the cardinal that led to him needing counseling.
"The emotional and psychological power Keith O'Brien had over me was incredible. He was utterly manipulative." Former priest "Lenny," who described how he had rejected the O'Brien's advances while he was a seminarian, said the newspapers were trying to force pieces of the story together.
"I was surprised at the suggestion that I was part of a gay cabal. And my partner of 26 years? I'd say she was quite surprised too," he said.
Details have leaked of a fifth complaint dating to September 2012. That priest reached an "agreement."
The four men, 'Lenny,' 'Peter,' 'John,' and 'Kenny,' made their complains against O'Brien five months later and were told to stay silent and that the cardinal would retire to Rome.
The men, fearing another cover-up, went public, leading the cardinal to resign and issue an apology.
"There have been times that my sexual conduct has fallen below the standards expected of me," the cardinal admitted.
"Father Peter" said the complaints were never political or timed to affect the papal conclave. Our complaints were made before the pope resigned. I am now more convinced than ever they played a part in his resignation. But this is not something that all happened last month. I sought help for this a decade ago. I was on antidepressants and I couldn't cope any more."
The four accusers knew one another but were not close friends. It was only after a chance reunion that Lenny and Peter discovered their common link to O'Brien.
"We couldn't have acted alone because Keith was too powerful," said Father Peter. "Gradually, we all found one another. We had each thought that Keith had a problem, but then we realised there was more to it. This was a man who was using his power wickedly."
The four complainants maintain they were not motivated by malice or revenge in any way.
"This was done for altruistic reasons," says Father Peter, who is still seeing a psychologist.
"I see the bigger picture: the cleansing of the church." Lenny said he had not received a formal response from the nuncio and did not know whether there would be an inquiry.
He said he believes the church would make them all disappear if it could.
"I think we are seeing evidence of this in the sheer anger of these statements. These may be men of the cloth but they are not men of the holy spirit," he said.