Part of the reason is the undoubted good work he did during most of his years in active ministry.
His scandal involved a consenting adult.
The scandal was that he broke his vows and used diocesan funds to help his son.
I wonder what would happen if a similar scandal came to light today?
It is the view of Pope Francis that if a priest fathers a child, he ought to leave his ministry and help raise his child.
Would he have done that?
The scandals that later came to light were vastly worse than the one involving Bishop Eamonn Casey.
They were crimes.
Bishop Casey's hypocrisy didn't compare with the horror of sexually abusing a child.
There is no question, however, that the Bishop Casey scandal put a large crack in the previously pristine image many faithful Catholics had of priests.
The child abuse scandals shattered that image completely for lots of them.
They entirely and completely toppled the priests, the religious, and the Church as a whole, from their pedestal.
Now, we have the outcry over the Tuam Mother and Babies Home.
Earlier this week, I gave a talk at Curraheen parish in Cork.
I spoke about why those of us who are still practising Catholics hang on despite everything.
I said that when people believe in something in itself, they will hang in there despite the scandals that might engulf it.
No one who likes football stops playing it, or supporting it, because of betting scandals, drug scandals, or sex scandals, including the recent revelations of the sex abuse of minors by football coaches in England.
Instead, they seek to reform the running of football and make it better.
Naturally, you expect a higher standard of religion.
But if religion means more than football, and obviously it does to its own adherents, that is all the more reason to hang on in there and seek reforms.
A better comparison might be with the UN.
For its strongest supporters, the UN represents the dream of eventual peace and unity in the world, the universal reach of human rights law.
The UN is every bit as universalist in its ambitions as Christianity.
But it has been beset by every possible scandal, including the sexual abuse of children by so-called 'peacekeepers'.
Do those people who believe in what the UN stands for walk away from it because of its many (and under-publicised) scandals?
The best of them seek to reform it and make it better. The worst of them cover up the scandals, just like in the Church.
So, is there anything in the Church worth hanging in there for?
Yes. And it can be summed up in two words: Jesus Christ.
In the final analysis, Christianity is only worthless if Jesus is worthless. And almost no one believes that.
The Church runs into trouble when it departs from his teachings and example. For example, why did it take to throwing stones at unmarried mothers with such gusto when Jesus said, "Let he who is without sin cast the first stone"?
In a way, the very word 'Church' has become a problem. When people hear that word, they think of the institution, of the bishops and the priests and religious. Jesus is not the first thing that comes into their minds.
When people think of the institution first when they think of the Church, that is a major issue.
In the final analysis, what is the Church? It's not the institution. It is the community of the followers of Jesus. There are many such communities. That's why there are so many different churches, ranging from the Catholic Church to the Presbyterian Church and everything in between.
But they have something very important in common - whether they do it well, badly, or indifferently, they are all trying to follow Jesus.
To put it another way, Christians are hanging on in there because they can see the bigger picture.
They know what Christianity is supposed to be about - and it's not about the institution first and foremost, it's about a person, and that person is Jesus, who is admired even by many people who don't have a religious bone in their body.
Christianity has often taken a very authoritarian turn and succumbed to the punitive morality of the Pharisees, which Jesus forthrightly and repeatedly condemned.
It's not the only idea that has sometimes been hideously twisted out of shape. Think of socialism, for example.
At the same time, Christianity has inspired many people to live lives of great and heroic service to others, usually quietly and without notice.
This why, 15 centuries after St Patrick brought the faith to the whole of Ireland, it is still worth hanging on to.
It is why those of us who still remain Christian keep going - despite the scandals.