The priest at St Colman's Catholic Church in Kilkeel gave permission for a wall to be built at the boundary of a graveyard where it is believed scores of stillborn babies were buried.
Up until the 1970s, stillborn babies in Ireland were buried outside consecrated ground because they were not baptised.
The Keenan family buried their child, James, as close to the cemetery as they could after he was stillborn on 29 November 1969.
The family always left flowers at the spot. A memorial was later built for him and all the other stillborn babies in the area.
But on Friday, as two of James' sisters were passing the graveyard, they saw a digger tearing up the ground.
"It is unbelievable that they feel that they can dig human remains up and put them on a landfill.
"It's bad enough that he had to be buried in a ditch because he wasn't baptised," James' sister, Eileen Tate, told UTV.
"My father had to dig a hole and bury his son in a box and now I don't know where he is", she added.
"He matters, he matters to us. He's our blood, our brother. He deserves to rest in peace."
Parish priest Father Michael Murray, who gave the go-ahead for the church's neighbour to build a wall, says he regrets the mix-up and wants to meet the family to try and rectify the situation.
"It was all done in good faith and I suppose in any cemetery you just don't know.
"I tried to be as careful as a possible and spoke to the local undertaker to make sure this area hadn't been used for that," said Fr Michael Murray.
It is believed there are scores of similar sites across Northern Ireland.
A spokesman for the Catholic Church has urged other families to come forward and tell clergy where their loved ones were buried so this never happens again.SIC: U.TV