Pope Benedict XVI will conduct an open-air Mass in front of 70,000 people at Cofton Park on Sunday 19 September.
The park is open from 2am with around 20,000 pilgrims expected from the Archdiocese of Birmingham.
But its spokesman Peter Jennings says some parishioners have withdrawn their application for a place at the Mass.
Draconian security measures
Mr Jennings, who previously called the security plans 'draconian' told BBC WM: "I'm not criticising the security in any way - the government have to decide not the Catholic Church.
"I felt the security was draconian - meaning very rigorous - which i was, but I'm expressing the anguish and anxiety that's been expressed to us by many of the older people in the Archdiocese of Birmingham.
"They are very concerned about getting up in the middle of the night and getting on coaches.
Same fee for everyone
"The security is extremely rigorous and has put a large number of people off. We are getting people who are saying 'I put my name down for a pilgrim invitation, but I want to take it off."
Mr Jennings also confirmed that all pilgrims would have to pay a £25 contribution fee to attend the Mass, regardless of how far they are travelling.
"That has been set in conjunction with the church and the government," he said.
SIC: BBC