He is among a group of MPs and peers warning that the population should not be allowed to go beyond 70 million.
Lord Carey said immigration angered many people and could lead to violence, and that the system had to focus more on maintaining "values".
Labour says the system works, but the Tories want caps on incoming workers.
All the main parties are sceptical about setting population targets which they believe is unrealistic and counter-productive.
Last year the Office for National Statistics said, if current trends continued, the UK population would rise by 10 million to more than 71.6 million by 2033 - the fastest rise in a century.
'Unprecedented'
Two-thirds of that increase would be caused, directly or indirectly, by migration to the UK, it suggested.
The Balanced Migration Group - made up of 20 parliamentarians, including Lord Carey, former Commons Speaker Baroness Boothroyd, five Labour MPs and 10 Conservative MPs - is backing a campaign calling for curbs on immigration.
Lord Carey told BBC Radio 4's Today programme: "We are not calling for a ban or anything like that because we value people from abroad."
"What I think I'm concerned about is not saying we must put a limit on people who are non-Christian populations. That's not the point. We welcome everybody and that's always been the generous spirit of the United Kingdom."
But, he said, immigrants must "understand" the UK's culture, including parliamentary democracy "which is built upon Christian heritage", "our commitment to the English language" and an understanding of the country's history.
The system should not "give preference to any particular group", he said, but added that points-based immigration could take these cultural aspects into consideration.
'Competing groups'
Lord Carey added: "If there's going to be an implementation of that points system, it must focus much more on values rather than religions...
"If there are competing groups wanting to come in, some groups which may have a greater understanding, an espousal to that, may be given preference under a points system, but that's not what I'm arguing and certainly not what the cross-party group is arguing."
Lord Carey told BBC Radio 5 live: "We've got to be more outspoken. What I'm calling for is a debate, a debate without any rancour."
He added that immigration was an issue that mattered to "ordinary working-class people" and that it was important to tackle "that kind of resentment which could build and is building up already".
Lord Carey said too much population growth in the UK could foster "dangerous social conditions", with some minority ethnic groups, such as young Muslim men, suffering "disproportionate" unemployment.
Labour MP Field and Tory MP Nicholas Soames, the co-chairmen of the migration group, said: "Poll after poll shows the public to be deeply concerned about immigration and its impact on our population.
"It is time parties turned their rhetoric into reality by making manifesto commitments to prevent our population reaching 70 million by 2029."
70 million 'unlikely'
The government must "restore control" over the UK's borders and "break the present almost automatic link between coming to Britain and later gaining citizenship", the group said.
Cabinet ministers have tried to do more to address public concerns about immigration, saying the issue must not become the preserve of the BNP.
Last November, Gordon Brown promised to "tighten" the UK's immigration rules by reducing the number of professions which can recruit from outside Europe while making it harder for illegal workers to enter the UK by obtaining student visas.
He said new restrictions were having an effect, adding the 70 million projection was unlikely to materialise.
The Conservatives have said they would keep the government's points-based system but place an overall annual limit on numbers and try to attract more highly qualified migrants.
The Lib Dems say they would ensure migrants were directed to parts of the country where they are most needed, where they will be welcomed and there are the resources to accommodate them.
Net migration - the number of people who come to live in Britain minus those who leave - fell by more than a third in 2008 but critics say this was driven by eastern Europeans returning home and immigration levels must fall to levels of the early 1990s.
+++++++++++++++++++++++++++++++++++++++++++++
Disclaimer
No responsibility or liability shall attach itself to us or to the blogspot ‘Clerical Whispers’ for any or all of the articles placed here.
The placing of an article hereupon does not necessarily imply that we agree or accept the contents of the article as being necessarily factual in theology, dogma or otherwise.
SIC: BBC