Campaigners have called on the Government to impose a time limit on the detention of immigrants after a highly critical review found that too many were being locked up.
Former prisons ombudsman Stephen Shaw also raised concerns about breaches of human rights laws in cases that suggested "problems with attitude and cynicism" on the part of staff.
There were some 3,000 people being detained - a mixture of asylum seekers, ex-offenders and those who had been deemed not to have a legal right to remain in the UK - while the number of people detained "at one time or another during the year" exceeds 30,000, Mr Shaw said.
The Home Office said the "vast majority" of those in detention were people who have "made their way to the UK unlawfully or breached their conditions of entry, have failed to make their case for asylum or are foreign criminals".
Immigration Minister James Brokenshire told MPs the Government would act on a number of Mr Shaw's recommendations about preventing the detention of victims of sexual offences, including rape, and those with mental illness.
He said: "The Government accepts Mr Shaw's recommendations to adopt a wider definition of those at risk, including victims of sexual violence, individuals with mental health issues, pregnant women, those with learning difficulties, post-traumatic stress disorder and elderly people, and to recognise the dynamic nature of vulnerabilities.
"It will introduce a new 'adult at risk' concept into decision-making on immigration detention with a clear presumption that people who are at risk should not be detained, building on the existing legal framework."
A more detailed mental health needs assessment will be carried out in immigration removal centres, reporting in March, and the Government will also introduce a "new approach to the case management of those detained".
But Mr Brokenshire rejected "arbitrary time limits" on detention, claiming that could lead to abuse of the system.
He said: "A stronger focus on and momentum towards removal, combined with a more rigorous assessment of who enters detention through a new gate-keeping function, will ensure that the minimum possible time is spent in detention before people leave the country without the potential abuse of the system that arbitrary time limits would create."
Among the 64 recommendations, the report called for greater internet access including to Skype and "social media sites like Facebook" to help immigrants detained in the UK keep in touch with their families.
Mr Shaw recommended that the Cedars facility close to Gatwick Airport, which provides pre-departure accommodation for families being removed from the UK, should be closed or have its use changed "as a matter or urgency" because of the cost.
He said detaining each family must cost "many tens of thousands of pounds" but "up to half are actually released rather than being removed" and "the current use of the centre is simply unacceptable at a time of financial austerity".
Mr Shaw wrote that his findings were "challenging" but "many are not new".
"Most of those who have looked dispassionately at immigration detention have come to similar conclusions: there is too much detention; detention is not a particularly effective means of ensuring that those with no right to remain do in fact leave the UK; and many practices and processes associated with detention are in urgent need of reform," he said.
"In my view, a smaller, more focused, strategically planned immigration detention estate, subject to the many reforms I have outlined in this report, would both be more protective of the welfare of vulnerable people and deliver better value for the taxpayer.
"Immigration detention has increased, is increasing and - whether by better screening, more effective reviews, or formal time limit - it ought to be reduced."
He said he was "acutely concerned" to discover that there had been six cases - although one had been overturned on appeal - where the Home Office had been found to breach Article 3 of the European Convention on Human Rights, which prohibits torture and inhumane treatment.
A review by Jeremy Johnson QC "suggests that the cases may indicate problems with attitude and cynicism on the part of some staff", the report said.
Refugee Council head of advocacy Lisa Doyle said: "This review shines a welcome spotlight on the hidden, abhorrent and often unlawful treatment of vulnerable people within Britain's shadowy immigration estate.
"The fact that Britain quietly continues to imprison rape victims, children and torture survivors without a time limit is completely unacceptable.
"The 64 recommendations for substantial change should lead the Government to acknowledge that wholesale reform is needed.
"Quite simply, Britain detains far too many people for far too long and such unnecessary imprisonment ruins lives, wastes money and is an affront to human dignity and justice."
Liberal Democrat home affairs spokesman Alistair Carmichael said: "The UK is one of only a few countries across the world which does not have a time limit of depriving people of their freedom.
"This can lead to enormous trauma and have a damaging impact on people's mental health, not to mention the millions it costs the UK taxpayer to keep these people locked up.
"For the first time we have cross party support for a time limit and I am disappointed that the Government has not taken the opportunity to radically shake up the system for the better."
Amnesty International UK's Steve Symonds said: "Most people wouldn't know it, but the UK locks up thousands in immigration detention, including rape victims, torture survivors and people with serious mental illness. This can and frequently does have a terrible impact on their mental health.
"In some cases the harm it does can be catastrophic. The system destroys people's lives, and is utterly unacceptable.
"It is high time UK Government ministers listened to the experts, who time and again have warned of the terrible consequences of the UK's excessive use of detention."
Shadow immigration minister and former director of public prosecutions Sir Keir Starmer said: "It is clear from this report that the rules meant to protect vulnerable people are not working.
"The Government should now take action and implement the recommended ban on the detention of pregnant women, and change the current rules that are failing to protect vulnerable people.
"We need a firm and humane immigration system, and this independent report shows the Government is failing to provide this."