Families of the Winterbourne View victims have accused the Government, NHS and local authorities of an "appalling failure" over the rehousing of vulnerable people with disabilities.
They say there has been a "betrayal" of people who are at risk of abuse and neglect after ministers claimed the care of about 3,000 people with learning disabilities and autism should be reviewed in light of the Winterbourne View case three years ago.
Undercover filming by BBC's Panorama disclosed the pattern of serious abuse at the private hospital near Bristol and six workers were jailed for ill-treatment and neglect.
The families wrote in a letter to The Daily Telegraph: "Today we have seen the appalling failure of the Government, the NHS and local authorities to meet their own deadline for moving people with a learning disability out of places like Winterbourne View."
They also wrote: "The time for talking and excuses is over. The Prime Minister must take personal responsibility and address this failure of national government, local government and the NHS."
Care and Support Minister Norman Lamb said: "The terrible abuses at Winterbourne View were a wake up call across the whole health and care system but progress to bring about change has been unacceptably slow.
"I want to see a real change of pace in the next few months to move people out of institutions where appropriate, and, with the right support, back into the community. NHS England has been tasked to develop a clear plan to move things forward quickly."
Complaints have been made that some families are having to travel hundreds of miles to the units where their children have been housed.
Mr Lamb said he would "investigate" the distressing situation faced by the family of 13-year-old Josh Wills who suffers from severe autism. He is cared for in Birmingham, some 260 miles away from his family in Cornwall.
Mr Lamb said: "It is completely wrong that children with learning disabilities should be cared for so far from home. It is distressing both for them and for their families. The NHS must make sure it provides good quality care close to people's homes and we are working with the NHS to help this happen.
"I have heard first-hand from Josh's parents about his case and I am appalled that this remains unresolved. I was horrified by what Josh's parents told me and asked for this to be investigated further."