The harassment wasn't bad when the registered sex offenders moved to the Wood hill neighborhood in February.
The small group of men occasionally heard taunts from children and teens when they left their five-bedroom rented home.
But the Catholic charity that helped provide the housing said the taunts escalated to threats and vandalism that prompted the men to relocate last weekend.
"It really became impossible for the men who are living there to feel safe," said Ginny Ramsey, co-director of the Catholic Action Center. "When that happens, it is not wise to keep them there."
This is the second time this year that the Catholic Action Center has had to alter plans to provide shelter to registered sex offenders. The charity tried to buy a house on Detroit Avenue in January for sex-offender housing, but the group declined to complete the purchase after neighbors vigorously opposed it.
The Catholic Action Center, which serves Lexington's homeless population, has seen more homeless sex offenders since more stringent limits were imposed on where they can live, Ramsey said. Under restrictions enacted in 2006, convicted sex offenders may not live within 1,000 feet of schools, public playgrounds or licensed day-care centers.
The sex offenders need supervised housing "so they can get on their feet and be contributing members of society," Ramsey said.
The law that outlines the sex offenders' residency restrictions fails to outline where they can live, said Lexington attorney Rebecca DiLoreto, who is familiar with the sex-offender restrictions.
"All the law did was make it nearly impossible for these folks to find a place to live," she said, "and we have to have a place for them to live for their entire natural lives after they've served their sentence."
The Catholic Action Center contacted the Woodhill neighborhood association before it rented the home. Association members said they didn't think a community meeting was necessary to discuss the sex-offender housing, Ramsey said.
The five residents, three of whom were registered sex offenders, told Ramsey the harassment escalated in April.
Kids called them rapists and threw rocks at them, which caused some bruises on one of the residents. Then someone broke into the house and broke a table, a lamp and other furniture. One of the residents left.
Ramsey isn't sure whether the men reported the threats to police.
"They were trying hard to get along," she said. "They wouldn't complain."
The Catholic Action Center decided to move the men out of the home at the end of May for their safety. Ramsey declined to identify the men or give an address of the home.
"We realized they were living in fear, and that wouldn't do anybody any good," Ramsey said.
DiLoreto said people who harass registered sex offenders think they have the authority to take the law into their own hands.
"If we don't have leadership that says this isn't the way you should treat these people, you'll see that kind of action," she said.
One of the sex offenders now lives in West Virginia. Another found somewhere else to live. The third is staying in a motel.
"What we're trying to do is give them a chance to get back in society, not to be behind locked doors," Ramsey said.