State audit report says schools are rating themselves too high in program reviews

An aspect of Kentucky's public school accountability system is under increased scrutiny after the state Department of Education found that schools were scoring themselves too high in self-evaluations.

One result is that the education department in 2015-16 is auditing three times the number of schools it audited last spring.

In Kentucky, 20 percent of a school's and district's overall accountability score comes from a school's self-evaluation of how well it is teaching subjects such as arts and humanities and writing that don't lend themselves to paper and pencil tests. The self-evaluations — called program reviews — are state-mandated.

In a pilot audit of eight schools across the state, the education department found instances in which schools scored themselves too high in areas such as kindergarten through third grade, writing, practical living/career studies, and arts and humanities.

"This finding indicates that schools do not understand the procedure for evaluating their programs," an audit report presented in June to the Kentucky Board of Education said.

The audit team disagreed with the school self-evaluation score 63 percent of the time, with every disagreement being an "overscore" by the school, said Karen Kidwell, program standards division director.

Kidwell said, for example, a school might have rated itself as "proficient," but the state determined in the audit that the school was actually in the "needs improvement" category.

"It's human nature," Associate Commissioner Amanda Ellis told the board in June. "You want to give yourself the benefit of the doubt."

Susan Weston, an independent education consultant based in Danville who has worked for the Prichard Committee for Academic Excellence and other education groups, said the findings present a "quite serious problem."

"It's quite a big deal because we currently don't have a system that is working," Weston said. "The big problem is not that they put down the wrong number. The big problem is if everyone is saying that their program is great, they are not saying, 'how can we make them better.' "

"It's what's better for kids that we ought to keep our attention on," she said.

State officials are reminding schools that one purpose of the program review is to improve the quality of teaching and learning.

Several state school board members said at their June meeting they were concerned by the findings.

Board chairman Roger Marcum said Friday in an interview that he was concerned about whether program reviews provided an accurate reflection of the quality of programs.

"We'll consider all options, but we want to see what we can do to improve the process," Marcum said.

The eight schools were selected by state officials based on a number of demographic and achievement indicators. Beginning with this school year, the state will audit 24 schools. State officials should be doing even more audits, Weston said, but the education department does not have the resources for that.

Weston said the state should consider having people in the community with expertise in certain fields conduct independent audits so that the scores of more schools could be scrutinized.

The audit report said the state might need to consider changing school ratings and accountability results "when schools are found to be over-rating and are not responsive to recommendations."

No schools have faced sanctions because of the audit, but Ellis told the board that in the future, overscoring could be considered a testing violation.

"It's concerning to us, but we don't think it's not something that can't be fixed," said Kidwell. She said schools could benefit from more training on the program reviews.

Ellis told the Kentucky school board members that the state would provide better guidance to schools. Also Kidwell said the schools in the pilot were open to the audit team's suggestions and wanted to improve their programs.

Bourbon County High School in Paris was one of the schools audited.

"It was a learning process for both entities," and Neely Traylor, Bourbon County curriculum and instruction supervisor.

"We wanted to see if we were aligned with what they think proficient is and they gave us feedback to say, 'This is why it would not be proficient,'" said Traylor.

"We still feel like we are (proficient) but we can collect better evidence to support that rating," Traylor said.

Madison Middle School in Richmond was also audited.

No one intentionally computed incorrect scores, said Steve Evans, Madison County's district assessment coordinator. Evans was also Madison Middle Schools' principal last school year.

School staff and the audit team were looking at the same material and coming up with different scores, because there are 'value judgments" and gray areas, Evans said in an interview.

"We felt like our scores were accurate,' Evans said "I can see their side and I think they can see ours."