NEWS

AZ dumps election vendor after failures in 2012, 14

Rob O'Dell, and Yvonne Wingett Sanchez

The secretary of state has not renewed the contract of its elections vendor
The state had numerous problems properly displaying election data in 2012 and 2014
Arizona paid the vendor %24945%2C000 since 2010. It will now do election night results in-house

The secretary of state has dumped the vendor responsible for publishing Arizona's election results online after persistent problems on Election Day in 2012 and 2014.

The state's election website was slow, difficult to load and produced error messages during the 2012 primary and general elections. In 2014, former Secretary of State Ken Bennett and the election vendor SOE Software promised improvements, but the problems persisted with outages and errors that frustrated voters and candidates.

Matt Roberts, spokesman for the secretary of state, said the contract was not renewed in March, because of SOE Software's poor election-night performances, "communication issues," and the system's limited customization features.

"Specifically in those first 20 minutes (on election night), where it's very important that we get it right and display results in those minutes, where everyone wants to see them, and there are issues with it, that's a problem for us," Roberts said. "We were pretty disappointed with our performance in those elections."

In a statement, Bo O'Dea, vice president of marketing for the company that owns SOE, wrote: "After five years of working with the State of Arizona, SOE's Election Night Reporting (ENR) contract will expire naturally. While ENR did experience slowdowns during the 2014 elections, since that time we addressed the core issue, invested heavily in architecture enhancements, hired an independent technology firm to test our changes and have successfully reported results for over 40 elections across the country."

Instead of hiring another vendor, Secretary of State Michele Reagan, the state's elections chief, is building an election-night reporting system in-house. Prior to its contract with SOE Software, the office ran election-night reporting internally.

The state hired Tampa-based SOE Software following a competitive process in 2010 and paid the company $945,000, according to figures provided by the secretary of state.

Bennett, who gave the green light to hire the vendor, was criticized for the system's poor performance during the August 2012 primary.

Bennett was out of town during Arizona's 2012 primary to attend the Republican National Convention in Tampa. There he also met with SOE officials to discuss the problems and try to ensure they would be fixed before the general election. But problems occurred in the 2012 general election as well.

E-mails obtained by The Arizona Republic through a public records request show that two years later, before the 2014 August primary election, state officials anticipated more problems with SOE's system.

The day before the primary, Assistant State Election Director Kris Kingsmore wrote to SOE representatives that, "we are extremely concerned that we are heading into a repeat of 2012. As we all remember that was a disaster for us."

Kingsmore wrote she was concerned about page load times. "I cannot stress it enough — this MUST be fixed headed into election night."

It wasn't.

After the polls closed, SOE's election system posted incorrect data for Maricopa County. In addition, Cochise County election results were declared unreliable and pulled from the website temporarily, an issue election officials blamed on Cochise County.

Yavapai County's election results weren't posted until nearly 6 a.m. the following day, despite e-mails showing SOE and the secretary of state had tested loading Yavapai's results prior to the primary election.

Kingsmore again wrote to SOE saying that a company statement of responsibility for errors in the Maricopa County data didn't go far enough, especially because state officials had warned the company of problems on June 27.

"Since SOE gave inaccurate information during the initial configuration, then introduced the error during the reconfiguration, and then failed to effectively communicate the requirement for testing before the new file was uploaded, we feel SOE should take full responsibility," Kingsmore said.

In the 2014 general election, the site would not load for more than 20 minutes after voting results began rolling in at 8 p.m., and the site was completely unresponsive for an hour later that night.

E-mails show a frantic back-and-forth between SOE and state officials, and Arizona officials pressing SOE to take responsibility for the breakdowns. SOE sent a statement asserting the problems were fixed by 8 p.m., but state officials disputed it.

Kingsmore responded: "This was NOT resolved at 8 p.m. We are adamant about 8:30 p.m. We can share media tweets and reports with you if that helps us all agree on the time. I'm not sure why you want to split hairs on that. The same problem is occurring again, right now at 10:25 p.m."

SOE later sent a statement saying that a network file-sharing service had failed and vowed to make sure they would never happen again.

DJ Quinlan, former head of Arizona Democratic Party, said in 2012 he experienced a "complete loss of access to the site."

"The whole site seemed like it just couldn't handle the traffic," Quinlan said.

He said he has concerns about the Secretary of State's Office handling election reporting in-house, noting that the office needs technical expertise and a commitment to openness in building the system.

On Tuesday, Roberts said one of Reagan's priorities is to improve the reporting system. "Nobody is more sensitive to our efforts to display election results in a timely effort than we are," he said.

The office has hired and dedicated two employees to develop and oversee the system; another hire is expected soon and additional employees will help develop the system. The office has budgeted $550,000 to develop and maintain the system, Roberts said.

He said the system will "meet the expectations of ourselves and also the elections community and voters."

"Our disappointment that we experienced in '12 and '14 was significant," he said. "It's not good enough for us to experience some of those problems time after time and not make a change."