AI Implementation Reality Check: Quality Issues and Consequences Ahead

Enterprise organizations are rushing to implement artificial intelligence without proper understanding of its limitations, creating a situation where many are "pretending that they know" how to effectively deploy AI technologies, according to experts from AI advisory service Codestrap.

"No one knows right now what the right reference architectures or use cases are for their institution," said Dorian Smiley, co-founder and CTO of Codestrap. "A lot of people are pretending that they know. But there's no playbook to pull from."

The Fundamental Problem with AI in Business

Smiley and CEO Connor Deeks, both former PwC consultants who now run their own AI advisory firm, argue that companies have gotten ahead of themselves in their AI adoption. "From the large language model perspective, people aren't really addressing the fallibility of the underlying text," Deeks explained.

If organizations were building AI systems from first principles, they would look drastically different from what's currently offered on the market, Deeks contends. The widespread narrative about AI eliminating entire professions is similarly unfounded, according to the experts.

"We don't subscribe to any of that," Deeks said regarding predictions of mass job displacement. "For the most part, [businesses] don't want to believe that everyone's going to be fired and there's not going to be anyone underneath them, particularly in the technology or information organizations inside these institutions."

Measuring the Wrong Things

A core issue in AI implementation is the reliance on inappropriate metrics to evaluate effectiveness. "Even within the coding, it's not working well," Smiley stated. "Code can look right and pass the unit tests and still be wrong."

Organizations typically measure AI success using metrics like lines of code and number of pull requests, which Smiley identifies as "liabilities" rather than measures of engineering excellence. Proper metrics should include deployment frequency, lead time to production, change failure rate, mean time to restore, and incident severity.

"We don't know what [AI-specific metrics] are yet," Smiley admitted, though he suggested measuring "tokens burned to get to an approved pull request" as one potential approach.

To illustrate the problem, Smiley pointed to an attempt to rewrite SQLite in Rust using AI that passed all unit tests and appeared correct superficially. "It's 3.7x more lines of code that performs 2,000 times worse than the actual SQLite," he explained. "Two thousand times worse for a database is a non-viable product. It's a dumpster fire. Throw it away. All that money you spent on it is worthless."

Fundamental AI Limitations

The underlying limitations of current AI technologies contribute significantly to these problems. "We know what the limitations of the model are," Smiley said. "It's hard to teach them new facts. It's hard to reliably retrieve facts. The forward pass through the neural nets is non-deterministic... meaning you're going to get a different answer every time."

Moreover, AI systems lack inductive reasoning capabilities and cannot verify their own work. "A model cannot check its own work. It doesn't know if the answer it gave you is right. Those are foundational problems no one has solved in LLM technology," Smiley emphasized.

Business Incentives Misaligned with Quality

A particularly problematic aspect is how business incentives often conflict with proper AI implementation. At large consulting firms, partners seek increased revenue and higher margins, which leads them to substitute AI for human labor rather than using AI as a complement with human oversight.

"The incentive for the director is to stop talking to the associates, because the associates don't know anything," Smiley explained. "[The director is going to] use AI to do the work of the associates. For the associate, the incentive is to get the work done faster and go to the beach. All these incentives are not aligned in a way that makes AI complementary to the business and deliver outcomes."

Consequences on the Horizon

The combination of technical limitations and misaligned incentives is setting businesses up for significant problems. Smiley expects "problems related to code quality that surface in eight to nine months for people who are heavy users of AI."

Deeks foresees a growing number of lawsuits as organizations begin to face consequences from AI-generated errors. "People are going to continue to start to feel the pressure of 'I have to adopt this stuff, I have to make AI decisions,'" he said. "They're going to put this stuff into production... And that accelerated collapse is then going to cost a lot of people their jobs."

Other emerging consequences include:

• Pricing pressure: Companies are demanding discounts when they learn service providers are using AI tools. "Even KPMG pressured another accounting firm to lower their price because they've been saying they use AI," Deeks noted. "Customers are now saying things like, 'Oh you're producing your PowerPoint decks with AI. Well I want to pay you less.'"

• Insurance challenges: Large insurers are becoming wary of underwriting policies that cover companies against AI risk. "Insurance underwriters are seriously trying now to remove coverage in policies where AI is applied and there's no clear chain of responsibility," Smiley warned.

Deeks revealed that insurers are already lobbying state-level regulators to carve out AI-related liabilities from business insurance policies. "That kills the whole system," he said.

Recommended Approach for AI Implementation

Given these challenges, Smiley recommends that organizations begin with experimentation and iteration in a feedback loop. "The first step for organizations considering AI is experimenting and iterating in a feedback loop," he advised.

Organizations should establish proper metrics before implementing AI at scale, focusing on outcomes rather than superficial measures like lines of code. They should also implement human oversight processes that account for AI limitations and potential errors.

"We need to be clearer about what AI means for finance, for underwriting, and for actual business and the practical operation of business systems," Deeks concluded. "Can we actually have a conversation about it? Is anyone going to talk about the opposite of AGI [artificial general intelligence] and how it's going to take over everything in a utopian future?"

Businesses that approach AI implementation with realistic expectations, proper metrics, and appropriate safeguards will be better positioned to navigate the coming reckoning and emerge with effective AI strategies that deliver genuine value.