AI Uncovered with homie AI: When AI Has to Prove Itself in a Real Customer Project

AI Uncovered Vol. 1 brought development, testing, and skills together. The focus was on practical experience: What drives AI projects forward? Where do they get stuck? And what do we only learn once AI is used in the real world?

The room at the appmatics office in Cologne was packed. To kick off appmatics’ new event series, the agenda featured three perspectives: Build. Test. Qualify.

Afterwards, the questions, discussions, and collaborative work on new ideas continued over pizza and drinks.

AI projects rarely fail because of the AI

The event began with Julia Kirchhefer, Chief of Product at homie AI. During the Build session, she shared insights into developing production-ready AI assistants for retail and industry—and the challenges that arise on the journey from the initial idea to launch.

One common stumbling block appears right at the beginning: the objective is defined too broadly. “We want to use AI to help our customers” may sound reasonable at first, but it leaves crucial questions unanswered.

What exactly should the AI help with? Which queries should it answer? And what specifically should improve for customers or employees?

Successful AI projects therefore begin with a clearly defined use case and realistic expectations. Only once the underlying problem is understood can a solution be created that provides genuine value in everyday use.

Going live is not the finish line

Many teams want to solve every conceivable edge case before launch. That is understandable, but it can delay the go-live unnecessarily.

Even extensive internal testing cannot fully replicate how real customers ask questions. They phrase things differently, introduce new topics, and quickly reveal where knowledge is missing.

Going live early does not mean compromising on quality. Instead, it means launching within a clearly defined scope, closely analyzing how the assistant is used, and continuously improving it based on real conversations.

Which questions arise most frequently? Where do misunderstandings occur? What information is missing? Many of these answers only emerge through real-world use.

Turning conversations into customer insights

At homie AI, customer interactions are therefore not only answered but also systematically analyzed. Chat Analytics summarizes conversations, identifies topics, captures sentiment, and highlights recurring questions.

Projects often reveal that around 40 percent of questions are repeated. The five most common topics alone can account for approximately 60 percent of all conversations.

This gives companies a clear basis for prioritization. If a topic occurs frequently and is also associated with negative sentiment, it signals a particularly urgent need for action.

In this way, the AI assistant improves step by step. At the same time, it generates valuable insights for product management, customer service, sales, and content teams. Individual customer questions become knowledge that can be used throughout the organization.

Good technology requires clear ownership

In addition to a specific objective, a production-ready AI assistant needs up-to-date data and clearly defined responsibilities.

Who maintains the knowledge base? Who evaluates new queries? Who decides which topics should be improved first? And who oversees the system after it goes live?

Production AI is not a one-off IT project. It evolves alongside users’ questions, the product range, and the company’s requirements. This requires people who take responsibility—both on the provider’s side and within the company.

Testing and skills are essential

In the second session, Florian Pohle from appmatics demonstrated how AI can already support testing today. His key point: teams do not need to automate the entire process immediately. It makes more sense to start where genuine bottlenecks exist. Responsibility for quality remains with people.

Prof. Dr. Mario Winter then turned the focus to the skills required. Anyone developing and testing AI-based systems needs not only technical expertise but, above all, sound judgment, collaboration, and a clear understanding of the technology’s limitations.

An evening that continued long after the final slide

AI Uncovered Vol. 1 did not end with a single formula for success. Instead, it provided a fairly clear picture of what production AI requires: a specific use case, reliable data, clear responsibilities, and the courage to learn from real users early on.

Fittingly, the evening was far from over after the three presentations. Many conversations revolved around participants’ own projects, unresolved questions, and experiences from their day-to-day work. These discussions revealed just how different the starting conditions can be—and how similar many of the challenges remain.

AI Uncovered brought these perspectives together: from development and quality assurance to the skills required within teams. After all, a compelling demo can show a great deal. But whether an AI solution truly works is only determined where it is used every day.