How To Build A Real Time AI Conversation Co-Pilot

A regional insurance agency was losing deals it should have won, not because the products were wrong, but because newer agents froze in the middle of live calls. I am Madhuranjan Kumar, and I want to walk you through how one agency turned that weakness around by building a real time conversation co-pilot, a quiet assistant that listens to a call, transcribes it instantly, and suggests the next smart question before the moment passes. This is the full story of that build, including the numbers that made the case to keep it.

The morning three deals slipped away

The agency had ten agents handling inbound calls about life, home, and auto policies. On a single Monday, three separate calls went sideways in the same way. A caller mentioned a new baby and a bigger house, and the agent, focused on the quote in front of them, never asked about updating life coverage or bundling home and auto. Each caller thanked the agent, said they would think about it, and hung up. Nobody did anything wrong on paper. The agents were polite, accurate, and fast. They just missed the obvious follow up because a live call asks you to do three hard things at once: listen closely, ask the right qualifying question, and recall the exact policy detail that fits.

The owner had seen this pattern for months. New agents took roughly four months to reach the call quality of a veteran, and during that ramp the agency was quietly leaking revenue on every missed cross sell. The owner did not want a script that made agents sound like robots. What they wanted was the feeling of a senior colleague sitting beside every new hire, nudging them at the right moment. That is exactly what a conversation co-pilot promises, and for the first time it was genuinely buildable.

Why this was impossible until recently

For years this idea failed on one word: speed. A co-pilot that takes thirty seconds to answer is useless in a conversation that moves every few seconds. Two things had to get fast at the same time, and until recently at least one of them was always too slow.

The first is transcription. Standard speech to text tools are built for finished recordings. You hand them a whole file and wait, which is fine for a podcast and hopeless for a live call. The second is the suggestion itself. A large, powerful model can write a beautiful answer, but it thinks too slowly for a moment that has already moved on. The agency could not have built this two years earlier because the transcription lagged and the models lagged. By the time they started, fast speech models and smaller, quicker language models both existed, and the whole thing finally felt instant instead of clumsy.

Building the transcription that keeps up

The team started with the ears of the system. Instead of waiting for the whole call to finish, they captured the audio in short chunks, every two to five seconds, transcribed each chunk, and stitched the pieces together so text appeared almost live on the agent's screen. That got them ninety percent of the way, and then they hit the classic problem: chunking chops words in half. When a chunk boundary lands in the middle of a sentence, gluing the pieces together produces garbled text that confuses the model downstream.

The fix was word level timestamps. By knowing exactly when each word started and ended, the system could compare the overlapping edges of two chunks and clean the seam, so a word cut between chunks was reassembled instead of duplicated or dropped. This one detail was the difference between a transcript the model could reason over and a transcript full of noise. The agency spent most of its early effort here, because everything downstream depends on clean, near live text.

Making the suggestions land in seconds

With a reliable live transcript, the team turned to the brain. Their first instinct was to use the biggest, smartest model available, and it produced wonderful suggestions that arrived far too late. So they switched to a smaller, faster model, and the trade was worth it. In a live call, a good suggestion in two seconds beats a perfect suggestion in twenty. Speed mattered more than raw power.

They also had to manage the growing transcript. On a long policy review, feeding the entire conversation back to the model on every request got slower and slower and eventually risked hitting the model's limit. The team borrowed the same trick good agents use for memory. They kept the last several lines word for word, since the recent context is what matters most, and replaced the older history with a running summary that refreshed periodically. On top of that, they stripped redundant filler out of the transcript before sending it, so the model read only what mattered. Less text in, faster suggestion out. That combination kept response time flat even on calls that ran twenty minutes.

The first two weeks on real calls

The agency did not roll this out to everyone at once. They put it in front of two newer agents first and watched. Before the co-pilot, the internal measure of response delay, the time from the system hearing something worth acting on to a useful suggestion appearing, sat around thirty five seconds in their early prototype, which was clearly too slow to use live. By the fourth week of tuning the chunking, the model choice, and the transcript trimming, that delay dropped to about eight seconds. By the twelfth week it was down to roughly two seconds, fast enough that agents forgot it was there and simply glanced at the screen when they wanted a nudge.

The workflow was simple by design. Before a call, the agent loaded context: the caller's situation if known, the products the agency offers, and the qualifying criteria that matter. During the call, the transcript scrolled in near real time, and whenever the agent wanted, they tapped a button to get the next suggested question or the fact they were reaching for. When the caller mentioned the new baby and the bigger house, the co-pilot surfaced the exact prompt to ask about updated life coverage and a home and auto bundle, and reminded the agent of the specific rider that fits a growing family. The agent still owned the conversation. The tool just made sure nothing relevant slipped by under pressure.

The numbers that justified keeping it

I always frame these figures as illustrative, because every agency is different, but the shape of the result is what mattered to the owner. Take two comparable agents each handling around two hundred inbound calls a month. Before the co-pilot, roughly one in eight of those calls had an obvious cross sell or coverage gap that went unasked. That is about twenty five missed opportunities a month per agent. If even a fifth of those turn into a real policy add at a modest annual premium, the lost revenue across a ten person team is not small, and it compounds every month.

With the co-pilot catching the moments agents used to miss, the two pilot agents started asking the coverage question that had been going unasked. The owner did not need a perfect number to make the decision. Seeing new hires ask veteran level questions in their second month instead of their fifth month was enough. The ramp that used to take four months visibly compressed, which meant every new agent started contributing sooner. That alone paid for the build.

Where the co-pilot ends and the human begins

The team drew one hard line and never crossed it. The co-pilot supports the expert, it does not pretend to be one. The whole reason this idea got a bad name early on was people trying to use it to fake expertise they did not have, like feeding someone answers in an interview. That is the wrong use, and it produces brittle, dishonest conversations. Pointed at genuine coaching, the same technology raises the quality of the call instead of faking it. The agent still gives the advice, still reads the human on the other end, and still owns the relationship. The co-pilot just makes sure the right question gets asked and the right detail gets remembered.

That distinction also kept the tool trusted internally. Agents did not feel replaced, they felt backed up, which is why adoption stuck.

How the co-pilot changed the rest of the funnel

The interesting part came after the calls got better. Cleaner, more consistent conversations meant richer notes flowing into the CRM and website stack, where follow up automation could pick up every coverage gap the co-pilot flagged and schedule the next touch. The agency also learned which questions actually moved callers, and fed that language back into its Facebook and Instagram ad campaigns and landing pages, so the messaging that worked live also worked in the ads that drove the calls. Even the SEO and organic search content improved, because the team finally had a clear map of the real questions customers asked, and could write pages that answered them directly. One tool aimed at live calls quietly upgraded the whole path from ad click to closed policy.

The mistakes the agency avoided the second time around

The first prototype was almost abandoned, and it is worth saying why, because the traps are the same ones that sink most co-pilot projects. The earliest version tried to be too clever. It pushed suggestions onto the screen constantly, unprompted, and agents found the stream of interruptions more distracting than helpful during a live call. The fix was to make the co-pilot silent by default and only speak when the agent tapped for it. A quiet assistant that answers when asked beats a chatty one that talks over your thinking.

The second trap was chasing accuracy at the cost of speed. The team briefly reverted to a larger model because its suggestions read better, and adoption cratered within days, because a suggestion that arrives after the moment has passed is worse than no suggestion at all. They learned to protect latency as the single most important metric and to accept slightly plainer wording in exchange for speed. In a live conversation, timing is the feature.

The third trap was letting the transcript context grow unmanaged. On the longest policy reviews, the early build slowed to a crawl near the twenty minute mark because it was resending the entire conversation each time. Only after adding the summary plus recent lines approach did response time stay flat regardless of call length. The lesson generalizes to any live AI tool: you have to actively manage what you feed the model, because the naive approach of sending everything gets slower and more expensive with every passing minute.

Finally, the agency almost over scoped the rollout. The temptation was to launch to all ten agents at once and measure everything. Starting with two agents instead let the team tune the tool against real calls, catch the interruption and latency problems early, and build internal trust before a wider release. By the time the other agents got access, the co-pilot already felt reliable, so adoption was easy rather than forced.

What I would tell an owner considering this

If your business runs on conversations, this is worth taking seriously. Start by wiring together fast transcription and a fast model behind a simple interface. Capture short audio chunks and stitch them with word level timestamps for accuracy. Choose a small, quick model so suggestions arrive in seconds. Give the user a context box to describe the call and the help they want. Manage the transcript with a summary plus the recent lines so it stays fast on long calls, and trim the filler before each request. Then show the live transcript and let the user tap for the next suggested question or answer.

The agency in this story did not need a giant budget or a research team. It needed clarity about the one moment it kept losing, and the patience to tune latency until the tool felt instant. You can build this yourself if you enjoy the engineering, or you can have someone assemble the transcription, the fast model, and the context handling into a clean tool your team simply opens on every call. Either way, the payoff is the same as it was for that agency: better conversations, faster onboarding, and far fewer opportunities that quietly walk out the door.