Voice Agents for Business: What They Can Do, What They Can't and How to Start

Voice agents are no longer science fiction but productive tools: they answer calls, book appointments and qualify prospects. This guide gives you an honest overview of what works, what doesn't and how to get started.

In short: A voice agent is an AI that conducts phone conversations in real time by combining speech recognition, a language model and speech synthesis. It is best suited to recurring conversations such as call answering, appointment booking and initial qualification, and less suited to emotional or legally sensitive conversations. Start with a single, clearly defined use case and a clean handover to humans.

What is a voice agent?

A voice agent conducts conversations by phone or voice interface in real time. The technology consists of three links:

1. Speech recognition (speech-to-text): The caller's speech is converted into text.
2. Language model: It understands the context and formulates a response.
3. Speech synthesis (text-to-speech): The response is spoken in a natural voice.

What matters is the overall latency: every noticeable pause feels unnatural on a call. That is why all three links work in streaming mode: Twilio Media Streams, for example, transmit the audio of live calls to the application in near real time via WebSocket, and Deepgram's live speech recognition delivers interim transcripts while the caller is still speaking. This is exactly how anicall.io is built: an end-to-end streaming pipeline of Twilio, Deepgram, Groq with Llama 4 and ElevenLabs.

Modern voice agents can:

• be interrupted (barge-in)
• ask follow-up questions
• respond to changes in mood
• hand conversations over to staff
• carry out actions, such as booking appointments, updating the CRM or sending emails

As soon as a voice agent carries out actions, it is an AI agent in the narrower sense and needs clear permissions and approvals. The basics are described in the guide AI agents in business.

The five strongest use cases

1. Inbound calls around the clock

Typical sectors: trades and crafts businesses, medical practices, law firms, gyms, estate agents

The voice agent answers every call, including at night and at weekends. It answers frequent questions about opening hours, prices and services, forwards emergencies to a human immediately and takes callback requests with a complete summary.

Example calculation with assumptions: Suppose a business misses two calls a day, i.e. 44 a month over 22 working days. If one in ten of these turns into an order worth €500, around €2,200 in revenue is lost each month. Plug in your own figures; above all, the calculation shows how quickly missed calls add up.

2. Appointment booking without the back-and-forth

Instead of email ping-pong, the voice agent calls, finds a free slot via the calendar integration and confirms it directly. Typical in healthcare, consulting, coaching and B2B sales.

3. Lead qualification

Every incoming lead is called promptly, qualified against fixed criteria and prioritised. Only prospects with a genuine need reach the sales team. Combined with lead scoring, such as anilead.io provides on a scale from 0 to 100, sales gets a clear order of priority.

4. Outbound campaigns

Payment reminders, appointment confirmations and satisfaction surveys can be handled in parallel and consistently with voice agents. Keep German unfair competition law in mind: under Section 7 of the Act against Unfair Competition (UWG), advertising calls to consumers are only permitted with prior express consent; for businesses, at least presumed consent is required.

5. First-level customer support

A large share of support requests often revolves around a few recurring topics. A voice agent that answers these reliably frees up the team for more complex cases. For the answers to be correct, it should draw on a verified knowledge base instead of improvising. How to build such a knowledge base with source references is explained in the guide Knowledge AI with RAG.

What voice agents cannot do

I see too many presentations that portray voice agents as a cure-all. The honest list:

Voice agents reach their limits with:

• very complex, emotionally charged complaints
• specialist fields without a maintained knowledge base
• strong regional dialects, even though recognition is improving
• conversations that require genuine empathy, such as bereavements or serious medical news
• legally sensitive information with liability risk

The rule of thumb: Use voice agents where availability and consistency matter more than nuance and empathy.

Legal obligations

In addition to competition law, the transparency obligation in Article 50 of the EU AI Act applies: people interacting with an AI system must be informed of this unless it is obvious. The agent should therefore identify itself as AI at the start. If calls are recorded or transcribed, you also need a legal basis under the GDPR and a data processing agreement with the provider. The overview Using AI telephony in compliance with the GDPR (in German) summarises which rules apply, from recording and cold calling to the Betriebsrat (works council).

Starting in three phases

Phase 1: Pilot

• choose a single, clearly defined use case, such as appointment confirmations only
• test with real calls, but a limited volume
• define success metrics: drop-off rate, success rate, satisfaction

Phase 2: Integration and fine-tuning

• CRM and calendar integration
• improve prompts based on real conversations
• set up a clean handover to staff for escalations

Phase 3: Expansion

• open up further use cases
• build reporting and monitoring
• keep learning from conversation analytics

Costs at a glance

Billing is usually per call minute, made up of telephony, speech recognition, language model and speech synthesis; if you combine the building blocks yourself, you also bear integration and maintenance. Ready-made solutions bundle all of this; anicall.io, for example, starts at €0.13 per minute. The calculation method, list prices and assumptions are in the article What does AI telephony cost?.

If you want to deploy voice agents, you will find our approach under Voice AI. How such a product is built from scratch is shown in the field report Building a voice agent SaaS (in German). Book a call now.

From my own products

At anicall.io, our AI receptionist for inbound calls and AI setter for outbound, two design decisions have made more difference than any voice. The first concerns knowledge boundaries: the agent is built to answer only from verified sources. Anything it doesn't know ends up as an approval task for the team instead of the agent improvising.

The second concerns the handover. A qualified conversation is worthless if nobody knows afterwards what was discussed. So plan from the outset how the outcome and context of a call arrive in the CRM and calendar. At anicall.io, there is an MCP server and HMAC-signed webhooks for this. My lesson: a voice agent delivers the greatest value when the handover afterwards is as well organised as the conversation itself.

Frequently asked questions

What is the difference between a voice agent and a chatbot?

A voice agent conducts spoken conversations in real time, usually on the phone. A chatbot communicates via text. Voice agents are technically more demanding, because speech recognition, natural speech synthesis and very short response times have to work together, but they feel more natural to callers.

Does a voice agent have to identify itself as AI?

Yes, unless it is obvious to callers. Article 50 of the EU AI Act requires that people be informed when they are interacting with an AI system. In practice, the agent therefore introduces itself as AI at the start of the call and offers a handover to a human if needed.

Can I set up a voice agent without AI expertise?

Platforms such as Vapi, Bland or Retell let you set up a simple pilot without in-depth AI knowledge. For integrations with CRM and calendar, your own knowledge bases and legally compliant operation, however, experience with voice AI and telephony is very helpful.

Which use cases are best for a first voice agent?

Recurring conversations where availability and consistency matter more than nuance: answering inbound calls around the clock, booking appointments, qualifying leads, outbound reminders and confirmations, and first-level support. Start with a single, clearly defined use case, such as appointment confirmations only.

What does a voice agent cost?

Billing is usually per call minute and covers telephony, speech recognition, language model and speech synthesis; building it yourself adds integration and maintenance. Ready-made solutions bundle these components; anicall.io, for example, starts at €0.13 per minute.