Project during Internship at
Xquisite
AI March 2025
Project Overview
A system for real-time spoken conversations in
Bahasa Indonesia. It combines speech
recognition, natural language processing, and
text-to-speech to create a direct audio-to-audio
interaction. Users ask questions aloud. The system retrieves the most relevant answers and responds with synthesized speech, allowing users to converse with a responsive virtual assistant.
How it works? (the pipeline)
The process begins when a user asks a question
verbally. The audio is transcribed into text,
which is then converted into vector embeddings
to capture its meaning. These embeddings are
searched against a vector database, and the top
three most relevant answers are returned.
A large language model refines these answers
into a coherent script. The script is then converted into audio and spoken back to the user, enabling a speech-to-speech interaction without a visual avatar.
Video Demonstration
For the demo, the system was configured to
provide information about
Mandiri Sekuritas
in real time. Users ask verbally, and the model
generates spoken responses directly.
Use Cases
Voice-based Customer Support
The chatbot can act as a 24/7 audio-first
assistant for handling customer inquiries. By
replacing long call queues or rigid IVR menus
with natural conversation, it provides faster
and more humanlike support.
Accessibility and Inclusion
A speech-only system is useful for users who may
prefer or require audio communication instead of
text or visual formats. This expands
accessibility for visually impaired users and
others who benefit from voice interaction.
Hands-free Assistance
In contexts such as driving, field work, or
smart home environments, speech-to-speech
interaction offers a practical solution. Users
can ask questions and receive answers instantly
without needing a screen.
