Our research interests include speech processing, natural language processing, multimedia
information retrieval, machine learning, and pattern recognition. Our research goal is to
develop methods for analyzing, extracting, recognizing, indexing, and retrieving information
from audio data, with special emphasis on speech and music.
In the field of speech, research has been focused mainly on speaker recognition, spoken language
recognition, voice conversion, and spoken document retrieval/summarization. Our recent achievements
include locally linear embedding-based approaches for voice conversion and post-filtering, discriminative
autoencoders for speech/speaker recognition, and novel paragraph embedding methods for spoken document
retrieval/summarization. Our ongoing research includes audio-visual speaker recognition and speech enhancement,
subspace neural networks for spoken language/dialect/accent recognition, many-to-one/non-parallel voice conversion,
and neural network-based spoken document retrieval/summarization and question answering.
In the music field, research has been focused mainly on vocal melody extraction and automatic generation of
music video. Our recent achievements in this field include an acoustic-phonetic F0 modeling framework for
vocal melody extraction and an emotion-oriented pseudo song prediction and matching framework for automatic
music video generation. We have successfully implemented a complete automatic music video generation system
that can automatically edit a long user-generated video into a music-compliant short professional-like video.
Our ongoing research includes continuous improvement of our own technologies and systems, cover song identification,
and automatic generation of set list for concert video, so as to facilitate the management and retrieval of a large
music database. Future research directions also include singing voice synthesis, speech to singing voice conversion,
and music structure analysis/summarization.