Abstract: The main objective of the indexing process is to assign labels to the audio-visual data in order to describe its content.Audio indexing plays a key role in this process.In this paper,a speech-based manmachine labeling system for media asset management is presented.The system recognizes the phrases spoken by the human annotator automatically and assists him to mark up shots of subjects in a video sequence.The phrases are segmented from short sentences and modeled by the elliptical basis function(EBF) networks.Experimental results indicate that the speech-based labeling system is practical and has great promise for media asset management.