Polyphonic note tracking using multi modal retrieval of musical events

This paper describes our work on the retrieval of polyphonic notes from a musical performance. Current state of the art transcription and score following systems extract features from the audio signal which are used to estimate what has been played in order to transcribe or find a position within a musical score. We propose that by extracting features from a video signal and fusing them with the audio features, the robustness of a system can be improved. We offer a framework which can be used to integrate these two modalities, and review preliminary work.