Music Signal Processing for Cochlear Implants

Cochlear implants (CI) restore hearing abilities in profoundly hearing-impaired or deaf people by bypassing the impaired auditory periphery and stimulating electrically the auditory nerve via an array of 12 to 22 electrodes, which is inserted in the cochlea. The electric stimulation pattern is generated by an external sound processor which decomposes an incoming speech or audio signal into frequency subbands whose envelopes are used to modulate a series of biphasic pulses. While this procedure can restore high degrees of speech intelligibility, music perception remains poor for most CI users. In particular, severe distortions of pitch and timbre occur which can be attributed to the low number of electrodes in current CIs and the spread of electric excitation in the conductive perilymph fluid within the cochlea. Hence, CI users generally perceive music as very complex and less enjoyable.

To mitigate perceived distortions of music in CI listeners, we have developed methods for reducing the spectral complexity of music signals by applying dimensionality reduction techniques in the time-frequency domain. Figure 1 shows the spectrograms of an exemplary classical music piece (clarinet accompanied by strings) before processing (left) and after computing block-wise frequency-domain reduced-rank approximations (right). Clearly, the spectral complexity is reduced by attenuating low-variance harmonics of both the accompaniment and the leading voice while retaining the most prominent harmonics of the leading voice. A subjective evaluation with CI listeners yielded a statistically significant preference score of 74% for the proposed processing methods compared to the unprocessed case.

To evaluate music processing schemes for spectral complexity reduction, we have also developed  instrumental measures of perceived music complexity and auditory distortion, which  have shown a high degree of consistency with the results of preference listening tests with CI users.


Figure 1: Spectrograms before (left) and after (right) spectral complexity reduction. Markers indicate the fundamental frequency of the melody instrument.


Nagathil, A., Schlattmann, J.-W., Neumann, K., Martin, R. (2018). "Music Complexity Prediction for Cochlear Implant Listeners Based on a Feature-based Linear Regression Model", J. Acous. Soc. Am. (JASA), 144(1), pp. 1-10, July 2018.

Nagathil, A., Weihs, C., Neumann, K., Martin, R. (2017). "Spectral Complexity Reduction of Music Signals Based on Frequency-domain Reduced-rank Approximations: An Evaluation with Cochlear Implant Listeners," J. Acous. Soc. Am. (JASA), 142(3), pp. 1219-1228, September 2017.

Nagathil, A., Schlattmann, J.-W., Neumann, K., Martin, R. (2017). “Evaluation of Spectral Music Complexity Reduction Methods for Cochlear Implant Listeners by Means of a Perceptual Music Quality Prediction Model”, Conference on Implantable Auditory Prostheses (CIAP), Lake Tahoe, USA,  July 2017

Nagathil, A., Schlattmann, J.-W., Neumann, K., Martin, R. (2017). "A Feature-based Linear Regression Model for Predicting Perceptual Ratings of Music by Cochlear Implant Listeners," in Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), New Orleans, USA, March 2017.

Nagathil, A., Weihs C., Neumann, K., Martin, R. (2016). “Frequency-domain Reduced-rank Approximations of Music Signals for the Improvement of Music Perception in Cochlear Implant Listeners,” ARCHES Meeting/ICanHear Conference, Zurich, Switzerland, November 2016

Nagathil, A., Weihs, C., Martin, R. (2016). “Spectral Complexity Reduction of Music Signals for Mitigating Effects of Cochlear Hearing Loss,” IEEE/ACM Trans. Audio, Speech, and Language Processing, vol. 24, no. 3, pp. 445-458, March 2016.

Nagathil, A., Weihs C., Martin R. (2015). “Signal Processing Strategies for Improving Music Perception in the Presence of a Cochlear Hearing Loss,” in Proc. Jahrestagung der Deutschen Gesellschaft für Audiologie (DGA), Bochum, Germany, ISBN 978-3-9813141-5-1.