MIR workshop 2008 notes

This page is intended to supplement the lecture material found in the class - providing extra tutorials, support, references for further reading, or demonstration code snippets for those interested in a given topic. Please contribute to this growing list of resources. Do you have a great explanation of how a technique works? Found a great Java applet that illustrates a concept? Discovered a great survey of the field for a particular area? Please add it for the benefit of future students. Thanks!

I encourage you to ADD links and sections - but please do not REMOVE headings or items from the page.

Timing and Segmentation

Onset Detection

• Papers:
• Code:

Beat Extraction

Online available beat tracking systems (compiled by Olmo Cornelis, University College Ghent, October 2010, posted on the MUSIC-IR list):

Queen Mary ‘s sonic visualiser:
http://www.sonicvisualiser.org/
with vamp plugin set:
http://vamp-plugins.org/download.html
http://www.vamp-plugins.org/download.html

Simon Dixon's BeatRoot (Java) 
http://www.eecs.qmul.ac.uk/~simond/beatroot/ 

Dan Ellis' algorithm:
http://labrosa.ee.columbia.edu/projects/coversongs/
D. Ellis (2007) Beat Tracking by Dynamic Programming, Journal of New Music Research, Special Issue on Beat and Tempo Extraction, vol. 36 no. 1, March 2007, pp. 51-60. (10pp)

Fabien Gouyon’s IBT in MARSYAS 
marsyas.info
(2 versions, 1 causal (real time) and 1 non causal)

Matthew Davies' beat-tracking (in SuperCollider by Nick Collins as "BeatTrack")
http://tinyurl.com/344ubus 

Adam Stark and Matthew Davies’ real-time beat tracker (as a Max/MSP external):
http://www.elec.qmul.ac.uk/digitalmusic/people/adams/bsa/
http://138.37.35.209/digitalmusic/papers/2009/StarkDaviesPlumbley09-dafx.pdf

Andrew Robertson has published "B-Keeper" drum tracker in Max/MSP: 
http://b-keeper.org/

Paul Brossier's Aubio C/C++ library (tempo tracking, onset detection):
http://aubio.org/

Not available online: 
- Klapuri et al. [2006] - Analysis of the Meter of Acoustic Musical Signals  
- Hainsworth [2004] - Techniques for the Automated Analysis of Musical Audio (PhD Thesis)
- Goto [2001] - An audio-based real-time beat tracking system for music with or without drum-sounds
- Schierer [1998] - Tempo and beat analysis of acoustic musical signals

Some online tools found by browsing:
http://www.zplane.de/index.php?page=home
http://www.beatunes.com/index.html
http://www.abyssmedia.com/bpmcounter/
http://users.compaqnet.be/CubicCarrot/index.htm
http://www.mp3doctor.com/free_bpm_detect.html
http://www.mixmeister.com/bpmanalyzer/bpmanalyzer.asp
http://www.softjock.com/bpmproscan.html
http://www.potionfactory.com/tangerine

Tempo Extraction

• Papers:
  • Tempo and beat analysis of acoustic musical signals
  • Determination of the meter of musical scores by autocorrelation
  • Tempo and beat analysis of acoustic musical signals
  • An Audio-based Real-time Beat Tracking System for Music With or Without Drum-sounds*

Feature Extraction

Low Level Features

• Zero Crossing, Temporal centroid, Log Attack time, Attack slope), Spectral features (Centroid, Flux, RMS, Rolloff, Flatness, Kurtosis, Brightness),Spectral bands, Log spectrogram
• Chroma bins
• MFCC
  • Auditory Toolbox (code and docs)
• MPEG-7

Higher-level features

• Key Estimation
• Chord Estimation
• Genre (genre, artist ID, similarity)
• "Fingerprints"

Visualizing and Sonifying Feature data

• Matt Hoffman's feature sonification work

Analysis / Decision Making

Classification

• Heuristic Analysis
• Distance measures (Euclidean, Manhattan, etc.)
• k-NN
• SVM / One-class SVM
  • Resources:
    • The interactive Matlab SVM Demo that I demonstrated on Lecture 5 comes from here
    • A nice SVM java applet to demo the concepts
    • Andrew Moore's SVM Powerpoint Lecture
    • User community of SVM enthusiasts
    • A practical guide to SVM classification
    • SVM Practical (How to get good results without cheating)
    • One-class SVM posting
  • Code:
    • libSVM (standalone, matlab, c, etc)
    • libSVM tools

Clustering and probability density models

• Density distance measures (centroid distance, EMD, KL-divergence, etc)
• k-Means
• Clustering Demo

Clustering

• GMM
  • Simple review of probability with introduction of Bayes Rules
  • Good description of conditional probabilities
  • EM explained
  • Expectation-Maximization Java Applet
  • Lab featuring real-world GMM examples for singing detection
  • Dan Ellis' Speech and Audio Processing Lectures

HMM

• High-level introduction to HMM
• “A tutorial on hidden markov models and selected applications in speech recognition” Lawrence Rabiner, Proc. IEEE, 77(2), Feb 1989.
• A self-directed introduction / lab for HMMs
• Matlab Introduction to HMM functions

There are a couple of MATLAB toolboxes for implementing HMMs and Bayesian nets, both by Kevin Murphy:

• HMM toolbox
• BNT

Nested classifier / Anchor-space / template-based systems

• ?

Model / Data Preparation Techniques

• Data Preparation
  • PCA / LDA
  • Scaling data
• Model organization
  • concept, design, data set construction and organization

Evaluation Methodology

• Cross Validation
• Information Retrieval metrics (precision, recall, F-Measure)

Real-world applications

Audio Segmentation

Automatic Audio Segmentation: Segment Boundary and Structure Detection in Popular Music

Audio Fingerprinting

• P. Cano, E. Batlle, T. Kalker, and J. Haitsma, “A review of algorithms for audio fingerprinting,” in IEEE International Workshop on Multimedia Signal Processing (MMSP), pp. 169 – 173, December 2002. 4 pages.
• "On the comparison of audio fingerprints for extracting quality parameters of compressed audio"
• Finding Structure in Audio for Music Information Retrieval
• "Computer Vision for Music Identification" Y. Ke, D. Hoiem, and R. Sukthankar

The Last.fm fingerprinter uses this approach, code can be checked out from: svn://svn.audioscrobbler.net/recommendation/MusicID/lastfm_fplib

• DAn Ellis' Matlab implementation based on Avery Wang's Shazam algorithm which works out to be pretty robust (in limited testing)

Drum Transcription

• DrumTracker

Audio Similarity

• A. Berenzweig, B. Logan, D. Ellis, and B. Whitman. A large-scale evaluation of acoustic and subjective music similarity measures. In Proceedings of 4th International Symposium on Music Information Retrieval, Baltimore, Maryland, 2003.

Music Recommendation / Playlisting

• ISMIR 2007 Music Recommendation Tutorial
• Social Tag info from Last.fm

Getting Involved in the MIR Community

• ISMIR
• Audio Engineering Society and the Semantic Audio Analysis Technical Committee
• MUSIC-IR list

Research Databases / Collections of Ground truth data and copyright-cleared music

• CAL500
• RWC
• USPop
• Magnatunes

General MIR Datasets

• http://grh.mur.at/misc/mir_datasets.html

Download links for the ISMIR 2004 genre classification contest training set:

• http://ismir2004.ismir.net/genre_contest/index.htm
• http://www.iua.upf.es/mtg/ismir2004/contest/Training_Tracks1.tar.gz
• http://www.iua.upf.es/mtg/ismir2004/contest/Training_Tracks2.tar.gz

Tags:

• http://SocialMusicResearch.org/data
• http://SocialMusicResearch.org/code

More:

• OLPC Sound Sample Archive (8.5 GB) [1]
• RWC Music Database (n DVDs) [available in Stanford Music library]
• RWC - Sound Instruments Table of Contents
• http://staff.aist.go.jp/m.goto/RWC-MDB/rwc-mdb-i.html
• Univ or Iowa Music Instrument Samples

From Georg Holzmann: 
LIST OF PUBLIC AVAILABLE MIR DATASETS
Downloadable Datasets:
- University of Iowa musical instruments samples:
   http://theremin.music.uiowa.edu/MIS.html
   Instrument samples recorded by the University of Iowa

- ISMIR2004 Audio Description Contest Dataset:
   http://ismir2004.ismir.net/ISMIR_Contest.html
   Datasets for
   - Genre Classification/Artist Identification
   - Melody Extraction
   - Tempo Induction
   - Rhythm Classification

- Graham's Melody Extraction Dataset:
   http://www.ee.columbia.edu/~graham/mirex_melody/
   http://labrosa.ee.columbia.edu/projects/melody/
   Audio files with correspondig pitch data

- MIREX06 Audio Tempo Extraction and Beat Tracking Datasets:
 
http://www.music-ir.org/mirex/2006/index.php/Audio_Tempo_Extraction#Practice_Data

- QBSH: A Corpus for Designing QBSH (Query by Singing/Humming) Systems
   http://neural.cs.nthu.edu.tw/jang2/dataSet/childSong4public/QBSH-corpus/

- Uni Dortmund Music Audio Benchmark Data Set:
   http://www-ai.cs.uni-dortmund.de/audio.html
   Songs from different genres and with tags (from garageband.com)

- Latin Music Database:
   http://www.ppgia.pucpr.br/~silla/lmd/
   3.160 music pieces in MP3 Format classified in 10 diferent musical genres
   (only features online)


Orderable Datasets:
- RWC Music Database:
   http://staff.aist.go.jp/m.goto/RWC-MDB/
   (many CDs)
   Datasets for
   - Pop Music & Royalty-Free Music
   - Classical Music
   - Jazz Music
   - Music Genre
   - Musical Instrument Sound

   Additional: AIST RWC Annotations
   http://staff.aist.go.jp/m.goto/RWC-MDB/AIST-Annotation/
   Additional annotations to the RWC database (beat, melody, ...)

- McGill University Master Samples:
   http://www.music.mcgill.ca/resources/mums/html/
   3 DVDs with instrument samples

- USPOP2002 Pop Music data set:
   http://labrosa.ee.columbia.edu/projects/musicsim/uspop2002.html
   (3 DVDs)
   MFCC features from 706 albums and 8764 tracks (400 artists)
   with style tags

- ENST-Drums:
   http://perso.telecom-paristech.fr/~gillet/ENST-drums/
   An extensive audio-visual database for drum signals processing


Free Online Music:

- magnatune.com creative commons music:
   http://magnatune.com/info/press/coverage/ccblog

- http://www.garageband.com/
   Public domain recordings

- http://epitonic.com/
   "high quality free and legal mp3 music"

- http://www.jamendo.com/
   Creative commons licensed music

- http://musicbrainz.org/
   Get music metadata

- http://www.freesound.org/
   Collaborative database of Creative Commons licensed sounds
   (not focused on songs)

Webservices:
- Networked Environment for Music Analysis:
   http://nema.lis.uiuc.edu/
   A webservices system for submitting code, running it against virtual collections
   (full use in 2010)

- MIREX DIY Framework:
   http://www.music-ir.org/mirexdiy/
   http://www.dlib.org/dlib/december06/downie/12downie.html
   (useable ?)

MIR Software and Toolboxes

Incomplete but growing list (courtesy of Joern Loviscach):
* MARSYAS
* jAudio
* Chuck
* The Sonic Visualizer/Annotator
* CLAM
* Music-to-Knowledge (M2K)
* MIRtoolbox
* MA toolbox
* Psysound
* Praat
* IPEM
* EchoNest
* libxtract
* MuBu
* Soundspotter
* timbreID
* openSMILE
* MPEG-7 XM
* MPEG-7 Audio Encoder
* MPEG-7 Audio Analyzer
* Sphinx 4 - Java-based open-source speech recognizer  http://cmusphinx.sourceforge.net/sphinx4/#capabilities

MIR Topic Areas

From Simon Dixon, Music-IR list, Dec 2008.

MIR Systems
- Content-based Querying
- Classification (genre/style/mood)
- Recommendation / playlist generation
- Fingerprinting / DRM
- Score following / Audio alignment
- Transcription / Annotation
- Tempo induction / Beat tracking
- Summarisation
- Streaming
- Text/web mining
- Optical music recognition
- Database systems / indexing / query languages

Human issues
- user interfaces, user models
- emotion, aesthetics
- perception, cognition
- social issues
- legal and ethical issues
- business issues
- methodological and philosophical issues

Data and metadata
- audio
- MIDI
- score
- text/web
- KR schemes, standards and protocols
- libraries and collections
- test sets and evaluation

Musical knowledge
- Melody and motives
- Harmony, chords and tonality
- Rhythm, beat, tempo and form
- Timbre, instrumentation and voice
- Genre, style and mood
- Performance
- Composition
- Ethnomusicology