Difference between revisions of "Gesture Signal Processing and Music"
From CCRMA Wiki
(→Gesture and Measuring Movement) |
(→Accelerometers: How to analyze data) |
||
| (4 intermediate revisions by the same user not shown) | |||
| Line 19: | Line 19: | ||
** Proportional to force (Newton's 2nd law of motion: F = ma ) | ** Proportional to force (Newton's 2nd law of motion: F = ma ) | ||
* Jerk: j = da/dt | * Jerk: j = da/dt | ||
| + | |||
| + | == Interlude #1: Open Sound Control == | ||
| + | * Open Sound Control: http://opensoundcontrol.org/ | ||
| + | * iPod Touch has 3 axis accelerometer | ||
| + | * We will use TouchOSC to transmit accelerometer data as OSC messages over wifi to Max/Pd | ||
| + | ** http://hexler.net/software/touchosc | ||
| + | |||
| + | == Interlude #2: Filtering of Audio Signals == | ||
| + | * Can view any sound as being composed of sinusiodal waves at different frequencies | ||
| + | * A filter removes energy at only certain frequencies: | ||
| + | **A low-pass filter removes high freqs. | ||
| + | **A high-pass filter removes low freqs. | ||
| + | ** A band-pass only lets certain middle frequencies through. | ||
| + | * Is differentiation a low-pass or high-pass filter? What about integration? | ||
| + | |||
| + | == Back to movement and music == | ||
| + | === Accelerometers: How to analyze data === | ||
| + | * How to distinguish orientation vs movement? | ||
| + | * Can get jerk via difference (as approximation of differentiation) | ||
| + | * How to get velocity from acceleration (or position from velocity)? | ||
| + | ** True integrator (will eventually overflow): <pre>y = y_prev + x.</pre> | ||
| + | ** Leaky integrator as approximation of integration: <pre>y = a*y_prev + (1-a)*x</pre> | ||
| + | * Thresholding - detecting specific events | ||
| + | ** max: past. pd: threshold | ||
| + | * Classification through training of Machine Learning algorithms | ||
| + | **e.g. Rebecca Fiebrink's Wekinator: http://wekinator.cs.princeton.edu/ | ||
| + | |||
| + | === Mapping to Sound === | ||
| + | * How do we create a musically expressive instrument from sensor data? | ||
| + | * We need to map the data from one domain to another appropriately: | ||
| + | ** linear: to map [a b] to [c d] use <pre>z = (s-a)*(d-c)/(b-a) + c</pre> | ||
| + | ** exponential: more appropriate in many situations since perceptual qualities are often logarthmic. | ||
| + | ** any arbitrary function | ||
| + | * max: scale, pd: ??? | ||
| + | * How do we decide the best mapping? (i.e. How do we select what physical parameter controls which musical parameter?) | ||
| + | **this is an art, not a science! | ||
Latest revision as of 23:43, 11 October 2009
Contents
Gesture and Measuring Movement
- What is a gesture? Physical. Bodily: hands, face, posture. Non-verbal. What do gestures communicate? Is a gesture more like a button or a handle?
- What is a musical gesture?
Techniques for measuring human movement
- Camera-based
- 2D image processing
- 3D Motion Capture using many cameras
- Goniometry: measuring joint angles with physically attached sensors
- Laetitia Sonami's Lady's Glove: http://www.youtube.com/watch?v=HYTrNOmSRSo&feature=player_embedded
- Electromyography: measures muscle activation by measuring electrical activity
- Pamela Z's controller (which I believe uses emg): http://www.pamelaz.com/VociMov.html
- With accelerometers attached to the body (e.g. wiimote, iphone)
Really basic physics
- Position: x
- Velocity: v = dx/dt
- Acceleration: a = dv/dt
- Proportional to force (Newton's 2nd law of motion: F = ma )
- Jerk: j = da/dt
Interlude #1: Open Sound Control
- Open Sound Control: http://opensoundcontrol.org/
- iPod Touch has 3 axis accelerometer
- We will use TouchOSC to transmit accelerometer data as OSC messages over wifi to Max/Pd
Interlude #2: Filtering of Audio Signals
- Can view any sound as being composed of sinusiodal waves at different frequencies
- A filter removes energy at only certain frequencies:
- A low-pass filter removes high freqs.
- A high-pass filter removes low freqs.
- A band-pass only lets certain middle frequencies through.
- Is differentiation a low-pass or high-pass filter? What about integration?
Back to movement and music
Accelerometers: How to analyze data
- How to distinguish orientation vs movement?
- Can get jerk via difference (as approximation of differentiation)
- How to get velocity from acceleration (or position from velocity)?
- True integrator (will eventually overflow):
y = y_prev + x.
- Leaky integrator as approximation of integration:
y = a*y_prev + (1-a)*x
- True integrator (will eventually overflow):
- Thresholding - detecting specific events
- max: past. pd: threshold
- Classification through training of Machine Learning algorithms
- e.g. Rebecca Fiebrink's Wekinator: http://wekinator.cs.princeton.edu/
Mapping to Sound
- How do we create a musically expressive instrument from sensor data?
- We need to map the data from one domain to another appropriately:
- linear: to map [a b] to [c d] use
z = (s-a)*(d-c)/(b-a) + c
- exponential: more appropriate in many situations since perceptual qualities are often logarthmic.
- any arbitrary function
- linear: to map [a b] to [c d] use
- max: scale, pd: ???
- How do we decide the best mapping? (i.e. How do we select what physical parameter controls which musical parameter?)
- this is an art, not a science!
