Gesture Signal Processing and Music: Difference between revisions

Revision as of 07:37, 12 October 2009

What is a gesture? Physical. Bodily: hands, face, posture. Non-verbal. What do gestures communicate? Is a gesture more like a button or a handle?
What is a musical gesture?

Camera-based
- 2D image processing
- 3D Motion Capture using many cameras
Goniometry: measuring joint angles with physically attached sensors
- Laetitia Sonami's Lady's Glove: http://www.youtube.com/watch?v=HYTrNOmSRSo&feature=player_embedded
Electromyography: measures muscle activation by measuring electrical activity
- Pamela Z's controller (which I believe uses emg): http://www.pamelaz.com/VociMov.html
With accelerometers attached to the body (e.g. wiimote, iphone)

Position: x
Velocity: v = dx/dt
Acceleration: a = dv/dt
- Proportional to force (Newton's 2nd law of motion: F = ma )
Jerk: j = da/dt

Open Sound Control: http://opensoundcontrol.org/
iPod Touch has 3 axis accelerometer
We will use TouchOSC to transmit accelerometer data as OSC messages over wifi to Max/Pd
- http://hexler.net/software/touchosc

Can view any sound as being composed of sinusiodal waves at different frequencies
A filter removes energy at only certain frequencies:
- A low-pass filter removes high freqs.
- A high-pass filter removes low freqs.
- A band-pass only lets certain middle frequencies through.
Is differentiation a low-pass or high-pass filter? What about integration?

How to distinguish orientation vs movement?
Can get jerk via difference (as approximation of differentiation)
How to get velocity from acceleration (or position from velocity)?
- True integrator (will eventually overflow):
  - ```
  y = y_prev + x.
```
- Leaky integrator as approximation of integration:
  - ```
  y = a*y_prev + (1-a)*x
```
Thresholding - detecting specific events
- max: past. pd: threshold
Classification through training of Machine Learning algorithms
- e.g. Rebecca Fiebrink's Wekinator: http://wekinator.cs.princeton.edu/

@@ Line 32: / Line 32: @@
 **A high-pass filter removes low freqs.
 ** A band-pass only lets certain middle frequencies through.
+* Is differentiation a low-pass or high-pass filter?  What about integration?
+== Back to movement and music ==
+=== Accelerometers: How to analyze data ===
+* How to distinguish orientation vs movement?
+* Can get jerk via difference (as approximation of differentiation)
+* How to get velocity from acceleration (or position from velocity)?
+** True integrator (will eventually overflow):
+***<pre>y = y_prev + x.</pre>
+** Leaky integrator as approximation of integration:
+***<pre>y = a*y_prev + (1-a)*x</pre>
+* Thresholding - detecting specific events
+** max: past. pd: threshold
+* Classification through training of Machine Learning algorithms
+**e.g. Rebecca Fiebrink's Wekinator: http://wekinator.cs.princeton.edu/
+=== Mapping to Sound ===