Latest revision as of 03:33, 26 February 2023

Programming Etude #3: "Wekinate Your World"

Music and AI (Music356/CS470) | Winter 2023 | by Ge Wang

In this programming etude, you are using Wekinator to create three interactive AI utilities/toys in your everyday life! These don't have to be useful; in fact, whimsical is good! Absurd is good! Playful is wonderful!

Due Dates

Final Deliverable: webpage due Monday (2/27, 11:59pm)
In-class Presentation: Tuesday (2/28)

Discord Is Our Friend

direct any questions, rumination, outputs/interesting mistakes to our class Discord

Things to Think With

some relevant readings (including from your Critical Response #2 assignment)

Tools to Play With

get and play with Wekinator
get the latest bleeding edge secret chuck build (2023.02.20 or later!)
- macOS this will install both command line chuck and the graphical IDE miniAudicle, and replace any previous ChucK installation.
- Windows you will need to download and use the bleeding-edge command line chuck (for now, there is no bleeding-edge miniAudicle for Windows); can either use the default cmd command prompt, or might consider downloading a terminal emulator.
- Linux you will need to build from source, provided in the linux directory
NOTE: to return your chuck back to a pre-bleeding-edge state, you can always install the latest official ChucK release

Getting Started with Wekinator

a terrific online course on Kadenze from Dr. Rebecca Fiebrink on interactive ML for music using Wekinator:
- "Machine Learning for Musicians and Artists"
- in particular, the videos for the following sessions can be particularly helpful for this assignment:
  - Session 5: Sensors and Features: Generating Useful Inputs for Machine Learning
  - Session 6: Working with Time
  - Session 7: Session 7: Developing a Machine Learning Practice; Wrap-up
- These videos are clear, concise, and practical! We highly recommend taking a look as you explore Wekinator and this homework!
- NOTE: to view these videos, you will need to 1) sign up for a Kadenze account, 2) upgrade for the (free) 7-day premium membership trial; for anyone wanting to reference the course beyond the trial, Ge will reimbursement one-month of premium membership.
Wekinator (and its underlying idea of creating expressive mapping interactively and by examples to continually refine the model) can be used in many different forms:
- (primary approach) the Wekinator application (GUI) is designed to be used in conjunction with other software (ChucK, Pure Data, Processing, Arduino etc.), connected via the Open Sound Control (OSC) communication protocol
  - the system architecture looks like INPUT >--(OSC)--> Wekinator app >--(OSC)--> OUTPUT where INPUT and OUTPUT can be different softwares that communicate with Wekinator app using OSC:
  - INPUT: provides the input data (e.g., mouse locations, sensor data, gamepad/gametrak input, video frames) for training, updating, and playing with Wekinator models
  - OUTPUT: based the output parameters of Wekinator, renders in real-time (e.g., sound synthesis, graphics rendering, parameters to control a game or hardware like waving a robot arm)
  - in this approach, Wekinator app's TRAIN mode iterates between adding new observations (specific input to output examples) to training the model; Wekinator app's RUN mode predicts output based on new input
- (secondary approach) the Wekinator object in ChucK allows you to work with Wekinator using code and without the need for network communication
  - Wekinator can TRAIN (from input-output observations) and/or RUN (predict output from new input)
  - or use Wekinator.loadData() to load input-output observations from an ARFF file, either saved in a Wekinator project (currentData.arff) or saved using Wekinator.saveData()
  - helpful in deploying a Wekinator system without using OSC; e.g., (step 1) capture observations in the Wekinator app, save the data; (step 2) load the data in the Wekinator object in ChucK, train, and run without OSC. Also maybe helpful for those wanting to deploy using WebChucK (which doesn't yet support OSC)
for this assignment, we recommend trying both before deciding what to use for each of your three system designs
Resources for learning and working with Wekinator
- check out some example projects people have created using Wekinator
- for a general overview of how Wekinator works, here are basic instructions
- get started with using Wekinator walkthrough
- for a more detailed reference, check out Wekinator detailed instructions
- example code for working with Wekinator app, OSC, and ChucK
- example code for working with the Wekinator object within ChucK
- here are a trove of examples for working with Wekinator across many languages

Your Task

observe and identify three activities or tasks you like to "wekinate" (i.e., to design a interactive AI system for—for this assignment, keep them small)
of your three systems, at least one should be an interactive and expressive musical instrument designed with Wekinator for regression
experiment (a lot) with different ideas (possibly more than the three you end up with)!
once again, these don't have to be useful; instead, playfulness, whimsicality, and/or expressiveness are virtues here
deploy these in some everyday context; feel free to involve those around you (e.g., roommates! friends! enemies!) in testing out your AI systems!
- "hey I have to do this thing for a class; would you kindly try/play/listen-to this thing..."
for each system, create a short video showing the system in action

A Few Bad Ideas!

Using camera input and sound output, create a day / night classifier (or sonifier)—I mean, who has time to look out the window?
An instrument using one or more continuous input (such as mouse, gamepad, gametrak, Kinect, wiimote, VR controller with or without the VR) controlling a multi-parameter synthesis (like one of the STK instruments like Bowed)—try playing this for your roommate, or your cat (and film the performance and audience reception)
using Wekinator's dynamic time warping, train a few audio or other gestures, and create a voice or gesture command system that does meaningless things—inject a little AI-mediated chaos into your day
Use Wekinator to train an instrument for controlling/exploring your audio mosaics (the output would be an N-dimensional/dimensionally-reduced vector)
for more ideas, check out these projects (some of these are large-scale projects; for this programming etude, remember to keep the scope of your Wekinate creations small)!

Reflections

write ~300 words of reflection on your etude. It can be about your process or the products. Tell us about your attempt to deploy them.

Final Deliverables

create a CCRMA webpage for this etude
- the URL should live at https://ccrma.stanford.edu/~YOURUSERID/356/hw3 or https://ccrma.stanford.edu/~YOURUSERID/470/hw3
- alternately, you may use Medium or another publishing platform (but please still link to that page from your CCRMA webpage)
your webpage is to include
- a title and description of what you made (free free to link to this wiki page)
- three video recordings corresponding to your three interactive AI systems in action
- all relevant code and data files for all three systems
- brief report on how you created these
- your 300-word reflection
- any acknowledgements (people, code, or other things that helped you through this)
submit to Canvas only your webpage URL

@@ Line 2: / Line 2: @@
 [https://ccrma.stanford.edu/courses/356/ Music and AI (Music356/CS470)] | Winter 2023 | by Ge Wang
-<div style="text-align: left;">[[Image:Mosaiconastick.jpg|400px]]</div>
+<div style="text-align: left;">[[Image:Wekinator-wavebot.jpg|400px]]</div>
-In this programming etude, you are using Wekinator to create three interactive AI utilities/toys in your everyday life! These don't have to be useful; in fact, whimsical is good! Absurd is probably good! Playful is wonderful!
+In this programming etude, you are using Wekinator to create three interactive AI utilities/toys in your everyday life! These don't have to be useful; in fact, whimsical is good! Absurd is good! Playful is wonderful!
 === Due Dates ===
@@ Line 14: / Line 14: @@
 === Things to Think With ===
-* some relevant readings (including from your Critical Response #2 assignment
+* some relevant readings (including from your Critical Response #2 assignment)
 ** [https://ccrma.stanford.edu/courses/356/readings/2014-power-to-the-people.pdf "Power to the People: The Role of Humans in Interactive Machine Learning"]
 ** [https://artful.design/hai/ "Humans in the Loop: The Design of Interactive AI Systems"]
@@ Line 20: / Line 20: @@
 === Tools to Play With ===
-* get and play with [http://www.wekinator.org/ Wekinator]
+* get and play with [http://www.wekinator.org/ '''Wekinator''']
-* get the latest [https://ccrma.stanford.edu/courses/356/bin/chuck/ '''bleeding edge secret <code>chuck</code> build'''] (2023.01.23 or later!)
+* get the latest [https://ccrma.stanford.edu/courses/356/bin/chuck/ '''bleeding edge secret <code>chuck</code> build'''] (2023.02.20 or later!)
 ** '''macOS''' this will install both command line <code>chuck</code> and the graphical IDE miniAudicle, and replace any previous ChucK installation.
 ** '''Windows''' you will need to download and use the bleeding-edge command line <code>chuck</code> (for now, there is no bleeding-edge miniAudicle for Windows); can either use the default <code>cmd</code> command prompt, or might consider downloading a [https://www.puttygen.com/windows-terminal-emulators terminal emulator].
 ** '''Linux''' you will need to build from source, provided in the <code>linux</code> directory
-** '''all platforms''' for this project, you will be using the command line version of chuck.
 * NOTE: to return your chuck back to a pre-bleeding-edge state, you can always install the latest [https://chuck.stanford.edu/ official ChucK release]
-=== Sample Code ===
+=== Getting Started with Wekinator ===
-* [http://www.wekinator.org/examples/ Wekinator examples]
+* a terrific online course on Kadenze from Dr. Rebecca Fiebrink on interactive ML for music using Wekinator:
-* [https://ccrma.stanford.edu/courses/356/code/wekinate/ '''sample code in ChucK''']
+** '''[https://www.kadenze.com/courses/machine-learning-for-musicians-and-artists/info "Machine Learning for Musicians and Artists"]'''
+** in particular, the videos for the following [https://www.kadenze.com/courses/machine-learning-for-musicians-and-artists-v/sessions sessions] can be particularly helpful for this assignment:
+*** Session 5: Sensors and Features: Generating Useful Inputs for Machine Learning
+*** Session 6: Working with Time
+*** Session 7: Session 7: Developing a Machine Learning Practice; Wrap-up
+** These videos are clear, concise, and practical! We highly recommend taking a look as you explore Wekinator and this homework!
+** NOTE: to view these videos, you will need to 1) sign up for a Kadenze account, 2) upgrade for the (free) 7-day premium membership trial; for anyone wanting to reference the course beyond the trial, Ge will reimbursement one-month of premium membership.
+* Wekinator (and its underlying idea of creating expressive mapping interactively and by examples to continually refine the model) can be used in many different forms:
+** (primary approach) the '''Wekinator application (GUI)''' is designed to be used in conjunction with other software (ChucK, Pure Data, Processing, Arduino etc.), connected via the <code>Open Sound Control (OSC)</code> communication protocol
+*** the system architecture looks like <code>'''INPUT >--(OSC)--> Wekinator app >--(OSC)--> OUTPUT'''</code> where INPUT and OUTPUT can be different softwares that communicate with Wekinator app using OSC:
+*** INPUT: provides the input data (e.g., mouse locations, sensor data, gamepad/gametrak input, video frames) for training, updating, and playing with Wekinator models
+*** OUTPUT: based the output parameters of Wekinator, renders in real-time (e.g., sound synthesis, graphics rendering, parameters to control a game or hardware like waving a robot arm)
+*** in this approach, '''Wekinator app''''s '''TRAIN mode''' iterates between adding new observations (specific input to output examples) to training the model; Wekinator app's '''RUN mode''' predicts output based on new input
+** (secondary approach) the '''<code>Wekinator</code>''' object in <code>ChucK</code> allows you to work with Wekinator using code and without the need for network communication
+*** '''<code>Wekinator</code>''' can TRAIN (from input-output observations) and/or RUN (predict output from new input)
+*** or use <code>'''Wekinator'''.loadData()</code> to load input-output observations from an ARFF file, either saved in a Wekinator project (currentData.arff) or saved using <code>'''Wekinator'''.saveData()</code>
+*** helpful in deploying a Wekinator system without using OSC; e.g., (step 1) capture observations in the '''Wekinator app''', save the data; (step 2) load the data in the '''<code>Wekinator</code>''' object in ChucK, train, and run without OSC. Also maybe helpful for those wanting to deploy using WebChucK (which doesn't yet support OSC)
+* for this assignment, we recommend trying both before deciding what to use for each of your three system designs
+* Resources for learning and working with Wekinator
+** check out some [http://www.wekinator.org/example-projects/ '''example projects'''] people have created using Wekinator
+** for a general overview of how Wekinator works, here are [http://www.wekinator.org/instructions/ '''basic instructions''']
+** get started with using [http://www.wekinator.org/walkthrough/ '''Wekinator walkthrough''']
+** for a more detailed reference, check out [http://www.wekinator.org/detailed-instructions/ '''Wekinator detailed instructions''']
+** example code for working with [https://ccrma.stanford.edu/courses/356/code/wekinate/app/ '''Wekinator app, OSC, and ChucK''']
+** example code for working with the [https://ccrma.stanford.edu/courses/356/code/wekinate/object/ '''<code>Wekinator</code> object within ChucK''']
+** here are a [http://www.wekinator.org/examples/ '''trove of examples'''] for working with Wekinator across many languages
 === Your Task ===
-* observe and identify three activities or tasks you like to "wekinate" (i.e., design an small interactive AI system)
+* observe and identify '''three activities or tasks''' you like to "wekinate" (i.e., to design a interactive AI system for—for this assignment, keep them small)
-* of the three, at least one should be an expressive musical instrument using Wekinator for regression
+* of your three systems, '''at least one should be an interactive and expressive musical instrument designed with Wekinator for regression'''
-* once again, these don't have to be useful; instead, playfulness, whimsicality, and/or expressiveness are the virtues here
+* experiment (a lot) with different ideas (possibly more than the three you end up with)!
-* deploy these in some everyday context; feel free to involve those around you (e.g., roommates! friends! enemies!) in testing out these small AI systems!
+* once again, these don't have to be useful; instead, playfulness, whimsicality, and/or expressiveness are virtues here
+* '''deploy''' these in some everyday context; feel free to involve those around you (e.g., roommates! friends! enemies!) in testing out your AI systems!
+** "hey I have to do this thing for a class; would you kindly try/play/listen-to this thing..."
+* for each system, create a '''short video''' showing the system in action
+=== A Few Bad Ideas! ===
+* Using camera input and sound output, create a day / night classifier (or sonifier)—''I mean, who has time to look out the window?''
+* An instrument using one or more continuous input (such as mouse, gamepad, gametrak, Kinect, wiimote, VR controller with or without the VR) controlling a multi-parameter synthesis (like one of the STK instruments like Bowed)—''try playing this for your roommate, or your cat (and film the performance and audience reception)''
+* using Wekinator's dynamic time warping, train a few audio or other gestures, and create a voice or gesture command system that does meaningless things—''inject a little AI-mediated chaos into your day''
+* Use Wekinator to train an instrument for controlling/exploring your audio mosaics (the output would be an N-dimensional/dimensionally-reduced vector)
+* for more ideas, check out [http://www.wekinator.org/example-projects/ these projects] (some of these are large-scale projects; for this programming etude, remember to keep the scope of your Wekinate creations small)!
 === Reflections ===
 * write ~300 words of reflection on your etude. It can be about your process or the products. Tell us about your attempt to deploy them.
-=== Milestone Deliverables ===
-submit a webpage for the project so far, containing:
-* a brief report of what you did / try / observed in Phase One, and a brief description of your experiments on in Phase Two so far
-* a demo video (doesn't have to be polished) briefly documenting your experiments/adventures in Phase Two, and a very preliminary sketch of Phase Three (a creative statement or performance using your system)
-* code and feature and usage instructions needed to run your system
-* list and acknowledge the source material (audio and any video) and people who have helped you along the way; source audio/video do not need to be posted (can submit these privately in Canvas)
-* In class, we will view your webpage/demo video and give one another feedback for this milestone.
 === Final Deliverables ===