Create a machine that asks a question of the world. Your machine should either measure/detect something interesting, measure/detect something in an interesting way, or create an interesting provocation by bringing an uncommon measurement/detection to our attention. The focus here is on the selection and collection of intriguing data, rather than on the production of an attractive interpretation or visualization. What overlooked dynamics or invisible rhythms can you discover?
In this speed project, you are asked to create a “situated eye” – a “contextualized classifier” – a “purposeful detector” – a “poetic surveillant“. You are asked to create a camera-based system:
- which is located in a specific place;
- which is trained to detect a specific thing (or things);
and which responds to what it sees, in an interesting way.(NOPE)
This is a speed project, due Monday 11/6. The project values your inventiveness about where you point your camera, your cleverness in choosing a subject on which to train an AI system, and your care in actually training the AI system. Although you will be working in p5.js, almost no code is required or expected.
Who, What, When, Where, Why
What is an interesting subject to study with a video camera?
For this subject, what are some interesting observations you can automate?
How might a system respond in an interesting way to these observations?
You will be provided a webcam, and potentially a tripod and/or USB extension cable.
- You are asked to — choose a subject, i.e., point the camera at something/someplace/someone interesting. Your system might respond to trees, machines, vehicles, places, animals. You can point your camera at people, but it is understood that you will not violate anyone’s privacy.
- You are asked to — train a neural network to analyze your subject. You can train a classifier (to recognize certain discrete patterns), or a regressor (to measure your subject along a continuum). You will actually train a real system. Although you will work in p5.js, there is no software deliverable expected (e.g. no OpenProcessing sketch to submit).
- You are asked to — document your neural network doing its job, successfully classifying or measuring your subject. Your document can take the form of a video or animated GIF.
- You are asked to — speculatively describe a custom app that would respond to these measurements or classifications. You don’t actually have to make the software—but it should be possible for you to do so in theory. Your description will take the form of a 100-150 word written Discord post in the #situated-eye channel. Your post should include your video/GIF documentation.
Here’s a dumb example—you can do better: Put the camera in your fridge. Train it to recognize when the fridge is open and closed. Perhaps you can train it to recognize the face of which roommate is opening the fridge. Use this system to create a system that records which of your roommates opened the fridge, when. Other examples: Cheese by Christian Moeller; Suicide Box by Natalie Jeremijenko.
Design Considerations
- Escape the physical context of the laptop. Don’t limit yourself to the physical constraints of your laptop’s webcam, and the implicit assumptions it imposes on where a camera can be (on a table, in a room, at waist height, with a level pitch). Give extremely careful consideration to where your camera is located, and/or what it observes. Is your camera on a building? In a refrigerator? Above a pet dish? Part of a microscope? Pointed at the sky, or at the floor? Looking at custom cardboard game pieces on a table?
- When describing your speculative system, remember that your system might respond in real-time (like a game, etc.), or it might serve as a system for recording, logging, or counting what it observes. Keep in mind that you can save files (data, images) to disk. Your system might respond audiovisually (i.e. with graphics and/or sound), and/or it might send a signal over the internet.
Template Code
Demonstrations of the following AI systems will be given in class. It is strongly recommended you use one of these.
Classification:
- p5 + Categorization of Pose Data (Rock/Paper/Scissors) (ML5 v.20)
- p5 + TeachableMachine: Classification of Image Data (ML5 v.0.12)
Regression:
- p5 + ML: Very Simple Regression (ML5 v.20)
- p5 + Pose Data Regression (ML5 v.20)
- p5 + Image Regression (ML5 v.0.12)
Thoughts
Census historian James C. Scott points out that measurement is a political act. Artists like Natalie Jeremijenko collect measurements in order to prompt evidence-driven discussion; others, like Mimi Onuoha, point out that what is not measured is equally revealing of a culture’s biases and indifferences (the study of which is called agnotology). In the weird world of quantum physics, the term “observer effect” refers to the idea that the very act of measurement changes the subject being measured. Measurement, or the collection of data, alters the world and the way we see it.
Data collection has become a key practice across many fields. “Citizen science” is an educational and political movement that enlists everyday people in scientific activities and often focuses on monitoring local environmental conditions through distributed DIY sensing. For example, in the aftermath of the Fukushima disaster, radiation sensors were distributed to a concerned public, who transmitted readings to a central server.
Scholars Catherine D’Ignazio and Lauren Klein outline ways to responsibly work with data, taking philosophical ideas from feminist thought and applying them to data collection and visualization practices. The principles of feminist data visualization include acknowledging that data represents an incomplete perspective; emphasizing the context and the situation in which data was collected; and providing a way for those represented in the data to respond to it.
There is often something absurd, poignant, or whimsically futile about the act of measurement —an attempt to reduce an infinitely complex experience to a handful of numbers. In the arts, measurement can explicitly remind us that our understanding of reality is only ever an approximation.