Artists build on top of science. Today’s cutting edge math and computer science research becomes tomorrow’s breakthrough creative projects.
Computer vision algorithms, machine learning techniques, and 3D topology are becoming vital prerequisites to doing daily work in creative fields from interactive art to generative graphics, data visualization, and digital fabrication. If they don’t grapple with these subjects themselves, artists are forced to wait for others to digest this new knowledge before they can work with it. Their creative options shrink to those parts of this research selected by Adobe and Autodesk for inclusion in prepackaged tools.
This class is designed to help you start seizing the results of this research for your own creative work. You’ll learn how to explore the published academic literature for techniques that will help you build your dream projects. You’ll learn how to apply these techniques across multiple media. And you’ll learn how to use these techniques to make your own projects more sophisticated and more original.
This class has two major goals. First, it aims to teach you a series of mathematical and computer science techniques that you can use to build better projects.
Each class we’ll explore a technique from one of these research fields. We’ll learn to understand the original research: how does the technique work? What problem is it trying to solve? What could it be useful for? And we’ll learn how to apply the technique in Processing to multiple different media, from text to images, from 3D models to movies and music.
The second goal is more ambitious. This class aims to help you launch your own relationship with the research feilds that will best enable your own creative goals. Are you interested in building interactive sculptures that track large groups of people outdoors? Are you interested in writing a program that visualizes the differences in style of different poets? Are you interested in building a video installation that automatically learns what parts of the room visitors tend to stand in? This class will give you an introduction to the areas of research that will help you make these creative ambitions a reality.
The class will be taught in Processing. Basic ICM-level familiarity with Processing is expected as well as a commitment to stretching your programming skills no matter your current level.
Weekly Project After each class you should do a project using the techniques and code we learned in class. Your project could involve applying the techniques to new sources of data, using the interactions they enable in new ways or contexts, or (especially awesome) combining them with something else we’ve already learned. By midnight on the night before the next class, you should send a link to the class mailing list with the documentation of your project. We’ll discuss some of the projects in class.
Weekly Paper Summary Before each class you should select a paper from the Makematics database of papers or one you’ve found in your own searchings. Write a blog post about it that describes, in your own words, what the paper does. You don’t need to capture exactly how it works, but try to describe what techniques it uses and what effects it achieves. This description can be short, no more than a handful of sentences. Reproducing images from the papers is highly encouraged. Then write a quick description of how you would use this technique in a project. Feel free to flesh out the project with sketches, etc — whatever it takes to make the idea vivid for us. Email a link to your blog post to the class mailing list by midnight the night before class.
Final Project For a final project you should use what you’ve learned in class to do an initial prototype of your dream project (see the Special Assignment below under Week 1). If you cannot figure out how you can use what we’ve done in class to prototype your dream project, email me or come talk to me and I’ll help you work something out or find an additiona technique that can help. The final projects will be presented on the last day of class.
Marching Squares is one of the most venerable and widely-cited techniques in computer graphics. It’s a technique for finding continous contours around the areas of a grayscale image that have a given color. It’s the basis for the Photoshop “magic wand” tool.
This week, we’ll learn how marching squares works and we’ll look at some exciting uses of it. First, we’ll see how to use marching squares to turn any continuous color image, from Kinect depth images to weather data, from election maps to website eye-tracking data, into a series of layers that can be digitally fabricated to produce a 3D object. Secondly, we’ll follow along with some work happening in Ken Perlin and NYU’s Vision Learning Group that uses marching squares to track the fingers of a user’s hand with the kinect. We’ll see how you can turn a marching squares-derived contour into the locations of individual fingers.
I’ll provide you with Processing libraries that implement both the marching squares algorithm in general as well as the finger-tracking application in particular. (Many thanks to Murphy Stein for the code here.)
This opening week will also include an introduction to the class. Why should artists and designers wrestle with copmuter science research? How can you get started in such a seemingly impenetrable and advanced field? We’ll also go over basic class procedures — homework, grading, final projects — as well as taking an overview of the material we’ll be covering in the class.
Special Assignment What is your dream project? What project have you always fantasized about building but always considered out of your reach? The purpose of the Makematics approach of mining computer science research for creative techniques is to enable you to come closer to acheiving these dream projects. For next class, prepare short descriptions of two projects you’ve always dreamed of doing. Write them up (as a blog post or in an email) and send them to the class mailing list.
Classification is the problem of sorting given data into a finite number of pre-existing categories. It is the basis for most machine learning applications. This week, we’ll learn about two forms of supervised learning: linear classification and support vector machines. Both of these techniques begin with a set of training data that has been hand-labeled by some person. They each then attempt to generalize from that training data in order to classify new, un-labeled data.
This week begins a two-class run on classification. This week we’ll focus on numerical data and we’ll learn the basics of linear classiciation. We’ll also see some of the limits of linear classification and we’ll introduce Support Vector Machines.
Support Vector Machines (SVM) is a powerful machine learning technique for classifying images, text, and other data into groups. SVM starts with human-classified data that demonstrates a series of categories. Once the algorithm is trained on this data, it can then place new unknown examples into the same categories based on their resemblance to the training data across a number of factors.
This week, we’ll start with the basics of SVM, training a model with 2D numerical data. Next class we’ll move on to fancier SVM tricks such as representing images and other forms of more complex data for use in training sets.
Linear classification and SVMs are widely used to detect spam, identify objects in images, performing hand-writing recognition, etc.
This week we’ll dive much more deeply into Support Vector Machines. We’ll start off by reviewing some of the theory. We’ll discuss the limitations of linear classification: it’s sensitivity to outliers and it’s inability to separate certain data sets. We’ll look at how SVMs overcome these problems. We’ll learn what a support vector is and we’ll discuss “the kernel trick”.
After the theory, we’ll look at applications. We’ll cover the idea of “vectorization”: how to turn texts, images, and other media into data that can be classified using SVMs. Specifically, we’ll look at working with images using color histograms and Histogram of Oriented Gradients. And we’ll look at working with text using a technique called “bag of words”.
We’ll look at applications of SVMs for hand gesture recognition, distinguishing birds from squirrels, recognizing toys, and detecting spam.
Principal Component Analysis (PCA) is a statistical technique for finding patterns in complex data. PCA takes complex data such as all of the pixels in an image, all of the points in a 3D model, or all the documents in a large database and uses statistical analysis to find the aspects of the data that are most meaningful, i.e. that determine most of the differences between the individual items in the set.
PCA can be used to detect areas of images that have been photoshopped, perform facial recognition, determine the orientation of 3D models, determine the political bias of news stories, etc.
Dynamic Programming is a general problem solving technique that automates the process of breaking a problem down into sub-problems that can be solved individually and then combining the results into a general conclusion. It is especially useful when the sub-problems occur repeatedly as it stores the results of each sub-solution for re-use.
Dynamic programming can be used to detect the beat of a piece of music, to automatically eliminate or repeat areas of an image so it can be resized without changing its most important parts (seam carving), and to help a robot plan a path through an environment.
Bayes' Rule is a probabilistic technique lets you figure out the most likely causes of observed evidence based on the likelihood of those effects. Or, as Geoff Bohling puts it, Bayes' Rule “lets us turn information about the probability of different effects from each possible cause:
into information about the probable cause given the observed effects:"
Bayes' rule can be used to filter spam, to automatically translate texts between languages, to identify people from their gait, etc.