Squirrel and Bird Classifier in Processing from Greg Borenstein on Vimeo.

Download PSVM / Read the documentation / PSVM source on Github

SVM is a Processing library that implements Support Vector Machines, a popular machine learning technique. It is based on LibSVM, an open source SVM library.

The SVM algorithm analyzes a set of labeled sample data in order to classify new sample data. To use SVM, you train the algorithm by providing it with example data that you have grouped into a series of categories. Then, when you provide the algorithm with new, unknown data, it assigns that data to one of your given categories based on its resemblance to the known training data.

Support Vector Machines are commonly used to detect spam, to recognize objects in images, and to detect trends in numerical data. PSVM includes examples for working with all of these, including examples of finding squirrels and birds in images and detecting hand gestures in live video.

Installation and Usage

To install Isolines, download the library and unzip it into your Processing libraries folder. Restart Processing and PSVM should show up under the “File > Examples” menu under “Contributed Libraries”.

Training

The first step in using PSVM is to train the model based on data that is already classified. In this example, we’ll use multi-class data, i.e. a set of 2D points that are already labeled as being in one of three sets. Once we’ve trained the model on those points, it will be able to predict into which set other points will fall. We’ll display those predictions by coloring in all of the pixels of our sketch to indicate which set the model would put them in. The final result will look like this:

PSVM machine leaning for Processing multi-class example

import psvm.*;

SVM model;
float[][] trainingPoints;
int[] labels;

Table data;

PGraphics modelDisplay;
boolean showModel = false;

void setup(){
  size(500,500);
  
  // displaying the model is very slow, so we'll
  // do it in a PGraphics so we only have to do it once
  modelDisplay = createGraphics(500,500);
  
  // load the data from the csv
  data = new Table(this, "points.csv");
  
  // we'll have one training point for each line in our csv
  // each training point will have two entries (x and y)
  trainingPoints = new float[data.getRowCount()][2];
  // we need one label for each training point incdicating
  // what set the point is in (1, 2, or 3)
  labels = new int[data.getRowCount()];
  
  // loop through the CSV rows
  // to create the trainingPoints and labels
  int i = 0;
  for (TableRow row : data) {
    float[] p = new float[2];
    // scale the data from 0-1 based on the
    // range of the data
    p[0] = row.getFloat(0)/500; 
    p[1] = row.getFloat(1)/500;
    trainingPoints[i] = p;
    labels[i] = row.getInt(2);    
    i++;
  }
  
  // initialize our model and problem objects
  model = new SVM(this);
  SVMProblem problem = new SVMProblem();
  // we need one feature for each axis of the data
  // so in this case x and y means 2 features
  problem.setNumFeatures(2);
  // load the problem with the labels and training data
  problem.setSampleData(labels, trainingPoints);
  // train the model
  model.train(problem);
  
  drawModel();
}

// this function colors in each pixel of the sketch
// based on what result the model predicts for that x-y value
// it saves the results in a PGraphics object
// so that it can be displayed everytime beneath the data
void drawModel(){
  // start drawing into the PGraphics instead of the sketch
  modelDisplay.beginDraw();
  // for each row
  for(int x = 0; x < width; x++){
    // and each column
    for(int y = 0; y < height; y++){
      
      // make a 2-element array with the x and y values
      double[] testPoint = new double[2];
      testPoint[0] = (double)x/width;
      testPoint[1] = (double)y/height;
      
      // pass it to the model for testing
      double d = model.test(testPoint);
      
      // based on the result, draw a red, green, or blue dot
      if((int)d == 1){
        modelDisplay.stroke(255,0,0);
      } else if ((int)d == 2){
        modelDisplay.stroke(0, 255 ,0);
      } else if ((int)d == 3){
        modelDisplay.stroke(0, 0, 255);
      }
      
      // which will fill up the entire area of the sketch
      modelDisplay.point(x,y);
  
    }
  }
  // we're done with the PGraphics
  modelDisplay.endDraw();
}

void draw(){
  // show our model background if we want
  if(showModel){
    image(modelDisplay, 0, 0);
  } else {
    background(255);
  }
  
  stroke(255);
  
  // show all of the training points
  // in the right color based on their labels
  for(int i = 0; i < trainingPoints.length; i++){
    if(labels[i] == 1){
      fill(255,0,0);
    } else if(labels[i] == 2){
      fill(0,255,0);
    } else if(labels[i] == 3){
      fill(0,0,255);
    }
    
    ellipse(trainingPoints[i][0] * 500, trainingPoints[i][1]* 500, 5, 5);
  }
}

void keyPressed(){
  if(key == ' '){
    showModel = !showModel;
  }
  // save out the model file for use
  // in future classification
  if(key == 's'){
      model.saveModel("model.txt");
  }
}

// on mouse click, for any given point
// test it against the model and print the result set
void mousePressed(){
  double[] p = new double[2];
  p[0] = (double)mouseX/width;
  p[1] = (double)mouseY/height;
  println((int)model.test(p));
}

Using a Trained Model

One of the best features of SVM is that, once you’ve trained a model, you can reuse it to predict future data without needing to have all of the training data on-hand. The training example above included the ability to save out such a model file. Start by running that sketch and saving out the model file. Then, create a new sketch and copy the “model.txt” file over to its data folder. This example demonstrates how to load that model file and use it to test new incoming data.

import psvm.*;

SVM model;

void setup() {
  size(500, 500);  
  model = new SVM(this);
  // load the model from a model file
  // the second argument is how many features the problem has 
  model.loadModel("model.txt",2);
}

void draw(){
  background(255);

  // scale the mouse point to 0-1
  float[] p = new float[2];
  p[0] = (float)mouseX/width;
  p[1] = (float)mouseY/height;
  
  // test the point,
  // convert the result to an int
  // and set the fill color based on the result
  int result = (int)model.test(p);
  if(result == 1){
    fill(255,0,0);
  } else if(result == 2){
    fill(0,255,0);
  } else if(result == 3){
    fill(0,0,255);
  }
  
  ellipse(mouseX, mouseY, 10, 10);
}

Working with Images

There are many ways to turn images into vectors in order to train a Support Vector Machine. PSVM includes examples of two different methods: color histograms and Histogram of Oriented Gradients.

Squirrel and Bird Classifier Using Color Histograms

Squirrel and Bird Classifier in Processing from Greg Borenstein on Vimeo.

/*

This sketch uses image color and Support Vector Machines
to distinguish pictures of birds from pictures of squirrels.
It calculates a color histogram for each picture and uses
that as the feature vector for training an SVM.

The SVM is trained using the images in the folders named
"birds" and "squirrels" and then tested on the images
in the folder named "test" to establish a success percentage.

The test images are then displayed with their histograms
and their classification result.

*/

import psvm.*;

SVM model;

// declare Histogram object
Histogram histogram;
// "bin" the histogram into 32 separate
// color groups
int numBins = 32;

PImage testImage;
double testResult;

int[] labels; // 1 = squirrel, 2 = bird
float[][] trainingData;
String[] testFilenames;

void setup() {
  size(600, 600);
  textSize(32);
  
  // get the names of all of the image files in the "squirrel" folder
  java.io.File squirrelFolder = new java.io.File(dataPath("squirrels"));
  String[] squirrelFilenames = squirrelFolder.list();

  // get the names of all of the image files in the "birds" folder
  java.io.File birdFolder = new java.io.File(dataPath("birds"));
  String[] birdFilenames = birdFolder.list();

  // initialize labels and trainingData arrays
  // one label for each training image
  labels = new int[squirrelFilenames.length + birdFilenames.length];
  // one vector for each training image
  // each vector has 3 entries for each bin
  // one each for R, G, and B
  trainingData = new float[labels.length][numBins*3];
  
  // create the histogram object and tell it how many
  // bins we want
  histogram  = new Histogram();
  histogram.setNumBins(numBins);

  // build vectors for each squirrel image
  // and add the appropriate label
  for (int i = 0; i < squirrelFilenames.length; i++) {
    println("loading squirrel " + i);
    trainingData[i] = buildVector(loadImage("squirrels/" + squirrelFilenames[i]));
    labels[i] = 1;
  }

  // build vectors for each bird image
  // and add the appropriate label
  for (int i = 0; i < birdFilenames.length; i++) {
    println("loading bird " + i);
    trainingData[i + squirrelFilenames.length] = buildVector(loadImage("birds/" + birdFilenames[i]));
    labels[i+ squirrelFilenames.length] = 2;
  } 

  // setup our SVM model
  model = new SVM(this);
  SVMProblem problem = new SVMProblem();
  // each vector will have three features for each bin
  // in our histogram: one each for red, green, and blue
  // components
  problem.setNumFeatures(numBins*3);
  // set the data and train the SVM
  problem.setSampleData(labels, trainingData);
  model.train(problem);
  
  // load in the test images
  java.io.File testFolder = new java.io.File(dataPath("test"));
  testFilenames = testFolder.list();
  
  // run the evaluation (see function for more)
  evaluateResults();
  // loads and tests a new image from the test folder
  loadNewTestImage();
}

// This function calculates the histogram for a PImage
// and scales it so that the sum of each RGB entry is 1
// the results are used as the feature vector for SVM
float[] buildVector(PImage img) {
  histogram.setImage(img);
  histogram.calculateHistogram();
  histogram.scale(0, 0.33);
  return histogram.getRGB();
}

void draw() {
  background(0);
  
  pushMatrix();
  scale(0.5);
  image(testImage, 0, 60);
  popMatrix();
  
  // testResult is set in loadNewTestImage()
  String message = "";
  if((int)testResult == 1){
    message = "Squirrel";
  } 
  else if((int)testResult == 2){
    message = "Bird";
  }
  
  fill(255);
  text(message, 10, 25);
  text("Percent classified correctly: " + nf(correctPercentage * 100,2,2), 20, height -40);
  stroke(255);
  histogram.drawRGB(0, height/2, width, height/2);
}

// Loads up a new random image.
// Runs in through the SVM for classification.
// Stores the results in the testResult variable.
void loadNewTestImage(){
  int imgNum = (int)random(0, testFilenames.length-1);
  testImage = loadImage("test/" + testFilenames[imgNum]); 
  testResult = model.test(buildVector(testImage));
}

void keyPressed(){
  loadNewTestImage();
}

float correctPercentage = 0.0;

void evaluateResults(){
  int numCorrect = 0;
  PImage img;
  
  for (int i = 0; i < testFilenames.length; i++) {
    img = loadImage("test/" + testFilenames[i]); 
    double r = model.test(buildVector(img));
    if(r == 1.0 && split(testFilenames[i], "-")[0].equals("Squirrel")){
      numCorrect++;
    }
    
    if(r == 2.0 && split(testFilenames[i], "-")[0].equals("Bird")){
      numCorrect++;
    }
  }
  
  correctPercentage = (float)numCorrect/testFilenames.length;
  
  println("Num Bins: " + numBins + " Percent Correct: " + correctPercentage);
}

And Histogram.pde, the class for calculating color histograms from a PImage:

/*

A helper class for calculating color histograms of images.
Calculates RGB as well as HSV histograms.

Should probably be its own library.

*/

class Histogram {
  int numBins;
  int[] pixels;
  boolean calculated;
  int min, max;

  Histogram() {
    this.numBins = 256;
    this.calculated = false;
    this.min = 0;
    this.max = 255;
  }

  void setNumBins(int numBins) {
    this.numBins = numBins;
  }

  void setImage(PImage img) {
    img.loadPixels();
    this.pixels = img.pixels;
    calculated = false;
  }

  void setPixels(int[] pix) {
    this.pixels = pix;
    calculated = false;
  }

  float[] rHist;
  float[] bHist;
  float[] gHist;

  float[] hHist;
  float[] sHist;
  float[] vHist;

  int binIndexForValue(int value) {
    return value/binSize();
  }

  void calculateHistogram() {
    rHist = new float[numBins];
    gHist = new float[numBins];
    bHist = new float[numBins];
    hHist = new float[numBins];
    sHist = new float[numBins];
    vHist = new float[numBins];

    for (int i = 0; i < this.pixels.length; i++) {
      int rBright = int(red(this.pixels[i]));
      int gBright = int(green(this.pixels[i]));
      int bBright = int(blue(this.pixels[i]));

      int hBright = int(hue(this.pixels[i]));
      int sBright = int(saturation(this.pixels[i]));
      int vBright = int(brightness(this.pixels[i]));

      rHist[binIndexForValue(rBright)]++;
      gHist[binIndexForValue(gBright)]++;
      bHist[binIndexForValue(bBright)]++;

      hHist[binIndexForValue(hBright)]++;
      sHist[binIndexForValue(sBright)]++;
      vHist[binIndexForValue(vBright)]++;
    }
  }

  void scale(float min, float max) {
    for (int i = 0; i < numBins; i++) {
      rHist[i] = map(rHist[i], 0, this.pixels.length, min, max);
      gHist[i] = map(gHist[i], 0, this.pixels.length, min, max);      
      bHist[i] = map(bHist[i], 0, this.pixels.length, min, max);
      hHist[i] = map(hHist[i], 0, this.pixels.length, min, max);
      sHist[i] = map(sHist[i], 0, this.pixels.length, min, max);
      vHist[i] = map(vHist[i], 0, this.pixels.length, min, max);
    }
  }


  void setRange(int min, int max) {
    this.min = min;
    this.max = max;
  }

  int binSize() {
    return ceil((float)(max - min) / numBins);
  }

  float[] getHSV() {
    float[] result = new float[numBins * 3];
    for (int i = 0; i < numBins; i++) {
      result[i] = hHist[i];
      result[i+numBins] = sHist[i];
      result[i+(2*numBins)] = vHist[i];
    }
    return result;
  }

  float[] getRGB() {
    float[] result = new float[numBins * 3];
    for (int i = 0; i < numBins; i++) {
      result[i] = rHist[i];
      result[i+numBins] = gHist[i];
      result[i+(2*numBins)] = bHist[i];
    }
    return result;
  }

  void drawRGB(int x, int y, int w, int h) {
    // int histMax = max(histData);

    fill(255, 0, 0);
    drawChannel(rHist, x, y, w, h/3);
    fill(0, 255, 0);
    drawChannel(gHist, x, y+h/3, w, h/3);
    fill(0, 0, 255);
    drawChannel(bHist, x, y+(2*h)/3, w, h/3);
  }

  void drawChannel(float[] data, int x, int y, int w, int h) {
    int binWidth = w/numBins;

    for (int i = 0; i < numBins; i++) {
      float mappedH = map(data[i], 0, 0.33, 0, h);
      rect((i*binWidth), y, w / numBins, -mappedH);
    }
  }
}

Interactive Hand Gesture Recognizer using Histogram of Oriented Gradients

Hand Gesture Recognizer in Processing from Greg Borenstein on Vimeo.

/*
HandGestureRecognizerInteractive by Greg Borenstein, October 2012
Distributed as part of PSVM: http://makematics.com/code/psvm

Depends on HoG Processing: http://hogprocessing.altervista.org/

Uses a trained Support Vector Machine to detect hand gestures in live video.
SVM is trained based on Histogram of Oriented Gradients on the 
Sebastien Marcel Hand Pose Dataset: http://www.idiap.ch/resource/gestures/
*/

import hog.*;
import psvm.*;
import processing.video.*;

// Capture object for accessing video feed
Capture video;
// size of the box we'll be looking in for hand gestures
int rectW = 150;
int rectH = 150;

SVM model;
PImage testImage;

void setup() {
  size(640/2 + 60, 480/2); 
  // capture video at half size for speed
  video = new Capture(this, 640/2, 480/2);
  video.start();   
  // declare our SVM object
  model = new SVM(this);
  // load the trained svm model from the file
  // our data has 324 dimensions because
  // that's what we get from doing Histogram of Oriented
  // Gradients on a 50x50 pixel image
  model.loadModel("hand_gesture_model.txt", 324);
  // initialize our PImage at 50x50
  // we'll use this to display the part
  // of the video feed we're searching
  testImage = createImage(50, 50, RGB);
}

// video event, necessary for getting live camera
void captureEvent(Capture c) {
  c.read();
}

void draw() {
  background(0);

  // copy the pixels in the incoming video
  // into our testImage. Only use the pixels
  // inside the 150x150 square at the center
  // also resize down to 50x50 (last two arguments)
  // (the subtractions are to make sure we get the pixels
  // in our red box)
  testImage.copy(video, video.width - rectW - (video.width - rectW)/2, video.height - rectH - (video.height - rectH)/2, rectW, rectH, 0, 0, 50, 50);

  // run Histogram of Oriented Gradients on the testImage
  // and pass the results to our model for testing
  double testResult = model.test(gradientsForImage(testImage)); 

  // display the video, the test image, and the red box
  image(video, 0, 0);
  image(testImage, width - testImage.width, 0);
  noFill();
  stroke(255, 0, 0);
  strokeWeight(5);
  rect(video.width - rectW - (video.width - rectW)/2, video.height - rectH - (video.height - rectH)/2, rectW, rectH);

  // use the result of our SVM test
  // to decide what text to put on the screen
  // based on what gesture is showing
  String result = "Gesture is: ";
  switch((int)testResult) {
  case 1:
    fill(255, 125, 125);
    result = result + "A";
    break;
  case 2:
    fill(125, 255, 125);
    result = result + "B";
    break;
  case 3:
    fill(125, 125, 255);
    result = result + "C";
    break;
  case 4:
    fill(125, 255, 255);
    result = result + "V";
    break;
  case 5:
    fill(255, 255, 125);
    result = result + "Five";
    break;
  case 6:
    fill(255);
    result = result + "Point";
    break;
  }
  text(result, 100, 20);
}

// Helper function that calculates the 
// Histogram of Oriented Gradients for
// a PImage, filled with a lot of HoG magic
float[] gradientsForImage(PImage img) {
  // settings for Histogram of Oriented Gradients
  // (probably don't change these)
  int window_width=64;
  int window_height=128;
  int bins = 9;
  int cell_size = 8;
  int block_size = 2;
  boolean signed = false;
  int overlap = 0;
  int stride=16;
  int number_of_resizes=5;

  // a bunch of unecessarily verbose HOG code
  HOG_Factory hog = HOG.createInstance();
  GradientsComputation gc=hog.createGradientsComputation();
  Voter voter=MagnitudeItselfVoter.createMagnitudeItselfVoter();
  HistogramsComputation hc=hog.createHistogramsComputation( bins, cell_size, cell_size, signed, voter);
  Norm norm=L2_Norm.createL2_Norm(0.1);
  BlocksComputation bc=hog.createBlocksComputation(block_size, block_size, overlap, norm);
  PixelGradientVector[][] pixelGradients = gc.computeGradients(img, this);
  Histogram[][] histograms = hc.computeHistograms(pixelGradients);
  Block[][] blocks = bc.computeBlocks(histograms);
  Block[][] normalizedBlocks = bc.normalizeBlocks(blocks);
  DescriptorComputation dc=hog.createDescriptorComputation();    

  return dc.computeDescriptor(normalizedBlocks);
}