Exhibit 3 Picking

Note: this exhibit relies on an extension that may not work in certain environments.

What We Did

• Picking refers to using WebGL to find which object is responsible for drawing a particular pixel, e.g., the object underneath a mouse pointer.
• There are a few ways to do this; in PPA2, the scene was simple enough that no special WebGL rendering was required. However, in complicated 3D scenes, this may not be the case. We'll show one approach to picking in arbitrary 3D scenes in this exhibit.
• Our approach:
  • Render two images at the same time.
    • The first is the image that you'll display to the user. It's the usual 3D rendering.
    • The second is an image where each pixel's color encodes the ID of the object that colored that pixel.
  • If you want to know which object is under the mouse cursor, you can transfer pixel data from the GPU to the CPU and extract the object ID from the second image.
• The above approach requires us to render two images at once. To do this in WebGL 1, we need an extension called WEBGL_draw_buffers (this extension in included by default in WebGL 2).

  var webglCanvas = $("#webglCanvas")
  var gl = initializeWebGL(webglCanvas);
  gl.getExtension("OES_texture_float");
  gl.webGlDrawBuffers = gl.getExtension("WEBGL_draw_buffers");
  

  Note that we save the return value of gl.getExtension("WEBGL_draw_buffers") in a field of the gl object. The return value contains more commands and constants that we will need later.
• To render to more than one texture at once, there are two steps:
  1. Write your fragment shader so that outputs to more than one texture.

     // The line below is needed to enable writing to more than one buffer.
     #extension GL_EXT_draw_buffers : require
     
     precision highp float;
     
     varying vec3 geom_color;
     
     // This is the ID of the primitive.
     uniform float id;
     
     void main() {
         // Instead of writing to gl_FragColor, we write to elements of gl_FragData.
         // gl_FragData[0] is the texture that is attached to the zeroth color attachment slot.
         // Here, we write the ID of the primitive to the zeroth color attachment.
         // Note that the alpha is 0.0; we will use this to see if any object at all is under the cursor.
         gl_FragData[0] = vec4(id, 0.0, 0.0, 0.0);
         // gl_FragData[1] is the texture that is attached to the first color attachment slot.
         // Here, we write the primitive color to the first color attachment.
         gl_FragData[1] = vec4(geom_color, 1.0);
     }
     

  2. Prepare the FBO to write to more than one texture.

     // Bind the FBO
     gl.bindFramebuffer(gl.FRAMEBUFFER, fbo);
     // Bind a buffer of the zeroth color attachment slot.
     gl.framebufferTexture2D(
         gl.FRAMEBUFFER,
         // Here, we use a value from the extension to make it the same as the next statement.
         // You can use gl.COLOR_ATTACHMENT0 as well because these two values are equal.
         gl.webGlDrawBuffers.COLOR_ATTACHMENT0_WEBGL,
         gl.TEXTURE_2D,
         // We attach the primitiveIndexTexture here because we will read from it later.
         // We can only transfer data from the zeroth color attachment back to the CPU.
         primitiveIndexTexture,
         0);
     // Bind another buffer to the first color attachment slot.
     gl.framebufferTexture2D(
         gl.FRAMEBUFFER,
         // This is the first color attachment slot.
         gl.webGlDrawBuffers.COLOR_ATTACHMENT1_WEBGL,
         gl.TEXTURE_2D,
         // We attach the render buffer here because we do not need to transfer it to the CPU.
         renderBuffer.getWriteBuffer(),
         0);
     // Since we are doing 3D rendering, we need the depth buffer.
     gl.framebufferRenderbuffer(
         gl.FRAMEBUFFER,
         gl.DEPTH_ATTACHMENT,
         gl.RENDERBUFFER,
         depthBuffer
     );
     // IMPORTANT: We need to tell WebGL to draw to two buffers.
     // By default, it only draws to the zeroth color attachment slot.
     // The function drawBuffersWEBGL is not available in the vanilla WebGL 1.0 context.
     // You only get it with the WEBGL_draw_buffers extension.
     gl.webGlDrawBuffers.drawBuffersWEBGL(
         // Here, we give it the list of slots we want to draw to.
         [
             gl.webGlDrawBuffers.COLOR_ATTACHMENT0_WEBGL, // gl_FragData[0]
             gl.webGlDrawBuffers.COLOR_ATTACHMENT1_WEBGL  // gl_FragData[1]
         ]
     );
     {
         gl.clearColor(0.75, 0.75, 0.75, 1.0);
         gl.clearDepth(1.0);
         gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);
     
         gl.enable(gl.DEPTH_TEST);
     
         // Code elided for brevity.
     
         gl.disable(gl.DEPTH_TEST);
     
         gl.flush();
     }
     
     //
     // Here we have code that reads from the zeroth color attachment.
     // This has to be done while the FBO is still bound and the texture still attached.
     // We will talk about this part later.
     //
     
     // To clean up, we detach the zeroth color attachment.
     gl.framebufferTexture2D(
         gl.FRAMEBUFFER,
         gl.webGlDrawBuffers.COLOR_ATTACHMENT0_WEBGL,
         gl.TEXTURE_2D,
         null,
         0);
     // Then the first color attachment.
     gl.framebufferTexture2D(
         gl.FRAMEBUFFER,
         gl.webGlDrawBuffers.COLOR_ATTACHMENT1_WEBGL,
         gl.TEXTURE_2D,
         null,
         0);
     // Then the depth attachment.
     gl.framebufferRenderbuffer(
         gl.FRAMEBUFFER,
         gl.DEPTH_ATTACHMENT,
         gl.RENDERBUFFER,
         null
     );
     gl.bindFramebuffer(gl.FRAMEBUFFER, null);
     renderBuffer.swap();
     

• With the code above, we render the ID of the primitives to the zeroth color attachment. To read this from within JavaScript, we use the gl.readPixels command while the FBO is still bound and the texture still attached.

  // First, we allocate space to store the pixel value.
  // Since we are using a floating point texture and each pixel has RGBA components,
  // we need an array of 4 floats to store the value of 1 pixel.
  var primitiveIndexPixel = new Float32Array(4);
  
      :
      :
      :
  
  // Use the gl.readPixels to read a rectangle of the zeroth color attachment.
  gl.readPixels(
      // Left edge of the rectangle.
      mousePos.x,
      // Bottom edge of the rectangle.
      mousePos.y,
      // Width of the rectangle.
      1,
      // Height of the rectangle.
      1,
      // Pixel format.
      gl.RGBA,
      // Type of each component of the pixel.
      gl.FLOAT,
      // The array to which the pixel data is stored.
      primitiveIndexPixel
  );
  
  // Display the values read in HTML.
  primitiveIndexTextureDiv.html("Value under mouse pointer = ("
      + primitiveIndexPixel[0] + ","
      + primitiveIndexPixel[1] + ","
      + primitiveIndexPixel[2] + ","
      + primitiveIndexPixel[3] + ")");
  
  // Check which primitive is under the cursor.
  if (primitiveIndexPixel[3] == 1.0) {
      hoverDiv.html("The mouse pointer is hovering over the background.");
  } else {
      if (primitiveIndexPixel[0] == 1.0) {
          hoverDiv.html("The mouse pointer is hovering over the red triangle.");
      } else if (primitiveIndexPixel[0] == 2.0) {
          hoverDiv.html("The mouse pointer is hovering over the green square.");
      } else {
          hoverDiv.html("This case should not happen!");
      }
  }
  

• Note that there are ways to implement picking that do not require the WEBGL_draw_buffers extension. For example, you could render the scene twice: once using an FBO with a GLSL program that outputs the object index, and once without an FBO that draws the scene as usual. You could then read pixel values from the FBO's render texture to extract the object index at a particular pixel.

Exercise

Can you implement picking without using the WEBGL_draw_buffers extension?