Exhibit 3 Picking

What Are We Doing?

• Picking refers to the action of using a mouse pointer to "pick" a 3D object inside a viewport.
• What is needed for picking is the ability to know which object is right under the mouse pointer.
  We'll show one approach to picking in this exhibit.
• Our approach:
  • Render two images.
    • First is the image that you'll display to the user. It's the usual 3D rendering.
    • Second is the image where each pixel's color encode the ID of the object at that pixel.
  • If you want to know which object is under the mouse cursor, just read the ID image at the position of the mouse pointer.
• The above approach requires us to render two images at once.
  This is not possible with vanilla WebGL.
  We need an extension called WEBGL_draw_buffers.
  This is how to enable that extension:

  var webglCanvas = $("#webglCanvas")
  var gl = initializeWebGL(webglCanvas);
  gl.getExtension("OES_texture_float");
  gl.getExtension("OES_texture_float_linear");
  gl.webGlDrawBuffers = gl.getExtension("WEBGL_draw_buffers");
  

  Note that we save the return value of gl.getExtension("WEBGL_draw_buffers") in a field of the gl object.
  The return value contains more commands and constants that we will need later.
• To render to more than one textures at once. There are two things you need to do.
  1. Write your fragment shader so that outputs to more than one textures.

     // The line below is needed to enable writing to more than one buffers.
     #extension GL_EXT_draw_buffers : require
     
     precision highp float;
     
     varying vec3 geom_color;
     
     // This is the ID of the primitive.
     uniform float id;
     
     void main() {
         // Instead of writing to gl_FragColor, we write to elements of gl_FragData.
         // gl_FragData[0] is the texture that is attached to the zeroth color attachment slot.
         // Here, we write the ID of the primitive to the zeroth color attachment.
         gl_FragData[0] = vec4(id, 0.0, 0.0, 0.0);
         // gl_FragData[1] is the texture that is attached to the first color attachment slot.
         // Here, we write the primitive color to the first color attachment.
         gl_FragData[1] = vec4(geom_color, 1.0);
     }
     

  2. Prepare the FBO to write to more than one textures.

     // Bind the FBO
     gl.bindFramebuffer(gl.FRAMEBUFFER, fbo);
     // Bind a buffer of the zeroth color attachment slot.
     gl.framebufferTexture2D(
         gl.FRAMEBUFFER,
         // Here, we use a value from the extension to make it the same as the next statement.
         // You can use gl.COLOR_ATTACHMENT0 as well because these two values are equal.
         gl.webGlDrawBuffers.COLOR_ATTACHMENT0_WEBGL,
         gl.TEXTURE_2D,
         // We attach the primitiveIndexTexture here because we will read from it later.
         // We can only read (to Javascript) from the zeroth color attachment.
         primitiveIndexTexture,
         0);
     // Bind another buffer of the first color attachment slot.
     gl.framebufferTexture2D(
         gl.FRAMEBUFFER,
         // This is the first color attachment slot.
         gl.webGlDrawBuffers.COLOR_ATTACHMENT1_WEBGL,
         gl.TEXTURE_2D,
         // We attach the render buffer here because we do not need to read from it to Javascript.
         renderBuffer.getWriteBuffer(),
         0);
     // Since we are doing 3D rendering, we need the depth buffer.
     gl.framebufferRenderbuffer(
         gl.FRAMEBUFFER,
         gl.DEPTH_ATTACHMENT,
         gl.RENDERBUFFER,
         depthBuffer
     );
     // IMPORTANT: We need to tell WebGL to draw to two buffers.
     // By default, it only draws to the zeroth color attachment slot.
     // The function drawBuffersWEBGL is not available in the vanilla WebGL 1.0 context.
     // You only get it with the WEBGL_draw_buffers extension.
     gl.webGlDrawBuffers.drawBuffersWEBGL(
         // Here, we give it the list of slots we want to draw to.
         [
             gl.webGlDrawBuffers.COLOR_ATTACHMENT0_WEBGL, // gl_FragData[0]
             gl.webGlDrawBuffers.COLOR_ATTACHMENT1_WEBGL  // gl_FragData[1]
         ]
     );
     {
         gl.clearColor(0.75, 0.75, 0.75, 1.0);
         gl.clearDepth(1.0);
         gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);
     
         gl.enable(gl.DEPTH_TEST);
     
         // Code elided for brevity.
     
         gl.disable(gl.DEPTH_TEST);
     
         gl.flush();
     }
     
     //
     // Here we have code that reads from the zeroth color attachment.
     // This has to be done while the FBO is still bound and the texture still attached.
     // We will talk about this part later.
     //
     
     // To clean up, we detach the zeroth color attachment.
     gl.framebufferTexture2D(
         gl.FRAMEBUFFER,
         gl.webGlDrawBuffers.COLOR_ATTACHMENT0_WEBGL,
         gl.TEXTURE_2D,
         null,
         0);
     // Then the first color attachment.
     gl.framebufferTexture2D(
         gl.FRAMEBUFFER,
         gl.webGlDrawBuffers.COLOR_ATTACHMENT1_WEBGL,
         gl.TEXTURE_2D,
         null,
         0);
     // Then the depth attachment.
     gl.framebufferRenderbuffer(
         gl.FRAMEBUFFER,
         gl.DEPTH_ATTACHMENT,
         gl.RENDERBUFFER,
         null
     );
     gl.bindFramebuffer(gl.FRAMEBUFFER, null);
     renderBuffer.swap();
     

• With the code above, we render the ID of the primitives to the zeroth color attachment.
  Now, we need to know the value of the pixel under the mouse pointer.
  To do so, we use the gl.readPixels command while the FBO is still bound and the texture still attached.
  The code is as follows.

  // First, we allocate space to contains the pixel value.
  // Since we use float texture and each pixel has RGBA components, we need an array of 4 floats.
  var primitiveIndexPixel = new Float32Array(4);
  
      :
      :
      :
  
  // Use the gl.readPixels to read a rectangle of the zeroth color attachment.
  gl.readPixels(
      // Left edge of the rectangle.
      mousePos.x,
      // Bottom edge of the rectangle.
      mousePos.y,
      // Width of the rectangle.
      1,
      // Height of the rectangle.
      1,
      // Pixel format.
      gl.RGBA,
      // Type of each component of the pixel.
      gl.FLOAT,
      // The array to store the value in.
      primitiveIndexPixel
  );
  
  // Display the values read in HTML.
  primitiveIndexTextureDiv.html("Value under mouse pointer = ("
      + primitiveIndexPixel[0] + ","
      + primitiveIndexPixel[1] + ","
      + primitiveIndexPixel[2] + ","
      + primitiveIndexPixel[3] + ")");
  
  // Check which primitive is under the cursor.
  if (primitiveIndexPixel[3] == 1.0) {
      hoverDiv.html("The mouse pointer is hovering over the background.");
  } else {
      if (primitiveIndexPixel[0] == 1.0) {
          hoverDiv.html("The mouse pointer is hovering over the red triangle.");
      } else if (primitiveIndexPixel[0] == 2.0) {
          hoverDiv.html("The mouse pointer is hovering over the green square.");
      } else {
          hoverDiv.html("This case should not happen!.");
      }
  }