# CS5625 PA2 Mesh Animation

Out: Thursday February 12, 2016

Due: Friday February 26, 2016 at 11:59pm

Work in groups of 2.

## Overview

In this programming assignment, you will implement:

- blend shapes for facial expressions, and
- linear blend skinning for articulated character animations.

## Task 1: Blend shapes

Run the pa2.PA2_BlendShapes class, and you will see that the program allows you to change between three characters:

KAITO | Hatsune Miku | Utane Uta |

Each charcter has a set of blend shapes or, as we shall call it in this assignment, "morphs," associated with it. A morph in this assigment modifies some facial features of the character, making him/her wink, for example. A specific morph can be selected by the combo box next in the bottom row. Each morph also has an associated "weight," which is a floating point number from 0 to 1. You can set this value using the spinner and the slider in the bottom row.

The basics of blend shapes were discussed in CS4620 in the animation lecture (slides 68–69). The idea is that each vertex comes with a 3D position for each of $N$ blend shapes, and the animation is controlled by per-frame weights $w_j$ for each blend shape, which sum to $1$. If $\mathbf{p}_{ij}$ is the position of vertex $i$ in blend shape $j$, and $w_j$ is the weight of blend shape $j$ in the current frame, then the position of vertex $i$ is $$\mathbf{p}_i' = \sum_j w_j\mathbf{p}_{ij}.$$

Because the blend shapes are the same over most of the character, in this assignment the blend shapes are stored as *displacements* from a neutral pose; that is
$$\mathbf{p}_{ij} = \mathbf{p}_i + \mathbf{d}_{ij}$$
so that the blended position is
$$\mathbf{p}_i' = \mathbf{p}_i + \sum_j w_j\mathbf{d}_{ij}.$$
Since the neutral shape has all displacements zero, it can be left out, and the weights no longer have to sum to $1$.

These displacements are zero for most vertices in each blend shape, so for each blend shape $j$ the intput file contains a list of the indices $i$ that have nonzero displacements, with values of $\mathbf{d}_{ij}$ just for those vertices. The data is stored in files like `student/data/pa2/models/mmd/KAITO/KAITO.dat`; search for the class name `SkeletalMeshMorph` to see how the data looks. Storing things this way also makes it easy to construct a list of the blend shapes that affect a particular vertex, and to process only those displacements in the vertex shader. The code to read these files and store the data in textures is provided. All this leads to a great decrease in storage and computation, at the cost of somewhat more complex indexing.

For this task, edit:

so that the shader implement morphing of vertices.Information about morphs come to the shader in pieces of three data structures.

- The first is the
`vert_morphWeights`texture rectangle. This is a texture of size $(\mbox{number of morphs}) \times 1$ such that the value of the $(j,0)$-pixel contains the weight of the $j$th morph. You can use the`getMorphWeight`function to fetch the weight of the morph with a given index (which starts from 0). - The second is the
`vert_morphDisplacements`texture rectangle. You can think of this as an array containing displcements to individual vertices from all the morphs, sorted by the indices of the vertex. (It will become apparent soon why we sort it this way.) You can recover the $i$th entry of the array using the`getMorphDisplacementInfo(int i)`function. The function returns a`vec4`whose x-, y-, and z-components denote the displacement being applied to the vertex position. The w-component stores the index of the morph having this displacement. - The third is the vertex data. Two relevent attributes are the
`vert_morphStart`and the`vert_morphCount`attributes. The`vert_morphCount`indicates the number of morphs the current vertex fall under influences of, and the`vert_morphStart`indicates the first index into the`vert_morphDisplacements`array of elements that hold displacements that the vertex is subjected to under various morphs.

Let $n$ denote the value of `vert_morphCount`. To compute a morphed vertex position, write a loop that starts fetching the `vert_morphDisplacements` array at indices `vert_morphStart`, `vert_morphStart`+1, `vert_morphStart`+2, $\dotsc$, `vert_morphStart`+$n$-1. Each of these texture fetching will give you information about the largest displacement that you can apply to the vertex and the index of the associated morph. Now, with the index of the morph at hand, you can find the weight of the morph from the `vert_morphWeights` texture. Once you know the weight, scale the displacement by the it and add the resulting weighted displacement to the vertex position. (For morphing, you only change the vertex position and do not have to care about the normal, the tangent, or the bitangent.)

A correct implementation of the shader should yield the following images

Morph | KAITO | Hatsune Miku | Utane Uta |

Eye: Wink (both) | |||

Mouth: a | |||

Eye: Small iris |

## Task 2: Linear blend skinning

The last piece of our character animation system is linear blend skinning, a technique that allows vertices of a mesh to deform according to an underlying skeleton. Linear blend skinning is discussed in slides 62–67 of the CS4620 animation lecture. The basic idea is that the pose of an animated character is specified by using per-frame linear transformations (usually rigid) for each of a set of *bones* in the skeleton, and the deformed positions of the mesh (the "skin") are computed by applying a combination of the bone transformations to the neutral vertex position. The influence of bone $j$ on vertex $i$ is given by a fixed (not per-frame) weight $w_{ij}$, so the deformed position is:
$$\mathbf{p}_i' = \sum_j w_{ij} \mathbf{P}_i.$$
Since each vertex is normally influenced by only a few bones, in this assignment we store the weights for a vertex $i$ by keeping a list of the bone indices $j$ for which the weights $w_{ij}$ are nonzero, togther with the values of those weights. Each vertex is only allowed to be influenced by up to four bones, so we store exactly four indices for each vertex, using the value $-1$ to indicate an unused index. You can see an example of the input data in the file `student/data/pa2/models/mmd/KAITO/KAITO.dat`, in the fields `boneIndices` and `boneWeights` of the `ConcreteSkeletalMesh`.

The animation data for each frame consists of a bone transformation for each bone and a morph weight for each blend shape; only nonzero morph weights are stored. See for example the single-frame animation in `student/data/pa2/motions/KAITO/stand01.dat`.

For this task, edit:

- student/src/shaders/forward/skeletal_mesh.vert, and
- the
`updateBoneXformTexture`method of the cs5625.gfx.mesh.SkeletalMeshPoseManager class.

The SkeletalMeshPoseManager class manages the "pose" of an articulated mesh. A pose is basically the collection of morph weights (each a floating point number from 0 to 1) and the configurations of skeletal bones (each bone has a corresponding displacement and a rotation). The morph weights are stored in the `morphWeights` field. The bone displacements and rotations associated with the pose are stored in the `boneDisplacements` and `boneRotations` fields, respectively. Morph weight processing has been completely implemented, so you do not need to worry about it. However, you will need to implement the computation of transformation matrices for each bone in this task.

The method `updateBoneXformTexture` computes for each bone a transformation matrix for use with linear blend skinning and stores the result in the `boneXforms` field so that the $i$th element stores the matrix for the $i$th bone. The bones themselves are stored in the `bones` field. Each bone is represented by the SkeletalMeshBone structure, and it has two piece of important information:

- the index of the parent bone (-1 if it has none) in the
`bones`field, and - the displacement from the child bone to the parent bone in
*rest position*. (If the bone has no parent, it is the world space position of the bone.)

We now discuss the computation of the matrices for linear blend skinning. Let:

- $T_i^{\mathrm{R}}$ denote the 4x4 translation matrix associated with the displacement of Bone $i$ to its parent in rest position.
- $T_i^{\mathrm{P}}$ denote the 4x4 translation matrix associated with the displacement stored in the $i$th element of
`boneDisplacements`. - $R_i^{\mathrm{P}}$ denote the 4x4 rotation matrix associated with the quaternion stored in the $i$th element of
`boneRotations`.

- First, compute the 4x4 transformation matrix associated with the translation from the origin to the world position of Bone $i_k$ in
*rest position*. This matrix is given by: $$M^{\mathrm{R}}_{i_k} = T^{\mathrm{R}}_{i_0} T^{\mathrm{R}}_{i_1} T^{\mathrm{R}}_{i_2} \dotsm T^{\mathrm{R}}_{i_{k-1}} T^{\mathrm{R}}_{i_k}.$$ - Second, compute the 4x4 transformation matrix associated with going from the origin to the world position of Bone $i_k$ according to the
*pose*. This matrix is given by: $$ M^{\mathrm{P}}_{i_k} = ( T^{\mathrm{R}}_{i_0} T^{\mathrm{P}}_{i_0} R^{\mathrm{P}}_{i_0} ) ( T^{\mathrm{R}}_{i_1} T^{\mathrm{P}}_{i_1} R^{\mathrm{P}}_{i_1} ) \dotsm ( T^{\mathrm{R}}_{i_{k-1}} T_{i_{k-1}}^{\mathrm{P}} R_{i_{k-1}}^{\mathrm{P}} ) ( T^{\mathrm{R}}_{i_k} T_{i_k}^{\mathrm{P}} R_{i_k}^{\mathrm{P}} ). $$ - The bone matrix for linear blend skinning is then given by: $$ M_{i_k} = M^{\mathrm{P}}_{i_k} ( M^{\mathrm{R}}_{i_k} )^{-1}. $$

After you have computed the matrices, the rest of the `updateBoneXformTexture` will take care of sending the information to the shader for you. The next part of this task is to write the shader to make use of this information. In skeletal_mesh.vert, we have provided the `getBoneXform` function such that calling `getBoneXform(i)` fetches the matrix `boneXforms.get(i)` that you previously computed.

To perform linear blend skinning, we first morph the vertex according to the blend shapes as was done in the last task. Let us denote the morphed vertex position by $\mathbf{p}$. The vertex can be influenced by a number of bones, and each of this bone has different amount of influence on the vertex, indicated by the bone's *weight*. The indices of the bones that influence the vertex is stored in the `vert_boneIndices` attributes, a `vec4`, meaning that there can be at most four bones that can influence the vertex. You can retrieve the index of the $j$th bone with the expression `vert_boneIndices[j]` where $j$ ranges from 0 to 3. Some of these indices may be -1, indicating that this is not a valid bone and should be ignored. The weights of the bones are stored in the attribute `vert_boneWeights`, and the weight of the $j$th bone can be retrieved by the expression `vert_boneWeights[j]`. Now, suppose that the indices of the bones are $j_0$, $j_1$, $j_2$, and $j_3$, and the associated weights are $w_{j_0}$, $w_{j_1}$, $w_{j_2}$, and $w_{j_3}$. Then, the linear blended vertex position is given by:
$$ \mbox{blended vertex position} = w_{j_0} M_{j_0} p + w_{j_1} M_{j_1} p + w_{j_2} M_{j_2} p + w_{j_3} M_{j_3} p.$$
Now, you also have to compute the blended tangent, bitangent, and normal. We suggest that you compute the tangent and bitangent using the formula similar to the above. Then, you can compute the normal by computing the cross product between the two vectors.

Lastly, note that the positions and tangent frame you computed in the last paragraph are in *object space*. You have to transform them with the appropriate matrices to pass them to the fragment shaders and the rasterization unit. However, the code to do this has already been provided for you in the shader.

You can test the full implementation of the character animation system by running the pa2.PA2_Animations class. Here, you still have the option of selecting among the three characters, but you can also subject them to 5 poses/animations. A correct implementation of the system should produce the following images:

Animation/Pose | KAITO | Hatsune Miku | Utane Uta |

Stand 1 | |||

Stand 2 | |||

Stand 3 | |||

Suki Yuki Maji Magic! (frame 3,000) | |||

Neko Mimi Switch (frame 3,000) |

For your reference, see the following Youtube videos for the full Suki Yuki Maji Magic! and Neko Mimi Switch motions.

Lastly, run pa2.PA2_DanceScene to see two characters dance as a duet. This scene uses the Blinn-Phong material, so make sure that you edit pa2.ForwardRenderer so that it can use the material as well.

## Credits and Copyrights

The character models used in this assignment were originally designed to be used with the MikuMikuDance animation software. The KAITO and Utane Uta models were created by マシシ, which, as far as I (Pramook) understand from his/her homepage, is OK with modification and redistribution of the models. The Hatsune Miku model was created by あにまさ. The model was distributed together with the MikuMikuDance software, and it seems to be editable and redistributable. The character KAITO and Hatsune Miku are copyright of Crypton Future Media Inc. They seem to be permitting the characters to be used for educational purposes.

The Stand 1, 2, and 3 poses were taken from a collection of poses created by ゴロゴロウ, who generously allow the poses to be used even for commercial purposes. The Suki Yuki Maji Magic and the Neko Mimi Switch motions were created by hino, who allows the motions to be used for non-commercial purposes and redistributed. The motion for the dance scene is one for the song 恋は気まぐれイリュージョン！！ and was created by CRACKDOWN, who allows the data to be modified and redistributed for non-commercial purposes as well.

We are grateful of all the creators for their efforts and setting permissive conditions for reuse of their works.