Tolerances in QMG

Many of the routines in QMG take a tolerance argument. This is a number between 0 and 1 and is used by QMG to determine when two items are the same and when they are different. If you leave the tol argument unspecified, it will to the value of the global variable gm_default_tol. This global variable is initialized to 1e-13 by the Matlab or Tcl/Tk startup routines.

QMG uses double precision arithmetic, so the smallest reasonable setting for this parameter is about 1e-16. But we found that roundoff error quickly builds, so 1e-13 seems reasonable. If routines start failing with peculiar error messages about points not in affine hulls or tolerance errors, you should increase the tolerance.

You also will need to increase the tolerance if you import a brep from some external package that works with lower precision.

Note that QMG routines automatically take into account the inherent size of the object you are working with and compute another tolerance called scaled_tol in most routines. For example, if your brep is a cube of side length 1e10, and your tolerance is 1e-11, then QMG will assume items in this brep are the same if they are within a distance 1e-1.

This documentation is written by Stephen A. Vavasis and is copyright (c) 1996 by Cornell University. Permission to reproduce this documentation is granted provided this notice remains attached. There is no warranty of any kind on this software or its documentation. See the accompanying file 'Copyright' for a full statement of the copyright.

Back to the QMG1.1 home page.

Stephen A. Vavasis, Computer Science Department, Cornell University, Ithaca, NY 14853, vavasis@cs.cornell.edu