# Randoop Manual

This is the manual for Randoop version 4.2.5, released December 2, 2020. The Randoop homepage is https://randoop.github.io/randoop/.

Contents:

- Introduction
  - Typical use of Randoop
- Installing Randoop
- Running Randoop
- Generating tests
  - Example: Generating tests for `java.util.Collections`
  - Specifying methods, constructors, and fields that may appear in a test
    - Which methods Randoop will call
  - Classifying tests
  - Error-revealing tests
  - Regression tests
    - Regression test failures
      - Verifying the cause of regression test failures
  - Stages of test generation, and limiting test generation
- Minimizing a failing JUnit test suite
- Customizing Randoop's behavior to your application
  - Command-line options
  - Avoiding calls to specific methods: the replacecall agent
    - Replacement exclusions
    - Defining replacements
      - Tips about implementing replacement classes
  - Specifying expected code behavior
    - Method specifications
      - Method signatures
      - Formal parameter names
      - Specifying conditions
      - Evaluation rules
    - Specifying representation invariant methods (such as `checkRep`)
  - Testing a specific class or method
    - Requiring tests to use a class: the covered-classes agent
  - Specifying additional primitive values
  - Nondeterminism
    - Finding the source of nondeterminism
    - Nondeterministic program under test
    - Changes to the program or its environment
    - Debugging nondeterminism in Randoop
- Getting help
  - Troubleshooting
    - Randoop does not run
    - Randoop cannot find a class-under-test
    - Randoop does not create enough tests, or creates no tests
    - Randoop does not terminate
    - Randoop produces different output on different runs
    - Randoop generated flaky tests
    - Randoop produces a regression test suite that does not pass
    - Tests behave differently in isolation or when reordered
  - Reporting a bug to the issue tracker
- Credits

Randoop.NET is a version of Randoop that works with Microsoft's .NET platform, such as the C# programming language.

There is a separate Randoop Developer's Manual.

# Introduction

Writing tests is important, but difficult and time-consuming. Randoop automatically generates unit tests for Java classes. Randoop has had many successful applications, especially with library classes (such as java.util). Randoop is used at companies like ABB and Microsoft, and on open-source projects.

Here is a JUnit test case generated by Randoop that reveals an error in OpenJDK (comments added by hand):

      // This test shows that the JDK collection classes
      // can create an object that is not equal to itself.
      @Test
      public static void test1() {

        LinkedList list = new LinkedList();
        Object o1 = new Object();
        list.addFirst(o1);

        // A TreeSet is an ordered collection. According to the API
        // documentation, this constructor call should throw a
        // ClassCastException because the list element is not Comparable. But
        // the constructor silently (and problematically) accepts the list.
        TreeSet t1 = new TreeSet(list);

        Set s1 = Collections.synchronizedSet(t1);

        // At this point, we have successfully created a set (s1)
        // that violations reflexivity of equality: it is not equal
        // to itself! This assertion fails at run time on OpenJDK.
        org.junit.Assert.assertEquals(s1, s1);
      }

Randoop outputs two kinds of tests:

- error-revealing tests that detect bugs in your current code, and
- regression tests that can be used to detect future bugs.

## Typical use of Randoop

Here is a typical way to use Randoop:

1. If Randoop outputs any error-revealing tests, fix the underlying defects, then re-run Randoop and repeat until Randoop outputs no error-revealing tests.
2. Add the regression tests to your project's test suite.
3. Run the regression tests whenever you change your project. These tests will notify you of changes to the behavior of your program.
4. If any test fails, minimize the test case and then investigate the failure.
   - If a test failure indicates you have introduced a code defect, fix the defect.
   - If a test failure indicates that the test was overly brittle or specific (for example, a method's output value has changed, but the new value is as acceptable as the old value), then disregard the test.
   If you disregarded any test (or if you added new code that you would like to test), then re-run Randoop to generate a new regression test suite that replaces the old one.

The paper "Scaling up automated test generation: Automatically generating maintainable regression unit tests for programs" gives additional tips for how to use Randoop-generated tests over the lifetime of a project.

A typical programmer will only ever examine very few Randoop tests — when they fail and reveal a bug or a regression failure — and then only minimized versions of them. A typical programmer will never modify Randoop tests by hand.

# Installing Randoop

Randoop runs on a Java 8 or Java 11 JVM.

Download and unzip the file `randoop-4.2.5.zip`. This manual uses `${RANDOOP_PATH}` to refer to the path of the unzipped archive, and `${RANDOOP_JAR}` to refer to the location of `randoop-all-4.2.5.jar` within the unzipped archive.

Alternately, if you want to view or work with the source code, follow the instructions in the Getting Started section of the Randoop Developer's Manual.

You may wish to join the randoop-discuss@googlegroups.com mailing list so that you are notified of new releases.

# Running Randoop

Run Randoop by invoking its main class `randoop.main.Main`:

    java randoop.main.Main command args...

Randoop supports three commands:

- gentests generates unit tests. For details, see Generating Tests. Example use:

      java -Xmx3000m -classpath myclasspath:${RANDOOP_JAR} randoop.main.Main gentests --testclass=java.util.TreeSet --output-limit=100

  (However, note that it is extremely unusual to use the `--testclass` command-line argument to specify just one class under test.)
- minimize minimizes a failing JUnit test suite. For details, see Minimizing a failing JUnit test suite. Example use:

      java -cp ${RANDOOP_JAR} randoop.main.Main minimize --suitepath=ErrorTest0.java --suiteclasspath=myclasspath

- help prints out a usage message. For details, see Getting Help. Example uses:

      java -classpath ${RANDOOP_JAR} randoop.main.Main help
      java -classpath ${RANDOOP_JAR} randoop.main.Main help gentests
      java -classpath ${RANDOOP_JAR} randoop.main.Main help minimize

(On Windows, adjust the classpath, such as using semicolon instead of colon as the separator.)

# Generating tests

By default, Randoop generates and then outputs two kinds of unit tests, written to separate files.

- Error-revealing tests are tests that fail when executed, indicating a potential error in one or more classes under test.
- Regression tests are tests that pass when executed, and can be used to augment a regression test suite.

Other generated tests, classified as invalid, are discarded.

This section of the manual gives an example of how to run Randoop, then describes test classification, error-revealing tests, and regression tests, and finally lists Randoop's command-line options.

## Example: Generating tests for `java.util.Collections`

Imagine we want to generate tests for the class `java.util.Collections`. (This example is a bit unrealistic! More likely you would want to generate tests for all of the JDK, or for parts of it like `java.lang` and `java.util`.)

Randoop's tests only use classes you specify. In order to effectively test Collections, you should probably also specify some helper classes, including classes that generate collections. For this example, we will add `java.util.TreeSet` to the mix.

Create a file `myclasses.txt` that lists the names of the classes under test:

    java.util.Collections
    java.util.TreeSet

and invoke Randoop as follows:

    java -classpath ${RANDOOP_JAR} randoop.main.Main gentests --classlist=myclasses.txt --time-limit=60

After 60 seconds, Randoop stops generating tests. The last thing Randoop prints out is the name of the JUnit files containing the tests it generated. You should see a message similar to the following:

    Created file: my/home/directory/RegressionTest0.java
    Created file: my/home/directory/RegressionTest.java
    Created file: my/home/directory/ErrorTest0.java
    Created file: my/home/directory/ErrorTest.java
    done.

Randoop creates two different test suites, in classes `ErrorTest` and `RegressionTest`. Compile and run the tests. (The classpath should include the code under test, the generated tests, and JUnit files `junit.jar` and `hamcrest-core.jar`. Classes in `java.util.*` are always on the Java classpath, so the myclasspath part is not needed in this particular example, but it is shown because you will usually need to supply it.)

    export JUNITPATH=.../junit.jar:.../hamcrest-core.jar
    javac -classpath .:$JUNITPATH ErrorTest*.java RegressionTest*.java -sourcepath .:path/to/files/under/test/
    java -classpath .:$JUNITPATH:myclasspath org.junit.runner.JUnitCore ErrorTest
    java -classpath .:$JUNITPATH:myclasspath org.junit.runner.JUnitCore RegressionTest

All of the tests in ErrorTest will fail, and all of the tests in RegressionTest will pass.

You also might want to minimize the error-revealing tests before running them:

    java -cp .:junit.jar:myclasspath:.../randoop/build/libs/randoop-all-4.2.5.jar \
      randoop.main.Main minimize --suitepath=ErrorTest0.java --suiteclasspath=.:junit-4.11.jar:myclasspath

## Specifying methods, constructors, and fields that may appear in a test

You need to tell Randoop what code it may call. In the common case of running Randoop on an entire program or library, it is usually enough to run Randoop on the `.jar` file. If you want to generate tests for only specific classes, then you need to do more work.

Suppose you want to test method `A.m`. You should direct Randoop to call not just `A.m`, but also methods that create `A.m`'s inputs. More concretely, if `A.m`'s parameter type is `B`, then you should tell Randoop to also call methods or constructors that create objects of type `B`. When a test calls `A.m`, all of the methods that set up state or create argument values are also being implicitly tested.

This command gives you an over-estimate (because it includes method call types as well as method argument types):

      jdeps -apionly -v -R -cp myclasspath MyClass.class \
        | grep -v '^[A-Za-z]' | sed -E 's/^.* -> ([^ ]+) .*$/\1/' | sort | uniq

You may want to adjust the command-line arguments. For example, you can pass a `.jar` file instead of a `.class` file. Or, you could remove `-R` (and possibly run it twice to get depth-2 instead of depth-1).

It is unusual to specify just one or a few classes under test. Usually, you want to test an entire application or library. Specifying too few classes under test can prevent Randoop from achieving good coverage.

### Which methods Randoop will call

Randoop only invokes a method or constructor M if all of the following are true:

1. M is declared in a class that is named by `--testjar`, `--classlist`, or `--testclass`, or M is a member named by `--methodlist`.
2. M does not match any of the patterns given via `--omit-methods` or `--omit-methods-file`.
3. All of the following are accessible from the package of the tests (set with `--junit-package-name`): M's class, M, and M's parameter types and return type.

A method or constructor M that satisfies the above conditions is called a "method under test". Randoop only calls methods under test; that is, these are the only methods that will appear in a test case.

Use raw type names such as `List`, not parameterized types such as `List<String>`, when specifying classes or methods.

Do not include methods that side-effect global state, for two reasons. (Methods that side-effect the state of their receiver or other arguments are fine.)

- Such methods create dependent tests that cannot be run in isolation or reordered. For example, if test A side-effects global state and test B reads global state, then whether test B passes depends on whether test A is run before test B.
- If you ask Randoop to test code that modifies your file system (such as File.delete()), then Randoop will generate tests that modify your file system! Be careful when choosing classes and methods to test.

## Classifying tests

Randoop generates many tests internally, and it classifies each generated test as error-revealing, expected behavior, or invalid. The classification depends primarily on whether the last statement throws an exception or violates a contract. You can also provide specifications of the methods under test to further help Randoop classify tests; see section "Specifying expected code behavior".

You can use command-line arguments to customize Randoop's rules for classifying a test that throws an exception.

- Checked exceptions: By default, Randoop classifies a test that throws a checked exception as expected behavior. You can override this default with the `--checked-exception` command-line option; specify `--checked-exception=error`, `--checked-exception=expected`, or `--checked-exception=invalid`.
- `OutOfMemoryError` or `StackOverflowError`. Options: `--oom-exception` and `--sof-exception`; default invalid
- `NullPointerException` when `null` is passed as an argument to any method in the test. Option: `--npe-on-null-input`; default expected
- `NullPointerException` when only non-`null` values are provided as an argument. Option: `--npe-on-non-null-input`; default invalid
- Unchecked exceptions other than `OutOfMemoryError` and `NullPointerException`, including AssertionError. Option: `--unchecked-exception`; default expected

By default, a thrown exception is not considered to indicate an error in the code under test. (Equivalently, none of the above command-line arguments defaults to `ERROR`.) This is because a method may have a precondition that is unknown to Randoop. For example, a binary search implementation might require that its input is sorted. Randoop should not mark the method as buggy just because it throws an exception when Randoop passes it an ill-formed unsorted array. Another example is passing `null` to a method that requires a non-null argument, which might result in a NullPointerException, AssertionError, or other exception. Randoop has no way of knowing each method's precondition. In order to avoid outputting an "error-revealing test" that doesn't reveal an error but merely misuses the software under test, by default Randoop assumes that any exception thrown by a method is correct behavior in response to the values that it was passed.

## Error-revealing tests

When Randoop includes a error-revealing test in file `ErrorTest.java`, the test shows that the code violates its specification or contract. For example, the test shown in the Introduction creates a `TreeSet` that violates reflexivity of equality: namely, for every object `o`, `o.equals(o)` should return `true`. The test reveals an error in the `TreeSet` constructor, which does not properly check its arguments (and possibly a second error in the `TreeSet.equals` method, which should always check if its argument is the same object as itself and return `true` in such a case).

The test in the Introduction is nicely succinct: it shows a small number of method calls leading up to the assertion violation. Because Randoop's generation is pseudo-random, the actual test revealing the error can be significantly longer, and contain many irrelevant calls that are not necessary to reveal the contract violation. You may wish to minimize the test case to ease debugging.

### What kinds of errors does Randoop check for?

When Randoop calls a method that creates an object, Randoop verifies that the object is well-formed. Currently, Randoop checks for the following contracts:

- Contracts over Object.equals():
  - Reflexivity: `o.equals(o)` == `true`
  - Symmetry: `o1.equals(o2)` == `o2.equals(o1)`
  - Transitivity: `o1.equals(o2)` && `o2.equals(o3)` ⇒ `o1.equals(o3)`
  - Equals to null: `o.equals(null)` == `false`
  - it does not throw an exception
- Contracts over Object.hashCode():
  - Equals and hashcode are consistent: If `o1.equals(o2)==true`, then `o1.hashCode() == o2.hashCode()`
  - it does not throw an exception
- Contracts over Object.clone():
  - it does not throw an exception, including `CloneNotSupportedException`
- Contracts over Object.toString():
  - it does not throw an exception
  - it does not return null
- Contracts over Comparable.compareTo() and Comparator.compare():
  - Reflexivity: `o.compareTo(o)` == 0 (implied by anti-symmetry)
  - Anti-symmetry: `sgn(o1.compareTo(o2))` == `-sgn(o2.compareTo(o1))`
  - Transitivity: `o1.compareTo(o2)>0` && `o2.compareTo(o3)>0` ⇒ `o1.compareTo(o3)>0`
  - Substitutability of equals: `x.compareTo(y)==0` ⇒ `sgn(x.compareTo(z)) == sgn(y.compareTo(z))`
  - Consistency with `equals()`: `(x.compareTo(y)==0) == x.equals(y)` (this contract can be disabled)
  - it does not throw an exception
- Contracts over `checkRep()` (that is, any nullary method annotated with `@CheckRep`):
  - it does not throw an exception
  - if its return type is `boolean`, it returns `true`

Violation of any of these contracts is highly likely to indicate an error.

You can add additional contracts to Randoop, that will be checked for every object Randoop creates. Define a contract in directory `randoop/src/main/java/randoop/contract/`, and reference it in the constructor for `OperationModel`. You can also provide specifications of methods in the tested code to help Randoop classify tests; see section "Specifying expected code behavior".

## Regression tests

The tests in the `RegressionTest*.java` files assert the current behavior of the code under test: values returned or exceptions thrown. After you change your code, you can run these regression tests, which will alert you if your code changes affect the external behavior of the classes.

Here is an example of a regression test for the TreeSet class:

    // This test passes when executed
    @Test
    public void test10() throws Throwable {

      java.util.TreeSet var0 = new java.util.TreeSet();
      java.lang.Short var1 = new java.lang.Short((short)100);
      boolean var2 = var0.contains(var1);

      // Regression assertion (captures the current behavior of the code)
      org.junit.Assert.assertEquals(false, var2);
    }

This test would fail if you introduced an error that caused `contains` to return true on an empty set.

A regression test can also assert what exception is currently thrown, and it will fail if the code is changed to no longer throw that exception.

### Regression test failures

If a regression test fails even though you have not changed your program, then the test is flaky.

If a regression test fails after you change your program, there are three possible causes. You need to debug the test failure to determine which is the cause, and then take the appropriate action.

- You have introduced an error — your code changes are incorrect.
  In this case, you should correct your mistake and re-run the test suite.
- You have intentionally changed the visible behavior of your program. The regression test has corroborated the change.
  In this case, you need to update the tests. Most commonly, you will just discard them and generate new ones.
- The tests are too sensitive, and they have failed even though you made an inconsequential change. For example, maybe the tests depend on the order of elements in a hash table.
  In this case, you need to re-run Randoop, informing it not to depend on observations that may vary from run to run. You can also see the techniques suggested in the paper Scaling Up Automated Test Generation.

Usually, a change to your program causes many test failures, not just one. This is because Randoop generates many tests for each part of your program's functionality. Don't panic! In most cases, all of the failures have the same cause.

#### Verifying the cause of regression test failures

You may wish to verify the cause of the failures. A way to do this is to change the part of the program, that you suspect is the cause, back to its previous behavior. Once you are satisfied that you understand the failures, you can discard the tests and have Randoop generate new ones. For example, suppose that you believe that all the test failures are caused by a change to method `m`. You can change `m` back to its previous implementation and run the tests. If they all succeed, you can change `m` back to its new implementation and re-generate the tests. (As an alternative to making a temporary change to your source code, you can use the `replacecall` feature of the Randoop agent to replace the new functionality by the old functionality.)

## Stages of test generation, and limiting test generation

Here is how Randoop attempts to generate a test:

1. Randoop selects a method to call. (This is called a "step".) The selected method will be the last method call in the new test.
2. Randoop chooses arguments to the method call, from among values that have been computed by previous tests.
   If Randoop cannot find values that are different from those in every previous call to the method, then Randoop starts over choosing arguments.
   Otherwise, Randoop has created a test candidate — a new, unique sequence of operations that ends with the selected method. The candidate's code is the concatenation of the code to produce the arguments (except that for primitives, just the value itself is used), followed by the method call.
3. Randoop executes the test candidate and classifies its behavior. If its behavior is invalid, then Randoop discards it and starts over.
4. Randoop outputs error-revealing and regression tests, in separate suites.

The user can provide limits on any part of the test generation process, or on Randoop's run time. Test generation stops when any of the following occurs (this is controlled by method `AbstractGenerator.shouldStop`):

- The time limit (`--time-limit`) is reached.
- The number of attempts to generate a sequence reaches the step limit (`--attempted-limit`).
- The number of generated sequences reaches the input limit (`--generated-limit`).
- The number of error-revealing and regression tests reaches the output limit (`--output-limit`).
- One error-revealing test is generated, and `--stop-on-error-test=true` was supplied.
- A custom `Stopper` was provided and indicates that generation should stop (see the Randoop code).
- A `RandoopListenerManager`'s `IEventListener` it indicates that generation should stop (see the Randoop code).

Randoop does not optimize to generate the best suite within these limits. Rather, Randoop just halts when a limit is reached.

Furthermore, see the command-line options for threading for additional per-test time limits.

# Minimizing a failing JUnit test suite

Minimized test cases are usually easier to diagnose. Randoop includes a minimizer that, given a JUnit test suite, minimizes each failing test case. Each minimized test fails in the same way as the original, having the same failing assertions with the same stack trace. The minimizer works by removing and simplifying methods and statements in its input file, without changing the test suite's failure behavior.

There are two ways to use the minimizer:

- Supply `--minimize-error-test`, and Randoop minimizes each error-revealing test before outputting it.
- Use Randoop's `minimize` command. The minimizer can be used with any JUnit test suite: it can minimize failing tests that Randoop did not create.

For both ways to use the minimizer, see the documentation for the command-line options for failing test suite minimization.

The rest of this section discusses Randoop's `minimize` command.

To minimize all failing tests in `ErrorTest1.java`, run Randoop as follows:

    java -cp randoop-all-4.2.5.jar randoop.main.Main minimize \
      --suitepath=ErrorTest0.java --suiteclasspath=junit.jar:commons-lang.jar --testsuitetimeout=30

The minimizer produces a Java file named `InputTestSuiteMinimized.java` where `InputTestSuite.java` is the file being minimized.

The minimizer creates a new output file, rather than modifying the original test suite in place, because sometimes the minimization loses important information. For example, when a test case fails due to a `NullPointerException`, the minimizer might replace the right hand side of an expression with the value `null`.

# Customizing Randoop's behavior to your application

If you run Randoop without any command-line arguments or customizations to your environment, it will produce helpful tests. However, Randoop will produce much more helpful tests if you spend a little bit of time to tell Randoop about your application, and that is how we recommend using Randoop.

## Command-line options

- Code under test: which classes and members may be used by a test
  - --testjar=filename `[+]`. Treat every class in the given jar file as a class to test. The jarfile must be on the classpath.

    See the notes about specifying methods that may appear in a test.

  - --classlist=filename. File that lists classes to test. All of their methods are methods under test.

    In the file, each class under test is specified by its binary name on a separate line. See an example. These classes are tested in addition to any specified using `--testjar` and `--testclass`.

    Using `--classlist` is less common than `--testjar`. See the notes about specifying methods that may appear in a test.

  - --omit-classes=regex `[+]`. A regex that indicates classes that should not be used in tests, even if included by some other command-line option. The regex is matched against fully-qualified class names. If the regular expression contains anchors "`^`" or "`$`", they refer to the beginning and the end of the class name.
  - --omit-classes-file=filename `[+]`. A file containing a list of regular expressions that indicate classes not to call in a test. These patterns are used along with those provided with `--omit-classes`.
  - --testclass=string `[+]`. The fully-qualified raw name of a class to test; for example, `--testclass=java.util.TreeSet`. All of its methods are methods under test. This class is tested in addition to any specified using `--testjar` or `--classlist`.

    It is unusual to specify just one or a few classes to test. See the notes about specifying methods that may appear in a test.

  - --methodlist=filename. A file containing a list of methods and constructors to call in tests, each given as a fully-qualified signature on a separate line.

    These methods augment any methods from classes given by the `--testjar`, `--classlist`, and `--testclass` options.

    See an example file.

    Using `--methodlist` is less common, and more error-prone, than `--testjar`, `--classlist`, or `--testclass`. See the notes about specifying methods that may appear in a test.

  - --omit-methods=regex `[+]`. A regex that indicates methods that should not be called directly in generated tests. This does not prevent indirect calls to such methods from other, allowed methods.

    Randoop will not directly call a method whose fully-qualified signature matches the regular expression, or a method inherited from a superclass or interface whose signature matches the regular expression.

    If the regular expression contains anchors "`^`" or "`$`", they refer to the beginning and the end of the signature string.

    Methods replaced by the `replacecall` agent are also automatically omitted.

  - --omit-methods-file=filename `[+]`. A file containing a list of regular expressions that indicate methods that should not be included in generated tests. These patterns are used along with those provided with `--omit-methods`, and the default omissions.
  - --omit-field=string `[+]`. A fully-qualified field name of a field to be excluded from test generation. An accessible field is used unless it is omitted by this or the `--omit-field-file` option.
  - --omit-field-file=filename. File that contains fully-qualified field names to be excluded from test generation. An accessible field is used unless it is omitted by this or the `--omit-field` option.
  - --only-test-public-members=boolean. Restrict tests to only include public members of classes.

    When this is false, the setting of `--junit-package-name` and package accessibility is used to determine which members will be used in tests. [default false]

  - --silently-ignore-bad-class-names=boolean. Ignore class names specified by user that cannot be found [default false]
  - --flaky-test-behavior=enum. What to do if Randoop generates a flaky test. A flaky test is one that behaves differently on different executions.

    Flaky tests are usually due to calling Randoop on side-effecting or nondeterministic methods, and ultimately, the solution is not to call Randoop on such methods; see section Nondeterminism in the Randoop manual. [default OUTPUT]

    - HALT Randoop halts with a diagnostic message. You can determine the responsible methods, fix or exclude them, and re-run Randoop.
    - DISCARD Discard the flaky test. This option should be a last resort. It is inefficient and unproductive for Randoop to produce and discard a lot of flaky tests.
    - OUTPUT Output the flaky test, but with flaky assertions commented out. When the value is `OUTPUT`, Randoop also suggests methods under test that might have caused the flakiness. You should investigate them, fix or exclude them, then re-run Randoop.
  - --nondeterministic-methods-to-output=int. How many suspected side-effecting or nondeterministic methods (from the program under test) to print. [default 10]
- Which tests to output
  - --no-error-revealing-tests=boolean. Whether to output error-revealing tests. Disables all output when used with `--no-regression-tests`. Restricting output can result in long runs if the default values of `--generated-limit` and `--time-limit` are used. [default false]
  - --no-regression-tests=boolean. Whether to output regression tests. Disables all output when used with `--no-error-revealing-tests`. Restricting output can result in long runs if the default values of `--generated-limit` and `--time-limit` are used. [default false]
  - --no-regression-assertions=boolean. Whether to include assertions in regression tests. If false, then the regression tests contain no assertions (except that if the test throws an exception, it should continue to throw an exception of the same type). Tests without assertions can be used to exercise the code, but they do not enforce any particular behavior, such as values returned. [default false]
  - --check-compilable=boolean. Whether to check that generated sequences can be compiled. If true, the code for each generated sequence is compiled, and the sequence is only kept if the compilation succeeds without error. This check is useful because the assumptions in Randoop generation heuristics are sometimes violated by input methods, and, as a result, a generated test may not compile. This check does increases the runtime by approximately 50%. [default true]
  - --require-classname-in-test=regex. Classes that must occur in a test. Randoop will only output tests whose source code has at least one use of a member of a class whose name matches the regular expression.
  - --require-covered-classes=filename. File containing binary names of classes that the tests must use, directly or indirectly. This option only works if Randoop is run using the covered-class javaagent to instrument the classes. A test is output only if it uses at least one of the class names in the file. A test uses a class if it invokes any constructor or method of the class, directly or indirectly (the constructor or method might not appear in the source code of the test). Included classes may be abstract.
  - --minimize-error-test=boolean. If true, Randoop outputs both original error-revealing tests and a minimized version. Setting this option may cause long Randoop run times if Randoop outputs and minimizes more than about 100 error-revealing tests; consider using `--stop-on-error-test=true`. Also see the test case minimization options. [default false]
- Test classification
  - --checked-exception=enum. If a test throws a checked exception, should it be included in the error-revealing test suite (value: ERROR), regression test suite (value: EXPECTED), or should it be discarded (value: INVALID)? [default EXPECTED]
    - ERROR Occurrence of exception reveals an error.
    - EXPECTED Occurrence of exception is expected behavior.
    - INVALID Occurrence of exception indicates an invalid test.
  - --unchecked-exception=enum. If a test throws an unchecked exception other than `ConcurrentModificationException`, `NoClassDefFoundError`, `NullPointerException`, `OutOfMemoryError`, and `StackOverflowError`, should the test be included in the error-revealing test suite (value: ERROR), regression test suite (value: EXPECTED), or should it be discarded (value: INVALID)?

    The arguments `--cm-exception`, `--ncdf-exception`, `--npe-on-null-input`, `--npe-on-non-null-input`, `--oom-exception`, and `--sof-exception` handle special cases of unchecked exceptions. [default EXPECTED]

    - ERROR Occurrence of exception reveals an error.
    - EXPECTED Occurrence of exception is expected behavior.
    - INVALID Occurrence of exception indicates an invalid test.
  - --cm-exception=enum. If a test throws a `ConcurrentModificationException` exception, should it be included in the error-revealing test suite (value: ERROR), regression test suite (value: EXPECTED), or should it be discarded (value: INVALID)? [default INVALID]
    - ERROR Occurrence of exception reveals an error.
    - EXPECTED Occurrence of exception is expected behavior.
    - INVALID Occurrence of exception indicates an invalid test.
  - --ncdf-exception=enum. If a test throws a `NoClassDefFoundError` exception, should it be included in the error-revealing test suite (value: ERROR), regression test suite (value: EXPECTED), or should it be discarded (value: INVALID)? [default INVALID]
    - ERROR Occurrence of exception reveals an error.
    - EXPECTED Occurrence of exception is expected behavior.
    - INVALID Occurrence of exception indicates an invalid test.
  - --npe-on-null-input=enum. If a test that passes `null` as an argument throws a `NullPointerException`, should the test be be included in the error-revealing test suite (value: ERROR), regression test suite (value: EXPECTED), or should it be discarded (value: INVALID)? [default EXPECTED]
    - ERROR Occurrence of exception reveals an error.
    - EXPECTED Occurrence of exception is expected behavior.
    - INVALID Occurrence of exception indicates an invalid test.
  - --npe-on-non-null-input=enum. If a test that never passes `null` as an argument throws a `NullPointerException`, should the test be included in the error-revealing test suite (value: ERROR), regression test suite (value: EXPECTED), or should it be discarded (value: INVALID)? [default ERROR]
    - ERROR Occurrence of exception reveals an error.
    - EXPECTED Occurrence of exception is expected behavior.
    - INVALID Occurrence of exception indicates an invalid test.
  - --oom-exception=enum. If a test throws an `OutOfMemoryError` exception, should it be included in the error-revealing test suite (value: ERROR), regression test suite (value: EXPECTED), or should it be discarded (value: INVALID)? [default INVALID]
    - ERROR Occurrence of exception reveals an error.
    - EXPECTED Occurrence of exception is expected behavior.
    - INVALID Occurrence of exception indicates an invalid test.
  - --sof-exception=enum. If a test throws a `StackOverflowError` exception, should it be included in the error-revealing test suite (value: ERROR), regression test suite (value: EXPECTED), or should it be discarded (value: INVALID)? [default INVALID]
    - ERROR Occurrence of exception reveals an error.
    - EXPECTED Occurrence of exception is expected behavior.
    - INVALID Occurrence of exception indicates an invalid test.
  - --specifications=filename `[+]`. Read file of specifications; see manual section "Specifying expected code behavior".
  - --use-jdk-specifications=boolean. Use built-in specifications for JDK classes and for classes that inherit from them, as if they had been supplied using the `--specifications` command-line argument. [default true]
  - --ignore-condition-compilation-error=boolean. Make Randoop proceed, instead of failing, if the Java condition text of a specification cannot be compiled. [default false]
  - --ignore-condition-exception=boolean. Make Randoop treat a specification whose execution throws an exception as returning `false`. If true, Randoop treats `x.f == 22` equivalently to the wordier `x != null && x.f == 22`. If false, Randoop halts when a specification throws an exception. [default false]
- Side-effect-free methods
  - --side-effect-free-methods=filename. File containing side-effect-free methods, each given as a fully-qualified signature on a separate line. Specifying side-effect-free methods has two benefits: it makes regression tests stronger, and it helps Randoop create smaller tests.
- Limiting test generation
  - --time-limit=int. Maximum number of seconds to spend generating tests. Zero means no limit. If nonzero, Randoop is nondeterministic: it may generate different test suites on different runs.

    The default value is appropriate for generating tests for a single class in the context of a larger program, but is too small to be effective for generating tests for an entire program.

    Randoop may run for longer than this because of long-running tests. The elapsed time is checked after each test, not during its execution. [default 100]

  - --attempted-limit=int. Maximum number of attempts to generate a test method candidate. [default 100000000]
  - --generated-limit=int. Maximum number of test method candidates generated internally. [default 100000000]
  - --output-limit=int. The maximum number of regression and error-revealing tests to output. If there is no output, this limit has no effect. There is no output when using either `--dont-output-tests` or `--no-error-revealing-tests` together with `--no-regression-tests`.

    In the current implementation, the number of tests in the output can be substantially smaller than this limit. One reason is that Randoop does not output subsumed tests, which appear as a subsequence of some longer test. [default 100000000]

  - --maxsize=int. Do not generate tests with more than this many statements. [default 100]
  - --stop-on-error-test=boolean. Stop generation as soon as one error-revealing test has been generated. [default false]
- Values used in tests
  - --null-ratio=double. Use null with the given frequency as an argument to method calls.

    For example, a null ratio of 0.05 directs Randoop to use `null` as an input 5 percent of the time when a non-`null` value of the appropriate type is available.

    Unless --forbid_null is true, a `null` value will still be used if no other value can be passed as an argument even if --null-ratio=0.

    Randoop never uses `null` for receiver values. [default 0.05]

  - --forbid-null=boolean. Do not use `null` as input to methods or constructors, even when no other argument value can be generated.

    If true, Randoop will not generate a test when unable to find a non-null value of appropriate type as an input. This could result in certain class members being untested. [default false]

  - --literals-file=string `[+]`. A file containing literal values to be used as inputs to methods under test, or "CLASSES".

    Literals in these files are used in addition to all other constants in the pool. For the format of this file, see documentation in class `randoop.reflection.LiteralFileReader`. The special value "CLASSES" (with no quotes) means to read literals from all classes under test.

  - --literals-level=enum. How to use literal values that are specified via the `--literals-file` command-line option. See: `ClassLiteralsMode`. [default CLASS]
    - NONE Do not use literals specified in a literals file.
    - CLASS A literal for a given class is used as input only to methods of that class.
    - PACKAGE A literal is used as input to methods of any classes in the same package.
    - ALL Each literal is used as input to any method under test.
  - --method-selection=enum. Randoop generates new tests by choosing from a set of methods under test. This controls how the next method is chosen, from among all methods under test. [default UNIFORM]
    - UNIFORM Select methods randomly with uniform probability.
    - BLOODHOUND The "Bloodhound" technique from the GRT paper prioritizes methods with lower branch coverage.
  - --string-maxlen=int. Maximum length of strings in generated tests, including in assertions. Strings longer than 65KB (or about 10,000 characters) may be rejected by the Java compiler, according to the Java Virtual Machine specification. [default 1000]
- Varying the nature of generated tests
  - --alias-ratio=double. Try to reuse values from a sequence with the given frequency. If an alias ratio is given, it should be between 0 and 1.

    A ratio of 0 results in tests where each value created within a test input is typically used at most once as an argument in a method call. A ratio of 1 tries to maximize the number of times values are used as inputs to parameters within a test. [default 0.0]

  - --input-selection=enum. Randoop generates new tests by combining old previously-generated tests. This controls how the old tests are chosen, from among all existing tests. [default UNIFORM]
    - SMALL_TESTS Favor shorter sequences. This makes Randoop produce smaller JUnit tests.
    - UNIFORM Select sequences uniformly at random.
  - --clear=int. Clear the component set each time it contains the given number of inputs.

    Randoop stores previously-generated tests in a "component" set, and uses them to generate new tests. Setting this variable to a small number can sometimes result in a greater variety of tests generated during a single run. [default 100000000]

  - --clear-memory=long. Clear the component set each time Randoop uses this much memory.

    Setting this variable to a smaller number may prevent an out-of-memory exception or a run that is slow due to thrashing and garbage collection. [default 4000000000]

- Outputting the JUnit tests
  - --testsperfile=int. Maximum number of tests to write to each JUnit file. [default 500]
  - --error-test-basename=string. Base name (no ".java" suffix) of the JUnit file containing error-revealing tests [default ErrorTest]
  - --regression-test-basename=string. Base name (no ".java" suffix) of the JUnit file containing regression tests [default RegressionTest]
  - --junit-package-name=string. Name of the package for the generated JUnit files. Enables testing non-public members. Tests can be restricted to public members only by also using the option `--only-test-public-members`.
  - --junit-before-each=string. Name of file containing code text to be added to the `@Before`-annotated method of each generated test class. Code is uninterpreted, and, so, is not run during generation. Intended for use when run-time behavior of classes under test requires setup behavior that is not needed for execution by reflection. (The annotation `@Before` is JUnit 4, and `@BeforeEach` is JUnit 5.)
  - --junit-after-each=string. Name of file containing code text to be added to the `@After` -annotated method of each generated test class. Intended for use when run-time behavior of classes under test requires tear-down behavior that is not needed for execution by reflection. Code is uninterpreted, and, so, is not run during generation. (The annotation `@After` is JUnit 4, and `@AfterEach` is JUnit 5.)
  - --junit-before-all=string. Name of file containing code text to be added to the `@BeforeClass`-annotated method of each generated test class. Intended for use when run-time behavior of classes under test requires setup behavior that is not needed for execution by reflection. Code is uninterpreted, and, so, is not run during generation. (The annotation `@BeforeClass` is JUnit 4, and `@BeforeAll` is JUnit 5.)
  - --junit-after-all=string. Name of file containing code text to be added to the `@AfterClass`-annotated method of each generated test class. Intended for use when run-time behavior of classes under test requires tear-down behavior that is not needed for execution by reflection. Code is uninterpreted, and, so, is not run during generation. (The annotation `@AfterClass` is JUnit 4, and `@AfterAll` is JUnit 5.)
  - --junit-output-dir=string. Name of the directory in which JUnit files should be written.
  - --dont-output-tests=boolean. Run test generation without output. May be desirable when running with a visitor.

    NOTE: Because there is no output, the value of `--output-limit` will never be met, so be sure to set `--generated-limit` or `--time-limit` to a reasonable value when using this option. [default false]

  - --junit-reflection-allowed=boolean. Whether to use JUnit's standard reflective mechanisms for invoking tests. JUnit's reflective invocations can interfere with code instrumentation, such as by the DynComp tool. If that is a problem, then set this to false and Randoop will output tests that use direct method calls instead of reflection. The tests will include a `main` method and will execute methods and assertions, but won't be JUnit suites. [default true]
- Runtime environment
  - -D --system-props=string `[+]`. System properties that Randoop will set similarly to `java -D`, of the form "x=y".
  - --jvm-max-memory=string. How much memory Randoop should use when starting new JVMs. This only affects new JVMs; you still need to supply `-Xmx...` when starting Randoop itself. [default 3000m]
- Controlling randomness
  - --randomseed=int. The random seed to use in the generation process. If you want to produce multiple different test suites, run Randoop multiple times with a different random seed. By default, Randoop is deterministic: you do not need to provide this option to make Randoop deterministic. [default 0]
  - --deterministic=boolean. If true, Randoop is deterministic: running Randoop twice with the same arguments (including `--randomseed`) will produce the same test suite, so long as the program under test is deterministic. If false, Randoop may or may not produce the same test suite. To produce multiple different test suites, use the `--randomseed` command-line option. [default false]
- Logging, notifications, and troubleshooting Randoop
  - --progressdisplay=boolean. Run noisily: display information such as progress updates. [default true]
  - --progressintervalmillis=long. Display progress message every <int> milliseconds. -1 means no display. [default 60000]
  - --progressintervalsteps=long. Display progress message every <int> attempts to create a test; -1 means none [default 1000]
  - --debug-checks=boolean. Perform expensive internal checks (for Randoop debugging) [default false]
  - --log=filename. A file to which to log lots of information. If not specified, no logging is done. Enabling the logs slows down Randoop.
  - --selection-log=filename. A file to which to log selections; helps find sources of non-determinism. If not specified, no logging is done.
  - --operation-history-log=filename. A file to which to write operation usage, when Randoop exits.
  - --print-non-compiling-file=boolean. True if Randoop should print generated tests that do not compile, which indicate Randoop bugs. [default false]
- Advanced extension points
  - --visitor=string `[+]`. Install the given runtime visitor. See class randoop.ExecutionVisitor.
- Test case minimization
  - --suitepath=string. The Java file whose failing tests will be minimized.
  - --suiteclasspath=string. Classpath that includes dependencies needed to compile and run the JUnit test suite being minimized.
  - --minimizetimeout=int. The maximum number of seconds allowed for the entire minimization process. [default 600]
  - --testsuitetimeout=int. The maximum number of seconds allowed for the entire test suite to run. [default 30]
  - --verboseminimizer=boolean. Produce verbose diagnostics to standard output if true. [default false]
- Threading
  - --usethreads=boolean. If true, Randoop executes each test in a separate thread and kills tests that take too long to finish, as determined by the --call-timeout command-line argument. Tests killed in this manner are not reported to the user, but are recorded in Randoop's log. Use the `--log` command-line option to make Randoop produce the log.

    Use this option if Randoop does not terminate, which is usually due to execution of code under test that results in an infinite loop or that waits for user input. The downside of this option is a BIG (order-of-magnitude) decrease in generation speed. The tests are not run in parallel, merely in isolation. [default false]

  - --call-timeout=int. After this many milliseconds, a non-returning method call, and its associated test, are stopped forcefully. Only meaningful if `--usethreads` is also specified. [default 5000]

`[+]` marked option can be specified multiple times

A "binary name" is like a fully-qualified name, but uses "$" to separate nested classes from their containing classes, rather than ".".
A Randoop "fully-qualified signature" is

- `package-name.classname.method-name(argument-list)` for a method, or
- `package-name.classname(argument-list)` for a constructor

where `package-name` is a period-separated list of identifiers, and `argument-list` is a comma-separated list of fully-qualified Java raw types, without spaces. Recall that a raw type does not contain type parameters. Compared to a regular type, it removes type parameter declarations such as `<T>`, and it and replaces any use of a type parameter by its upper bound.

## Avoiding calls to specific methods: the replacecall agent

The code under test might make calls to methods that you do not want executed. Here are some examples.

- `System.exit` will terminate Randoop if called during test generation.
- `JOptionPane.showConfirmDialog` opens a modal dialog box requiring user interaction, which interferes with automated testing.
- The no-argument `Random()` constructor uses a randomly-chosen seed, so tests might behave differently on each execution.

It is not enough to use `--omit-methods` to prevent direct calls to these methods, because the methods may still be called indirectly. To prevent all calls to these methods (including direct calls), use the `replacecall` agent.

The Randoop `replacecall` agent transforms the code under test, replacing each call to an undesired method by a call to some other method. To use it:

- When running Randoop, pass the following command-line options to `java`:

      -Xbootclasspath/a:${RANDOOP_PATH}/replacecall-4.2.5.jar -javaagent:${RANDOOP_PATH}/replacecall-4.2.5.jar

- When running the generated tests, add the file `replacecall-4.2.5.jar` to the boot class path, and pass the same command-line options to `java`:

      -Xbootclasspath/a:${RANDOOP_PATH}/replacecall-4.2.5.jar -javaagent:${RANDOOP_PATH}/replacecall-4.2.5.jar

Randoop uses a default set of replacements. You can define additional replacements.

### Replacement exclusions

There are some classes that the replacecall agent should not transform. Suppose you are using Gradle to run Randoop or the generated tests. Randoop's replacement of `System.exit` will affect the behavior of the build.

The replacecall agent never modifies

- Randoop, including the agent
- JUnit and Hamcrest
- Gradle
- Classes in `rt.jar`, except in packages `java.awt` and `javax.swing`

(The exact rules appear in the Randoop source code in class `CoveredClassTransformer`.)

You can exclude additional packages from being transformed. Create an exclusion file `replacecall-exclusions.txt` containing, for example:

    mypackage.mysubpackage.
    org.something.library.

and run the agent with the `--dont-transform=filename` command-line option:

    -javaagent:${RANDOOP_PATH}/replacecall-4.2.5.jar=--dont-transform=replacecall-exclusions.txt

The exclusion file is a list of package names, one package per line. (Blank lines and `//`-style comments are ignored.) If a package name does not end with a period, one will be added. The package name is used to match the prefix of the fully-qualified classname.

For diagnostic output (such as to see what classes are being transformed), run the agent with the `--debug` flag

    -javaagent:${RANDOOP_PATH}/replacecall-4.2.5.jar=--debug

### Defining replacements

To define your own method replacements, you will write mock implementations (example) and you will specify the replacements in a replacement file (example).

Each line of a replacement file is whitespace-separated package names (replace all calls in all methods of all classes in the package and its subpackages), or whitespace-separated class names (replace all methods in the class), or whitespace-separated method signatures without return types (replace the first method by the second).

Give the replacement file to the replacecall agent using the `--replacement-file` argument:

    -javaagent:${RANDOOP_PATH}/replacecall-4.2.5.jar=--replacement-file=replacecall-replacements.txt

A replacement method must be static and non-private. (Therefore, replacement classes don't need to be instantiated.)

- If the original method is static, the replacement method should have the identical signature.
- If the original method is non-static, then the replacement method should have an additional first parameter whose type is the containing class of the original method. For instance, a call to `java.awt.Component.show()` would be replaced by a call to a method with the signature `show(java.awt.Component)`. An example mock class is `randoop.mock.java.awt.Component`.
- If the original method is a constructor (an `<init>` method), then the replacement method should have the same parameter types, and its return type is the class that contains the constructor.

Please keep in mind that when you define a replacement, it can change the behavior of client code.

#### Tips about implementing replacement classes

To simplify your replacement file and keep your mock classes organized, you should choose a package prefix and keep the package structure of the original classes. Randoop's default replacements file uses `randoop.mock` as the prefix, so the binary name of the `JOptionPane` mock class is `randoop.mock.javax.swing.JOptionPane`. (You may need to exclude this package if any of the replacement methods use otherwise replaced methods.)

If the class containing the method you wish to replace is loaded by the bootstrap class loader, or if it is called from a class loaded by the bootstrap class loader, then your replacement classes must also be loaded by the bootstrap class loader. For Java 8 this list includes most of the Java runtime. For Java 9+, the new module system dramatically reduces the number of classes loaded by the bootstrap loader. See Module base for a list. If this condition applies to your replacement code (assume it is located in `myreplace.jar`):

- When running Randoop, add the additional command-line option to `java`:

      -Xbootclasspath/a:mypath/myreplace.jar

- When running the generated tests, add the file `myreplace.jar` to the boot class path:

      -Xbootclasspath/a:mypath/myreplace.jar

Randoop's default replacements are in sub-packages of `randoop.mock`. You should not use the `randoop.mock` prefix for your own mock classes.

Sometimes, a replacement for an instance method needs to access an object that is different than the receiver (for example, it might have the same or a different type than the receiver). One way to manage this is for your replacement class to maintain a map from instances of the original receiver type to instances that will be used in the replacement method implementations.

## Specifying expected code behavior

You can provide a specification of the expected behavior of code. Randoop will use the specification to better classify method calls as error-revealing, expected behavior, or invalid.

You can specify both the behavior of methods and constructors, and representation invariants for classes.

### Method specifications

A specification of a method (including constructors) indicates the circumstances when the method can be called, and how it should behave when called.

The `--specifications=file` command-line option names a file containing the method specifications. The file format is a JSON list with elements that are the serialized form of the OperationSpecification class. You may also supply a `.zip` file containing such JSON files.

Randoop comes with an example specification file, which provides specifications for class `net.Connection`.

A method specification consists of the method signature, parameter names, a list of pre-conditions, a list of (normal) post-conditions, and a list of throws-conditions (exceptional post-conditions):

      {
        "operation": method-signature,
        "identifiers": formal-parameters,
        "pre-conditions": pre-condition-list,
        "post-conditions": normal-post-condition-list,
        "throws-conditions": throws-condition-list
      }

where only the `operation` field is required.

#### Method signatures

A method signature uniquely describes a method. It consists of the declaring class, method name, and parameter types. Types are given in the format used by `Class.forName()`; for example, member classes are set off with "`$`", as in `pkg.Class$InnerClass`.

The method `net.Connection.send(int)` is represented as

      {
        "classname": "net.Connection",
        "name": "send",
        "parameterTypes": [ "int" ]
      }

#### Formal parameter names

The conditions in a specification use identifiers, or variable names, to refer to formal parameters and the result. This part of the specification gives the names. For example:

      {
        "parameters": [ "coded" ],
        "receiverName": "target",
        "returnName": "returnValue"
      }

indicates that the single formal parameter of the method is referred to by the name `coded`. The receiver name may be omitted (default: `receiver`), and the return value name may be omitted (default: `result`).

#### Specifying conditions

The specification contains 3 lists of conditions, one for each different type of condition.

Pre-condition

  A pre-condition indicates a requirement that must be satisfied by the caller. Its guard is a Boolean expression that is evaluated in the pre-state — that is, before a method starts to execute. If the caller does not satisfy the pre-condition, then the call is invalid.

  To specify that the argument to a call to `net.Connection.send(int)` must be positive, you could write the condition

          {
            "description": "the code must be positive",
            "guard": "coded > 0"
          }

  The condition has an optional field `description` that Randoop uses in comments in the generated tests.

  To require that the `Connection` is open and the code is non-negative, you could equivalently write a single pre-condition `receiver.isOpen() && code > 0` or two pre-conditions `receiver.isOpen()` and `code > 0`.

  A pre-condition is used to filter values, not to generate them. Randoop uses a precondition to prune, not expand, its random search.

(Normal) Post-condition

  A post-condition indicates a requirement that must be satisfied by the method implementation. A normal postcondition indicates what values the method returns. If the caller satisfies the method's pre-condition and the method returns normally (without throwing an exception), then the post-condition must be satisfied or else the method is buggy.

  You specify a normal post-condition as guard-property pair. (A property is given using the same format as a guard, but a property is evaluated in the post-state — that is, after a method has finished executing.) In the example file, the method `net.Connection.receive()` has the post-condition:

          {
            "guard": "true",
            "property": "result >= 0",
            "description": "returns non-negative received value"
          }

  The guards of post-conditions are evaluated in the order that the post-conditions are listed. The list

          [
            {
              "guard": "!receiver.isOpen()",
              "property": "result == -1",
              "description": "if this is not open, return -1"
            },
            {
              "guard": "true",
              "property": "result >= 0",
              "description": "otherwise, returns non-negative received value"
            }
          ]

  indicates that the error-case should be tested first, followed by the general case that has the vacuous guard `true`.

Throws-condition

  A post-condition indicates requirements that must be satisfied by the method implementation. A throws-condition is a post-condition that indicates an exception the method is expected to throw. If a call satisfies the guard of a throws-condition, the exception must be thrown or the method is buggy.

  You specify a throws-condition as a guard-exception pair. If a guard is satisfied, then the method is supposed to throw an exception of the named class, or a subclass of the named class. From the example file, the `net.Connection.open()` method has the throws-condition

            {
              "guard": "receiver.isOpen()",
              "exception": "java.lang.IllegalStateException",
              "description": "throws IllegalStateException if the connection is already open"
            }

  Similar to normal post-conditions, throws-conditions are given in a list. The list, however, is treated as a set (meaning unordered), and may contain duplicate guards and exceptions. A method whose behavior is specified as

            [
              {
                "guard": "a == null",
                "exception": "java.lang.NullPointerException",
                "description": "throws NullPointerException if a is null"
              },
              {
                "guard": "i < 0",
                "exception": "java.lang.IndexOutOfBoundsException",
                "description": "throws IndexOutOfBoundsException if i is negative"
              }
            ]

  is considered as correct if, when called with `a==null` and `i==-1`, it throws either of the specified exceptions.

#### Evaluation rules

Randoop classifies a test sequence according to its last method call.

1. If the guard of any pre-condition is violated before the call, then Randoop classifies the method call as invalid and the sequence is discarded.
2. If the guard of any throws-condition is satisfied before the call, the exception from the condition is added to a set `expected` of expected exceptions. If the call throws a checked exception, the sequence is classified as having expected behavior if the thrown exception is a member of the set `expected`, or is a subclass of a member of `expected`, and is otherwise classified as error-revealing. If the `expected` set is non-empty and the call terminates normally, the sequence is classified as error-revealing.
3. If the guard of a normal-precondition is satisfied before the call, the property for the first satisfied guard is checked after the call. If the call terminates normally, the sequence is classified as having expected behavior if the property is satisfied. Otherwise, the sequence is classified as error-revealing.
4. If the method is inherited, the method call is also checked against all specifications of overridden methods in the inheritance chain. If the call is invalid by any specification, the sequence is classified as invalid. If the call is error-revealing by any specification, the sequence is classified as error-revealing. Otherwise, all specifications are satisfied and the sequence has expected behavior.
5. Otherwise, the sequence is classified according to the rules of classifying tests.

Randoop does no check of whether the guards are exhaustive or disjoint.

Randoop ships with specifications for parts of the JDK, and these specifications are used by default, unless you disable command-line option `--use-jdk-specifications`.

### Specifying representation invariant methods (such as `checkRep`)

The "representation invariant" of a class is a property that all instances of the class must satisfy. For example, a field might contain a non-negative number, or two fields might contain lists of the same length. Sometimes, a program contains a method that checks the rep invariant, for debugging purposes. If the rep invariant is ever violated, the program has a bug.

By using the `@CheckRep` annotation, you can tell Randoop which methods in your classes under test are rep invariant methods. Randoop will call these methods; if the method ever fails, Randoop outputs the test as a failing test.

A method annotated with `@CheckRep` must have one of two allowed signatures:

- A public instance method with no arguments and return type boolean. In this case, Randoop will interpret a return value of `true` as the rep invariant being satisfied, and a return value of `false` or an exception escaping the method as the rep invariant being violated.
- A public instance method with no arguments and return type void. In this case, Randoop will interpret a normal return (no exceptions) as the rep invariant being satisfied, and an exception escaping the method as the rep invariant being violated.

## Testing a specific class or method

Randoop is designed to create tests for an entire project. This is usually what engineers want. In rare cases, you might want to create tests for a particular class or method. In general, it is better to run Randoop on your entire project, and Randoop will create useful, broad tests, including for the specific class or method. Randoop does not have a way to generate tests for a specific class or method. However, you can filter Randoop's output so that you only see some of its tests, using the `--require-covered-classes` or `--require-classname-in-test` command-line option.

If you want to test a particular class or method, do not attempt to use the `--methodList`, `--classlist`, or similar command-line options. They restrict the methods that appear anywhere in a test, and are likely to lead to Randoop not producing any output.

### Requiring tests to use a class: the covered-classes agent

Randoop generates tests that use methods, constructors, and fields of classes given using the `--testjar`, `--classlist`, and `--testclass` options. Any given test may involve one or more of the classes.

The `--require-covered-classes=filename` option makes Randoop discard any test that doesn't use at least one of the classes listed in the file.

Suppose you only want to test class A, but its methods takes arguments of type B and C. If you supply only A as a test class, Randoop cannot create objects of type B and C, so Randoop may be unable to call class A. Instead, you can supply A, B, and C as test classes, but instruct Randoop to only output the tests that use A.

Use of the `--require-covered-classes` option is unusual: you probably care about testing your entire application rather than just part of it. The option makes Randoop less efficient, because Randoop creates but discards tests.

For the `--require-covered-classes` option to work, the classes must be instrumented so Randoop can determine when they are covered. You can do so by giving the following command-line option for java

    -javaagent:${RANDOOP_PATH}/covered-class-4.2.5.jar

This will instrument any non-abstract constructor and method of loaded classes so that the use of a class can be detected. The instrumentation is not done on any JDK or JUnit classes. The `covered-class-4.2.5.jar` file can be found in the unzipped distribution archive, or can be downloaded from the latest release.

## Specifying additional primitive values

Randoop chooses arguments to method calls from the result of any previous method call, and from a set of seed values. These are the built-in seed values:

- `byte`: `-1`, `0`, `1`, `10`, `100`
- `short`: `-1`, `0`, `1`, `10`, `100`
- `int`: `-1`, `0`, `1`, `10`, `100`
- `long`: `-1`, `0`, `1`, `10`, `100`
- `float`: `-1`, `0`, `1`, `10`, `100`
- `double`: `-1`, `0`, `1`, `10`, `100`
- `char`: `'#'`, `' '`, `'4'`, `'a'`
- `java.lang.String`: `""`, `"hi!"`

There are two ways of specifying additional seed values to Randoop.

- Command line. Use the `--literals-file` and/or `--literals-level` command-line option.
- Annotated code. Write the `@TestValue` annotation on a field. The field should:
  - include the `@TestValue` annotation,
  - be declared `public` and `static`,
  - be of a primitive (or `String`) type, or an array of primitives or `String`s, and
  - be initialized to contain one or more seed values.
  You can write `@TestValue` on a field in a class test, or you can create a helper class that just defines fields, and include the class in the list of classes given to Randoop to test.

  Below is an example helper class that demonstrates the use of the @TestValue annotation. For Randoop to use the primitive values described in the class, you would add this class to the list of classes under test (e.g. via the option `--testclass=example.TestValueExamples`).

        package example;
        import randoop.*;

        public class TestValueExamples {

          @TestValue
          public static int i = 0;

          @TestValue
          public static boolean b = false;

          @TestValue
          public static byte by = 3;

          @TestValue
          public static char c = 'c';

          @TestValue
          public static long l = 3L;

          @TestValue
          public static float f = (float) 1.3;

          @TestValue
          public static double d = 1.4;

          @TestValue
          public static String s1 = null;

          @TestValue
          public static String s2 = "hi";

          @TestValue
          public static int[] a1 = new int[] { 1, 2, 3 };

        }

Randoop does not generate random string or primitive values. You can include, as a method under test, a method that creates values. Its output will be used as an input to methods in your code.

## Nondeterminism

Randoop is deterministic: if you run Randoop twice using the `--deterministic` command-line argument, Randoop outputs the same tests in the same order.

If you want Randoop to produce different test suites, run Randoop twice supplying different values for the `--randomseed` command-line argument. This is the recommended practice, to yield an overall test suite with better coverage.

### Finding the source of nondeterminism

If Randoop produces a different output when you run it twice with the `--deterministic` command-line argument, then there is some source of nondeterminism.

If Randoop produces flaky tests and `--flaky-test-behavior` is not set to HALT, Randoop will output a list of methods suspected to be the cause of nondeterminism in decreasing order of likelihood. You should investigate these methods, fix nondeterministic ones or exclude them via `--omit-methods` or `--omit-methods-file` command-line argument, and re-run Randoop until flaky tests are not generated. If you discover a nondeterministic method, let the Randoop maintainers know so we can include it in Randoop's default `omitmethods-defaults.txt` or `JDK-nondet-methods.txt`.

The remainder of this section gives several reasons you may discover.

### Nondeterministic program under test

If the program under test is nondeterministic, Randoop may produce different tests on different runs. More seriously, Randoop may produce flaky tests that sometimes pass and sometimes fail.

If the program under test is nondeterministic, you have several options:

1. Change the program to be deterministic. In addition to making the program more testable, this is likely to make the program easier to debug. Here are some ways to eliminate nondeterminism in a sequential Java program:
   - Hashing. Even if your code does not call `hashCode()` directly, it may be nondeterministic if some `hashCode()` method is deterministic.
     - `Object.hashCode()` depends on where the object is laid out in memory, which depends on the operating system and on when the garbage collector runs. Most classes should `equals()` (the default is reference equality), and any class that overrides `equals()` must override `hashCode()`, for consistency with `equals()`.
     - `Object.toString()` depends on `Object.hashCode()`. Most classes should override `Object.toString()` to give more informative output (and usually not to depend on the hash code).
     - `Arrays.deepHashCode()` depends on on `Object.hashCode()`.
     - For determinism, don't instantiate the `Object` class, as in `new Object()`; instead, use some class that overrides `Object.hashCode()`. For example, DeterministicObject returns a number indicating the creation order of the object.
     - Some collections have arbitrary iteration order, such as `HashMap` and `HashSet`. Either sort their results, or replace the data structures by `LinkedHashMap` and `LinkedHashSet`.
   - Some APIs produce results in arbitrary order, such as File.list() and File.listFiles(). Sort their results before using them.
   - Don't call `new Random()` without an argument. Instead, use (for example) `new Random(0)`.
   - Don't use time and date functions to obtain the current time.
2. Transform the program at run time to make changes like the ones above, replacing the method's behavior with a different implementation (such as a deterministic one or a mock). You can do this by using Randoop's replacecall agent.
3. Prevent Randoop from directly calling problematic methods. Use the `--omit-methods` or `--omit-methods-file` command-line argument.
4. Instruct Randoop to attempt to discard or repair any flaky tests that it generates. Use the `--flaky-test-behavior` command-line argument. This is a last resort because Randoop may output a small test suite due to discarding many of the generated tests. When possible, is better to avoid generating flaky tests rather than discarding them after generating them.

You can also see the techniques suggested in the paper Scaling Up Automated Test Generation.

### Changes to the program or its environment

If the program or its environment changed between runs, Randoop may produce different test suites..

Changing the JDK counts as a change to the program. For example, consider a class

    public class Corner extends JComponent { ... }

In JDK 7, `Corner`'s `getAccessibleContext` method is inherited from `JComponent`. In JDK 8, `Corner`'s `getAccessibleContext` method is inherited from `java.awt.Component`. Depending on the JDK in use, Randoop discovers a different set of operations, because `Corner.class.getMethods()` returns different results depending on the JDK. Given different sets, Randoop may make different choices and produce different tests.

There can be implementation changes even within a single JDK major version, such as two different versions of JDK 8. These differences can lead to different Randoop outputs, or to tests that passed on one JDK implementation but fail on another.

### Debugging nondeterminism in Randoop

If the above techniques don't help you, you can diagnose nondeterminism by running Randoop twice with the command-line options

      --deterministic --log=randoop-log.txt --selection-log=selection-log.txt

Compare the logs across runs, and the first place that they differ will help you find the nondeterminism.

If you find that Randoop is nondeterministic because of a bug in Randoop, please report it so that we can investigate and correct the problem.

# Getting help

You can get help on Randoop in the following ways:

- Manual: what you are currently reading.
- Randoop's `help` command: For help on usage, use Randoop's `help` command. To print Randoop's main help message, run:

      java -classpath ${RANDOOP_JAR} randoop.main.Main help

  To print help for a command (such as `gentests` or `minimize`), supply it:

      java -classpath ${RANDOOP_JAR} randoop.main.Main help gentests

- Mailing list: If you don't find your question answered, email either `randoop-discuss@googlegroups.com` or `randoop-developers@googlegroups.com` (never both) with your question. (You might not see your question on the mailing lists immediately, because your message may need to be approved.) If your question relates to an execution of Randoop, be sure to give enough information to reproduce the execution; see the instructions for submitting a bug report.
- Bug reporting: If you have discovered a problem with Randoop, we would like to fix it. Please see the instructions for submitting a bug report.

## Troubleshooting

### Randoop does not run

If the command to run Randoop has the output

    Cannot find or load randoop.main.Main

then Randoop is not properly included in the classpath. One possibility is that you have included the Randoop jar file in the path specified by the `CLASSPATH` variable, but have given the location of the classes-under-test with the `-classpath` option at the command line. In this case, the `java` command will only use the classpath from `-classpath`.

It is recommended that both Randoop and the classes under test be included as the argument to the `-classpath` option to the `java` command. It only makes sense to set `CLASSPATH` to run Randoop if the location of classes-under-test will be the same each time you run it. See the Java Tutorial on path and classpaths.

If the command to run Randoop has the output

    Cannot find the Java compiler. Check that classpath includes tools.jar

then there is a problem with your Java installation. You perhaps have the JRE on your path before the JDK. Ensure that `JAVA_HOME` appears at the front of your path. If you run the Windows operating system, see the instructions for setting a custom Java path after installing JDK 8. Another possible cause of the problem is using an unsupported version of Java.

### Randoop cannot find a class-under-test

If when Randoop is run, something like the following is output

    No class found for type name "example.ExampleClass"

    Unable to load class "example.ExampleClass" due to exception: java.lang.ClassNotFoundException

then the classes under test are not in the classpath. The most likely case is that you have not included the path to these classes. Another possible cause is that you did not supply the classpath to Randoop properly.

If you are running Randoop on Windows, parts of the classpath should be separated by a semicolon ";" instead of the colon ":" shown in the examples in this document. See the Java Tutorial on paths and classpaths for platform-specific information on paths.

### Randoop does not create enough tests, or creates no tests

if Randoop's generated tests do not cover some part of your code (or if Randoop outputs "No tests were created. No JUnit class created."), then you need to change Randoop's parameters and re-run it. Spending just a little bit of effort may have a big impact on the quality of Randoop's output.

Consider some specific method that Randoop does not call, and determine why Randoop does not call that method. Run Randoop with logging enabled. Here is an incomplete list of possible problems and solutions:

1. The method is not a candidate to be called by Randoop. Randoop creates a "model", or a list of methods that it might call. This list appears in the Randoop logs after the text "Operations:". If the method is not in that list, determine why. Two common reasons are that the method is not public or that it is excluded by an `--omit-methods` or `--omit-methods-file` argument (or the default omissions).
2. Randoop never tries to call the method. That is, Randoop never randomly selects this method, from among all the methods in the model. You should run Randoop with a larger timeout. Alternately, you could run Randoop with fewer classes, or exclude some classes or methods from consideration. You might also avoid using `--usethreads`, which can make Randoop run very slowly.
3. Randoop tries to call the method, but is unable to create inputs (arguments to the method). You should give Randoop more classes to analyze, namely ones that create the objects that the method needs as input.
4. Randoop calls the method, but the inputs it uses are not valid/useful inputs. You should investigate why Randoop is unable to create the needed inputs. Maybe you need to give Randoop more classes to analyze (as in the previous bullet point) or provide additional primitive values.

If Randoop gets stuck, it outputs "*** Randoop has spent over 10 seconds executing the following test.", but it does not output any tests. Randoop creates and executes one test at a time, so a long-running sequence will prevent Randoop from creating new tests. Two possible reasons for a long-running test are:

- Randoop is executing a test that contains nonterminating behavior, or that takes very long to execute.

  The best solution is to determine the long-running method, then fix it (if it is buggy) or exclude Randoop from calling it. Here are ways to determine which method took a long time to execute:

  - Examine the stack traces that Randoop prints.
  - Examine the test that Randoop was running.
  - Run the test that Randoop was running.
  - Run Randoop with command-line option `--usethreads`, and with logging enabled. Examine the logs, which report which methods were terminated because they ran for too long.

  Another possibility is to run Randoop with the command-line option `--usethreads=true`, without fixing the method or excluding it from Randoop.

- Java has run out of memory and is thrashing. This explanation is likely if Randoop had become progressively slower before getting stuck; examine Randoop's progress output to see if this is the case. Give Java more memory by running `java` with, say, `-Xmx3000m`.

### Randoop does not terminate

When Randoop does not terminate, that is usually because Randoop is creating tests that do not terminate (or that execute for a very long time). Here are three things you can do.

- Determine the test that is currently executing. Use the `--log=filename` command-line option, and the currently-executing test should be at the end of the file.
- Determine which calls are executing by examining the stack trace. Use the `jstack` command, which you can run as `jstack pid`. Get the process id by running `jcmd`. (On Unix you can also type `Ctrl-\`; that is, hold down the Control key, then press the Backslash key.) If the same methods appear in multiple such stack traces, then you can exclude them from consideration by Randoop or prevent the JVM from calling the method.
- Run Randoop with the command-line option `--usethreads=true`. This will cause Randoop to discard tests in which any directly-called method takes `too long` to run.

If you cannot fix the problem that causes the long execution, you might want to exclude the method from testing.

### Randoop produces different output on different runs

See the manual section on nondeterminism.

### Randoop generated flaky tests

A test is flaky if it behaves differently on different executions.

There are two main reasons for flaky tests. One is global side effects: a Randoop-generated test might call a method that sets a static field or creates/deletes a file. (Calling Randoop on a method that side-effects local state, such as a field of the receiver or of an argument, is fine.) Another reason is nondeterminism: the program under test might behave differently on different runs. Such behaviors are not appropriate for in unit tests, so you should not call Randoop on such methods.

If Randoop encounters a flaky test, Randoop halts so that you can correct your invocation of Randoop. Another section of the manual gives steps for determining the source of nondeterminism.

If you find a flaky test that does not result from side effects or nondeterminism in the program, then please help us improve Randoop by submitting an issue via the issue tracker.

### Randoop produces a regression test suite that does not pass

Randoop outputs regression tests that passed during test generation. If the regression tests fail when you run them, they are said to be flaky. This is due to nondeterminism in the program under test. See the manual section on nondeterminism.

### Tests behave differently in isolation or when reordered

Another symptom of calling Randoop on methods that have global side effects, in addition to flaky tests, is dependent tests: tests that behave differently depending on whether some other test has already run. For instance, test B might succeed if run after test A, but fail if run on its own.

You have two choices. (1) Always run Randoop's generated tests in the same order (without performing test selection, prioritization, or parallelization). (2) Avoid calling Randoop on methods with global side effects as described in the section on flaky tests.

Another possible symptom is that if you run Randoop's tests and then your original test suite, the original test suite might fail because of side effects performed by Randoop's tests. To avoid this problem, you can run Randoop's tests in a separate JVM, or run Randoop's tests last, or avoid calling Randoop on methods with global side effects as described in the section on flaky tests.

## Reporting a bug to the issue tracker

If you have discovered a problem with Randoop, we would like to fix it. Please submit an issue to the issue tracker. (Before doing so, search the open issues and the manual, especially the Troubleshooting section.)

You should include enough information to enable the Randoop developers to reproduce the problem, which will make it easier for them to diagnose the problem, fix Randoop, and verify the fix. This generally includes

- the version of Randoop (please use the most recent released version, or the current version from the GitHub version control repository),
- the exact commands you ran (please use the `-ea` option to `java` when running Randoop, which makes diagnosis easier),
- the complete output of the commands, and
- all the files that are necessary to run the commands.

Never submit just a screenshot or image of your computer screen. Instead, cut-and-paste the text of your command and its output. The text is easier to read, is searchable, and does not require transcribing from a screenshot in order to reproduce.

If you encountered the problem while using the Maven plug-in or running a command within an IDE, then please reproduce the problem from the command line, to help the Randoop developers find the underlying bug (or to isolate the problem to the plug-in or IDE configuration).

# Credits

The following individuals have contributed code to Randoop: Alessandra Gorla, Ayman Abdelghany, Ben Keller, Carlos Pacheco, Casey Xing, Jeff Perkins, Laura Inozemtseva, Mark Roberts, Martin Kellogg, Michael Ernst, Peter Kalauskas, René Just, Sai Zhang, Sonion Lee, Suzanne Millstein, Thad Guidry, Waylon Huang.

The feedback of Randoop users has been very valuable.