Hard and incomputable problems

Hard problems

We have been seeing various useful algorithms and data structures for solving problems. However, some problems are intractably hard, in the sense that they require an unreasonable amount of time or space to compute, or even cannot be solved in general at all.

The class of problems that are generally considered to be tractable are those that can be solved in polynomial time, which is to say O(n^k) for some k. In practice, algorithms that take polynomial time with k larger than 1 scale poorly as well, and algorithms that require k≥5 are not used in practice (and even k=3 and k=4 are often impractically slow).

We define the complexity class P as the set of all problems that can be solved by an algorithm taking polynomial time:

P = ⋃_k O(n^k)

Beyond polynomial time there is the class EXPTIME, which includes all algorithms that take time O(2^{n^k}) for some k. Algorithms in EXPTIME effectively hit a wall when the problem size becomes too large. For example, if a algorithm takes time 2ⁿ, increasing the problem size from n to size n+1 requires twice the time. Even if the algorithm is fast for small n, increasing n quickly reaches a size where even a small increase is far too expensive. Contrast this with the case of an O(n^k) algorithm, where going from n to n+1 means increasing the time only by a multiple of k/n, a factor that gets smaller as n increases. (To see why, note that ((n+1)/n)^k ≈ 1 + k/n when n ≫ k.)

Another important class is nondeterministic polynomial time, or NP. These are the problems for which a possible answer can be checked in polynomial time, or equivalently, which can be computed in polynomial time using an unbounded number of machines computing independently in parallel. Given an unbounded number of machines, one simply has each machine generate one of the possible answers and then check in polynomial time whether it is correct.

Examples of problems in NP are the following:

Graph coloring. What is the minimum number of colors needed to color a graph so no edge connects two vertices of the same color? Given a coloring, checking whether it is correct can be done in linear time.
Hamiltonian cycles. A Hamiltonian cycle is a cycle that includes every node in a directed graph. The problem is whether there is such a cycle. Clearly, given a candidate cycle, it can be checked in polynomial time.
SAT. Given a boolean formula using logical “and”, ”or”, and “not”, is there a way to choose values for variables in the formula such that the whole formula is true? Given candidate assignments to variables, only polynomial time is required to determine whether the formula evaluates to true.

These are three famous problems in NP. However, the best known deterministic algorithms to solve these problems require exponential time. It is not known whether there is a polynomial-time algorithm to solve these problems, though most computer scientists believe there is none.

These three problems have the interesting property that in polynomial time, any problem in NP can be encoded into any of them. Because they can express any problem in NP, these problems are said to be NP-complete. If we had a polynomial-time algorithm to solve an NP-complete problem, we could solve any problem in NP in polynomial time! This result would mean that the complexity classes P and NP were exactly the same. Most computer scientists believe that P and NP are not the same; that there is no algorithm that solves problems in NP in worst-case polynomial time. However, no one has managed to prove it. Showing that P≠NP is probably the best known unsolved problem of computer science.

It is also possible to classify algorithms in terms of the memory space they require. Algorithms in PSPACE require a polynomial amount of space. Algorithms in L require only a logarithmic amount of space in addition to the input data; in effect, they can use a constant number of pointers into the input data. For example, a recent surprising result is that graph reachability is in L.

Some relationships among complexity classes have been proved, such as the following inclusion relationships:

L ⊆ P ⊆ NP ⊆ PSPACE ⊆ EXPTIME

It also known that L≠PSPACE and that P≠EXPTIME. However, many important things are not known! We don't know where the dividing line is between these classes. It is not known whether P=NP, whether L=P, or whether NP=EXPTIME. The complexity of some important problems is not known either. For example, even though the security of RSA encryption rests on the difficulty of factoring numbers, it is not known whether factoring is in P. (It is, however, known that factoring can in principle be solved in polynomial time on a quantum computer, though no one has been able to build a useful quantum computer.)

Computability and decidability

Beyond hard problems, there are even incomputable problems that can't be solved by any algorithm running on a computer. In fact, we can prove that such problems exist. We will focus on decision problems where the goal is to compute a boolean result about some input, and we will show that some decision problems are undecidable by any algorithm.

An example of such a decision problem is the halting problem: does a program terminate when run on an arbitrary input? We will see that the halting problem is undecidable (i.e., incomputable) in general, assuming that our programming language is expressive enough to write an interpreter for itself.

We have seen in previous lectures and from the programming assignments that programs can be represented as a data structure as such as abstract syntax tree or bytecode. Let us assume there is a type "Program" that we can use to represent programs.

We want to know whether we can implement a method with the following specification:

/** Returns whether program p terminates when given inp as input. */
boolean terminates(Program p, Object inp);

That is, for every program p and input inp, it successfully returns either true or false. Note that although terminates() is just one method, it is allowed to use as many other classes and methods as it likes. We have the full power of Java at our disposal.

For simplicity we will consider only programs p that themselves implement a decision problem, and therefore have the following structure:

class p {
    boolean main(Object inp) { ... }
}

Again, the method main is allowed to use other classes and methods. However, we will only consider programs that receive no input from and send no output to the outside environment. The only input to the program is inp and the only output is the boolean result of main. If we can't determine whether such simple programs terminate, we of course have no hope of determining whether more complex programs do.

We have seen that it is possible to write an interpreter for a programming language. An interpreter for programs like p can be written with the following signature:

/** Simulate the execution of program p on input inp, returning
 * the same result as p would. If p would fail to terminate on
 * this input, so does interpret.
 */
boolean interpret(Program p, Object inp);

In other words, running p.main(inp) has exactly the same result as running interpret(p, inp).

However, the ability to implement interpret implies that we cannot implement terminates. To see why, let us assume that we can implement both of these methods.

Creating a contradiction

We introduced the idea of an adjective being autological previously. We can think of a decision program as an adjective. Let us say that a program p is autological if terminates and returns true whenever it is passed itself as an input: that is, p.main(p) == true.

Similarly, a program is heterological if it does not terminate or does not return true when passed itself as an argument. Using terminates, we can implement a test for heterologicity!

class H {
    boolean main(Program p) {
	if (terminates(p, p))
	    return !interpret(p,p);
	else
	    return true;
    }
}

Program H must terminate because it only calls interpret when interpret is guaranteed to terminate. Therefore H always returns either true or false.

But now consider what happens when we test whether H itself is heterological by evaluating H.main(H). We know this must terminate, so we are going to be in the “then” branch of the code. From the code, we see that H.main(H) == !interpret(H, H). But from the specification of interpret, we know that H.main(H) == interpret(H, H). So H cannot return either true or false, yet it must return one of them.

This contradiction means that our original assumption must be wrong. We cannot implement both terminates and interpret. Since we know (roughly) how to implement interpret in Java, we must not be able to solve the halting problem for Java. Conversely, if we have a programming language in which we can compute whether programs terminate, that programming language must not be expressive enough to implement an interpreter for itself. In practice, such programming languages tend to lose even more expressive power than that.

One conclusion is that some useful things are simply not computable by any programming languages we know how to build.

Implications for program analysis

This result has some practical implications. In particular, many different program analyses other than termination cannot be precisely decided, because if they could be, the halting problem would be decidable. As a result, these program analyses must be conservative, giving answers “true”, “false” or “not sure”. Type checking is one example. Ideally a type checker would tell you at compile time whether there was any input that could cause a program to have a type error at run time. But we could apply such a type checker to code like the following:

while (...) {
   // complex computation
}
int x = 32 + "hi";

This code has a dynamic type error if and only if there is an input that causes the while loop to terminate. If we had a precise type checker, we could use it to determine whether the while loop terminates—if it reports no type error, the while loop cannot terminate, and vice versa. Even if we can build a type checker that works precisely on some programs, in general there will be some programs that cannot be type-checked precisely.

By similar arguments, we see that we have to be conservative about many other facts we'd like to know about programs—for example, whether they are correct, whether they are secure, or whether they leak memory. All automatic tools for analyzing programs will either be incomplete, meaning that they reject some safe programs as possibly unsafe (this is a false positive), or else unsound, meaning that they accept some unsafe programs as safe (this is a false negative). However, incomplete and even unsound automatic tools can still be useful!

Notes by Andrew Myers, 5/3/12.