Recipient generates public/private key pair. Everyone learns the public key. Sender uses public key to encrypt; message can only be decrypted by recipient.
Signing is like encryption, except that the signer has the private key. Only the signer can generate the signature, but anyone with the public key can verify the signature.
Recipient generates two large primes, p and q. Computes m = pq. Chooses a unit [k] of Zϕ(m). Computes [k] − 1.
Public key: (m, k). Private key: [k] − 1.
Sender encodes message msg as element of Zm. Encrypts msg by forming the cyphertext c = [msg]k. Transmits the cyphertext.
Recipient computes c[k] − 1 = [msg][k][k] − 1 = [msg][1] = [msg].
To sign msg: compute sig = [msg][k] − 1.
To verify, compute [sig]k = [msg].
The key to security is that only the recipient knows p and q, so only the recipient can compute ϕ(pq), and thus only the recipient can compute [k] − 1.
If you could factor m, then you would know p and q, so you could compute the private key. Factoring is thought to be hard, i.e. to take exponential time in the number of bits of the key. Nobody has proven that factoring is hard.
Factoring non-deterministically is easy: if you can guess one of the factors, you can just divide to check. Not only is it not known whether factoring is hard, it is not known whether there are any problems that are easy to do non-deterministically but hard to compute deterministically.
To generate a large prime: choose a large number. Make sure it is odd. Test (probabilistically) whether it is prime. If not, add 2 and try again. There are lots of primes, so expected running time of this algorithm is low.
To generate k (which must be a unit mod ϕ(m)), choose a prime number. 17 is often used. [k] will be a unit as long as it doesn't divide p − 1 or q − 1; you can check this while generating p and q.
To compute [a]k, write k in binary; that is, write k = dn2n + dn − 12n − 1 + ⋯ + d222 + d121 + d020. Then [a]k = [a]dn2n[a]dn − 12n − 1⋯[a]4d2[a]2d1[a]d0 and each di is either 0 or 1. This reduces the problem to computing [a]2n for various n. You can compute those by repeatedly squaring [a].
You proved on the homework that [a] is a unit mod m if and only if gcd(a, m) = 1.
ϕ(m) is the number of units. If m is prime, then everything except [0] is relatively prime to m so ϕ(m) = m − 1.
If m = pq with p and q prime then ϕ(m) = (p − 1)(q − 1). Proof: start with pq total elements of Zpq. Subtract off p multiples of q and q multiples of p. You double counted 0. Use algebra to get (p − 1)(q − 1).