Evaluating ranked retrieval systems

We will discuss some measures for evaluating rank retrieval methods.

To start let's imagine we have an information need that is satisfied by a set Relevant of relevant documents and three rankings returned by three systems. We want a measure that tells us which one provides a better ranking. Note that what we mean by "better" can depend on the use case.

In [15]:
%matplotlib inline
import matplotlib.pyplot as plt
In [16]:
## 6 relevant documents:
Relevant=["A","B","C","D","E","F"]

## rankings returned by 3 systems, we use # for non=relevant results:
S1=["A",
    "#",
    "B",
    "#",
    "#",
    "C",
    "D",
    "#",
    "#",
    "#"]

S2=["A",
    "#",
    "B",
    "#",
    "C",
    "#",
    "#",
    "D",
    "#",
    "#"]

S3=["#",
    "#",
    "#",
    "#",
    "A",
    "B",
    "C",
    "D",
    "E",
    "F"]

calculating set precision and recall

In [17]:
def isRelevant(item):
    if item in Relevant:
        return True

def precision(Sys):
    return len([item for item in Sys if isRelevant(item)])/len(Sys)

def recall(Sys):
    return len([item for item in Sys if isRelevant(item)])/len(Relevant)

since precision is a set measure, it should be the same for S1 and S2 should be the same (they return the same results)

In [18]:
print(precision(S1))
print(precision(S2))
print(precision(S3))

0.4
0.4
0.6

same for recall:

In [19]:
print(recall(S1))
print(recall(S2))
print(recall(S3))

0.666666666667
0.666666666667
1.0

the first attempt to take the ranking into account is to calculate precision@k: the precision of the set formed by the first k returned documents

In [25]:
def precision_at_k(Sys,k):
   return precision(Sys[:k]) 
    
print (" ".join([str("Rank"), str("Item"), "P@k"]))
for S in [S1,S2,S3]:
    for rank,item in enumerate(S):
        rank+=1 
        print ("    ".join([str(rank), str(item), "{0:.2f}".format(precision_at_k(S,rank))]))
    print("===")

Rank Item P@k
1    A    1.00
2    #    0.50
3    B    0.67
4    #    0.50
5    #    0.40
6    C    0.50
7    D    0.57
8    #    0.50
9    #    0.44
10    #    0.40
===
1    A    1.00
2    #    0.50
3    B    0.67
4    #    0.50
5    C    0.60
6    #    0.50
7    #    0.43
8    D    0.50
9    #    0.44
10    #    0.40
===
1    #    0.00
2    #    0.00
3    #    0.00
4    #    0.00
5    A    0.20
6    B    0.33
7    C    0.43
8    D    0.50
9    E    0.56
10    F    0.60
===

We can do the same for recall

In [27]:
def recall_at_k(Sys,k):
   return recall(Sys[:k]) 

print (" ".join([str("Rank"), str("Item"), " P@k","   R@k"]))
for S in [S1,S2,S3]:
    for rank,item in enumerate(S):
        rank+=1 
        print ("    ".join([str(rank), str(item), "{0:.2f}".format(precision_at_k(S,rank)),"{0:.2f}".format(recall_at_k(S,rank))]))
    print("===")

Rank Item  P@k    R@k
1    A    1.00    0.17
2    #    0.50    0.17
3    B    0.67    0.33
4    #    0.50    0.33
5    #    0.40    0.33
6    C    0.50    0.50
7    D    0.57    0.67
8    #    0.50    0.67
9    #    0.44    0.67
10    #    0.40    0.67
===
1    A    1.00    0.17
2    #    0.50    0.17
3    B    0.67    0.33
4    #    0.50    0.33
5    C    0.60    0.50
6    #    0.50    0.50
7    #    0.43    0.50
8    D    0.50    0.67
9    #    0.44    0.67
10    #    0.40    0.67
===
1    #    0.00    0.00
2    #    0.00    0.00
3    #    0.00    0.00
4    #    0.00    0.00
5    A    0.20    0.17
6    B    0.33    0.33
7    C    0.43    0.50
8    D    0.50    0.67
9    E    0.56    0.83
10    F    0.60    1.00
===

And now we print the precision vs recall, i.e., the "precision-recall" plot

In [45]:
for S in [S1,S2,S3]:
    plt.plot([recall_at_k(S,rank) for rank in range(1,11)],[precision_at_k(S,rank) for rank in range(1,11)])

plt.xlabel("Recall")
plt.ylabel("Precision")
plt.xlim([0.05,1.1])
plt.ylim([0,1.1])
plt.legend(["S1","S2","S3"])
Out[45]:
<matplotlib.legend.Legend at 0x115ea3cd0>

We can see that S3 stands out as a system that favors recall and sacrifces precision to the top. However, we might still want to summarize the performance in a single number.

One way to do this is to calculate Average precision. This measure averages precisions calculated every time a new relevant document is retrieved.

ingredient: precision up to document d

In [30]:
def precision_up_to(Sys,d):
    if d not in Sys:
        return 0
    rank_d=Sys.index(d)+1
    ranked_below=Sys[:rank_d]
    return precision(ranked_below)
In [31]:
def average_precision(Sys):
    ap=0
    for d in Relevant:
        ap+=precision_up_to(Sys,d)
    return ap/len(Relevant)
In [32]:
for S in [S1,S2,S3]:
    print("{0:.2f}".format(average_precision(S)))

0.46
0.46
0.44