Computes information gain of single feature and target vector.
Usage
calc_ig(feature, target, len_target, pos_target)
Arguments
feature
feature vector.
target
target.
len_target
length of the target vector.
pos_target
number of positive cases in the target vector.
Value
A numeric vector of length 1 representing information gain in nats.
Details
The information gain term is used here (improperly) as a synonym of mutual
information. It is defined as:
$$IG(X; Y) = \sum_{y \in Y} \sum_{x \in X} p(x, y) \log \left(\frac{p(x, y)}{p(x) p(y)} \right)$$
In biogram package information gain is computed using following relationship:
\(IG = E(S) - E(S|F)\)
References
Cover TM, Thomas JA Elements of Information Theory, 2nd Edition
Wiley, 2006.