A NEW EXPONENTIAL DIRECTED DIVERGENCE INFORMATION MEASURE

Abstract

Depending upon the nature of the problem, different divergence measures are suitable. So it is always desirable to develop a new divergence measure. In the present work, new information divergence measure, which is exponential in nature, is introduced and characterized. Bounds of this new measure are obtained in terms of various symmetric and non- symmetric measures together with numerical verification by using two discrete distributions: Binomial and Poisson. Fuzzy information measure and Useful information measure corresponding to new exponential divergence measure are also introduced.

1. Introduction

Divergence measures are basically measures of distance between two probability distributions or compare two probability distributions. Divergence measure must increase as probability distributions move apart.

Divergence measures have been demonstrated very useful in a variety of disciplines such as Bayesian model validation [50], quantum information theory [33,35], model validation [4], robust detection [39], economics and political science [48,49], biology [38], analysis of contingency tables [18], approximation of probability distributions [11,29], signal processing [27,28], pattern recognition [2,7,10,26], color image segmentation [34], 3D image segmentation and word alignment [47], cost- sensitive classification for medical diagnosis [42], magnetic resonance image analysis [51] etc.

Also we can use divergence measures in fuzzy mathematics as fuzzy directed divergences and fuzzy entropies [1,20,25], which are very useful to find the amount of average ambiguity or difficulty in making a decision whether an element belongs to a set or not. Fuzzy information measures have recently found applications to fuzzy aircraft control, fuzzy traffic control, engineering, medicines, computer science, management and decision making etc. Divergence measures are also very useful to find the utility of an event [6,44], i.e., an event is how much useful compare to other event.

Without essential loss of insight, we have restricted ourselves to discrete probability distributions, so let , n ≥ 2 be the set of all complete finite discrete probability distributions. The restriction here to discrete distributions is only for convenience, similar results hold for continuous distributions as well. If we take pi ≥ 0 for some i = 1, 2, 3..., n, then we have to suppose that

Some generalized f- information divergence measures had been introduced, characterized and applied in variety of fields, such as: Csiszar’s f- divergence [12,13], Bregman’s f- divergence [8], Burbea- Rao’s f- divergence [9], Renyi’s like f- divergence [40], M- divergence [41], Jain- Saraswat f- divergence [22] etc.

Besides these, The f- divergence measure [3] with respect to two functions (f, g) is also introduced, which is

where g is an increasing function on R and f is real, continuous, and convex function on R+. We obtain many standard divergence measures by suitably defining the function f and g, such as: for f (t) = −t1−r, g (t) = − log (−t), 0 ≤ r ≤ 1, we get called Chernoff Coefficient and at well known the Bhattacharyya distance [5]. Similarly for , we obtain so called Generalized Matusita distance and at r = 1, we obtain the well known Variational distance or l1 distance [30]. Csiszar’s f- divergence is widely used due to its compact nature, which is given by

where f : (0, ∞) → R (set of real no.) is real, continuous, and convex function and P = (p1, p2, ..., pn), Q = (q1, q2, ..., qn) ∈ Γn, where pi and qi are probabilities.

Cf (P,Q) is a natural distance measure from a true probability distribution P to an arbitrary probability distribution Q. Typically P represents observations or a precise calculated probability distribution, whereas Q represents a model, a description or an approximation of P. Fundamental properties of Cf (P,Q) can be seen in literature [36], in detail.

Remark 1.1. For comparing multiple number of discrete probability distributions, following will be the Csiszar’s generalized f- divergence [15]

and following relation can be seen as well in the same literature

Divergences between more than two probability distributions are useful for discrimination and taxonomy.

Definition 1.1. Convex function: A function f (t) is said to be convex over an interval (a, b) if for every t1, t2 ∈ (a, b) and 0 ≤ λ ≤ 1, we have

and said to be strictly convex if equality does not hold only if λ ≠ 0 or λ ≠ 1. Geometrically, it means that if A, B, C are three distinct points on the graph of convex function f with B between A and C, then B is on or below chord AC.

Definition 1.2. Jensen inequality: Let f : I ⊂ R → R be differentiable convex on I0 (I0 is the interior of the interval I), ti ∈ I0 , λi > 0 ∀ i = 1, 2, ..., n and , then we have the following inequality.

If function is concave, then Jensen’s inequality will be reversed.

Corollary 1.3. After replacing λi with qi as and ti with for each i = 1, ..., n by assuming that the function is normalized, i.e., f (1) = 0, we get

The following theorem is well known in literature [13].

Theorem 1.4. If the function f is convex and normalized, i.e., f′′ (t) ≥ 0 ∀ t > 0 and f (1) = 0 respectively, then Cf (P,Q) and its adjoint Cf (Q, P) are both non-negative and convex in the pair of probability distribution (P,Q) ∈ Γn × Γn.

The following theorem is given by Taneja (2005), which relates two generalized f- divergence measures.

Theorem 1.5. Let f1, f2 : I ⊂ R+ → R be two convex differentiable and normalized functions, i.e., and f1 (1) = f2 (1) = 0 respectively and suppose the following assumptions.

(i) f1 and f2 are twice differentiable on (α, β), 0 < α ≤ 1 < β < ∞.

(ii) There exists the real constants m, M such that m < M and

If P,Q ∈ Γn, then we have the following inequalities

where Cf (P,Q) is given by (1).

 

2. New exponential divergence measure and properties

In this section, we introduce a new exponential divergence measure of Csiszar’s class and define the properties.

Let f : (0, ∞) → R be a real differentiable mapping, which is defined as

and

We can check that the function f1 (t) is exponential in nature and convex normalized because and f1 (1) = 0 respectively. Further f1 (t) is monotonically increasing in (0, ∞) as

After putting this exponential function in (1), we obtain

In view of corollary (1.3) and theorem (1.4), we see that Gexp (P,Q) is positive and convex for the pair of probability distribution (P,Q) ∈ Γn × Γn and equal to zero (Non- degeneracy) or attains its minimum value when pi = qi. We can also see that Gexp (P,Q) is non- symmetric divergence w.r.t. P and Q because Gexp (P,Q) ≠ Gexp (Q, P).

Remark 2.1. If function f (t) is convex in interval (0, ∞), then will be a convex function as well because , called conjugate of f (t). By putting this conjugate convex function in (1), we get , and we can see

is a symmetric exponential divergence.

Consequently, we obtain the following intra relations among new exponential divergences by applying remark (1.1), for comparing multiple number of discrete probability distributions taking normalized function

Remark 2.2. Bajaj and Hooda (2010) have defined ’useful’ fuzzy directed divergence of fuzzy set A from fuzzy set B. We can also define a new exponential measure of ’useful’ fuzzy directed divergence on the same lines, for this let A and B be two standard fuzzy sets with same supporting points x1, x2, ..., xn and with fuzzy vectors µA (x1) , ..., µA (xn) and µB (x1) , ..., µB (xn), then fuzzy information measure corresponding to new exponential measure (7), will be

Consequently, let ui > 0 be the utilities of the events Ei with probabilities pi and revised probabilities qi respectively, for all i = 1, 2, ..., n. Then Useful information measure corresponding to new exponential divergence measure (7), will be

If utilities are ignored, i.e., ui = 1 for each i, then we obtain the as usual Gexp (P,Q). Fuzzy information measures are very useful to find the amount of average ambiguity or difficulty in making a decision whether an element belongs to a set or not, whereas Useful information measure are very useful to find utility of an event, i.e., an event is how much useful compare to other event.

FIGURE 1.Convex function f1 (t)

 

3. Bounds of new divergence measure

To estimate the new exponential divergence Gexp (P,Q), it would be very interesting to establish some upper and lower bounds. So in this section, we obtain bounds of the exponential divergence measure (7) in terms of other symmetric and non- symmetric divergence measures.

Proposition 3.1. Let P,Q ∈ Γn and 0 < α ≤ 1 < β < ∞, then we have

where Gexp (P,Q) and ∆ (P,Q) are given by (7) and (15) respectively.

Proof. Let us consider

and

Since and f2 (1) = 0, so f2 (t) is strictly convex and normalized function respectively. By putting f2 (t) in (1), we get

where ∆ (P,Q) is called the Triangular discrimination [14].

Now, let

where are given by (6) and (14) respectively and

It is clear that g′ (t) > 0 for t > 0, therefore g (t) is strictly increasing function in interval (0, ∞). So

and

The result (12) is obtained by using (7), (15), (16), and (17) in inequality (4). □

Proposition 3.2. Let P,Q ∈ Γn and 0 < α ≤ 1 < β < ∞, then we have

where Gexp (P,Q) and h (P,Q) are given by (7) and (21) respectively.

Proof. Let us consider

and

Since and f2 (1) = 0, so f2 (t) is strictly convex and normalized function respectively. By putting f2 (t) in (1), we get

where h (P,Q) is called the Hellinger discrimination or Kolmogorov’s divergence [19].

Now, let

where are given by (6) and (20) respectively and

It is clear that g′ (t) > 0 for t > 0, therefore g (t) is strictly increasing function in interval (0, ∞). So

and

The result (18) is obtained by using (7), (21), (22), and (23) in inequality (4). □

In a similar procedure, we obtain the bounds of Gexp (P,Q) in terms of the other well known divergence measures. The results are as follows.

(a) If , then we have

where

is the Jensen- Shannon divergence or Information radius [9,43].

(b) If f2 (t) = (t − 1) log t, then we have

where

is the J- divergence or Jeffrey- Kullback divergence [24,31].

(c) If , then we have

where

is the Jain- Srivastava divergence [23].

(d) If , then we have

where

is the Symmetric chi- square divergence [17].

(e) If , then we have

where

is the Arithmetic- Geometric mean divergence [45].

(f) If , then we have

where

is the Kumar- Johnson divergence [32].

(g) If , then we have

where

is the Relative J- divergence [16].

(h) If f2 (t) = t log t, then we have

where

is the Kullback- Leibler divergence or Relative entropy or Directed divergence or Information gain [31].

(i) If , then we have

where

is the Relative Arithmetic- Geometric divergence [45].

(j) If f2 (t) = (t − 1)2, then we have

where

is the Chi- square divergence or Pearson divergence [37].

(k) If , then we have

where

is the Relative Jensen- Shannon divergence [43].

(l) If , then we have

where

is the Jain and Chhabra divergence [21].

Remark 3.1. Divergences (15), (21), (25), (27), (29), (31), (33), (35) are symmetric and divergences (37), (39), (41), (43), (45), (47) are non- symmetric with respect to probability distributions P,Q ∈ Γn.

 

4. Numerical verification of bounds

In this section, we take an example for calculating the divergences ∆ (P,Q), h (P,Q), G (P,Q) and Gexp (P,Q) and verify numerically the inequalities (12), (18), and (40) or verify the bounds of Gexp (P,Q).

Example 4.1. Let P be the binomial probability distribution with parameters (n = 10, p = 0.7) and Q its approximated Poisson probability distribution with parameter (λ = np = 7) for the random variable X. Then we have

By using Table 1, we get the followings.

Put the approximated values from (48) to (52) in inequalities (12), (18), and (40) respectively and get the following results

respectively. Hence verified the bounds of Gexp (P,Q) in terms of ∆ (P,Q), h (P,Q) and G (P,Q) for p = 0.7.

TABLE 1.Evaluation of discrete probability distributions for (n = 10, p = 0.7, q = 0.3)

Similarly, we can verify the bounds of Gexp (P,Q) in terms of other divergences or can verify the other inequalities for different values of p and q and for other discrete probability distributions as well, like; Negative binomial, Geometric, uniform etc.

In Figure 2, we have considered pi = (a, 1 − a), qi = (1 − a, a), where a ∈ (0, 1).

FIGURE 2.Comparison of divergence measures with new exponential divergence measure

It is clear from Figure 2 that the new exponential divergence Gexp (P,Q) has a steeper slope than ψ (P,Q), χ2 (P,Q), E (P,Q), ∆ (P,Q), h (P,Q), I (P,Q), J (P,Q), T (P,Q), and JR (P,Q).

 

5. Conclusion and discussion

To design a communication system with a specific message handling capability, we need a measure of information content to be transmitted. Divergence measures are for quantifying the dissimilarity among probability distributions. In this work we introduced a new exponential divergence measure and obtained the bounds by using Csiszar’s information inequality and verified the bounds numerically as well in the interval (α, β), 0 < α ≤ 1 < β < ∞. Fuzzy exponential information measure and Useful exponential information measure also introduced. Work on further generalizations of this new divergence measure is in progress and will be reported elsewhere, like: Application to the mutual information, other relations by using standard algebraic and exponential inequalities, square root of this new measure is a metric space etc.

We hope that this work will motivate the reader to consider the extensions of divergence measures in information theory, other problems of functional analysis and fuzzy mathematics.