Question

5.35 A study has ni independent binary observations (y yin) when Xx, Σί ni. Consider the model logit(m):: α+pm, where 1, , N, with n a. Show that the kernel of the likelihood function is the same treating the d b. For the saturated model, explain why the likelihood function is different for these c. Explain why the difference between deviances for two unsaturated models does d. Suppose that each n 1. Show that the deviance depends on fi but not y. ata as n Bernoulli observations or N binomial observations. two data forms. [Hint: The number of parameters differs.] Hence, the deviance reported by software depends on the form of data entry not depend on the form of data entry Hence, it is not useful for checking model fit (see also Exercise 4.18).

Answer 1

If our experiment is a single Bernoulli trial and we observe X = 1 (success) then the likelihood function is L(p ; x) = p . This function reaches its maximum at p^=1. If we observe X = 0 (failure) then the likelihood is L(p ; x) = 1 − p , which reaches its maximum at p^=0.

Suppose that X = (X₁, X₂, . . ., X_n) represents the outcomes of n independent Bernoulli trials, each with success probability p . The likelihood for p based on X is defined as the joint probability distribution of X₁, X₂, . . . , X_n. Since X₁, X₂, . . . , X_n are iid random variables, the joint distribution is

L(p;x)≈f(x;p)=∏i=1nf(xi;p)=∏i=1npx(1−p)1−x

Differentiating the log of L(p ; x) with respect to p and setting the derivative to zero shows that this function achieves a maximum at p^=∑i=1nxi/n. Since  ∑i=1nxi is the total number of successes observed in the n trials, p^ is the observed proportion of successes in the n trials. We often call p^ the sample proportion to distinguish it from p , the “true” or “population” proportion. Note that in some textbooks the authors may use π instead of p. For repeated Bernoulli trials, the MLE p^ is the sample proportion of successes.

ML for Binomial

Suppose that X is an observation from a binomial distribution, X ∼ Bin(n, p ), where n is known and p is to be estimated. The likelihood function is

L(p;x)=n!x!(n−x)!px(1−p)n−x

which, except for the factor n!x!(n−x)!, is identical to the likelihood from n independent Bernoulli trials with x=∑i=1nxi. But since the likelihood function is regarded as a function only of the parameter p, the factor n!x!(n−x)!is a fixed constant and does not affect the MLE. Thus the MLE is again p^=x/n, the sample proportion of successes.

You get the same value by maximizing the binomial loglikelihood function

l(p;x)=k+x log p+(n−x) log (1−p)

where k is a constant that does not involve the parameter p. In the future we will omit the constant, because it's statistically irrelevant.

The fact that the MLE based on n independent Bernoulli random variables and the MLE based on a single binomial random variable are the same is not surprising, since the binomial is the result of n independent Bernoulli trials anyway. In general, whenever we have repeated, independent Bernoulli trials with the same probability of success p for each trial, the MLE will always be the sample proportion of successes. This is true regardless of whether we know the outcomes of the individual trials X₁, X₂, . . . , X_n, or just the total number of successes for all trials X=∑i=1nXi

2).

The deviance of the saturated model is always given as zero", which is true, since the deviance, by definition (see Section 4.5.1 of Categorical Data Analysis (2nd Edition) by Alan Agresti) is the likelihood ratio statistic of a specified GLM to the saturated model. The constant aforementioned in the R documentation is actually twice the maximized log-likelihood of the saturated model.

1. The log-likelihood of the saturated model is in general non-zero.

2. The deviance (in its original definition) of the saturated model is zero.

3. The deviance output from softwares (such as R) is in general non-zero as it actually means something else (the difference between deviances).

.