7.2 Estimating Probabilities

• MLE (maximum likehood estimator)
  • sample$B$NIQEY$r$=$N$^$^MxMQ(B
  • sample$B?t$,B?$1$l$PB?$$$[$I(B,$B?.MjEY$,>e$,$k(B.
    the law of large numbers($BBg?t$NK!B'(B,$BCf?46K8BDjM}(B)
  • sample$B?t$,>/$J$$$H?.MjEY$,2<$,$k(B
    margin of error, coin toss $B$NNc(B($B;n9T2s?t$,>/$J$$$H%@%a(B)
  • Sparse Data
    Brown Courps $BC18l?t(B 100$BK|(B,$B0[$J$j8l?t(B 49,000
    $BJ?6Q(B 1$BC18l(B20$B2s$E$D=P8=$7$F$k$O$:$J$s$s$@$1$I(B... $B$,(B 5$B2s0J2<(B
    $B$3$N$h$&$J(B sparse $B$J4D6-$G$O==J,$J?dDj$,9T$($J$$(B
• ELE (expected likehood estimator)
  • V_i $B$r(B X=x_i $B$H$J$k2s?t$@$H$9$k$H(B, $B;v>](B x_i $B$,5/$3(B $B$k3NN($O(B
    
    $$\begin{eqnarray*}PROB(X = x_i) \simeq \frac{V_i}{\sum_i V_i} \end{eqnarray*}$$
    
    
  • data $B$,(B sparse $B$J>l9g(B,$BJ,Jl$,(B 0 $B$K$J$k2DG=@-$,9b$/(B, MLE $B$@$H3NN(CM$,Dj5A$G$-$J$$(B
  • $V_i = \vert x_i\vert + \epsilon(0.5)$ $B$H$9$k(B, $B$D$^$j$9$Y$F$NIQEY(B $B$,:GDc(B $\epsilon$ $B2s$"$k$H2>Dj(B
  • ($BNc(B1)
    40 $B$NIJ;lJ,N`(B, corpus $B$K0lEY$b=P8=$7$J$+$C$?C18l(Bw$B$,(B, $B$"$kIJ;l(B L_i $B$G$"$k3NN((B PROB(L_i|w) $B$r5a$a$k(B ( $\epsilon = 0.5$)
    
    $$\begin{eqnarray*}PROB(L_i\vert w) &=& 0.5/(40 \cdot 0.5) \\ &=& 0.025 \end{eqnarray*}$$
    
    ELM $B$@$H3NN(CM$r5a$a$k;v$,2DG=(B, MLE $B$@$HJ,Jl(B= 0 $B$H$J$j(B $B3NN(CM$rDj5A$G$-$J$$(B
  • ($BNc(B2)
    40$B$NIJ;lJ,N`(B, courps $B$K(B 5$B2s=P8=$7$?C18l(B w, $B$=$N$&$A(B noun 4$B2s(B, verb 1$B2s$N;~(B
    
    $$\begin{eqnarray*}MLE \cdots PROB(N\vert w) &=& 4/(0 \cdot 38 + 4 + 1) = 4/5 = 0.... ...t w) &=& (4+0.5)/(0.5\cdot38 + 4.5 + 1.5) = 4.5/25\\ &=& 0.18 \end{eqnarray*}$$
    
    
• Evaluation
  • courps $B$r(B traing set $B$H(B test set $B$K$o$1(B,training set $B$G3X(B $B=,$7$?7k2L$r(B test set $B$G;n$9(B
  • test set $B$NA*$P$lJ}$NJ,;6$r8:$i$9$?$a(B($B6vA3$K0-$$7k2L(B,$BNI$$(B $B7k2L$r$J$/$9$?$a(B)$B$K(B, traing set,test set $B$r=[4DE*$KJL$1$F(B, $BO"B3$7$?I>2A$r9T$&(B (cross validation)
    $BNc$($P(B, corpus $B$r(B 10$BEyJ,$7$F(B $B$=$N$&$A(B 9 $B$r(B traing, 1 $B$r(B test $B$H$9$k(B,
    test set $B$,(B 10$BDL$j$G$-$k$?$a(B 10$B2s$N;n83$,2DG=(B