Training error, Empirical error, Generalization errorapi
overfitting, underfittingspa
Hold-out: \(D=S \cup T, ~S\cap T=\emptyset , ~where ~S~means~\textit{Sample},~T~means~\textit{Training}.\) Stratified sampling.orm
Cross validation: \(D=\bigcup_k D_k,~D_i\cap D_j=\emptyset ~ (i\neq j),\) K-fold cross validation. \(when ~K=\# D, ~we ~call ~it~ LOO ~\): Leave-one-out.ci
Bootstraping: give samples set \(D\), \(\#D=m\), do stratified sampling with replacement on \(D\), each time take one case to \(D'\), Repeat \(m\) times, then we get a new set \(D' ~ (\#D' = m)\). \(D'\) becomes training set, \(D - D'\) becomes testing set. out-of-bag estimate.rem
Parameter tuning: step, validation set.get
Performance measure:it
Samples \(D=\{(x_1,y_1),...(x_m,y_m)\}\), \(x_i \to y_i\), learner \(f\), learning result \(f(x)\), real tag \(y\).io
Regression: mean squared error.
\[ E(f;D)=\frac{1}m\sum_{i=1}^m (f(x_i)-y_i)^2 ~. \\ \int_{x \sim d} (f(x)-y)^2p(x)dx ~. \]form
Error rate:
\[ E(f;D)=\frac{1}m \sum_m\mathbb{I}(f(x_i)\neq y_i) ~. \]
class
Accuracy rate:
\[ acc(f; D)=\frac{1}m\sum_m\mathbb{I}(f(x_i) = y_i) \]
\[ =1-E(f; D) \]
Precision and Recall:
\[ P=\frac{TP}{TP+FP}\\ R=\frac{TP}{TP+FN} \]
\[ \frac{1}{F_\beta} = \frac{1}{1+\beta^2}\left( \frac{1}{P}+\frac{\beta^2}{R} \right) \]
\[ macro-R=\frac{1}n \sum_nP_i \\ macro-P=\frac{1}n \sum_nR_i \\ macro-F_\beta = F_\beta(macro-R, macro-P) \]
\[ micro-R = \frac{\overline{TP}} {\overline{TP}+\overline{FP}}. \]
\[ FPR=\frac{FP}{TP+FP} ~. \]