Machine Learning Benno Stein
Contents I. Introduction
Objectives
Related Fields 1. Statistics
Literature Machine Learning:
Software Programming:
Software Statistics:
Chapter ML:I I. Introduction
Examples of Learning Tasks Car Shopping Guide
Examples of Learning Tasks Risk Analysis for Credit Approval
Examples of Learning Tasks Risk Analysis for Credit Approval
Examples of Learning Tasks Image Analysis
Examples of Learning Tasks Image Analysis
Specification of Learning Problems Definition 1 (Machine Learning
Remarks:
Specification of Learning Problems Learning Paradigms
Specification of Learning Problems Learning Paradigms
Specification of Learning Problems Example Chess: Kinds of Experience
Specification of Learning Problems Example Chess: Kinds of Experience
Specification of Learning Problems Example Chess: Kinds of Experience
Specification of Learning Problems Example Chess: Ideal Target Function γ
Specification of Learning Problems Example Chess: Ideal Target Function γ
Specification of Learning Problems Example Chess: From the Real World γ to a Model World y
Specification of Learning Problems Example Chess: From the Real World γ to a Model World y
Specification of Learning Problems Example Chess: From the Real World γ to a Model World y
Remarks:
Remarks (continued) :
Specification of Learning Problems
Specification of Learning Problems
Specification of Learning Problems How to Build a Classifier y
Specification of Learning Problems How to Build a Classifier y
Specification of Learning Problems How to Build a Classifier y
Specification of Learning Problems How to Build a Classifier y
Specification of Learning Problems How to Build a Classifier y
Specification of Learning Problems How to Build a Classifier y
Remarks:
Specification of Learning Problems LMS Algorithm for Fitting y
Remarks:
Specification of Learning Problems Design of Learning Systems
Specification of Learning Problems Design of Learning Systems
Specification of Learning Problems Related Questions
Specification of Learning Problems Related Questions
Chapter ML:II II. Machine Learning Basics
Regression Classification versus Regression
Regression Classification versus Regression
Regression The Linear Regression Model
Regression The Linear Regression Model
Remarks:
Remarks (continued) :
Regression One-Dimensional Feature Space
Regression One-Dimensional Feature Space
Regression One-Dimensional Feature Space
Regression One-Dimensional Feature Space
Regression One-Dimensional Feature Space
Regression One-Dimensional Feature Space
Regression One-Dimensional Feature Space
Regression Higher-Dimensional Feature Space
Regression Higher-Dimensional Feature Space
Regression Higher-Dimensional Feature Space
Regression Higher-Dimensional Feature Space
Regression Higher-Dimensional Feature Space
Remarks:
Regression Linear Regression for Classification
Regression Linear Regression for Classification
Regression Linear Regression for Classification
Regression Linear Regression for Classification
Regression Linear Regression for Classification
Regression Linear Regression for Classification
Regression Linear Regression for Classification
Regression Linear Regression for Classification
Regression Linear Regression for Classification
Regression Linear Regression for Classification
Regression Linear Regression for Classification
Regression Linear Regression for Classification
Regression Linear Regression for Classification
Regression Linear Regression for Classification
Regression The Linear Model Function: Variants
Regression The Linear Model Function: Variants
Regression The Linear Model Function: Variants
Regression The Linear Model Function: Variants
Regression The Linear Model Function: Properties of the Solution
Regression The Linear Model Function: Properties of the Solution
Regression
Regression n
Regression n
Remarks:
Chapter ML:II
Concept Learning: Search in Hypothesis Space A Learning Task
Remarks:
Concept Learning: Search in Hypothesis Space Objects
Concept Learning: Search in Hypothesis Space Objects
Concept Learning: Search in Hypothesis Space Usually, an example set D, D = {(x1, c(x1)), . . . , (xn, c(xn))}, contains positive
Remarks:
Concept Learning: Search in Hypothesis Space A Learning Task
Concept Learning: Search in Hypothesis Space A Learning Task
Concept Learning: Search in Hypothesis Space Order of Hypotheses
Concept Learning: Search in Hypothesis Space Order of Hypotheses
Concept Learning: Search in Hypothesis Space Order of Hypotheses
Remarks:
Remarks: (continued)
Remarks: (continued)
Concept Learning: Search in Hypothesis Space Inductive Learning Hypothesis
Concept Learning: Search in Hypothesis Space Find-S Algorithm
Remarks:
Concept Learning: Search in Hypothesis Space Find-S Algorithm
Concept Learning: Search in Hypothesis Space Find-S Algorithm
Concept Learning: Search in Hypothesis Space Find-S Algorithm
Concept Learning: Search in Hypothesis Space Find-S Algorithm
Concept Learning: Search in Hypothesis Space Discussion of the Find-S Algorithm
Concept Learning: Search in Version Space Definition 5 (Version Space)
Concept Learning: Search in Version Space Definition 5 (Version Space)
Remarks:
Concept Learning: Search in Version Space Definition 6 (Boundary Sets of a Version Space)
Concept Learning: Search in Version Space Definition 6 (Boundary Sets of a Version Space)
Concept Learning: Search in Version Space Candidate Elimination Algorithm
Concept Learning: Search in Version Space Candidate Elimination Algorithm
Remarks:
Concept Learning: Search in Version Space Candidate Elimination Algorithm (pseudo code)
Concept Learning: Search in Version Space Candidate Elimination Algorithm (pseudo code)
Concept Learning: Search in Version Space Candidate Elimination Algorithm (illustration)
Concept Learning: Search in Version Space Candidate Elimination Algorithm (illustration)
Concept Learning: Search in Version Space Candidate Elimination Algorithm (illustration)
Concept Learning: Search in Version Space Candidate Elimination Algorithm (illustration)
Concept Learning: Search in Version Space Candidate Elimination Algorithm (illustration)
Concept Learning: Search in Version Space Discussion of the Candidate Elimination Algorithm
Concept Learning: Search in Version Space Question 1: Selecting Examples from D
Concept Learning: Search in Version Space Question 1: Selecting Examples from D
Concept Learning: Search in Version Space Question 2: Partially Learned Concepts
Concept Learning: Search in Version Space Question 2: Partially Learned Concepts
Concept Learning: Search in Version Space Question 2: Partially Learned Concepts
Concept Learning: Search in Version Space Question 2: Partially Learned Concepts
Concept Learning: Search in Version Space Question 2: Partially Learned Concepts
Concept Learning: Search in Version Space Question 3: Inductive Bias
Concept Learning: Search in Version Space Question 3: Inductive Bias
Concept Learning: Search in Version Space Question 3: Inductive Bias
Concept Learning: Search in Version Space Question 3: Inductive Bias
Concept Learning: Search in Version Space Question 3: Inductive Bias
Chapter ML:II
Measuring Performance True Misclassification Rate
Measuring Performance True Misclassification Rate
Remarks:
Measuring Performance True Misclassification Rate: Probabilistic Foundation
Measuring Performance True Misclassification Rate: Probabilistic Foundation
Measuring Performance True Misclassification Rate: Probabilistic Foundation
Measuring Performance True Misclassification Rate: Probabilistic Foundation
Measuring Performance True Misclassification Rate: Probabilistic Foundation
Measuring Performance True Misclassification Rate: Probabilistic Foundation
Measuring Performance True Misclassification Rate: Probabilistic Foundation
Remarks:
Measuring Performance Training Error
Measuring Performance Training Error
Measuring Performance 2-Fold Cross-Validation (Holdout Estimation)
Measuring Performance 2-Fold Cross-Validation (Holdout Estimation)
Remarks:
Measuring Performance k-Fold Cross-Validation
Measuring Performance n-Fold Cross-Validation (Leave One Out)
Measuring Performance n-Fold Cross-Validation (Leave One Out)
Remarks:
Measuring Performance Bootstrapping
Remarks:
Measuring Performance Misclassification Costs
Remarks:
Chapter ML:III III. Decision Trees
Decision Trees Basics Specification of Classification Problems
Decision Trees Basics Decision Tree for the Concept “EnjoySport”
Decision Trees Basics Decision Tree for the Concept “EnjoySport”
Decision Trees Basics Definition 1 (Splitting)
Decision Trees Basics Definition 1 (Splitting)
Decision Trees Basics Definition 1 (Splitting)
Decision Trees Basics Definition 1 (Splitting)
Remarks:
Decision Trees Basics Definition 2 (Decision Tree)
Decision Trees Basics Definition 2 (Decision Tree)
Remarks:
Decision Trees Basics Notation
Remarks:
Decision Trees Basics Algorithm Template: Construction
Decision Trees Basics Algorithm Template: Classification
Remarks:
Decision Trees Basics When to Use Decision Trees
Decision Trees Basics On the Construction of Decision Trees
Decision Trees Basics Performance of Decision Trees
Decision Trees Basics Performance of Decision Trees
Decision Trees Basics Performance of Decision Trees: Size
Decision Trees Basics Performance of Decision Trees: Size
Decision Trees Basics Performance of Decision Trees: Size
Decision Trees Basics Performance of Decision Trees: Size
Decision Trees Basics Performance of Decision Trees: Size
Decision Trees Basics Performance of Decision Trees: Size
Decision Trees Basics Performance of Decision Trees: Classification Error
Decision Trees Basics Performance of Decision Trees: Classification Error
Decision Trees Basics Performance of Decision Trees: Classification Error
Remarks:
Decision Trees Basics Performance of Decision Trees: Misclassification Costs
Decision Trees Basics Performance of Decision Trees: Misclassification Costs
Decision Trees Basics Performance of Decision Trees: Misclassification Costs
Remarks:
Chapter ML:III III. Decision Trees
Impurity Functions Splitting
Impurity Functions Splitting
Impurity Functions Splitting
Impurity Functions Splitting
Impurity Functions Definition 4 (Impurity Function ι)
Impurity Functions Definition 5 (Impurity of an Example Set ι(D))
Impurity Functions Definition 5 (Impurity of an Example Set ι(D))
Remarks:
Impurity Functions Impurity Functions Based on the Misclassification Rate
Impurity Functions Impurity Functions Based on the Misclassification Rate
Impurity Functions Impurity Functions Based on the Misclassification Rate
Impurity Functions Impurity Functions Based on the Misclassification Rate
Impurity Functions Impurity Functions Based on the Misclassification Rate
Impurity Functions Impurity Functions Based on the Misclassification Rate
Impurity Functions Definition 7 (Strict Impurity Function)
Impurity Functions Definition 7 (Strict Impurity Function)
Remarks:
Impurity Functions Impurity Functions Based on Entropy
Remarks:
Impurity Functions Impurity Functions Based on Entropy
Impurity Functions Impurity Functions Based on Entropy
Impurity Functions Impurity Functions Based on Entropy
Remarks [::::::: Bayes :::
Remarks (continued) :
Impurity Functions Impurity Functions Based on Entropy
Impurity Functions Impurity Functions Based on Entropy
Impurity Functions Impurity Functions Based on Entropy
Impurity Functions Impurity Functions Based on Entropy
Impurity Functions Impurity Functions Based on Entropy
Impurity Functions Impurity Functions Based on Entropy
Impurity Functions Impurity Functions Based on Entropy
Impurity Functions Impurity Functions Based on the Gini Index
Impurity Functions Impurity Functions Based on the Gini Index
Impurity Functions Impurity Functions Based on the Gini Index
Impurity Functions Impurity Functions Based on the Gini Index
Chapter ML:III III. Decision Trees
Decision Tree Algorithms ID3 Algorithm
Decision Tree Algorithms ID3 Algorithm
Decision Tree Algorithms ID3 Algorithm
Decision Tree Algorithms ID3 Algorithm (pseudo code)
Decision Tree Algorithms ID3 Algorithm (pseudo code)
Decision Tree Algorithms ID3 Algorithm (pseudo code)
Decision Tree Algorithms ID3 Algorithm (pseudo code)
Remarks:
Decision Tree Algorithms ID3 Algorithm: Example
Decision Tree Algorithms ID3 Algorithm: Example
Decision Tree Algorithms ID3 Algorithm: Example
Remarks:
Decision Tree Algorithms ID3 Algorithm: Example
Decision Tree Algorithms ID3 Algorithm: Example
Decision Tree Algorithms ID3 Algorithm: Example
Decision Tree Algorithms ID3 Algorithm: Hypothesis Space
Decision Tree Algorithms ID3 Algorithm: Inductive Bias
Decision Tree Algorithms ID3 Algorithm: Inductive Bias
Decision Tree Algorithms ID3 Algorithm: Inductive Bias
Remarks:
Decision Tree Algorithms CART Algorithm
Decision Tree Algorithms CART Algorithm
Decision Tree Algorithms CART Algorithm
Decision Tree Algorithms CART Algorithm
Decision Tree Algorithms CART Algorithm
Chapter ML:III III. Decision Trees
Decision Tree Pruning Overfitting
Decision Tree Pruning Overfitting
Decision Tree Pruning Overfitting
Decision Tree Pruning Overfitting
Remarks:
Decision Tree Pruning Overfitting
Decision Tree Pruning Overfitting
Remarks:
Decision Tree Pruning Overfitting
Decision Tree Pruning Stopping
Decision Tree Pruning Pruning
Decision Tree Pruning Pruning
Decision Tree Pruning Pruning
Decision Tree Pruning Pruning
Decision Tree Pruning Pruning
Decision Tree Pruning Pruning: Reduced Error Pruning
Decision Tree Pruning Pruning: Reduced Error Pruning
Decision Tree Pruning Pruning: Reduced Error Pruning
Decision Tree Pruning Extensions
Chapter ML:IV IV. Statistical Learning
Probability Basics Area Overview
Probability Basics Definition 1 (Random Experiment, Random Observation)
Remarks:
Probability Basics Definition 2 (Sample Space, Event Space)
Probability Basics Definition 3 (Important Event Types)
Probability Basics Classical Concept Formation
Probability Basics Classical Concept Formation
Remarks:
Probability Basics Axiomatic Concept Formation
Probability Basics Axiomatic Concept Formation
Probability Basics Axiomatic Concept Formation
Probability Basics Axiomatic Concept Formation
Probability Basics Axiomatic Concept Formation
Remarks:
Probability Basics Conditional Probability
Probability Basics Conditional Probability
Probability Basics Conditional Probability
Probability Basics Conditional Probability
Remarks:
Probability Basics Independence of Events
Probability Basics Independence of Events
Chapter ML:IV
Bayes Classification Single Conditional Event
Bayes Classification Single Conditional Event
Bayes Classification Combined Conditional Events
Bayes Classification Combined Conditional Events
Remarks [Information gain for classification] :
Remarks (continued) :
Bayes Classification Naive Bayes
Bayes Classification Naive Bayes
Remarks:
Bayes Classification Naive Bayes
Bayes Classification Naive Bayes
Bayes Classification Naive Bayes
Remarks:
Bayes Classification Naive Bayes: Classifier Construction Summary
Bayes Classification Naive Bayes: Classifier Construction Summary
Remarks:
Bayes Classification Naive Bayes: Example
Bayes Classification Naive Bayes: Example
Bayes Classification Naive Bayes: Example
Bayes Classification Naive Bayes: Example
Bayes Classification Naive Bayes: Example
Bayes Classification Naive Bayes: Example
Chapter ML:VI VI. Neural Networks
Perceptron Learning The Biological Model
Perceptron Learning The Biological Model
Perceptron Learning History
Perceptron Learning The Perceptron of Rosenblatt
Perceptron Learning The Perceptron of Rosenblatt
Perceptron Learning The Perceptron of Rosenblatt
Remarks:
Perceptron Learning Specification of Classification Problems
Perceptron Learning Computation in the Perceptron
Perceptron Learning Computation in the Perceptron
Perceptron Learning Computation in the Perceptron
Remarks:
Perceptron Learning Weight Adaptation
Remarks:
Perceptron Learning Weight Adaptation: Illustration in Input Space
Perceptron Learning Weight Adaptation: Illustration in Input Space
Remarks:
Perceptron Learning Example
Perceptron Learning Example
Perceptron Learning Example: Illustration in Input Space
Perceptron Learning Example: Illustration in Input Space
Perceptron Learning Example: Illustration in Input Space
Perceptron Learning Example: Illustration in Input Space
Perceptron Learning Example: Illustration in Input Space
Perceptron Learning Example: Illustration in Input Space
Perceptron Learning Example: Illustration in Input Space
Perceptron Learning Example: Illustration in Input Space
Perceptron Learning Example: Illustration in Input Space
Perceptron Learning Example: Illustration in Input Space
Perceptron Learning Perceptron Convergence Theorem
Perceptron Learning Perceptron Convergence Theorem
Perceptron Learning Perceptron Convergence Theorem: Proof
Perceptron Learning Perceptron Convergence Theorem: Proof
Perceptron Learning Perceptron Convergence Theorem: Proof
Perceptron Learning Perceptron Convergence Theorem: Proof
Perceptron Learning Perceptron Convergence Theorem: Proof
Perceptron Learning Perceptron Convergence Theorem: Proof
Perceptron Learning Perceptron Convergence Theorem: Proof
Perceptron Learning Perceptron Convergence Theorem: Proof
Perceptron Learning Perceptron Convergence Theorem: Proof
Perceptron Learning Perceptron Convergence Theorem: Proof
Perceptron Learning Perceptron Convergence Theorem: Proof
Perceptron Learning Perceptron Convergence Theorem: Proof
Perceptron Learning Perceptron Convergence Theorem: Proof
Perceptron Learning Perceptron Convergence Theorem: Proof
Perceptron Learning Perceptron Convergence Theorem: Proof
Perceptron Learning Perceptron Convergence Theorem: Discussion
Perceptron Learning Perceptron Convergence Theorem: Discussion
Gradient Descent Classification Error
Gradient Descent Classification Error
Gradient Descent
Gradient Descent Weight Adaptation
Gradient Descent Weight Adaptation
Gradient Descent Weight Adaptation
Gradient Descent Weight Adaptation
Gradient Descent Weight Adaptation
Gradient Descent Weight Adaptation: Batch Gradient Descent
Remarks:
Gradient Descent Weight Adaptation: Delta Rule
Gradient Descent Weight Adaptation: Incremental Gradient Descent
Remarks:
Remarks (continued):
Chapter ML:VI
Multilayer Perceptron Definition 1 (Linear Separability)
Multilayer Perceptron Definition 1 (Linear Separability)
Multilayer Perceptron Separability
Multilayer Perceptron Separability
Multilayer Perceptron Separability
Remarks:
Multilayer Perceptron Computation in the Network
Multilayer Perceptron Computation in the Network
Multilayer Perceptron Computation in the Network
Multilayer Perceptron Computation in the Network
Multilayer Perceptron Computation in the Network
Multilayer Perceptron Computation in the Network
Multilayer Perceptron Computation in the Network
Remarks:
Multilayer Perceptron Classification Error
Multilayer Perceptron Weight Adaptation: Incremental Gradient Descent
Multilayer Perceptron Weight Adaptation: Incremental Gradient Descent
Multilayer Perceptron Weight Adaptation: Incremental Gradient Descent
Remarks:
Multilayer Perceptron Weight Adaptation: Momentum Term
Multilayer Perceptron Weight Adaptation: Momentum Term
Kapitel ML:IX
Motivating Ensemble Classification Generalisierungsfähigkeit von Klassifikatoren
Motivating Ensemble Classification No Free Lunch Theorems
Motivating Ensemble Classification Instabilität von Lernverfahren
Motivation Ensemble-Klassifikation Zusammenfassung
Bagging Bootstrap Aggregating
Remarks:
Bagging Bootstrap Aggregating
Bagging Algorithm:
Bagging Leistungsfähigkeit von Bootstrap Aggregating
Boosting Boosting Weak Classifiers
Boosting AdaBoost, Adaptive Boosting
Boosting Algorithm:
Boosting Leistungsfähigkeit von AdaBoost.M1
Remarks:
Boosting Algorithm:
Cascading Cascades of Classifiers
Cascading Cascades of Classifiers
Cascading Maßzahlen
Cascading Maßzahlen
Cascading Algorithm:
Cascading Anwendung: Face Detection (Main Loop)
Cascading Anwendung: Face Detection (Gesamtklassifikator)
Cascading Anwendung: Face Detection (Einzelklassifikator in Kaskade)
Cascading Anwendung: Face Detection (Einzelklassifikator in Ensemble)
Cascading Anwendung: Face Detection (alle Features)
Cascading Anwendung: Face Detection (ausgewählte Features)
Cascading Anwendung: Face Detection (Effizienz)
Cascading Anwendung: Face Detection (Effizienz beim Training)
Ensemble Classifier Trainingsmengen
Ensemble Classifier Grundlage der Entscheidung
Ensemble Classifier Literature