IDC410 Machine Learning

2022-01-11.md

Machine Learning

1. Data collected from a real world process in form of input output pairs.
2. ML algorithm gives an approximation to $f$ which gives output for input. $f$ is the target function
3. $\hat{f}$ is a family of functions from the Hypothesis Space and the function is chosen by finding the argmin of some cost function.

Structure of Data

• Most data can be represented as a table of input columns and an output column.
• Data can be of
  • Continuous Variable
    • We want a PDF
      • # parameters depends on PDF
  • Categorical Variable
    • We want the probability distribution for the variable
      • k-1 parameters
    • There is no obvious order on them
  • Ordinal Variable
    • Categorical with order

But there are an infinite number of input parameters, as the input may be Continuous.

Conditional Probability

• We want to estimate the Conditional Probability $P(y|\overrightarrow{x})$
• This is called a discriminative model.
• In the binary categorical output, it is a bernoulli distribution.

Bayes Theorem

$$P(Y\parallel X)=\frac{P(X\parallel Y)P(Y)}{P(X)}$$

Conditional Independance Assumption

Given an output Y, all input parametersare independant of each other, i.e

$P(X|y_i)=\prod _{j}P(x_j|y_i)$

By modelling $P(x_j|y_i)$ as a Gaussian function, we have a total of $2n$ parameters for every $y_i$. In the case of the binary output, there are a total of $2n+2n+1$ parameters only, which reduces the complexity of the problem. $P(x_j|y_1)=2n$ $P(x_j|y_2)=2n$ $P(y_1)=\theta ,P(y_2)=1-\theta$

This is called the Naive Bayes algorithm.

Since we are looking at the joint probability distribution, and if we know it, we can generate a sample. Hence this model is also known as a generative sample.

Defining without Conditional Independance

We can also model the entire problem as an N dimensional Gaussian without the [2022-01-11.md].

For this, we need an N dimensional $\overrightarrow{\mu }$ and a covarience matrix which has $\mathcal{O}(n^2)$

So here, we have a total of $\mathcal{O}(n^2)$. Hence, [2022-01-11.md] makes the parameter space linear in $n$.

Basically, constraining the Hypothesis space reduces the complexity of the ML problem.

Logistic Approach

We can further simplify the model further, by considering

$P(y_i|X)=\frac{1}{1+\exp (\sum {\beta }_i{x}_i)}$

This model has only $n+1$ parameters.

Equation of decision boundary

P(y_1|X) = P(y_2|X) $⟹0=\sum {\beta }_i{x}_i$

That is, our discrimination boundary is a plane. If the data cannot be cut by the plane, then we cannot use this assumption.

Constraint defined on our hypothesis set is known as a Language Bias, and it is our choice which Language Bias to pick.