Linear Independence

Linear independence is a property of a set of vectors where no vector in the set can be expressed as a linear combination of the others. In other words, the only way to combine the vectors to get zero is by using all zero coefficients. This means each vector contributes a unique "direction" that cannot be replicated by combining the other vectors.

To test for linear independence: (1) Set up the equation c₁v₁ + c₂v₂ + ... + cₙvₙ = 0; (2) Form a matrix with vectors as columns and row reduce to echelon form; (3) If every column has a pivot (full rank), the vectors are independent. Alternatively, calculate the determinant - if non-zero, the vectors are independent. For two vectors, simply check if one is a scalar multiple of the other.

A basis is a set of linearly independent vectors that spans the entire vector space. It's the minimum set of vectors needed to represent any vector in that space. For example, the standard basis for R³ consists of the three unit vectors (1,0,0), (0,1,0), and (0,0,1). Every vector in 3D space can be written as a unique combination of these basis vectors. The number of vectors in any basis equals the dimension of the space.

The span of a set of vectors is the collection of all possible linear combinations of those vectors. It represents all the points you can "reach" by scaling and adding the vectors together. For example, the span of a single non-zero vector is a line through the origin. The span of two independent vectors in 3D is a plane. If vectors span the entire space, any vector in that space can be written as their linear combination.

Linearly independent vectors cannot be expressed as combinations of each other - each adds a unique direction. Linearly dependent vectors have redundancy - at least one vector can be written as a combination of the others. For example, (1,0) and (0,1) are independent, but (1,0), (0,1), and (1,1) are dependent because (1,1) = (1,0) + (0,1). Dependent sets contain "unnecessary" vectors.

Linear independence is crucial in ML for several reasons: (1) Feature selection - independent features provide unique information, reducing redundancy; (2) Avoiding multicollinearity in regression, which causes unstable coefficients; (3) PCA and dimensionality reduction rely on finding independent principal components; (4) Neural network weight matrices need full rank for effective learning; (5) Understanding when models can find unique solutions versus infinite solutions.

No, in Rⁿ (n-dimensional real space), you cannot have more than n linearly independent vectors. This is because the dimension of the space limits how many independent directions exist. For example, in R² (a plane), at most 2 vectors can be independent. Any third vector will always be a combination of the first two. This is why a basis for Rⁿ always has exactly n vectors.

If the zero vector is included in a set, that set is automatically linearly dependent. This is because you can always write 0 = 1·0⃗ + 0·v₁ + 0·v₂ + ..., giving a non-trivial solution (the coefficient of the zero vector is 1, not 0). The zero vector adds no direction - it's the ultimate "redundant" vector. Therefore, any linearly independent set must exclude the zero vector.

For a square matrix formed by n vectors in Rⁿ, the determinant tells you about independence: if det ≠ 0, the vectors are linearly independent; if det = 0, they are dependent. Geometrically, the determinant measures the volume of the parallelepiped formed by the vectors. Zero volume means the vectors are "flat" - they lie in a lower-dimensional subspace, indicating dependence.

The rank of a matrix equals the maximum number of linearly independent columns (or rows). If you have n column vectors and the matrix has rank n, all columns are independent. If rank < n, some columns are dependent. For example, a 3×3 matrix with rank 2 has only 2 independent columns - the third is a combination of the other two. Rank essentially counts the "true" number of independent directions in your data.