Vector Spaces

A vector space is a mathematical structure consisting of a set of objects called vectors, along with two operations: vector addition and scalar multiplication. These operations must satisfy eight axioms (like closure, associativity, and the existence of identity elements) that ensure vectors can be combined and scaled in consistent, predictable ways. Common examples include \(\mathbb{R}^n\), polynomial spaces, and function spaces.

A subspace is a subset of a vector space that is itself a vector space under the same operations. To verify that a subset W is a subspace, you only need to check three conditions: (1) it contains the zero vector, (2) it is closed under vector addition (if u and v are in W, then u + v is in W), and (3) it is closed under scalar multiplication (if v is in W and c is a scalar, then cv is in W). Examples include planes and lines through the origin in \(\mathbb{R}^3\).

The dimension of a vector space is the number of vectors in any basis for that space. A basis is a set of linearly independent vectors that span the entire space. For example, \(\mathbb{R}^3\) has dimension 3 because any basis (like the standard basis i, j, k) contains exactly three vectors. The dimension tells you how many independent "directions" exist in the space and is a fundamental invariant - all bases of the same vector space have the same number of elements.

The eight vector space axioms are: (1) Closure under addition - adding two vectors gives another vector in the space; (2) Commutativity - u + v = v + u; (3) Associativity of addition - (u + v) + w = u + (v + w); (4) Additive identity - there exists a zero vector; (5) Additive inverse - every vector has a negative; (6) Closure under scalar multiplication; (7) Distributivity - c(u + v) = cu + cv and (c + d)v = cv + dv; (8) Compatibility - (cd)v = c(dv) and 1v = v.

A basis of a vector space is a set of vectors that is both linearly independent and spans the entire space. This means every vector in the space can be written as a unique linear combination of basis vectors. The standard basis for \(\mathbb{R}^3\) is {(1,0,0), (0,1,0), (0,0,1)}, but infinitely many other bases exist. Any two bases of the same vector space have the same number of vectors, which defines the dimension.

A vector is an individual element or object (like an arrow, a list of numbers, or a function), while a vector space is the entire collection of all such vectors together with the rules for adding them and scaling them. Think of it like the difference between a single point and the entire coordinate plane - the point is one element, but the plane (with its addition and scaling rules) is the vector space containing infinitely many such points.

The zero vector is the additive identity of a vector space - the unique vector that, when added to any other vector v, gives back v (i.e., v + 0 = v). In \(\mathbb{R}^n\), it's the origin (0, 0, ..., 0). The zero vector is crucial because it's required by the axioms, every subspace must contain it, and it's what you get when you multiply any vector by the scalar 0. It serves as the "neutral element" that makes the algebraic structure work properly.

The span of a set of vectors is the collection of all possible linear combinations of those vectors - every vector you can create by scaling and adding them together. For example, span{v} is a line through the origin (in the direction of v), and span{v, w} is typically a plane (if v and w aren't parallel). The span is always a subspace, and if vectors span the entire vector space, they can generate any vector in that space.

Vector spaces are fundamental to machine learning. Data points are represented as vectors in high-dimensional spaces, embeddings (like word2vec or BERT embeddings) live in vector spaces where similar items are close together, neural network weights form vector spaces that are optimized during training, and operations like PCA use linear algebra to reduce dimensionality. Understanding vector spaces helps explain why ML algorithms can find patterns - they're essentially doing geometry in high-dimensional vector spaces.

An inner product space is a vector space equipped with an additional operation called an inner product (or dot product), which takes two vectors and returns a scalar. This operation allows you to measure lengths (norms) and angles between vectors, enabling concepts like orthogonality and projections. The standard dot product in \(\mathbb{R}^n\) is the most common example. Inner product spaces are essential for applications like least squares fitting, Fourier analysis, and quantum mechanics.