52 Vector, Banach, and Hilbert spaces

Author

Affiliation

Aditya Mahajan

McGill University

Updated

February 11, 2026

52.1 Vector space

A linear vector space over reals is a set \(\ALPHABET V\) of elements called vectors satisfying the following conditions:
1. Vector addition: For any \(v, w \in \ALPHABET V\), there is a unique vector \(v + w \in \ALPHABET V\) (called the sum) that satisfies the following properties: for all \(u, v,w \in \ALPHABET V\), we have
  - Commutativity: \(v + w = w + v\)
  - Associativity: \((u + v) + w = u + (v + w)\)
  - Existence of an identity: There exists a unique vector \(\mathbf{0} \in \ALPHABET V\) such that \[v + \mathbf{0} = v, \quad \forall v \in \ALPHABET V\]
  - Existence of an inverse: For each \(v \in \ALPHABET V\), there exists \((-v) \in \ALPHABET V\) such that \[ v + (-v) = \mathbf{0}.\]
2. Scalar multiplication: For every scalar \(α \in \reals\) and vector \(v \in \ALPHABET V\), there exists a unique vector \(αv \in \ALPHABET V\) (called the product) that satisfies the following properties: for all \(v, w \in \ALPHABET V\) and \(α, β \in \reals\), we have
  - Distributed law over vectors: \(α ⋅ (v + w) = α ⋅ v + α ⋅ w\)
  - Distributed law over scalars: \((α + β) ⋅ v = α ⋅ v + β ⋅ v\)
  - Associative law for scalar multiplication: \(α ⋅ (β ⋅ v) = (α β) ⋅ v\)
  - Existence of multiplicative identity: \(1 ⋅ v = v\).
Some examples of a vector space are:
- The Eucledian space \(\reals^d\) is a vector space with the usual definition of addition and scalar multiplication.
- Let \(\ALPHABET X \subset \reals^d\) be an open set. Then the set \(C(\ALPHABET X)\) of all continuous functions of \(\ALPHABET X\) is a vector space under pointwise addition and multiplication, i.e., for any \(f, g \in C(\ALPHABET X)\), \(f+g\) is defined by \[ (f+g)(x) = f(x) + g(x), \quad x \in \ALPHABET X \] belongs to \(C(\ALPHABET X)\); and for \(α \in \reals\), \(α f\) defined as \[ (αf)(x) = α f(x), \quad x \in C(\ALPHABET X) \] also belongs to \(C(\ALPHABET X)\).
A linear subspace \(\ALPHABET M\) of a vector space \(\ALPHABET V\) is a subset of \(\ALPHABET V\) which is closed under the addition and scalar multiplication operations of \(\ALPHABET V\), i.e., for any \(v, w \in \ALPHABET M\) and any \(α \in \reals\) we have \(v + w \in \ALPHABET M\) and \(α v \in \ALPHABET M\).
As an example, consider the linear subspace \(\ALPHABET V = \reals^3\). Then \[ \ALPHABET M = \{ (x_1, x_2, 0) : x_1, x_2 \in \reals \} \] is a linear subspace of \(\ALPHABET V\). In contrast, \[ \ALPHABET M' = \{ (x_1, x_2, 1) : x_1, x_2 \in \reals \} \] is not a linear subspace.
\(\ALPHABET M\) is said to be a proper subspace if \(\ALPHABET M \neq \ALPHABET V\).

52.2 Normed spaces

Given a vector space \(\ALPHABET V\), a norm \(\NORM{⋅}\) is a function from \(\ALPHABET V\) to \(\reals\) with the following properties:
1. Non-negativity: \(\NORM{v} \ge 0\) for all \(v \in \ALPHABET V\), and \(\NORM{v} = 0\) if and only if \(v = 0\).
2. Positive homogeneity: \(\NORM{αv} = \ABS{α}\NORM{v}\) for any \(v \in \ALPHABET V\) and \(α \in \reals\).
3. Triangle inequality: \(\NORM{u + v} \le \NORM{u} + \NORM{v}\) for any \(u,v \in \ALPHABET V\).
Given a vector \(\ALPHABET V\), a semi-norm \(\ABS{⋅}\) is a function from \(\ALPHABET V\) to \(\reals\) with all the properties of norm except \(\ABS{v} = 0\) does not imply that \(v = 0\).
For \(\ALPHABET V = \reals^d\), and \(p \in [1, ∞)\), the \(\ell_p\) norm of \(x = (x_1, \dots, x_d) \in \ALPHABET V\) is defined as \[ \NORM{x}_p = \biggl( \sum_{i=1}^d \ABS{x_i}^p \biggr)^{1/p}. \]. It can be shown that \[ \lim_{p \to ∞} \NORM{x}_{p} = \max_{1 \le i \le d} \ABS{x_i} \eqqcolon \NORM{x}_{∞} \] which is called the \(\ell_{∞}\) norm or the sup-norm.
A vector space \(\ALPHABET V\) with a norm \(\NORM{⋅}\) is called a normed space.
Given a normed space \((\ALPHABET V, \NORM{⋅})\), we can talk about convergence. In particular, a sequence \(\{v_n\}_{n \ge 1}\) in \(\ALPHABET V\) converges to \(v^* \in \ALPHABET V\) if \[ \lim_{n \to ∞} \NORM{ v_n - v^*} = 0. \]
Two norms \(\NORM{⋅}_{(1)}\) and \(\NORM{⋅}_{(2)}\) are equivalent if there exist positive constants \(c_1,c_2\) such that \[ c_1 \NORM{v}_{(1)} \le \NORM{v}_{(2)} \le c_2 \NORM{v}_{(1)}, \quad \forall v \in \ALPHABET V. \]

With two equivalent norms, a sequence \(\{v_n\}_{n \ge 1}\) converges with respect to one norm if and only if it converges with respect to the other. Conversely, if every sequence that converges with respect to one norm also converges with respect to the other, then the two norms are equivalent.
All norms on a finite dimensional space are equivalent.
Norms over infinite dimensional space are not equivalent. For example, let \(\ALPHABET V\) be the space of continuous functions on \([0,1]\). Define the following two norms: \[\begin{align*} \NORM{v}_p &= \left[ \int_{0}^1 \ABS{v(x)}^p dx \right]^{1/p}, \quad 1 \le p < ∞ \\ \NORM{v}_{∞} &= \sup_{0 \le x \le 1} \ABS{v(x)}. \end{align*}\]

Now consider a sequence \(\{v_n\}_{n \ge 1}\) of functions, \(v_n \in \ALPHABET V\), defined by \[ v_n(x) = (1 - nx) \IND_{[0, \frac 1n]}(x). \] It is easy to show that \[ \NORM{v_n}^p = \left[ \frac{1}{n(p+1)}\right]^{1/p} \to 0 \] However, \[ \NORM{v_n}_{∞} = 1, \quad n \ge 1. \] Thus, \(\{v_n\}_{n \ge 1}\) converge with respect to the \(\ell_p\) norm, \(p \in [1, ∞)\), but not with respect to the \(\ell_{∞}\) norm.

52.3 Banach space

Let \(\ALPHABET V\) be a normed space. A sequence \(\{v_n\}_{n \ge 1}\), \(v_n \in \ALPHABET V\), is called a Cauchy sequence if \[ \lim_{n,m \to ∞} \NORM{v_n - v_m} = 0. \]
All convergent sequences are Cauchy sequences. In finite dimensional spaces, all Cauchy sequences converge, but the reverse is not true in infinite dimensional spaces. For example, the space \(C[0,1]\) of continuous functions equipped with \(\NORM{⋅}_p\) for some \(p \in [1, ∞)\). Consider the sequence \(\{v_n\}_{n \ge 1}\) defined by \[ v_n(x) = \begin{cases} 0, & 0 \le x < \frac 12 - \frac{1}{2n} \\ nx - \frac{n-1}{2}, & \frac 12 - \frac{1}{2n} \le x \le \frac 12 + \frac{1}{2n} \\ 1, & \frac 12 + \frac{1}{2n} < x \le 1. \end{cases} \] See Figure 52.1 for an illustration. It can be shown that \(v_n \to v^*\) where \[ v^*(x) = \begin{cases} 0 & 0 \le x < \frac 12 \\ 1 & \frac 12 < x \le 1 \end{cases} \] but no matter how we define \(v^*(\frac 12)\), \(v^*(x)\) is not continuous.

viewof n = Inputs.range([1, 50], {value: 2, step: 1, label: "n:"})

points = [
  {x: 0, y: 0}, 
  {x: 0.5 - 1/(2*n), y: 0}, 
  {x: 0.5 + 1/(2*n), y: 1}, 
  {x: 1, y: 1}
]

Plot.plot({
  y: { label: "v_n(x)", domain: [-0.1, 1.1] },
  x: { label: "x", domain: [0, 1] },
  grid: true,
  marks: [
    // Draw the function line
    Plot.line(points, {x: "x", y: "y", stroke: "steelblue", strokeWidth: 3}),
    // Add axes lines for clarity
    Plot.ruleY([0]),
    Plot.ruleX([0]),
  ]
})

(a)

(b)

Figure 52.1: A Cauchy sequence of continuous functions that does not converge to a continuous function

A normed space is said to be complete if every Cauchy sequence from the space converges to an elemnt in the space. A complete normed space is called a Banach space
If a normed space is not complete, we can “complete” it. For instance, the completion of the space \(C[0,1]\) of cotinuous function under the norm \(\NORM{⋅}_p\) is the space of measurable functions with finite \(\NORM{⋅}_p\). Such a space is often denoted by \(L^p[0,1]\).

52.4 Inner Product space

An inner product space over reals is a linear vector space \(\ALPHABET H\) over reals which is equipped with an inner product function \(\IP{⋅}{⋅}\) that satisfies the following conditions: for all \(x,y, z \in \ALPHABET H\) and \(α, β \in \reals\)

\(\IP{x}{y} = \IP{y}{x}\)
\(\IP{x}{x} \ge 0\) and \(\IP{x}{x} = 0\) if and only if \(x = 0\)
\(\IP{α ⋅ x + β ⋅ y}{z} = α \IP{x}{z} + β \IP{y}{z}\).

The norm of an inner product space is a function \(\NORM{⋅} \colon \ALPHABET H \to \reals_{\ge 0}\) defined by \[ \NORM{x} = \sqrt{\IP{x}{x}}. \]

It can be shown that the norm satisfies the following properties: for any \(x, y \in \ALPHABET H\), we have

The Cauchy-Schwarz inequality: \[\ABS{\IP{x}{y}} \le \NORM{x} ⋅ \NORM{y}.\]
The triangle inequality: \[\NORM{x + y} \le \NORM{x} + \NORM{y}.\]
The parallelogram equality: \[\NORM{x + y}^2 + \NORM{x - y}^2 = 2 \NORM{x}^2 + 2 \NORM{y}^2. \]

52.5 Hilbert space

Define a distance function \(d \colon \ALPHABET H × \ALPHABET H \to \reals\) to be \(d(x,y) = \NORM{x - y}\). It is easy to verify that \(d(⋅,⋅)\) satisfies:
- \(d(x,y) = d(y,x)\)
- \(d(x,y) = 0\) if and only if \(x = y\)
- \(d(x,z) \le d(x,y) + d(y,z)\).
Thus, \(d\) is a metric and, therefore \((\ALPHABET H, d)\) is a metric space.
It can be shown that the inner product is a continuous function from \(\ALPHABET H × \ALPHABET H\) into \(\reals\), i.e., if \(x_n \to x\) and \(y_n \to y\) as \(n \to ∞\) (in the metric topology on \(\ALPHABET H\)), then \(\IP{x_n}{y_n} \to \IP{x}{y}\) (in the Eucledian topology on \(\reals\)).
Consequently, if \(x_n \to x\) then \(\NORM{x_n} \to \NORM{x}\) as \(n \to ∞\).
A sequence \(\{x_n\}_{n \ge 1}\) in a metric space is called a Cauchy sequence if for every \(ε > 0\), there exists an \(N\) such that \[ d(x_n, x_m) < ε, \quad \forall n, m \ge N. \]
A pre-Hilbert space \(\ALPHABET H\) is said to be complete if for every Cauchy sequence \(\{x_n\}_{n \ge 1}\), \(x_n \in \ALPHABET H\), there exists an \(x \in \ALPHABET H\) such that \(x_n \to x\) as \(n \to ∞\).
A Hilbert space is an inner product space \(\ALPHABET H\) that is complete with respect to the metric \(d(x,y) = \NORM{x - y}\), \(x, y \in \ALPHABET H\).

52.6 Examples of Hilbert space

The space \(\reals^n\) is a finite-dimensional Hilbert space with the usual inner product \[ \IP{x}{y} = \TR(x y^\TRANS) = \sum_{i=1}^n x_i y_i. \] The completeness follows from the completeness of each of the coordinate spaces and the fact that inner product is a continuous function.
Any finite-dimensional pre-Hilbert space \(\ALPHABET H\) is complete. This can be shown by constructing a linear isometry from \(\ALPHABET H\) to \(\reals^n\) using Gram-Schmidt orthogonalization.
The space of continuous functions on \([0,1]\), denoted by \(C[0,1]\), may be made into an inner product space by defining \[ \IP{f}{g} = \int_{0}^1 f(t) g(t) dt. \] This space is not complete.
The space of measurable functions on \([0,1]\) with finite second moment (i.e., \(\int_{0}^1 (f(t))^2 dt < ∞\)), denoted by \(L^2([0,1])\), can also be made into an inner product space by defining \[ \IP{f}{g} = \int_{0}^1 f(t) g(t) dt. \] This space is complete and hence a Hilbert space.

52.7 Orthogonal Projection Theorem

In Eucledian spaces, the shortest path from a point to a plane is along the perpendicular from the point to the plane. This is sometimes called the projection theorem. The main result in the geometry of Hilbert spaces is a generalization of this result called the Orthogonal projection theorem. We start with some notation before describing the result.

Let \(\ALPHABET H\) be a Hilbert space

Two vectors \(x, y \in \ALPHABET H\) are said to be orthogonal (denoted by \(x \perp y\)) if \(\IP{x}{y} = 0\).
Given a subset \(\ALPHABET S\) of \(\ALPHABET H\) and a vector \(x \in \ALPHABET H\), we write \(x \perp \ALPHABET S\) if \(x \perp s\) for all \(s \in \ALPHABET S\).
Given two subsets \(\ALPHABET S\) and \(\ALPHABET T\) of \(\ALPHABET H\), we write \(\ALPHABET S \perp \ALPHABET T\) if all elements of \(\ALPHABET S\) are orthogonal to all elements of \(\ALPHABET T\).
Suppose there exist subspaces \(\ALPHABET A\) and \(\ALPHABET B\) of \(\ALPHABET H\) such that \(\ALPHABET H = \ALPHABET A + \ALPHABET B\), that is, for each \(x \in \ALPHABET H\) there exist \(a \in \ALPHABET A\) and \(b \in \ALPHABET B\) such that \(x = a + b\). If \(\ALPHABET A \perp \ALPHABET B\), then we write \(\ALPHABET H = \ALPHABET A \oplus \ALPHABET B\) and say \(\ALPHABET H\) is the direct sum of \(\ALPHABET A\) and \(\ALPHABET B\).

Theorem 52.1 (The Orthogonal Projection Theorem) Let \(\ALPHABET M\) be a proper subspace of a Hilbert space \(\ALPHABET H\) and \(x\) be a point of \(\ALPHABET H\). Then \(x\) can be uniquely represented in the form \[ x = y + z \] where \(y \in \ALPHABET M\) and \(z \perp \ALPHABET M\). Furthermore, for all \(w \in \ALPHABET M\), we have \[ \NORM{x - w} \ge \NORM{x - y} \] with equality if and only if \(w = y\).

This unique element \(y\) is called the orthogonal projection of \(x\) onto \(\ALPHABET M\) and denoted by \(\LEXP[ x \mid \ALPHABET M]\).

Note that if \(\ALPHABET M\) is not a proper subspace of \(\ALPHABET H\), we can simply take \(z = 0\).

Interpretation of the orthogonal projection theorem

Theorem 52.1 shows that the orthogonal projection of \(x\) only \(\ALPHABET M\) is equal to \[ \arg \min_{w \in \ALPHABET M} \NORM{x - w}. \] Thus, it is a generalization of the projection theorem of Eucledian spaces.

52.8 Linear Operators on Banach Spaces

A function or an operator \(L\) from one linear space \(\ALPHABET V\) to another \(\ALPHABET W\) is said to linear if
1. for all \(u, v \in \ALPHABET V\), \[ L(u + v) = L(u) + L(v). \]
2. For all \(v \in \ALPHABET V\) and \(α \in \reals\), \[ L(α v) = α L(v). \]
For a linear function, we often write \(Lv\) instead of \(L(v)\).
Let \((\ALPHABET V, \NORM{⋅}_{\ALPHABET V})\) and \((\ALPHABET W, \NORM{⋅}_{\ALPHABET W}\) be Banach spaces. Then, \(L \colon \ALPHABET V \to \ALPHABET W\) be a linear operator. The operator-norm of \(L\) is defined as \[ \NORM{L}_{\ALPHABET V,\ALPHABET W} = \sup_{0 \neq v \in V} \frac{\NORM{L v}_W}{\NORM{v}_V}. \]
The space \(\ALPHABET L(\ALPHABET V, \ALPHABET W)\) of linear operators from Banach space \(\ALPHABET V\) to Banach space \(\ALPHABET W\) is a linear space. The operator norm \(\NORM{⋅}_{\ALPHABET V,\ALPHABET W}\) is a norm on this space.
We will use \(\ALPHABET L(\ALPHABET V)\) to denote \(\ALPHABET L(\ALPHABET V,\ALPHABET V)\) and use \(\NORM{⋅}\) to denote the operator norm \(\NORM{⋅}_{\ALPHABET V, \ALPHABET V}\).
Let \(\ALPHABET V\) be a Banach space and \(L \in \ALPHABET L(V)\) such that \[ \NORM{L} < 1. \] Then, the Neumann series converges \[ \sum_{n=0}^∞ L^n = (I - L)^{-1} \] and \[ \NORM{(I - L)^{-1}} \le \frac{1}{1 - \NORM{L}}. \]

Exercises

Exercise 52.1 Let \(\ALPHABET H_0\) be a Hilbert space, \(\ALPHABET H_1\) be a Hilbert subspace of \(\ALPHABET H_0\), and \(\ALPHABET H_2\) be a Hilbert subspace of \(\ALPHABET H_1\). Let \(x,y,z \in \ALPHABET H\) be such that \[ y = \LEXP[ x \mid \ALPHABET H_1 ] \quad\text{and}\quad z = \LEXP[ y \mid \ALPHABET H_2 ] \] Show that \[ z = \LEXP[ x \mid \ALPHABET H_2]. \]

Notes

The material presented in this section is standard. The order of presentation and the examples are adapted from Caines (2018).