Isometries and Orthogonal Transformations

Introduction

An isometry of a quadratic space is a linear isomorphism that preserves the quadratic form. The isometries of a fixed form organise themselves into the orthogonal group, and the reflections — the isometries that fix a hyperplane pointwise — generate it. This article develops that circle of ideas: the orthogonal group and the similarity group, the reflection in a non-isotropic vector, the equal-norm lemma, the theorem of Cartan–Dieudonné that every isometry is a product of reflections, and the special orthogonal group of determinant one.

The base is a field $F$ of characteristic not $2$ unless stated otherwise, and the space $V$ is finite-dimensional, so that every isometry is a linear automorphism with an inverse and a determinant. The vocabulary of quadratic forms is that of Quadratic Forms and Polarisation: a quadratic form $q$, its polar form $B$ with $q(v) = B(v, v)$, the radical, and non-degeneracy. The orientation of a space, the determinant map as a map of groups, the notion of angle, and the rotation groups of the plane and of space are not covered here; here the determinant is used only to define the special orthogonal group and to count reflections. The orthogonal Lie algebra is treated, and the corresponding matrix groups.

Isometries

Definition

Let $V$ be a finite-dimensional $F$-space with quadratic form $q$ and polar form $B$. A linear isometry of $q$ is an $F$-linear isomorphism $T : V \to V$ such that

$$ q(Tv) = q(v) \qquad (v \in V). $$

When $2$ is invertible, this is equivalent to

$$ B(Tu, Tv) = B(u, v) \qquad (u, v \in V). $$

Proposition. Let $2$ be invertible. Then $T$ preserves $q$ if and only if $T$ preserves $B$.

Proof. If $q(Tv) = q(v)$ for all $v$, then for all $u, v$,

$$ 2B(Tu, Tv) = q(Tu + Tv) - q(Tu) - q(Tv) = q\bigl(T(u + v)\bigr) - q(Tu) - q(Tv) = q(u + v) - q(u) - q(v) = 2B(u, v), $$

and invertibility of $2$ gives $B(Tu, Tv) = B(u, v)$. The converse is the special case of the identity $q(v) = B(v, v)$.

The Orthogonal Group

Definition. The set of all linear isometries of $(V, q)$ is the orthogonal group of the form,

$$ \operatorname{O}(V, q) = \{T \in GL(V) : q(Tv) = q(v)\ \text{for all } v \in V\}. $$

It is a subgroup of the general linear group $GL(V)$: the identity preserves $q$; if $S$ and $T$ preserve $q$ then so does $ST$, because $q(STv) = q(Tv) = q(v)$; and if $T$ preserves $q$ then so does $T^{-1}$, because $q(v) = q(T(T^{-1}v)) = q(T^{-1}v)$.

Proposition. The orthogonal group depends only on the isometry class of $q$. If $T : (V, q) \to (V', q')$ is an isometry, then conjugation by $T$ is an isomorphism $\operatorname{O}(V, q) \to \operatorname{O}(V', q')$.

Proof. For $S \in \operatorname{O}(V, q)$ the conjugate $TST^{-1}$ preserves $q'$, since $q'(TST^{-1}v') = q(ST^{-1}v') = q(T^{-1}v') = q'(v')$, using $q' \circ T = q$; the assignment is a group homomorphism with inverse given by conjugation by $T^{-1}$.

Remark (the ring case). Over a commutative ring the same definitions make sense, and $\operatorname{O}(V, B)$ is the subgroup of $GL(V)$ preserving $B$, where $GL(V)$ consists of the endomorphisms whose determinant is a unit. If the module is not free, the isometry group is not a matrix group. The proofs above use only bijectivity, not invertibility of a determinant, so they carry over unchanged.

Similarities

Definition. A similarity of $(V, q)$ is an $F$-linear isomorphism $T$ for which there is a constant $c \in F^\times$, the multiplier, with

$$ q(Tv) = c\,q(v) \qquad (v \in V). $$

The similarities form the similarity group $GO(V, q)$, and $T \mapsto c$ is a homomorphism $GO(V, q) \to F^\times$ with kernel $\operatorname{O}(V, q)$.

The multiplier is unique when $q \neq 0$, by evaluating on a vector with $q(v) \neq 0$. Equivalently, a similarity satisfies $B(Tu, Tv) = c\,B(u, v)$ when $2$ is invertible. Taking determinants of $T^T G T = c\,G$ gives

$$ (\det T)^2 = c^{\,n}, \qquad n = \dim V, $$

for a non-degenerate form. In particular an isometry ($c = 1$) has $(\det T)^2 = 1$, so over a field of characteristic not $2$ every isometry has determinant $+1$ or $-1$.

Example. On $\mathbb{R}^2$ with $q(x, y) = x^2 + y^2$, the map $T = \lambda\,\mathrm{id}$ is a similarity with multiplier $c = \lambda^2$, and its determinant is $\lambda^2$, so $(\det T)^2 = \lambda^4 = c^2 = c^n$ with $n = 2$.

The Multiplier and the Discriminant

The determinant identity of the previous paragraph is the first of two constraints on the multiplier, the second coming from the discriminant; together they show that similarities are rare over an odd-dimensional space.

Proposition. Let $q$ be non-degenerate on $V$ of dimension $n$ and let $T$ be a similarity with multiplier $c$. Then

$$ (\det T)^2 = c^{\,n}, \qquad c^{\,n} \in (F^\times)^2 . $$

If $n$ is odd, $c$ is a square in $F$. If $n$ is even the second condition is automatic.

Proof. The determinant identity was computed above. For the divisibility of the discriminant, $T$ is an isometry from $(V, q)$ to $(V, cq)$, so $cq \cong q$; the discriminant is multiplicative in the scaling, $\Delta(cq) = c^{\,n}\Delta(q)$, and is an isometry invariant, so $c^{\,n}\Delta(q) = \Delta(q)$ and hence $c^{\,n}$ is a square because $\Delta(q) \neq 0$ for a non-degenerate form. The group $F^\times/(F^\times)^2$ has exponent two, so its element $[c]$ satisfies $[c]^{n} = [c]$ for odd $n$; from $[c^{n}] = 1$ we get $[c] = 1$, that is $c$ is a square. For even $n$ the element $c^{n} = (c^{n/2})^2$ is a square without further hypothesis.

So the multipliers form a subgroup $\mu \subseteq F^\times$ containing the squares, and there is an exact sequence

$$ 1 \longrightarrow \operatorname{O}(V, q) \longrightarrow GO(V, q) \xrightarrow{\ c\ } \mu \longrightarrow 1, $$

in which the middle arrow is $T \mapsto c$. For odd $n$ the group $\mu$ consists of squares, so a similarity of an odd-dimensional space has a square multiplier; for even $n$ the discriminant imposes no condition at all, and the multiplier need not be a square.

Example. On $\mathbb{Q}^3$ with $q = x^2 + y^2 + z^2$ the identity $(\det T)^2 = c^3$ forces $c^3$ to be a square; since $3$ is odd this forces $c$ to be a square, so $c = 2$ is not a multiplier and no similarity of $q$ has multiplier $2$. On $\mathbb{Q}^2$ with $q = x^2 + y^2$ the discriminant is no obstruction and non-squares do occur as multipliers: the matrix $T = \begin{pmatrix} 1 & 1 \\ -1 & 1\end{pmatrix}$ satisfies $T^{T}T = 2I$, hence $q(Tv) = 2q(v)$, so $c = 2$ is a multiplier. The multiplier $c = 3$ is not, because a similarity of $q$ satisfies $T^{T}T = c\,I$ and therefore has $c = a^2 + b^2$ a sum of two squares.

Reflections

The Reflection in a Non-Isotropic Vector

Definition. Let $v \in V$ with $q(v) \neq 0$. The reflection in $v$ is the linear map

$$ \tau_v(x) = x - \frac{2B(x, v)}{q(v)}\,v. $$

The vector $v$ is non-isotropic, and the reflection is well defined because $q(v) \neq 0$.

Proposition. The map $\tau_v$ is an isometry with $\tau_v(v) = -v$, $\tau_v(x) = x$ whenever $B(x, v) = 0$, and $\tau_v^2 = \mathrm{id}$. Its fixed subspace is the hyperplane $v^\perp$, and its determinant is $-1$.

Proof. The vector $v$ has $B(v, v) = q(v)$, so $\tau_v(v) = v - \frac{2q(v)}{q(v)}v = -v$. If $B(x, v) = 0$ then $\tau_v(x) = x$, so $\tau_v$ fixes $v^\perp$ pointwise and acts as $-1$ on the complementary line $Fv$; hence $\tau_v^2 = \mathrm{id}$ and $\tau_v$ has the eigenvalue $-1$ once and $+1$ on $v^\perp$, so $\det \tau_v = -1$. For the isometry property, expand using the polar form:

$$ q(\tau_v x) = q\Bigl(x - \frac{2B(x, v)}{q(v)}v\Bigr) = q(x) - \frac{4B(x, v)}{q(v)}\,B(x, v) + \frac{4B(x, v)^2}{q(v)^2}\,q(v) = q(x). $$

Definition. A hyperplane reflection is a reflection $\tau_v$; its fixed hyperplane is $v^\perp$ and it negates the line $Fv$.

Example (the Euclidean case). For $V = \mathbb{R}^n$ with $q(x) = \sum_i x_i^2$ and $v$ a unit vector, $\tau_v(x) = x - 2\langle x, v\rangle v$ is the usual mirror reflection in the hyperplane perpendicular to $v$. For $n = 2$ and $v = (1, 0)$ it is $\tau(x, y) = (-x, y)$.

Example (the hyperbolic case). Let $V = F^2$ with $q(x, y) = x^2 - y^2$ and $v = (1, 0)$. Then $q(v) = 1$ and the reflection in $v$ is $\tau_v(x, y) = (-x, y)$; it negates the line $\{(t, 0)\}$ and interchanges the two isotropic lines $\{(t, t)\}$ and $\{(t, -t)\}$, so it does not preserve an isotropic line.

The Equal-Norm Lemma

The following lemma is the combinatorial heart of the generation theorem. It is also the one-dimensional case of the extension phenomenon treated; it is proved here directly by reflections, so that its use there does not presuppose the general theorem.

Lemma (equal norms). Let $(V, q)$ be a non-degenerate quadratic space over a field of characteristic not $2$, and let $x, y \in V$ satisfy $q(x) = q(y) \neq 0$. Then there is an isometry carrying $x$ to $y$, and it is a product of at most two reflections.

Proof. If $q(x - y) \neq 0$, the reflection $\tau_{x-y}$ sends $x$ to $y$: indeed

$$ \tau_{x-y}(x) = x - \frac{2B(x, x - y)}{q(x - y)}(x - y), $$

and $2B(x, x - y) = 2q(x) - 2B(x, y) = q(x - y)$ because $q(x) = q(y)$, so the coefficient is $1$ and $\tau_{x-y}(x) = x - (x - y) = y$. If $q(x - y) = 0$, then $q(x + y) = 2q(x) + 2q(y) - q(x - y) = 4q(x) \neq 0$, and the same computation with $x + y$ gives $\tau_{x+y}(x) = -y$; since $\tau_y(y) = -y$, the composition $\tau_y \circ \tau_{x+y}$ carries $x$ to $y$. In both cases at most two reflections suffice.

The Cartan–Dieudonné Theorem

Statement

Theorem (Cartan–Dieudonné). Let $(V, q)$ be a non-degenerate quadratic space of dimension $n$ over a field of characteristic not $2$. Then every isometry of $V$ is a product of at most $n$ hyperplane reflections.

The bound $n$ is sharp: the isometry $-\mathrm{id}$ is not a product of fewer than $n$ reflections, because a product of $k$ reflections fixes the intersection of their fixed hyperplanes, a subspace of dimension at least $n - k$, and $-\mathrm{id}$ fixes only $0$. The identity requires no reflection at all, and for other isometries the count can be smaller than the bound.

Proof

The theorem is standard; we give the inductive reduction on which its proof rests. The key point is that an isometry fixing a non-isotropic vector preserves that vector's orthogonal complement, which is again a non-degenerate quadratic space of dimension one less.

Proof sketch. Let $\dim V = n$ and argue by induction on $n$. Let $T \in \operatorname{O}(V, q)$. If $T = \mathrm{id}$ there is nothing to prove. Otherwise choose $x$ with $Tx \neq x$. Suppose first that $x$ is non-isotropic and $q(Tx - x) \neq 0$; by the computation in the proof of the equal-norm lemma, the reflection $\tau_{Tx-x}$ sends $Tx$ to $x$, so $\tau_{Tx-x}T$ fixes the non-isotropic vector $x$. Since $B(x, Sw) = B(Sx, Sw) = B(x, w) = 0$ for $w \in x^\perp$ whenever $S$ fixes $x$, such an $S$ preserves the non-degenerate hyperplane $x^\perp$ and restricts to an isometry there; by induction the restriction is a product of at most $n - 1$ reflections of $x^\perp$, each extended by the identity on $Fx$ to a reflection of $V$. Together with the single reflection $\tau_{Tx-x}$ this expresses $T$ as a product of at most $n$ reflections.

When $q(Tx - x) = 0$ for the chosen non-isotropic $x$, one looks for another non-isotropic vector: if $q(Ty - y) \neq 0$ for some non-isotropic $y$, the argument above applies with $y$ in place of $x$ and the bound holds again. The case then left for separate analysis, and the one in which the counting of reflections is delicate, is the case in which $q(Ty - y) = 0$ for every non-isotropic $y$. Its extreme subcase, in which the vanishing holds for every $y \in V$, is analysed as follows. Then $N = T - \mathrm{id}$ satisfies $q(Ny) = 0$ for all $y$; polarising gives $B(Nu, Nv) = 0$ for all $u, v$, and expanding $B(Tu, Tv) = B(u, v)$ with $T = \mathrm{id} + N$ gives $B(Nu, v) + B(u, Nv) + B(Nu, Nv) = 0$, so that $B(Nu, v) = -B(u, Nv)$ and

$$ B(N^2u, v) = B\bigl(N(Nu), v\bigr) = -B(Nu, Nv) = 0 \qquad (u, v \in V), $$

whence $N^2 = 0$ because the form is non-degenerate. If $N \neq 0$ then $U = \operatorname{Im}N$ is a nonzero totally isotropic subspace, because $q(Ny) = 0$ for every $y$, and skewness of $N$ gives $(\operatorname{Im}N)^\perp = \ker N$. A product $\tau_a\tau_b$ of two reflections fixes $(Fa + Fb)^\perp$ pointwise and preserves the plane $Fa + Fb$; in the subcase at hand the product is unipotent, and the restriction of a unipotent endomorphism to an invariant subspace is unipotent, so the restriction to that plane is unipotent; a unipotent endomorphism of a two-dimensional space has minimal polynomial dividing $(x - 1)^2$, so $(T - \mathrm{id})^2 = 0$ on that plane and the rank there is at most one, so $\operatorname{rank}(T - \mathrm{id}) \leq 1$ for such a product. Hence a unipotent $T$ with $\operatorname{rank}(T - \mathrm{id}) \geq 2$ is not a product of two reflections, and since $\det T = 1$ and $T \neq \mathrm{id}$ it is not a product of fewer than four. If the form is anisotropic then $q$ vanishes only at $0$, so $N = 0$ and $T = \mathrm{id}$, which the hypothesis $T \neq \mathrm{id}$ excludes. The counting of reflections for the unipotent isometries of this subcase, including those with $\operatorname{rank}(T - \mathrm{id}) = 1$, is the delicate step of the theorem and is carried out in the standard sources, where it is shown to stay within the bound $n$. The intermediate case, in which $q(Ty - y) = 0$ for every non-isotropic $y$ but $q(Tx - x) \neq 0$ for some isotropic $x$, is reduced in the standard sources to the subcase just analysed by extending the isotropic $x$ to a hyperbolic pair and replacing $x$ by a non-isotropic vector of the hyperbolic plane it spans. The reader is referred to those sources for the detailed case analysis; the statement and the bound $n$ are exactly as cited there.

Remark. The theorem is a statement about generation by reflections; it does not claim that the number of reflections is an invariant of the isometry, only that it can be chosen at most $n$. The parity of the number is an invariant, and it is the subject of the next section.

The Degenerate Case

If $q$ is degenerate the situation changes: a reflection in a non-isotropic vector $v$ acts trivially on the radical, since $B(r, v) = 0$ for $r \in \operatorname{rad}(q)$, and hence so does every product of reflections, while a general isometry of a degenerate space may act nontrivially on the radical. What survives is the reduction to the non-degenerate quotient.

Proposition. An isometry $T$ of $(V, q)$ preserves $\operatorname{rad}(q)$ and induces an isometry $\bar{T}$ of the reduced space $(V/\operatorname{rad}(q), \bar{q})$, and every reflection of the reduced space is induced by a reflection of $V$. Hence $T$ is a product of at most $\dim V - \dim\operatorname{rad}(q)$ reflections of $V$ followed by an isometry inducing the identity on $V/\operatorname{rad}(q)$.

Proof. For $r \in \operatorname{rad}(q)$ and $x \in V$ the equality $q(T(x + r)) = q(x + r)$ reads, after cancelling $q(Tx) = q(x)$, $B(x, r) = 0$ and $q(r) = 0$,

$$ q(Tr) + 2B(Tx, Tr) = 0. $$

Taking $x = 0$ gives $q(Tr) = 0$, and then $2B(Tx, Tr) = 0$ for every $x$, so $Tr \in \operatorname{rad}(q)$ because $T$ is surjective and $2$ is invertible. Hence $T$ preserves the radical and descends to the quotient. A vector $\bar v$ with $\bar{q}(\bar v) \neq 0$ has a lift $v$ with $q(v) = \bar{q}(\bar v)$, because $q$ is constant on the cosets of the radical, and the reflection $\tau_v$ then induces the reflection $\tau_{\bar v}$ of the quotient, since $\bar{B}(\bar x, \bar v) = B(x, v)$ and $\bar{q}(\bar v) = q(v)$. The reduced form is non-degenerate, so Cartan–Dieudonné applies to $\bar{T}$, and lifting its factors writes $T$ as a product of reflections times an isometry inducing the identity on the quotient.

We state the non-degenerate case of Cartan–Dieudonné only; the refined structure of the isometries that induce the identity on $V/\operatorname{rad}(q)$ is part of Dieudonné's theory of the isometry group of a possibly degenerate space, and the Hermitian analogue is recorded.

The Special Orthogonal Group

Definition

Definition. The special orthogonal group of a non-degenerate quadratic space $(V, q)$ is the kernel of the determinant,

$$ \operatorname{SO}(V, q) = \{T \in \operatorname{O}(V, q) : \det T = 1\}. $$

It is a normal subgroup of $\operatorname{O}(V, q)$, because the determinant is a homomorphism.

Theorem. Let $(V, q)$ be non-degenerate over a field $F$ of characteristic not $2$. Then

$$ \operatorname{O}(V, q) / \operatorname{SO}(V, q) \cong \{+1, -1\} \cong \mathbb{Z}/2\mathbb{Z}, $$

provided the determinant assumes the value $-1$ on $\operatorname{O}(V, q)$; this holds whenever there is a hyperplane reflection, in particular for $n \geq 1$. The determinant map is not surjective onto $\{+1,-1\}$ over a general commutative ring, where $(\det T)^2 = 1$ may have more than two solutions.

Proof. Two isometries have the same determinant if and only if one is the other composed with an element of $\operatorname{SO}(V, q)$, so the quotient embeds in the group of units of $F$ cut out by $d^2 = 1$; over a field of characteristic not $2$ this group is $\{+1, -1\}$. The reflection $\tau_v$ has determinant $-1$, so the image is all of $\{+1,-1\}$.

The Parity of the Number of Reflections

Corollary. Let $T$ be an isometry of a non-degenerate quadratic space over a field of characteristic not $2$, written as a product of reflections. Then the parity of the number of reflections is $\det T$, so it is well defined: it is $+1$ for an even product and $-1$ for an odd one. In particular the isometries that are products of an even number of reflections are exactly the rotations $\operatorname{SO}(V, q)$.

Proof. Each reflection has determinant $-1$ and the determinant is multiplicative, so a product of $k$ reflections has determinant $(-1)^k$.

The corollary is the algebraic meaning of the distinction between a rotation (even reflection length, determinant $+1$) and a reflection-type isometry (odd reflection length, determinant $-1$). The geometric reading of this parity, and its dependence on the orientation of the space, is developed.

Example (the plane). On $\mathbb{R}^2$ with the standard form, $\tau_v$ for $v = (1,0)$ is $\tau(x, y) = (-x, y)$, and for $v = (0,1)$ it is $\tau(x, y) = (x, -y)$. Their product is $\tau_{(1,0)}\tau_{(0,1)}(x, y) = (-x, -y)$, the rotation through $\pi$, which is a product of two reflections and lies in $\operatorname{SO}(2)$. Every rotation of $\mathbb{R}^2$ is a product of two reflections, and every element of $\operatorname{O}(2)$ is a product of at most two.

Example (three reflections in the plane). On $\mathbb{R}^2$ a product of three reflections is a single reflection, because the determinant is $(-1)^3 = -1$ and the odd elements of $\operatorname{O}(2)$ are exactly the reflections; this is the finite version of the parity statement and shows that the bound of the Cartan–Dieudonné theorem is not the length in general.

The Orthogonal Group in Low Dimensions

The generation of the isometries by reflections can be made completely explicit in dimension one and two, and the two plane cases with distinct behaviour are the definite plane and the hyperbolic plane.

The Line

Let $\dim V = 1$ and $q(e) = a \neq 0$. An isometry satisfies $q(Te) = q(e)$, so if $Te = \lambda e$ then $\lambda^2 = 1$, and over a field of characteristic not $2$ this gives $\lambda = \pm 1$:

$$ \operatorname{O}(V, q) = \{\pm \mathrm{id}\} \cong \mathbb{Z}/2\mathbb{Z}, \qquad \operatorname{SO}(V, q) = \{\mathrm{id}\}. $$

The reflection $\tau_e$ is $-\mathrm{id}$, so the group is generated by reflections, and the bound of Cartan–Dieudonné is attained.

The Definite Plane

Let $V = \mathbb{R}^2$ with $q(x, y) = x^2 + y^2$. The columns of a matrix in $\operatorname{O}(2)$ are orthonormal, so for some $\theta$

$$ R(\theta) = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}, \qquad S(\theta) = \begin{pmatrix} \cos\theta & \sin\theta \\ \sin\theta & -\cos\theta \end{pmatrix}, $$

and

$$ \operatorname{SO}(2) = \{R(\theta) : \theta \in \mathbb{R}\}, \qquad \operatorname{O}(2) = \operatorname{SO}(2) \sqcup \operatorname{SO}(2)\cdot\tau_{(1,0)}. $$

Indeed $\det R(\theta) = 1$ and $\det S(\theta) = -1$, and $S(\theta) = R(\theta + \pi)\tau_{(1,0)}$; the second family consists of the reflections, $S(\theta)$ being the reflection in the line at angle $\theta/2$. So $\operatorname{O}(2)$ has two components, each a circle, and every element is a product of at most two reflections: a reflection in one, a rotation in two, by the plane proposition above.

The Hyperbolic Plane

Let $F$ have characteristic not $2$ and let $q(x, y) = x^2 - y^2$ on $F^2$, with Gram matrix $G = \operatorname{diag}(1, -1)$.

Proposition. Every $T \in \operatorname{O}(V, q)$ is of the form

$$ T = \begin{pmatrix} p & \eta r \\ r & \eta p \end{pmatrix}, \qquad p^2 - r^2 = 1, \quad \eta = \pm 1, $$

with $\det T = \eta$; conversely every such matrix lies in $\operatorname{O}(V, q)$.

Proof. Write $T = \begin{pmatrix} p & q' \\ r & t\end{pmatrix}$ and impose $T^{T} G T = G$, which is the three conditions

$$ p^2 - r^2 = 1, \qquad t^2 - q'^2 = 1, \qquad pq' = rt. $$

From $p^2 - r^2 = (p + r)(p - r) = 1$ the elements $\lambda = p - r$ and $\lambda^{-1} = p + r$ are nonzero, so $p = \tfrac{1}{2}(\lambda + \lambda^{-1})$ and $r = \tfrac{1}{2}(\lambda - \lambda^{-1})$; likewise, with $\mu = t - q'$, one has $t = \tfrac{1}{2}(\mu + \mu^{-1})$ and $q' = \tfrac{1}{2}(\mu - \mu^{-1})$. Substituting in $pq' = rt$ and cancelling the factor $\tfrac{1}{4}$ gives

$$ (\lambda + \lambda^{-1})(\mu - \mu^{-1}) = (\lambda - \lambda^{-1})(\mu + \mu^{-1}), $$

that is $2\lambda^{-1}\mu = 2\lambda\mu^{-1}$, hence $\mu^2 = \lambda^2$ and $\mu = \eta\lambda$ with $\eta = \pm 1$. Therefore $q' = \eta r$ and $t = \eta p$, which is the displayed form, and $\det T = pt - q'r = \eta(p^2 - r^2) = \eta$. Conversely, substituting the displayed form in the three conditions gives $p^2 - r^2 = 1$, $(\eta p)^2 - (\eta r)^2 = p^2 - r^2 = 1$ and $p\eta r = r\eta p$.

Over $\mathbb{R}$ the equation $p^2 - r^2 = 1$ is a hyperbola with two branches, $p \geq 1$ and $p \leq -1$, so $\operatorname{O}(1, 1)$ has four components, each homeomorphic to a line; the group is not connected and neither is $\operatorname{SO}(1, 1)$, which is the union of the two components with $\eta = 1$. This is the structural difference from the definite plane: for a definite form, and only then, the determinant detects the components.

Example (reflections and hyperbolic rotations). With $q = x^2 - y^2$ the reflections in the two basis vectors are

$$ \tau_{(1,0)} = \begin{pmatrix} -1 & 0 \\ 0 & 1\end{pmatrix}, \qquad \tau_{(0,1)} = \begin{pmatrix} 1 & 0 \\ 0 & -1\end{pmatrix}, $$

with $p = -1, r = 0, \eta = -1$ and $p = 1, r = 0, \eta = -1$; their product is $-\mathrm{id}$ with $p = -1$, $r = 0$, $\eta = 1$. The elements with $\eta = 1$ and $p > 0$ are the hyperbolic rotations

$$ H(u) = \begin{pmatrix} \cosh u & \sinh u \\ \sinh u & \cosh u\end{pmatrix}, \qquad H(u)H(v) = H(u + v), $$

which fix no nonzero vector when $u \neq 0$ and preserve the two isotropic lines spanned by $(1, 1)$ and $(1, -1)$; they form a subgroup isomorphic to $\mathbb{R}$, and $\operatorname{SO}(1, 1) \cong \mathbb{Z}/2\mathbb{Z} \times \mathbb{R}$ with the second factor the hyperbolic rotations and the first generated by $-\mathrm{id}$.

Proof of the group law. Multiplying the two matrices and using $\cosh(u + v) = \cosh u\cosh v + \sinh u\sinh v$ and $\sinh(u+v) = \sinh u\cosh v + \cosh u\sinh v$ gives $H(u+v)$; the inverse of $H(u)$ is $H(-u)$, so the map $u \mapsto H(u)$ is a homomorphism from $\mathbb{R}$ with kernel $\{0\}$.

Summary

A linear isometry of a quadratic space $(V, q)$ is a linear isomorphism $T$ with $q(Tv) = q(v)$ for all $v$, equivalently $B(Tu, Tv) = B(u, v)$ when $2$ is invertible. The isometries form the orthogonal group $\operatorname{O}(V, q)$, a subgroup of $GL(V)$; it depends only on the isometry class of $q$. A similarity is an isomorphism with $q(Tv) = c\,q(v)$ for a unique multiplier $c \in F^\times$ when $q \neq 0$; the similarities form $GO(V, q)$ with kernel $\operatorname{O}(V, q)$, and for a non-degenerate form they satisfy $(\det T)^2 = c^{\,n}$.

For a non-isotropic vector $v$, the reflection $\tau_v(x) = x - \frac{2B(x, v)}{q(v)}v$ is an isometry with $\tau_v(v) = -v$, fixed hyperplane $v^\perp$, $\tau_v^2 = \mathrm{id}$, and $\det \tau_v = -1$. The equal-norm lemma says that vectors $x, y$ with $q(x) = q(y) \neq 0$ are related by at most two reflections. Cartan–Dieudonné's theorem states that over a field of characteristic not $2$ and for a non-degenerate form, every isometry is a product of at most $n = \dim V$ reflections.

The special orthogonal group $\operatorname{SO}(V, q)$ is the group of isometries of determinant $+1$. It is normal in $\operatorname{O}(V, q)$, and over a field of characteristic not $2$ the quotient is $\mathbb{Z}/2\mathbb{Z}$ whenever a hyperplane reflection exists. The determinant of a product of $k$ reflections is $(-1)^k$, so the parity of the reflection length of an isometry is well defined and equals its determinant; the even products are exactly the elements of $\operatorname{SO}(V, q)$. Over a general commutative ring the determinant may take values in a larger group of solutions of $d^2 = 1$, and the quotient can be larger than $\mathbb{Z}/2\mathbb{Z}$.

A similarity with multiplier $c$ satisfies $(\det T)^2 = c^{\,n}$ and has $c^{\,n}$ a square, so for odd $n$ the multiplier is a square; the multipliers form a subgroup $\mu$ with $\operatorname{O}(V, q)$ as the kernel of $GO(V, q) \to \mu$.

In low dimensions the groups are completely explicit. On a line $\operatorname{O}(V, q) = \{\pm\mathrm{id}\}$. For the definite plane, $\operatorname{SO}(2)$ consists of the matrices $R(\theta)$ and the second component consists of the reflections $S(\theta) = R(\theta + \pi)\tau_{(1,0)}$. For the hyperbolic plane with $q = x^2 - y^2$, every isometry is $\begin{pmatrix} p & \eta r \\ r & \eta p\end{pmatrix}$ with $p^2 - r^2 = 1$ and $\eta = \pm 1$, of determinant $\eta$; over $\mathbb{R}$ the group $\operatorname{O}(1, 1)$ has four components and $\operatorname{SO}(1, 1) \cong \mathbb{Z}/2\mathbb{Z} \times \mathbb{R}$, with the hyperbolic rotations $H(u)$ as the identity component. Unlike the definite case, the determinant does not detect the components here.

Summary of Notation

Further Reading

• Jean Dieudonné, La géométrie des groupes classiques (Springer, 1971), for the Cartan–Dieudonné theorem and its degenerate and Hermitian analogues.
• Michael Artin, Algebra (Prentice Hall, 1991), for reflections and the generation of the orthogonal group.
• Serge Lang, Algebra (Springer, 3rd ed. 2002), for the orthogonal and similarity groups over fields.
• Larry C. Grove, Classical Groups and Geometric Algebra, Graduate Studies in Mathematics 39 (American Mathematical Society, 2002), for the reflection-length development of the classical groups.
• Winfried Scharlau, Quadratic and Hermitian Forms, Grundlehren der mathematischen Wissenschaften 270 (Springer, 1985), for the structure theory of the orthogonal group.