The Fixed-Point Subgroup of a Continuous Involution

Introduction

A continuous involution of a topological group has a fixed-point set, and the first question about it is whether it is merely closed or genuinely a subgroup. Closedness is free: the fixed set is the equalizer of two continuous maps and a Hausdorff group receives it as a closed set. Being a subgroup is a real condition, and the condition is commutativity — the fixed points multiply to a point of the fixed set exactly when the two factors commute, so the fixed set is a subgroup exactly when it is abelian. This article proves that criterion, identifies the subgroup that the involution always carries — the fixed subgroup of the associated involutive automorphism, which is the inverted set of the involution — and describes the quotient by it and the structure that the quotient inherits.

The article assumes the continuous involution, the associated involutive automorphism $\alpha = \sigma\iota$, the fixed and inverted sets, the dictionary $G^\sigma = I(\alpha)$, $I(\sigma) = G^\alpha$, and the closedness of both sets from Involutive Topological Groups; the abstract involutions, the fixed and inverted sets, the quotient behaviour and the split extension from Involutive Groups; and the local base, the homogeneity, the quotient topology and the compactness theorems from Topological Groups. The homogeneous space $G/G^\sigma$ as a symmetric space, with its geometry, is Part IV and is named once; the invariant integral is Part III and is not used. Nothing analytic and nothing geometric is proved.

Throughout, $G$ is a Hausdorff topological group with identity $e$, $\iota$ is the inversion, $\sigma$ is a continuous involution, $\alpha = \sigma\iota$ is the associated continuous involutive automorphism, $G^\sigma = \{g : \sigma(g) = g\}$ is the fixed-point set, $I(\sigma) = \{g : \sigma(g) = g^{-1}\}$ is the inverted set, and by the dictionary $G^\sigma = I(\alpha)$ and $I(\sigma) = G^\alpha$.

The Fixed-Point Set

Definition. The fixed-point set of the continuous involution $\sigma$ is

$$ G^\sigma = \{g \in G : \sigma(g) = g\} . $$

Theorem (the elementary properties). The set $G^\sigma$ contains $e$, it is closed under inversion, it is closed in $G$, and it is inverted elementwise by the associated automorphism: $\alpha(g) = g^{-1}$ for every $g \in G^\sigma$.

Proof. $\sigma(e) = e$ because an anti-automorphism preserves the identity; $\sigma(g^{-1}) = \sigma(g)^{-1} = g^{-1}$ for $g \in G^\sigma$; the set is the equalizer of the continuous maps $\sigma$ and $\mathrm{id}$, hence closed in the Hausdorff group. Finally $\alpha(g) = \sigma(\iota(g)) = \sigma(g^{-1}) = \sigma(g)^{-1} = g^{-1}$ for $g \in G^\sigma$, which is the identity $G^\sigma = I(\alpha)$ of the dictionary.

Proposition (centraliser of the inverted subgroup). The fixed-point set contains the two-torsion subgroup of any abelian subgroup that it contains; more precisely, if $A \subseteq G^\sigma$ is abelian then every element of $A$ is fixed by $\sigma$, and the subgroup generated by $A$ is contained in $G^\sigma$. In particular $G^\sigma$ is a subgroup exactly when it is abelian.

Proof. The subgroup generated by an abelian subset of $G^\sigma$ consists of products of fixed points, and such a product is fixed by the antimultiplicativity of $\sigma$ precisely when the factors commute; this is the criterion proved in the next section.

Example (the inversion). For $\sigma = \iota$ the fixed-point set is the set of elements of order at most two,

$$ G^\iota = \{g : g^2 = e\} . $$

If $G$ is abelian this is a subgroup; if $G$ is not abelian it need not be, and $S_3$ is the smallest witness: it contains the identity and three transpositions, and the product of two distinct transpositions is a three-cycle, which is not fixed by the inversion.

When the Fixed Set is a Subgroup

Theorem (the criterion). The fixed-point set $G^\sigma$ is a subgroup of $G$ if and only if it is abelian, and the two conditions are equivalent to the commutation of every pair of fixed points.

Proof. Suppose $G^\sigma$ is closed under the product and let $x, y \in G^\sigma$. Then $\sigma(xy) = \sigma(y)\sigma(x) = yx$, while $xy \in G^\sigma$ gives $\sigma(xy) = xy$; hence $xy = yx$ and $G^\sigma$ is abelian. Conversely, if $G^\sigma$ is abelian and $x, y \in G^\sigma$ then $\sigma(xy) = \sigma(y)\sigma(x) = yx = xy$, so $xy \in G^\sigma$; the set already contains $e$ and is closed under inversion, so it is a subgroup.

Corollary. The fixed-point set is a subgroup whenever the involution is the inversion of an abelian group, whenever $G$ has exponent two, and whenever $\sigma$ is the identity; it fails to be a subgroup for the inversion of any nonabelian group containing two noncommuting elements of order two, and for the word-reversal involution of a nonabelian free group, whose fixed set generates the group.

Proof. Each positive case makes $G^\sigma$ abelian by inspection. For the failure, two noncommuting elements of order two give a product not fixed by the inversion; the free group case is the abstract statement of Involutive Groups.

Proposition (stability of the associated automorphism on fixed points). If $G^\sigma$ is a subgroup then it is stable under $\alpha$, it is equal to its own inverted set computed inside $\alpha$, and the restriction of $\alpha$ to it is the inversion; if $G^\sigma$ is not a subgroup, none of these statements is available, and only the elementwise inversion of the theorem holds.

Proof. For $g \in G^\sigma$ one has $\alpha(g) = g^{-1}$, so $\alpha$ maps $G^\sigma$ into itself with the inversion as its restriction; the converse holds because an element of $G^\sigma$ inverted by $\alpha$ is fixed by $\alpha$, and $I(\alpha) = G^\sigma$ by the dictionary.

The Fixed Subgroup of the Automorphism

The subgroup that a continuous involution always carries is not the fixed set but its other face.

Theorem (the fixed subgroup). $H := G^\alpha = I(\sigma)$ is a closed subgroup of $G$, and it is the largest subgroup of $G$ on which $\sigma$ acts by the inversion. One has $H \subseteq G^\sigma$ exactly when every element of $H$ has order two, and then $H = G^\sigma$; consequently $G^\sigma$ coincides with $H$ if and only if it is an abelian subgroup of exponent two.

Proof. $\alpha$ is a continuous automorphism of order two, so its fixed set $G^\alpha$ is a closed subgroup; by the dictionary it equals $I(\sigma)$, the set of elements inverted by $\sigma$. Every subgroup on which $\sigma$ acts by the inversion consists of elements inverted by $\sigma$, hence lies in $I(\sigma) = H$, so $H$ is the largest such subgroup. For $h \in H$ one has $\alpha(h) = h$ and therefore $\sigma(h) = h^{-1}$; hence $h \in G^\sigma$ exactly when $h = h^{-1}$, that is $h^2 = e$. Thus $H \subseteq G^\sigma$ if and only if $H$ has exponent two, and then the two sets are equal, because an element $g \in G^\sigma$ has $\sigma(g) = g$, so $g \in I(\sigma)$ if and only if $g^2 = e$. The characterisation of equality with $G^\sigma$ is the criterion of the previous section together with this exponent-two condition.

Proposition (the two faces differ in general). The sets $G^\sigma$ and $H = I(\sigma)$ need not be comparable: for $S_3$ with the involution $\sigma = \iota\,c_{(1\,2)}$ one has $\lvert G^\sigma\rvert = 4$ and $\lvert H\rvert = 2$, so neither contains the other. The elementwise statement that always holds is the factorisation of the symmetric map

$$ \varphi : G \longrightarrow G^\sigma, \qquad \varphi(g) = g^{-1}\alpha(g) = g^{-1}\sigma(g)^{-1}, $$

which lands in the fixed-point set, is constant on the left cosets of $H$, and induces a continuous bijection of $G/H$ onto its image, the set of twisted squares; this is the coset theorem of Involutive Topological Groups, and the image need not be all of $G^\sigma$, as the same source records.

The Quotient

Definition. The involution acts on the quotient by the fixed subgroup $H = G^\alpha$ as follows. Since $\sigma(h) = h^{-1}$ for $h \in H$, the involution carries the left coset $gH$ to the right coset $H\sigma(g)$, and therefore induces a bijection

$$ \bar\sigma : G/H \longrightarrow H\backslash G, \qquad \bar\sigma(gH) = H\sigma(g), $$

between the space of left cosets and the space of right cosets of $H$.

Theorem. The bijection $\bar\sigma$ is continuous for the two quotient topologies and it is an involution in the sense that the reverse map is the involution applied to the reversed quotient; when $H$ is normal, $\bar\sigma$ is a continuous involution of the quotient group $G/H$, the quotient map is equivariant, and $\bar\sigma$ is the involution induced by $\sigma$ on the quotient.

Proof. The quotient maps $G \to G/H$ and $G \to H\backslash G$ are continuous and open, and $\bar\sigma$ is the composite of the continuous map $g \mapsto \sigma(g)$ with the appropriate quotient map; since $\sigma$ is a homeomorphism, the induced map is continuous with continuous inverse. If $H$ is normal then $G/H$ is a group, $H\sigma(g) = \sigma(g)H$, and $\bar\sigma(gH)H = \sigma(g)H$; the computation $\bar\sigma(xHyH) = \sigma(xy)H = \sigma(y)\sigma(x)H = \bar\sigma(yH)\bar\sigma(xH)$ shows that $\bar\sigma$ is an anti-automorphism, and $\bar\sigma^2 = \mathrm{id}$ because $\sigma^2 = \mathrm{id}$.

Theorem (the fixed set of the quotient involution). Let $H$ be normal. The fixed set of the induced involution $\bar\sigma$ on $G/H$ contains the image of $G^\sigma$, and it may be strictly larger; the image of the fixed-point set is a closed subset of the quotient, and it is a subgroup exactly when the fixed set of the quotient is abelian.

Proof. If $g \in G^\sigma$ then $\bar\sigma(gH) = \sigma(g)H = gH$, so the image is contained in the fixed set. The strictness is the quotient phenomenon of Involutive Groups, where $\mathbb{Z}/4\mathbb{Z}$ is the standard witness; the closedness is the continuity of the quotient map, and the subgroup statement is the criterion applied to $G/H$.

Theorem (the orbit space of the fixed subgroup). The fixed subgroup $H$ acts on $G$ on both sides, and the square of the involution factors through the two-sided quotient: the map

$$ G \longrightarrow H\backslash G/H, \qquad g \mapsto HgH , $$

identifies $g$ and $\sigma(g)$ up to the action of $H$ in the sense that $\sigma(HgH) = H\sigma(g)H$, so $\sigma$ induces an involution of the double coset space $H\backslash G/H$; when $G$ is compact, this double coset space is compact Hausdorff and the induced involution has a closed fixed set.

Proof. The double cosets are the orbits of the action of $H\times H$ by $(h,k)\cdot g = hgk^{-1}$; since $\sigma(H) = H$, the involution permutes them and descends to the quotient. For compact $G$ the quotient by a closed subgroup is compact Hausdorff, and the fixed set of the induced involution is closed by the equalizer argument.

Remark (the symmetric space). The space $G/G^\sigma$ of cosets of the fixed-point set, when $G^\sigma$ is a closed subgroup, carries the differential geometry of a symmetric space and the involution it inherits is the geodesic symmetry; that geometry is Part IV, Symmetric Spaces, and this article stops at the closed subspace $G^\sigma$ and the quotient topology of $G/H$.

Summary

The fixed-point set $G^\sigma$ of a continuous involution of a Hausdorff topological group contains $e$, is closed under inversion and is closed in $G$, because it is an equalizer of continuous maps; the associated automorphism $\alpha = \sigma\iota$ inverts each of its elements. It is a subgroup if and only if it is abelian, and the criterion is the commutation of every pair of fixed points; it fails for the inversion of a nonabelian group with two noncommuting elements of order two, of which $S_3$ is the smallest witness, and it holds for the inversion of an abelian group, for a group of exponent two, and for the identity involution. The subgroup that every continuous involution carries is the fixed subgroup $H = G^\alpha = I(\sigma)$, closed and the largest subgroup on which the involution acts by the inversion; it coincides with $G^\sigma$ exactly when the latter is an abelian subgroup of exponent two. The involution carries left cosets of $H$ to right cosets and induces a continuous bijection $G/H \to H\backslash G$, a continuous involution of the quotient group when $H$ is normal; the fixed set of the quotient involution contains the image of $G^\sigma$ and may be strictly larger, and the involution descends to the double coset space $H\backslash G/H$, compact and with closed fixed set when $G$ is compact. The geometric symmetric space $G/G^\sigma$ is Part IV and is named only.

Summary of Notation

Symbol	Meaning
$\sigma$	a continuous involution, an anti-automorphism of order two
$\iota$	the inversion; $\iota(g) = g^{-1}$
$\alpha = \sigma\iota$	the associated continuous involutive automorphism
$G^\sigma = \{g : \sigma(g) = g\}$	the fixed-point set, closed; a subgroup iff abelian
$I(\sigma) = \{g : \sigma(g) = g^{-1}\}$	the inverted set, always a closed subgroup
$G^\alpha = I(\sigma)$	the fixed subgroup of $\alpha$, the subgroup the involution carries
$I(\alpha) = G^\sigma$	the other face of the dictionary
$\varphi(g) = g^{-1}\alpha(g)$	the symmetric map into $G^\sigma$, with image the twisted squares
$H = G^\alpha$	the fixed subgroup, used as the modulus of the quotients
$\bar\sigma(gH) = H\sigma(g)$	the induced bijection $G/H \to H\backslash G$
$H\backslash G/H$	the double coset space, carrying an induced involution