Kavli Affiliate: Brian D. Nord
| First 5 Authors: Ashwin Samudre, Mircea Petrache, Brian D. Nord, Shubhendu Trivedi,
| Summary:
There has been much recent interest in designing symmetry-aware neural
networks (NNs) exhibiting relaxed equivariance. Such NNs aim to interpolate
between being exactly equivariant and being fully flexible, affording
consistent performance benefits. In a separate line of work, certain structured
parameter matrices — those with displacement structure, characterized by low
displacement rank (LDR) — have been used to design small-footprint NNs.
Displacement structure enables fast function and gradient evaluation, but
permits accurate approximations via compression primarily to classical
convolutional neural networks (CNNs). In this work, we propose a general
framework — based on a novel construction of symmetry-based structured
matrices — to build approximately equivariant NNs with significantly reduced
parameter counts. Our framework integrates the two aforementioned lines of work
via the use of so-called Group Matrices (GMs), a forgotten precursor to the
modern notion of regular representations of finite groups. GMs allow the design
of structured matrices — resembling LDR matrices — which generalize the
linear operations of a classical CNN from cyclic groups to general finite
groups and their homogeneous spaces. We show that GMs can be employed to extend
all the elementary operations of CNNs to general discrete groups. Further, the
theory of structured matrices based on GMs provides a generalization of LDR
theory focussed on matrices with cyclic structure, providing a tool for
implementing approximate equivariance for discrete groups. We test GM-based
architectures on a variety of tasks in the presence of relaxed symmetry. We
report that our framework consistently performs competitively compared to
approximately equivariant NNs, and other structured matrix-based compression
frameworks, sometimes with a one or two orders of magnitude lower parameter
count.
| Search Query: ArXiv Query: search_query=au:”Brian D. Nord”&id_list=&start=0&max_results=3