DiffusionNet: Discretization Agnostic Learning on Surfaces

2.1 Learning on Surfaces

Methods that learn on 3D surfaces directly typically fall into two major categories, based either on point cloud or triangle mesh representations.

Point-based methods.. A successful set of methods for learning on 3D shapes represented as point clouds was pioneered by the PointNet [Qi et al. 2017a] and PointNet++ [Qi et al. 2017b] architectures, which have been extended in many recent works, including PointCNN [Li et al. 2018], DGCNN [Wang et al. 2019], PCNN [Atzmon et al. 2018], and KPConv [Thomas et al. 2019], to name a few (see also Guo et al. [2020] for a recent survey). Moreover, recent efforts have also been made to incorporate invariance and equivariance of the networks with respect to various geometric transformations, e.g., Deng et al. [2018], Hansen et al. [2018], Li et al. [2021], Poulenard et al. [2019], Zhang et al. [2019], and Zhao et al. [2020]. The major advantages of point-based methods are their simplicity, flexibility, applicability in a wide range of settings, and robustness in the presence of noise and outliers. However, first, their overall accuracy can often be lower than that of methods that explicitly use surface (e.g., mesh) connectivity when it is available. Second, though effective on static mechanical objects and scenes, point-based methods may not be well suited for deformable (non-rigid) shape analysis, requiring extremely large training sets and significant data augmentation to achieve good results, e.g., for non-rigid shape matching applications [Groueix et al. 2018; Donati et al. 2020]. Globally supported point-based networks were recently considered in Peng et al. [2020]; our method naturally allows global support via learned diffusion.

Surface and graph-based methods.. To address the limitations of point-based approaches, several methods have been proposed that operate directly on mesh surfaces and thus can learn filters that are intrinsic and robust to complex non-rigid deformations. The earliest pioneering approaches in this direction generalize convolutions [Masci et al. 2015; Boscaini et al. 2016; Monti et al. 2017; Fey et al. 2018], typically using local surface parameterization via the logarithmic map. Unfortunately local parameterizations are only defined up to rotation in the tangent plane, leading to several methods that address this issue through design of equivariant surface networks [Poulenard and Ovsjanikov 2018; He et al. 2020; Wiersma et al. 2020; Yang et al. 2021; Haan et al. 2021; Mitchel et al. 2021]. Likewise, operating on vector-valued data in a local tangent space can expand the expressivity of the filter space [Wiersma et al. 2020; Mitchel et al. 2021; Beani et al. 2021]. Our method leverages learned gradient features (Section 3.4), which geometrically require only a local spatial gradient operation, and sidesteps the challenge of equivariant filters by using only inner products, which are naturally invariant. These gradient features are built on local differential operators, which have also been exploited in other recent methods (e.g., Eliasof and Treister [2020] and Jiang et al. [2018]).

Surface mesh structure has also been used in a variety of graphlike approaches that specifically leverage mesh connectivity [Hanocka et al. 2019; Verma et al. 2018; Lim et al. 2018; Gong et al. 2019; Feng et al. 2019; Milano et al. 2020; Hajij et al. 2020; Bodnar et al. 2021], the structure of discrete operators [Smirnov and Solomon 2021], or even random walks along edges [Lahav and Tal 2020], among others. While accurate, these methods can be costly on densely sampled shapes and often are not robust to significant changes in the mesh structure (Figure 3).

2.2 Spectral Methods

Our use of diffusion is also loosely related to techniques that operate in the spectral domain and often exploit the link between convolution and operations in a derived (e.g., Fourier or Laplace–Beltrami) basis, including Bruna et al. [2014], Levie et al. [2018], and Sun et al. [2020]. Such methods have a long history in graph-based learning and are well rooted in data analysis more broadly, including Laplacian eigenmaps [Belkin and Niyogi 2003], spectral clustering [Vallet and Lévy 2008], and diffusion maps [Coifman et al. 2005]. In geometry processing, spectral methods have been used for a range of tasks, including multi-resolution representation [Levy 2006], segmentation [Rustamov 2007], and matching [Ovsjanikov et al. 2012], among others [Vallet and Lévy 2008; Zhang et al. 2010].

Unfortunately, Laplacian eigenfunctions depend on each shape, and thus coefficients or learned filters from one shape are not trivially transferable to another. Levie et al. [2019] argue for transfer between discretizations of the same shape, but 3D geometric learning typically demands transfer between different shapes. Functional maps [Ovsjanikov et al. 2012] can be used to “translate” coefficients between shapes, and have been used, for example, in Yi et al. [2017] with spectral filter learning. Deep functional maps [Litany et al. 2017] propose to learn features in the primal domain, which are then projected onto the Laplace–Beltrami basis for functional map estimation. However, the features are still learned either with multi-layer perceptrons (MLPs) starting from pre-computed descriptors [Litany et al. 2017; Roufosse et al. 2019; Ginzburg and Raviv 2020; Halimi et al. 2019] or using point-based architectures [Donati et al. 2020].

Instead, we propose an approach that learns the parameters of a diffusion process that is directly transferable across shapes and, as we show below, can be used effectively in applications like non-rigid shape matching. We also stress that DiffusionNet is not spectral in nature and only uses spectral operations as an acceleration technique for evaluating diffusion efficiently.

We also note that our use of the Laplacian in defining the diffusion operator is related to methods based on polynomials of the Laplacian [Kostrikov et al. 2018; Defferrard et al. 2016], CayleyNets [Levie et al. 2018] and their recent application in shape matching using ACSCNNs [Li et al. 2020b]. However, we demonstrate that complex polynomial filters can be replaced with simple learned diffusion and, moreover, that gradient features can inject orientation information into the network, improving performance and robustness.

Similarly to our approach, diffusion for smooth communication has been explored on graphs [Klicpera et al. 2019; Xu et al. 2019], images [Liu et al. 2016], and point clouds [Hansen et al. 2018]. In contrast, our method directly learns a diffusion time per-feature (which significantly improves performance, Table 7), incorporates a learned gradient operation, and is applied directly to mesh surfaces.

Pooling. In surface learning, it is nontrivial to define pooling, especially on meshes where it often amounts to mesh simplification [Hanocka et al. 2019]. Various recent operations have been proposed for point cloud [Lin et al. 2020; Hu et al. 2020], mesh [Milano et al. 2020; Zhou et al. 2020], or even graph pooling [Ma et al. 2020; Li et al. 2020a]. A key advantage of our approach is that it automatically supports global spatial support without any downsampling operation, simplifying implementation and improving learning.

3.1 Pointwise Perceptrons

On a mesh or point cloud with \( V \) vertices, we consider a collection of \( D \) scalar features defined at each vertex. Our first basic building block is a pointwise function \( f : \mathbb {R}^D \rightarrow \mathbb {R}^D \), which is applied independently at every vertex to transform the features. We represent these pointwise functions as a standard MLP with shared weights across all vertices. Although these MLPs can fit arbitrary functions at each point, they do not capture the spatial structure of the surface or allow any communication between vertices, so a richer structure is needed.

Past approaches for communication have ranged from global reductions to explicit geodesic convolutions—instead, we will demonstrate that a simple learned diffusion layer effectively propagates information, without the need for potentially costly or error-prone computations.

3.2 Learned Diffusion

In the continuous setting, diffusion of a scalar field \( u \) on a domain is modeled by the heat equation, (1) \( \begin{equation} \tfrac{d}{dt} u_t = \Delta u_t, \end{equation} \) where \( \Delta \) is the Laplacian (or, more formally, the Laplace–Beltrami operator). The action of diffusion can be represented via the heat operator \( H_t \), which is applied to some initial distribution \( u_0 \) and produces the diffused distribution \( u_t \); this action can be defined as \( H_t(u_0) = \exp (t \Delta) u_0 \), where \( \exp \) is the operator exponential. Over time, diffusion is an increasingly-global smoothing process: for \( t=0 \), \( H_t \) is the identity map, and as \( t \rightarrow \infty \) it approaches the average over the domain.

We propose to use the heat equation to spatially propagate features for learning on surfaces; its principled foundations ensure that results are largely invariant to the way the surface is sampled or meshed. To discretize diffusion, one replaces \( \Delta \) with the weak Laplace matrix \( L \) and mass matrix \( M \). Here, \( L \) is a positive semi-definite sparse matrix \( L \in \mathbb {R}^{V \times V} \) with the opposite sign convention such that \( M^{-1} L\!\approx \!-\Delta \). The number of entries in \( L \) and \( M \) are generally \( O(V) \), scaling effectively to large inputs (Table 6). On triangle meshes, we will use the cotan-Laplace matrix, which is ubiquitous in geometry processing applications [MacNeal 1949; Pinkall and Polthier 1993; Crane et al. 2013]; for point clouds, we will use the related Laplacian from Sharp and Crane [2020]. This matrix has also been defined for voxel grids [Caissard et al. 2019], polygon meshes [Bunge et al. 2020], tetrahedral meshes [Alexa et al. 2020], and so on. The weak Laplace matrix is accompanied by a mass matrix \( M \), such that the rate of diffusion is given by \( -M^{-1} L u \). Here, \( M \) will be a “lumped” diagonal matrix of areas associated with each vertex.

We define a learned diffusion layer \( h_t : \mathbb {R}^{V} \rightarrow \mathbb {R}^{V} \), which diffuses a feature channel \( u \) for learned time \( t \in \mathbb {R}_{\ge 0} \). In our networks, \( h_t(u) \) is applied independently to each feature channel, with a separate learned time \( t \) per channel. Learning the diffusion parameter is a key strength of our method, allowing the network to continuously optimize for spatial support ranging from purely local to totally global and even choose different receptive fields for each feature (Figure 4). We thus sidestep challenges like manually choosing the support radius of a convolution or sizes for a pooling hierarchy.

In the language of deep learning, diffusion can be viewed as a kind of smooth mean (average) pooling operation with many benefits: It has a geometrically principled meaning, its support ranges from purely local to totally global via the choice of diffusion time, and it is differentiable with respect to diffusion time, allowing spatial support to be automatically optimized as a network parameter.

A note on generality.. Remarkably, eschewing traditional representations of convolutions in favor of diffusion does not reduce the expressive power of our networks. This is supported by the following theoretical result (that we prove in Appendix A), which shows that radially symmetric convolutions are contained in the function space defined by diffusion followed by a pointwise map:

This fact is significant, because it suggests that simple and robust diffusion can, without loss of generality, be used to replace complicated operations such as radial geodesic convolution.

Importantly, we will also extend our architecture beyond radially symmetric filters by incorporating gradient features (Section 3.4).

3.3 Computing Diffusion

Many numerical schemes could potentially be used to evaluate the diffusion layer, \( h_t(u) \), from direct solvers [Chen et al. 2008] to hierarchical schemes [Vaxman et al. 2010; Liu et al. 2021]. In particular, we seek schemes that are efficient as well as differentiable, to enable network training. Here we describe two simple methods considered in our experiments. The first scheme we consider is an implicit timestep, which is straightforward but requires solving large sparse linear systems, and the second is spectral expansion, which uses only efficient dense arithmetic at evaluation time but requires some modest precomputation. Both are easily implemented using common numerical libraries, and we observe that networks trained with either approach have similar accuracy. Efficiency is evaluated in Section 5.6; we generally recommend spectral acceleration.

3.3.2 Spectral Acceleration.

An alternate approach is to leverage a closed-form expression for diffusion in the basis of low-frequency Laplacian eigenfunctions [Vallet and Lévy 2008; Zhang et al. 2010]. Once the eigenbasis has been precomputed, diffusion can then be evaluated for any time \( t \) via elementwise exponentiation. Truncating diffusion to a low-frequency basis incurs some approximation error, but we find that this approximation has little effect on our method (Figure 13), perhaps because diffusion quickly damps high-frequency content regardless.

View Figure

For weak Laplace matrix \( L \) and mass matrix \( M \), the eigenvectors \( \phi _i \in \mathbb {R}^V \) are solutions to (3) \( \begin{equation} L \phi _i = \lambda _i M \phi _i, \end{equation} \) corresponding to the first \( k \) smallest-magnitude eigenvalues \( \lambda _1, \ldots , \lambda _k \). We normalize them so that \( \phi _i^T M \phi _i = 1 \). These eigenvectors are easily precomputed for each shape of interest via standard numerical packages [Lehoucq et al. 1998]; the inset figure shows several example functions \( \phi _i \) for a surface of a human shape.

Let \( \Phi := [\phi _i] \in \mathbb {R}^{V \times k} \) be the stacked matrix of eigenvectors, which form an orthonormal basis with respect to \( M \). We can then project any scalar function \( u \) to obtain its coefficients \( c \) in the spectral basis via \( c \leftarrow \Phi ^T M u \) and recover values at vertices as \( u \leftarrow \Phi c \). Conveniently, diffusion for time \( t \) is easily expressed as an elementwise scaling of spectral coefficients according to \( c_i \leftarrow e^{-\lambda _i t} c_i \). The diffusion layer \( h_t(u) \) is then evaluated by projecting onto the spectral basis, evaluating pointwise diffusion, and projecting back (4) \( \begin{equation} h_t(u) := \Phi \begin{bmatrix}e^{-\lambda _0 t} \\ e^{-\lambda _1 t} \\ \dots \end{bmatrix} \odot (\Phi ^T M u), \end{equation} \) where \( \odot \) denotes the Hadamard (elementwise) product. This operation is efficiently evaluated using dense linear algebra operations like elementwise exponentiation and matrix multiplication and is easily differentiable with respect to \( u \) and \( t \).

Remarks. We emphasize that DiffusionNet can still learn high-frequency outputs despite the use of a low-frequency basis (e.g., in Section 5.4). Intuitively, diffusion is used for communication across points, for which a low-frequency approximation is typically sufficient, while MLPs and gradient features learn high-frequency features as needed for a task. Additionally, we note that DiffusionNet is not a spectral learning method—spectral coefficients are never used to represent filters or latent data, and thus no issues arise due to differing eigenbases on different shapes. Spectral acceleration is merely one possible numerical scheme to compute diffusion.

3.4 Spatial Gradient Features

Our learned diffusion layer enables propagation of information across different points on a shape, but it supports only radially symmetric filters about a point. The last building block in our method enables a larger space of filters by computing additional features from the spatial gradients of signal values at vertices (Figure 6). Specifically, we construct features from the inner products between pairs of feature gradients at each vertex, after applying a learned scaling or rotation.

Evaluating gradients.. We will express the spatial gradient of a scalar function on a surface as a 2D vector in the tangent space of each vertex. These gradients can be evaluated by a standard procedure, choosing a normal vector at each vertex (given as input or locally approximated) and then projecting neighbors into the tangent plane—either 1-ring neighbors on a mesh or \( m \)-nearest neighbors in a point cloud. The gradient is then computed in the tangent plane via least-squares approximation of the function values at neighboring points (see Mukherjee and Wu [2006] for analysis). These gradient operators at each vertex can be assembled into a sparse matrix \( G \in \mathbb {C}^{V \times V} \), which is applied to a vector \( u \) of real values at vertices to produce gradient tangent vectors at each vertex. This matrix does not depend on the features and can be precomputed once for each shape. We use complex numbers as a convenient notation for tangent vectors in an arbitrary reference basis in the tangent plane of each point (as in Knöppel et al. [2013] and Sharp et al. [2019], etc.). If the normals are consistently oriented, then the imaginary axis is chosen to form a right-handed basis in 3D with respect to the outward surface normal.

Learned pairwise products.. Equipped with per-vertex spatial gradients of each channel, we learn informative scalar features by evaluating an inner product between pairs of feature gradients at each vertex after a learned linear transformation. Inner products are invariant to rotations of the coordinate system, so these features are invariant to the choice of tangent basis at vertices, as expected. Putting it all together, given a collection of \( D \) scalar feature channels, for each channel \( u \) we first construct its spatial gradient as \( z_u \in \mathbb {C}^{V} \), a vector of local 2D gradients per-vertex, (5) \( \begin{equation} z_u := G u, \end{equation} \) and then at each vertex \( v \), we stack the local gradients of all channels to form \( w_v \in \mathbb {C}^{D} \) and obtain real-valued features \( g_v \in \mathbb {R}^D \) as (6) \( \begin{equation} g_v := \textrm {tanh}(\textrm {Re} (\overline{w}_v \odot A w_v)), \end{equation} \) where \( A \) is a learned square \( D \times D \) matrix, and taking the real part \( \textrm {Re} \) after a complex conjugate \( \overline{w}_v \) is again just a notational convenience for dot products between pairs of 2D vectors. This means the \( i{\rm th} \) entry of the output at vertex \( v \) is given by the dot product \( g_v(i) = \textrm {tanh}(\textrm {Re} \lbrace \sum _{j=1}^D \overline{w}_v(i) A_{ij} w_v(j) \rbrace) \), so that each inner product is scaled by a learned coefficient \( A_{ij} \). The outer \( \textrm {tanh}(\cdot) \) nonlinearity is not fundamental, but we find that it stabilizes training.

The choice of \( A \) as a complex or real matrix has a subtle relationship with the orientation of the underlying surface. Multiplying \( w_v(j) \) by a complex scalar both rotates and scales local gradient vectors before taking inner products (recall that complex multiplication can be interpreted as a rotation and scaling in the complex plane). In contrast, real \( A \) only allows scaling. However, the direction of rotation (clockwise or counter-clockwise) depends on the choice of the outward normal and hence on orientation—so surfaces with consistently oriented normals gain a richer representation by learning a complex matrix, whereas surfaces without consistent orientation (e.g., raw point clouds) should restrict to real \( A \).

In Figure 5, a small synthetic experiment (detailed in Appendix C) demonstrates how these learned rotations allow our method to disambiguate bilateral symmetry even in a purely intrinsic representation, a common challenge in non-rigid shape correspondence.

4.1 Input Features

DiffusionNet takes a vector of scalar values per vertex as input features. Here, we consider two simple choices of features; others could easily be included when available. Most directly, we simply use the raw 3D coordinates of a shape as input; rotation augmentation can be used to promote rigid invariance when inputs are not consistently aligned. When rigid or even non-rigid invariance is desired, we instead use the heat kernel signatures (HKS) [Sun et al. 2009] as input; these signatures are trivially computed from the spectral basis in Section 3.3.2. Due to the intrinsic nature of our approach, with HKS as input, the networks are invariant to any orientation-preserving isometric deformation of the shape. Higher-order descriptors such as SHOT [Tombari et al. 2010] seem unnecessary and may be unstable under remeshing [Donati et al. 2020].

5.1 Classification

We first apply DiffusionNet to classify meshes in the SHREC-11 dataset [Lian et al. 2011], which has 30 categories of 20 shapes each. We demonstrate that DiffusionNet learns successfully even in the presence of limited data. As in the other cited results, we train on just 10 samples per class; our results are averaged over 10 trials of the experiment with random training splits. We fit a cross-entropy loss with a label smoothing factor of 0.2 (see discussion in Goyal et al. [2021]). We use a 32-width DiffusionNet for hks features and 64-width DiffusionNet for xyz features with rotation augmentation. DiffusionNet achieves the highest reported accuracy when applied directly on the original dataset models or to the simplified models common in recent mesh-based learning work (Table 1).

5.2 Segmentation

Molecular segmentation.. We evaluate a 128-width DiffusionNet on the task of segmenting RNA molecules into functional components, using the dataset introduced by Poulenard et al. [2019]. This dataset consists of 640 RNA surface meshes of about 15k vertices each extracted from the Protein Data Bank [Berman et al. 2000], labelled at each vertex according to 259 atomic categories, with a random 80-20 train-test split. We learn these labels directly on the raw meshes, as well as on point clouds of 4096 uniformly sampled points as in past work [Poulenard et al. 2019]. For comparison, we cite point cloud results reported in Poulenard et al. [2019] and additionally train methods related to ours, SplineCNN [Fey et al. 2018] and a Dirac Surface Network [Kostrikov et al. 2018] on meshes. We also attempted to train MeshCNN [Hanocka et al. 2019] and HSN [Wiersma et al. 2020] but found the former prohibitively expensive, while the latter could not successfully preprocess the data. Our method achieves state-of-the-art accuracy on both the mesh and point cloud variants of the problem (Table 2, Figure 8). Learning directly on the mesh yields greater accuracy, perhaps because no information is lost when sampling a point cloud, and surface structure is preserved.

Table 2.

View Table

Table 2. RNA Surface Segmentation

Human segmentation.. We train a 128-width DiffusionNet with dropout to segment the human body parts on the composite dataset of Maron et al. [2017], containing models from several other human shape datasets [Bogo et al. 2014; Anguelov et al. 2005; Adobe 2016; Vlasic et al. 2008; Giorgi et al. 2007]. Additionally, we cite a variety of reported results from other approaches on this task, as reported by the respective original authors and by Wiersma et al. [2020]. For clarity, we distinguish between variants of this task in past work that used simplified meshes and soft ground truth; more details are presented in Appendix C. Our model is quite effective using both rotation-augmented raw coordinates or heat kernel signatures as input (Table 3).

Table 3.

View Table

Table 3. Human Part Segmentation

5.3 Functional Correspondence

Functional maps compute a correspondence between a pair of shapes by finding a linear transformation between spectral bases, aligning some set of input features [Ovsjanikov et al. 2012]. Recent work has shown that learned features can improve performance, e.g., Donati et al. [2020] and Litany et al. [2017]. Here, we demonstrate that using DiffusionNet as a feature extractor outperforms other recent approaches, yielding to state-of-the-art correspondence results in both the supervised and weakly supervised variants of the problem. We emphasize that the spectral representation in functional maps is unrelated to the spectral acceleration from Section 3.3.2, which is merely a scheme for evaluating diffusion; DiffusionNet itself does not learn in the spectral domain.

Our experiments follow the setup of Donati et al. [2020], training and evaluating on both SCAPE [Anguelov et al. 2005; Ren et al. 2018] and FAUST [Bogo et al. 2014], including training on one dataset and evaluating on the other. In the supervised setting, we fit dataset-provided correspondences, and we generate rigid-invariant models by randomly rotating all inputs for training and testing. For the weakly supervised setting, we use the dataset and losses advocated in Sharma and Ovsjanikov [2020], where rigid alignment of the input is used as weak supervision, without known correspondences. In all cases, we extract point-to-point maps between test shapes and evaluate them against ground truth dense correspondences; for simplicity, we compare all methods without post-processing the maps, though we also report accuracies after postprocessing with ZoomOut for our method [Melzi et al. 2019]. In addition to citing results obtained using the KPConv feature extractor [Thomas et al. 2019] by Donati et al. [2020], we also train HSN [Wiersma et al. 2020], ACSCNN [Li et al. 2020b], and our own 128-width DiffusionNet with dropout. We also tried MeshCNN [Hanocka et al. 2019], but it proved to be prohibitively expensive at 14hours per epoch.

As shown in Table 4, DiffusionNet yields state-of-the-art results for non-rigid shape correspondence in the both the supervised and weakly supervised settings, especially when transferring between datasets. One might wonder whether the improvement in this task truly stems from our DiffusionNet architecture or from the use of HKS features. As shown in the same table, training KPConv with HKS as features shows DiffusionNet yields significant improvements regardless. Figure 9 visualizes the resulting correspondences on a challenging test pair, where only DiffusionNet achieves a high-quality correspondence when generalizing after training on a different dataset.

5.4 Discretization Agnostic Learning

A key benefit of DiffusionNet compared to many past approaches is that its outputs are robust to changes in the discretization of the input (e.g., different meshes of the same shape or a mesh vs. a point cloud). This property is essential for practical applications where meshes for inference are likely to be tessellated differently from the training set, and so on. Other invariants (e.g., rigid invariance) can be encouraged via data augmentation, but for discretization it is impractical to generate augmented inputs across all possible sampling patterns. Below, we demonstrate that DiffusionNet generalizes quite well across discretizations without any special regularization or augmentation, especially compared to recent mesh-based methods.

We study discretization invariance using a popular formulation of the shape correspondence task on the FAUST human dataset [Bogo et al. 2014], where each vertex of a mesh is to be labelled with the corresponding vertex on a template mesh. Importantly, these input meshes are already manually aligned templates, with exactly identical mesh connectivity. Past work has achieved near-perfect accuracy in this problem setup (e.g., Fey et al. [2018] and Li et al. [2020b]); however we suggest that these models primarily overfit to mesh graph structure. In contrast, DiffusionNet learns an accurate and general function of the shape itself, despite the synthetic setup.

To experimentally quantify this effect, we construct a version of the FAUST test set after remeshing with several strategies: orig is the original test mesh, iso is a uniform isotropic remeshing, dense refines the mesh in randomly sampled regions, qes first refines the meshes and then applies quadric error simplification [Garland and Heckbert 1997], and cloud is a point cloud with normals sampled from the surface (Figure 10). Unrelated remeshings have appeared in Poulenard and Ovsjanikov [2018] and Wiersma et al. [2020], but the procedure therein left large regions of the mesh unchanged. Ground truth for evaluation is defined via nearest-neighbor on the original test mesh. We train on the 3D coordinates of the 80 standard FAUST registered meshes but evaluate on the remeshed set to mimic the practical scenario where the training set contains meshes tessellated via some particular common strategy, yet the fitted model must be applied to totally different meshes encountered in the wild. We also train several other methods—details are presented in Appendix C.

Table 5 and Figure 11 show how other mesh-based approaches degrade rapidly under remeshing; only DiffusionNet yields accurate correspondences that are largely stable under remeshing and resampling. Some point-based methods avoid the dependence on connectivity yet do not match the overall accuracy of the surface-based DiffusionNet.

5.5 Transfer across Representations

Not only are the outputs of DiffusionNet consistent across remeshing and resampling of a shape, but furthermore the same network can be directly applied to different discrete representations. The only geometric data required for DiffusionNet are the Laplacian, mass, and spatial gradient matrices, which are easily constructed for many representations. Because all geometric operations in the network are defined in terms of these standard matrices, fitted network weights retain the same meaning across different representations. This enables us to train on one representation and evaluate on another, as seen in Figure 1 and Section 5.4, cloud, without any special treatment or fine-tuning. In the future, DiffusionNet opens the door to heterogeneous training sets that intermingle mesh, point cloud, and other surface data from various sources.

5.6 Efficiency and Robustness

Runtime. DiffusionNet requires only standard linear algebra operations for training and inference and is thus straightforward and efficient on modern hardware. As an example, DiffusionNet with spectral acceleration trains on the 14k-vertex RNA meshes (Section 5.2) in 38 ms per input and requires 2.2 GB of GPU memory. Preprocessing is performed on the CPU once for each input; for these RNA meshes, preprocessing takes 5.4 seconds and generates 12 MB of data each, composed mainly of the Laplacian eigenbasis \( \Phi \) for spectral acceleration. Table 6 summarizes the runtime performance of DiffusionNet and several other recent methods on the human segmentation task (Section 5.2), including preprocessing, training with gradient computation, and inference. All timings are measured on a 24-GB Titan RTX GPU and dual Xeon 5120 2.2-GHz CPUs.

Table 6.

View Table

Table 6. Runtime of Mesh-Based Learning Methods

Scaling. Most significantly, DiffusionNet’s efficiency enables direct learning on common mesh data without dramatic simplification, in contrast to other recent mesh-based schemes. As an example, the segmentation meshes from Maron et al. [2017] have up to 13k vertices, yet recent approaches simplify/downsample to roughly 1k vertices for training, as shown in Figure 2 [Hanocka et al. 2019; Wiersma et al. 2020; Lahav and Tal 2020; Mitchel et al. 2021]. In contrast, our networks easily run at full resolution on this and other datasets, paving the way for adoption in practice and improving accuracy due to preserved details (e.g., in Table 2). We even demonstrate DiffusionNet on a large, 184k vertex raw scan mesh from FAUST—again no special treatment is needed (Table 6 and Figure 1).

Fig. 2.

View Figure

Fig. 3.

View Figure

Fig. 4.

View Figure

Robustness. DiffusionNet is also very robust to poor-quality input data; diffusion is a stable smoothing operation, and our method does not require any complex geometry processing operations such as geodesic distance [Masci et al. 2015], edge collapse [Hanocka et al. 2019], parallel transport [Wiersma et al. 2020], or managing pooling hierarchies with upsampling/ downsampling. Even the gradient matrix \( G \) is the result of a stable least-squares fit. If desired, then techniques like the intrinsic Delaunay Laplacian on meshes can be used to further increase robustness [Bobenko and Springborn 2007; Sharp and Crane 2020], though we do not find it necessary in our experiments. We demonstrate in Figure 1 that DiffusionNet can be applied directly to a low-quality, nonmanifold raw scan mesh without any issues.