International Journal for Numerical Methods in Engineering
Volume 125, Issue 14 e7481
REVIEW ARTICLE
Open Access

On sparse regression, Lp-regularization, and automated model discovery

Jeremy A. McCulloch, 

Jeremy A. McCulloch

Department of Mechanical Engineering and Bioengineering, Stanford University, Stanford, California, USA

Search for more papers by this author
Skyler R. St. Pierre, 

Skyler R. St. Pierre

Department of Mechanical Engineering and Bioengineering, Stanford University, Stanford, California, USA

Search for more papers by this author
Kevin Linka, 

Kevin Linka

Institute for Continuum and Material Mechanics, Hamburg University of Technology, Hamburg, Germany

Search for more papers by this author
Ellen Kuhl, 

Corresponding Author

Ellen Kuhl

Department of Mechanical Engineering and Bioengineering, Stanford University, Stanford, California, USA

Correspondence

Ellen Kuhl, Department of Mechanical Engineering and Bioengineering, Stanford University, 452 Escondido Mall, Stanford, CA 94305, USA.

Email: [email protected]

Search for more papers by this author
First published: 08 April 2024
https://doi.org/10.1002/nme.7481
Citations: 3
This manuscript is dedicated to Robert L. Taylor on the occasion of his 90th birthday and to his many contributions to the finite element method and his open source software FEAP. Congratulations, Bob: tang,, 1!

Abstract

Sparse regression and feature extraction are the cornerstones of knowledge discovery from massive data. Their goal is to discover interpretable and predictive models that provide simple relationships among scientific variables. While the statistical tools for model discovery are well established in the context of linear regression, their generalization to nonlinear regression in material modeling is highly problem-specific and insufficiently understood. Here we explore the potential of neural networks for automatic model discovery and induce sparsity by a hybrid approach that combines two strategies: regularization and physical constraints. We integrate the concept of Lp regularization for subset selection with constitutive neural networks that leverage our domain knowledge in kinematics and thermodynamics. We train our networks with both, synthetic and real data, and perform several thousand discovery runs to infer common guidelines and trends: L2 regularization or ridge regression is unsuitable for model discovery; L1 regularization or lasso promotes sparsity, but induces strong bias that may aggressively change the results; only L0 regularization allows us to transparently fine-tune the trade-off between interpretability and predictability, simplicity and accuracy, and bias and variance. With these insights, we demonstrate that Lp regularized constitutive neural networks can simultaneously discover both, interpretable models and physically meaningful parameters. We anticipate that our findings will generalize to alternative discovery techniques such as sparse and symbolic regression, and to other domains such as biology, chemistry, or medicine. Our ability to automatically discover material models from data could have tremendous applications in generative material design and open new opportunities to manipulate matter, alter properties of existing materials, and discover new materials with user-defined properties.

1 MOTIVATION

The ability to discover meaningful constitutive models from data would forever change how we understand, model, and design new materials and structures. Massive advancements in data science are now bringing us closer than ever towards this goal.1, 2 Throughout the past three years, numerous research groups have begun to harness the potential of neural networks and fit constitutive models to experimental data,3-14 an approach that is now widely known as constitutive neural networks.15 While initial studies have used neural networks exclusively as black box regression operators,16 recent approaches are increasingly recognizing their potential to discover not only the model parameters, but also the model itself.17 The paradigm of automated model discovery was first formalized in the context of nonlinear dynamical systems more than a decade ago to discover Lagrangians and Hamiltonians for oscillators,18 pendula, biological processes,19 or turbulent fluid flows.20 It is now rapidly gaining popularity in the context of constitutive modeling,21 and several promising techniques have emerged to decipher constitutive relations between stresses and strains,11 and even integrate them automatically into a finite element analysis.22-24 These not only include constitutive neural networks,15 but also sparse regression,25 genetic programming in the form of symbolic regression,26 and variational system identification.27

The holy grail in automated model discovery is to identify generalizable and truly interpretable models with physically meaningful parameters.2, 28, 29 Ideally, we want to discover a concise, yet simple and interpretable model with only a few relevant terms that best explains experimental data, while remaining robust to outliers and noise. In terms of statistical learning, this translates model discovery into a subset selection or feature extraction task. Subset selection and shrinkage methods are by no means new; in fact, they have been extensively studied for many decades.30-33 In the context of linear regression, these methods have become standard textbook knowledge.34 In the context of nonlinear regression, when analytical solutions are rare, subset selection is much more nuanced, general recommendations are difficult, and feature extraction becomes highly problem-specific.35 To be clear, this limitation is not exclusively inherent to automated model discovery with constitutive neural networks—it applies to distilling scientific knowledge from data in general.19 This includes alternative model discovery approaches like sparse regression,20, 25 or symbolic regression.19, 26, 36 The key question to the success of discovering new knowledge from data is: how do we robustly discover the best interpretable model with a small subset of relevant terms? And, probably equally importantly: What is the trade-off between interpretability and prediction accuracy? To frame these questions more broadly, let us first revisit the notions of regression and neural networks in the context of constitutive modeling:

Regression. Regression is a statistical method to examine the relationship between a dependent variable, in constitutive modeling in solid mechanics the stress σ $$ \boldsymbol{\sigma} $$ , and one or more independent variables, in this case the strain ε $$ \boldsymbol{\varepsilon} $$ , using a model that depends on a set of model parameters θ $$ \boldsymbol{\theta} $$ . Here regression has two main objectives: characterizing the form and strength of the relationship between stress and strain to enable predictions, and providing insights into how stress and strain are correlated.34 Popular types of regression are logistic regression, assuming for example a binary relationship; linear regression,37 assuming a relationship that is linear in the model parameters θ $$ \boldsymbol{\theta} $$ , or nonlinear regression, assuming a relationship that is nonlinear in the model parameters θ $$ \boldsymbol{\theta} $$ , as we do throughout this manuscript. Regression is the cornerstone of statistical learning.35 It provides tools to decipher relationships within data, but its application to constitutive modeling requires attention to physical constraints including objectivity, symmetry, incompressibility, polyconvexity, or thermodynamic consistency.38-43 As a natural consequence, we cannot just use any set of functions to build our constitutive model: while polynomial functions between stresses and strains associated with a linear regression would be ideal from an optimization point of view, these models may violate thermodynamic constraints, which favor exponential or power functions associated with a nonlinear regression.

Linear regression. Linear regression37 seeks to model the relationship between a dependent variable, in our case the stress σ $$ \sigma $$ , and a set of one or more independent variables, in our case the set of strains ε i $$ {\varepsilon}_i $$ at different load levels i $$ i $$ , using a function that depends linearly on the model parameter θ = {   E   } $$ \theta =\left\{\;\mathrm{E}\;\right\} $$ , in this case the elastic modulus or Young's modulus. The regression estimates this parameter by minimizing the difference between the predicted stress, σ i = E ε i $$ {\sigma}_i=\mathrm{E}\kern0.3em {\varepsilon}_i $$ , for given strains ε i $$ {\varepsilon}_i $$ and stiffness E $$ \mathrm{E} $$ , and the experimentally measured stresses σ ^ i $$ {\hat{\sigma}}_i $$ , divided by the number of data points n data $$ {n}_{\mathrm{data}} $$ . A common measure for this difference is the mean squared error based on the L 2 $$ {L}_2 $$ -norm,34 | |   (   ∘   )   | | = | |   (   ∘   )   | | 2 = |   (   ∘   ) 2   | 1 / 2 $$ \left\Vert\;\left(\circ \right)\;\right\Vert ={\left\Vert\;\left(\circ \right)\;\right\Vert}_2={\left|\;{\left(\circ \right)}^2\;\right|}^{1/2} $$ , for which the minimization problem becomes
L ( E ; ε ) = 1 n data ∑ i = 1 n data | | E ε i − σ ^ i | | 2 → min E … estimate model parameter E . $$ L\left(\mathrm{E};\varepsilon \right)=\frac{1}{n_{\mathrm{data}}}\sum \limits_{i=1}^{n_{\mathrm{data}}}{\left\Vert \kern0.3em \mathrm{E}\kern0.3em {\varepsilon}_i-{\hat{\sigma}}_i\kern0.3em \right\Vert}^2\to \underset{\mathrm{E}}{\min}\kern2em \dots \mathrm{estimate}\ \mathrm{model}\ \mathrm{parameter}\kern2.56804pt \mathrm{E}. $$ ()
When phrased as least square's problems, provided linear regression problems have a convex objective function with a unique global minimum. For our example (1), we can find it by evaluating the vanishing derivative, ∂ L / ∂ E = ∑ i = 1 n data 2 ε i   (   E   ε i − σ ^ i ) ≐ 0 $$ \partial L/\mathrm{\partial E}={\sum}_{i=1}^{n_{\mathrm{data}}}2{\varepsilon}_i\;\left(\;\mathrm{E}\;{\varepsilon}_i-{\hat{\sigma}}_i\right)\doteq 0 $$ . Here, the minimization problem is not only linear in the model parameter E $$ \mathrm{E} $$ , but also in the dependent variable ε $$ \varepsilon $$ , and we obtain an explicit solution for the elastic modulus, E = ∑ i n data ε i   σ ^ i   /   ∑ i n data ε i   ε i $$ \mathrm{E}={\sum}_i^{n_{\mathrm{data}}}{\varepsilon}_i\;{\hat{\sigma}}_i/{\sum}_i^{n_{\mathrm{data}}}{\varepsilon}_i\;{\varepsilon}_i $$ . For linear regression with multiple model parameters θ $$ \boldsymbol{\theta} $$ , for which the minimization problem is a linear function in the dependent variables ε $$ \varepsilon $$ , we obtain a coupled system of equations, ∂ L / ∂ θ ≐ 0 $$ \partial L/\partial \boldsymbol{\theta} \doteq \mathbf{0} $$ , with an explicit solution for the parameter vector θ $$ \boldsymbol{\theta} $$ . For linear regression with one or multiple model parameters θ $$ \boldsymbol{\theta} $$ , for which the minimization problem is a nonlinear function in the dependent variables ε $$ \varepsilon $$ , we obtain a similar set of one or more equations ∂ L / ∂ θ ≐ 0 $$ \partial L/\partial \boldsymbol{\theta} \doteq \mathbf{0} $$ , which may require an iterative solution for the parameter vector θ $$ \boldsymbol{\theta} $$ . Importantly, any regression that is linear in the model parameters θ $$ \boldsymbol{\theta} $$ —independent of whether it is linear, polynomial, or generally nonlinear in the independent variables ε i $$ {\varepsilon}_i $$ —is considered a linear regression problem that, when phrased as a least square's problem with appropriate data, results in a convex quadratic function in the model parameters θ $$ \boldsymbol{\theta} $$ with a unique global minimum.
Nonlinear regression. Nonlinear regression44 seeks to model the relationship between a dependent variable, in our case the Piola stress P $$ \boldsymbol{P} $$ , and a set of one or more independent variables, in our case the set of deformation gradients F i $$ {\boldsymbol{F}}_i $$ at different load levels i $$ i $$ , using a function that depends nonlinearly on a set of model parameters θ $$ \boldsymbol{\theta} $$ .45 The regression estimates these parameters by minimizing the difference between the predicted stress, P ( F i , θ ) $$ \boldsymbol{P}\left({\boldsymbol{F}}_i,\boldsymbol{\theta} \right) $$ , for given deformation gradients F i $$ {\boldsymbol{F}}_i $$ and model parameters θ $$ \boldsymbol{\theta} $$ , and the experimentally measured stresses P ^ i $$ {\hat{\boldsymbol{P}}}_i $$ , divided by the number of data points n data $$ {n}_{\mathrm{data}} $$ . Similar to linear regression, we can measure for this difference as the mean squared error based on the L 2 $$ {L}_2 $$ -norm,34 | |   (   ∘   )   | | = | |   (   ∘   )   | | 2 = |   (   ∘   ) 2   | 1 / 2 $$ \left\Vert\;\left(\circ \right)\;\right\Vert ={\left\Vert\;\left(\circ \right)\;\right\Vert}_2={\left|\;{\left(\circ \right)}^2\;\right|}^{1/2} $$ , for which the minimization problem becomes
L ( θ ; F ) = 1 n data ∑ i = 1 n data | | P ( F i , θ ) − P ^ i | | 2 → min θ … estimate model parameters θ . $$ L\left(\boldsymbol{\theta}; \boldsymbol{F}\right)=\frac{1}{n_{\mathrm{data}}}\sum \limits_{i=1}^{n_{\mathrm{data}}}{\left\Vert \kern0.3em \boldsymbol{P}\Big({\boldsymbol{F}}_i,\boldsymbol{\theta} \Big)-{\hat{\boldsymbol{P}}}_i\kern0.3em \right\Vert}^2\to \underset{\boldsymbol{\theta}}{\min}\kern2em \dots \mathrm{estimate}\ \mathrm{model}\ \mathrm{parameters}\kern2.56804pt \boldsymbol{\theta} . $$ ()
In general, nonlinear regression problems have a non-convex objective function with multiple local minima. Solving non-convex optimization problems requires iterative algorithms that are at risk of converging to a local minimum instead of the global minimum, and their solution is often highly sensitive to the initial conditions that we select for the parameter vector. Depending on the nature of the problem, the solution we find may involve a large and dense parameter vector θ $$ \boldsymbol{\theta} $$ , and overfitting may occur when the number of parameters is larger than the number of data points, n para > n data $$ {n}_{\mathrm{para}}>{n}_{\mathrm{data}} $$ . Notably, even for many data points, we may face overfitting when the data are noisy or not rich enough to sufficiently activate all the parameters. For example, with tension and compression tests alone, we cannot estimate model parameters for shear.

Sparse regression. Sparse regression is a special type of regression that seeks to prevent overfitting by inducing sparsity in the parameter vector θ $$ \boldsymbol{\theta} $$ by setting a large number of parameters to zero.33 Sparse regression is particularly useful in high-dimensional settings, since it generates models with a small subset of non-zero parameters,34 which tends to make the model more interpretable.35 Historically, the need for sparse regression emerged prominently with the advent of high-dimensional datasets for which the number of parameters can easily exceed the number of independent observations.46 A prominent example is SINDy, an algorithm for sparse identification in nonlinear dynamics that promotes sparsity through sequential thresholded least-squares by iterating between a partial least-squares fit and a thresholding step to sequentially drop the least relevant terms of a model.20 Importantly, while these sparsification algorithms converge well in linear regression associated with convex objective functions,47 their convergence is no longer guaranteed in nonlinear regression with non-convex objective functions. The advantages of sparse regression are improved interpretability by reducing the parameter set to only a few non-zero terms; feature selection by identifying the most relevant terms; and reduced risk of overfitting by promoting simpler models. These advantages come at a price: the disadvantages of sparse regression are selection bias by enforcing sparsity of the parameter estimates; additional hyperparameters that need to be tuned and require additional attention; and risk of misspecification by excluding relevant parameters if sparsity is enforced too aggressively. In conclusion, sparse regression offers a powerful toolset for high-dimensional modeling, but introduces a trade-off between interpretability and prediction accuracy.

Neural networks. Neural networks are a class of models and algorithms that can approximate a wide range of functions.48 Their versatility not only makes them a powerful tool for classification, reinforcement learning, and generative tasks, but also for regression problems,49 in our context, for regression in constitutive modeling.16 Neural networks consist of input, hidden, and output layers with several nodes in each layer. Their parameters are the network weights θ = { w i , j } $$ \boldsymbol{\theta} =\left\{{w}_{i,j}\right\} $$ , where i = 1 , … , n lay $$ i=1,\dots, {n}_{\mathrm{lay}} $$ is the number of hidden layers and j = 1 , … , n nod $$ j=1,\dots, {n}_{\mathrm{nod}} $$ is the number of nodes per layer. During training, neural networks effectively perform a regression as they learn their parameters by minimizing a loss function L $$ L $$ that penalizes the error between model and data. Similar to the classical nonlinear regression in Equation (2), we can characterize this error as the mean squared error, the L 2 $$ {L}_2 $$ -norm of the difference between the stress predicted by the model P ( F i ) $$ \boldsymbol{P}\left({\boldsymbol{F}}_i\right) $$ and the experimentally measured stress P ^ i $$ {\hat{\boldsymbol{P}}}_i $$ , divided by the number of data points n data $$ {n}_{\mathrm{data}} $$ to train the model,
L ( θ ; F ) = 1 n data ∑ i = 1 n data | | P ( F i , θ ) − P ^ i | | 2 → min … learn network weights θ = { w i , j } . $$ L\left(\boldsymbol{\theta}; \boldsymbol{F}\right)=\frac{1}{n_{\mathrm{data}}}\sum \limits_{i=1}^{n_{\mathrm{data}}}{\left\Vert \kern0.3em \boldsymbol{P}\Big({\boldsymbol{F}}_i,\boldsymbol{\theta} \Big)-{\hat{\boldsymbol{P}}}_i\kern0.3em \right\Vert}^2\to \min \kern2em \dots \mathrm{learn}\ \mathrm{network}\ \mathrm{weights}\kern2.56804pt \boldsymbol{\theta} =\left\{{w}_{i,j}\right\}. $$ ()
However, in contrast to traditional regression tools that have a fixed functional form, neural networks can easily adapt their shape which allows them to model complex functions,5 either linear50 or nonlinear29 in the model parameters θ $$ \boldsymbol{\theta} $$ . The advantages of neural networks are their universal approximation that allows them to approximate any continuous function for a sufficiently large number of weights,48 and their inherent flexibility that allows them to model high-dimensional nonlinear relationships like the constitutive behavior we seek to model here. Their disadvantages are their computational complexity, especially for densely connected architectures with multiple hidden layers; risk of overfitting sparse or noisy data; and lack of interpretability that generally worsens with the number of layers and makes plain neural networks unsuitable for model discovery tasks.

Sparse neural networks. Sparse neural networks use a special type of network architecture for which a large number of weights are zero. This reduces the number of active connections between the nodes of consecutive layers. Sparsity can be induced during training by using special algorithms, or after training by pruning.51 The concepts of sparse neural networks and weight or node pruning—inspired by brain development and synaptic pruning—have gained increasing attention with the rise of deep learning and the need for computational efficiency.52 The advantages of sparse neural networks are their computational efficiency and faster inference times; reduced risk of overfitting by promoting smaller model sizes; and their potential for regularization and improved generalization. The disadvantages are increased training complexity to induce sparsity; and risk of decreased performance through overly aggressive sparsification or pruning.

The objective of this review is to explore neural networks for which sparsity is induced by a hybrid approach of two combined strategies: regularization and physical constraints. Towards this goal, first, we review popular subset selection and shrinkage methods to induce sparsity within the general framework of L p $$ {L}_p $$ regularization in Section 2. Then, we discuss how to leverage physical constraints to induce sparsity within the framework of constitutive neural networks in Section 3. Finally, we propose a hybrid approach of L p $$ {L}_p $$ regularized constitutive neural networks for automated model discovery and discuss the interpretability and prediction accuracy of their discovered models by means of a library of illustrative examples in Section 4. We conclude by comparing the different approaches and providing guidelines and recommendations in Section 5.

2 L   p $$ {}_p $$ REGULARIZATION

The concept of L p $$ {L}_p $$ regularization or bridge regression was first introduced three decades ago in the context of chemometrics with the goal to shrink the parameter space in chemical data analysis.30 The method has re-gained attention as a powerful tool to promote sparsity in system identification,20 and, most recently, in discovering constitutive models from data.25 L p $$ {L}_p $$ regularization is a generalized regularization technique that uses the L p $$ {L}_p $$ norm of the parameter vector θ $$ \boldsymbol{\theta} $$ , the sum of the p $$ p $$ th power of the norm of its i = 1 , … , n para $$ i=1,\dots, {n}_{\mathrm{para}} $$ coefficients w i $$ {w}_i $$ , raised to the inverse power, | |   θ   | | p = [   ∑ i = 1 n para |   w i   | p   ] 1 / p $$ {\left\Vert\;\boldsymbol{\theta}\;\right\Vert}_p={\left[\;{\sum}_{i=1}^{n_{\mathrm{para}}}{\left|\;{w}_i\;\right|}^p\;\right]}^{1/p} $$ . Bridge regression constrains the regression (2) by constraining this L p $$ {L}_p $$ norm to be smaller than a non-negative constant ϵ ≥ 0 $$ \epsilon \ge 0 $$ ,
L ( θ ; F ) = 1 n data ∑ i = 1 n data | | P ( F i , θ ) − P ^ i | | 2 → min subject to | | θ | | p p = ∑ i = 1 n para | w i | p ≤ ϵ . $$ L\left(\boldsymbol{\theta}; \boldsymbol{F}\right)=\frac{1}{n_{\mathrm{data}}}\sum \limits_{i=1}^{n_{\mathrm{data}}}{\left\Vert \kern0.3em \boldsymbol{P}\Big({\boldsymbol{F}}_i,\boldsymbol{\theta} \Big)-{\hat{\boldsymbol{P}}}_i\kern0.3em \right\Vert}^2\to \min \kern2em \mathrm{subject}\ \mathrm{to}\kern2em {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_p^p=\sum \limits_{i=1}^{n_{\mathrm{para}}}{\left|\kern0.3em {w}_i\kern0.3em \right|}^p\le \epsilon . $$ ()
It proves convenient to reformulate this constrained regression problem as a penalized regression problem, by penalizing the regression (2) with a penalty term that consists of the L p $$ {L}_p $$ norm | | θ | | p p $$ {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_p^p $$ , multiplied by a non-negative penalty parameter α ≥ 0 $$ \alpha \ge 0 $$ ,
L ( θ ; F ) = 1 n data ∑ i = 1 n data | | P ( F i , θ ) − P ^ i | | 2 + α | | θ | | p p → min with | | θ | | p p = ∑ i = 1 n para | w i | p . $$ L\left(\boldsymbol{\theta}; \boldsymbol{F}\right)=\frac{1}{n_{\mathrm{data}}}\sum \limits_{i=1}^{n_{\mathrm{data}}}{\left\Vert \kern0.3em \boldsymbol{P}\Big({\boldsymbol{F}}_i,\boldsymbol{\theta} \Big)-{\hat{\boldsymbol{P}}}_i\kern0.3em \right\Vert}^2+\alpha \kern0.3em {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_p^p\to \min \kern2em \mathrm{with}\kern2em {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_p^p=\sum \limits_{i=1}^{n_{\mathrm{para}}}{\left|\kern0.3em {w}_i\kern0.3em \right|}^p. $$ ()
The constrained and penalized regression problems (4) and (5) are equivalent, which implies that for a given ϵ ≥ 0 $$ \epsilon \ge 0 $$ , there exists an α ≥ 0 $$ \alpha \ge 0 $$ such that the two problems share the same solution θ $$ \boldsymbol{\theta} $$ .53 The flexible power p $$ p $$ not only allows us to recover classical regularization techniques like L 2 $$ {L}_2 $$ or L 1 $$ {L}_1 $$ regularization as special cases, but also to interpolate smoothly between these different methods.30 The advantages of L p $$ {L}_p $$ regularization are its inherent flexibility by allowing for a continuum of popular regularization techniques for varying powers p $$ p $$ ; and its potential to effectively promote sparsity. Its disadvantages are the added complexity associated with the selection of hyperparameters, specifically the penalty parameter α $$ \alpha $$ and the power p $$ p $$ ; and the potential computational challenges associated with specific choices for p $$ p $$ . Figure 1 illustrates the contours of the regularization term, for varying powers p $$ p $$ as p = [ 0 . 25 , 0 . 5 , 0 . 75 , 1 , 1 . 5 , 2 , 4 , 8 ] $$ p=\left[0.25,0.5,0.75,1,1.5,2,4,8\right] $$ , evaluated for two parameters, w 1 $$ {w}_1 $$ and w 2 $$ {w}_2 $$ . The top row illustrates the effect of powers smaller than or equal to one, p ≤ 1 $$ p\le 1 $$ ; the bottom row illustrates the effect powers larger than one, p > 1 $$ p>1 $$ . In the following, we highlight the most popular special cases of L p $$ {L}_p $$ regularization, their history, and their advantages and disadvantages.
Details are in the caption following the image
FIGURE 1
Open in figure viewerPowerPoint
Lp regularization. Contours of regularization term, L p = α   ∑ i = 1 n para | |   θ   | | p p $$ {L}_p=\alpha\;{\sum}_{i=1}^{n_{\mathrm{para}}}{\left\Vert\;\boldsymbol{\theta}\;\right\Vert}_p^p $$ with | | θ | | p p = | w i | p $$ {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_p^p={\left|\kern0.3em {w}_i\kern0.3em \right|}^p $$ , for varying powers, p = [ 0 . 25 , 0 . 5 , 0 . 75 , 1 , 1 . 5 , 2 , 4 , 8 ] $$ p=\left[0.25,0.5,0.75,1,1.5,2,4,8\right] $$ , evaluated for two parameters, w 1 $$ {w}_1 $$ and w 2 $$ {w}_2 $$ . For p ≤ 1 $$ p\le 1 $$ , top row, with the special case of L 1 $$ {L}_1 $$ regularization or lasso represented through the pyramid, L p $$ {L}_p $$ regularization promotes sparsity by setting some weights exactly to zero, but is no longer strictly convex and can have multiple local minima. For p > 1 $$ p>1 $$ , bottom row, with the special case of L 2 $$ {L}_2 $$ regularization or ridge regression represented through the ellipsoid, L p $$ {L}_p $$ regularization promotes stability by reducing outliers, while the regularization term remains convex.
L   2 $$ {}_{\mathbf{2}} $$ regularization or ridge regression. L 2 $$ {L}_2 $$ regularization, commonly known to as ridge regression, was introduced more than half a century ago to address multicollinearity in regression analysis,31 and has gained attention for its ability to stabilize parameter estimates, especially when the parameters are closely correlated.34 It uses the L 2 $$ {L}_2 $$ norm of a vector, the Euclidian norm, the sum of the vector components squared, | |   θ   | | 2 = [   ∑ i = 1 n para |   w i   | 2   ] 1 / 2 $$ {\left\Vert\;\boldsymbol{\theta}\;\right\Vert}_2={\left[\;{\sum}_{i=1}^{n_{\mathrm{para}}}{\left|\;{w}_i\;\right|}^2\;\right]}^{1/2} $$ . Notably, the L 2 $$ {L}_2 $$ norm does not weigh all entries of the vector equally. Instead, it squares the vector entries which makes it highly sensitive to outliers as it penalizes the squared magnitude of the individual parameters w i $$ {w}_i $$ . Ridge regression supplements the regression (2) with a penalty term that consists of the L 2 $$ {L}_2 $$ norm multiplied by a penalty parameter α $$ \alpha $$ ,
L ( θ ; F ) = 1 n data ∑ i = 1 n data | | P ( F i , θ ) − P ^ i | | 2 + α | | θ | | 2 2 → min with | | θ | | 2 2 = ∑ i = 1 n para | w i | 2 . $$ L\left(\boldsymbol{\theta}; \boldsymbol{F}\right)=\frac{1}{n_{\mathrm{data}}}\sum \limits_{i=1}^{n_{\mathrm{data}}}{\left\Vert \kern0.3em \boldsymbol{P}\Big({\boldsymbol{F}}_i,\boldsymbol{\theta} \Big)-{\hat{\boldsymbol{P}}}_i\kern0.3em \right\Vert}^2+\alpha \kern0.3em {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_2^2\to \min \kern2em \mathrm{with}\kern2em {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_2^2=\sum \limits_{i=1}^{n_{\mathrm{para}}}{\left|\kern0.3em {w}_i\kern0.3em \right|}^2. $$ ()
Its advantages are stability in multicollinearity by offering stable parameter estimates even in the presence of highly correlated predictors; managing outliers and preventing overfitting by quadratically penalizing extreme coefficients; and computational efficiency even for large datasets. Its disadvantages are introducing bias, which may result in underestimating certain coefficients and effects; and its inability to induce sparsity, which makes it unsuitable for our current focus of subset selection. Figure 1 shows that, for the special case of L 2 $$ {L}_2 $$ regularization, the regularization term adopts a convex ellipsoidal shape that promotes stability by reducing outliers.
L   1 $$ {}_{\mathbf{1}} $$ regularization or lasso. L 1 $$ {L}_1 $$ regularization was initially introduced as a method to analyze seismograms in geophysics almost four decades ago,32 and has become widely known under the name of lasso, short for least absolute shrinkage and selection operator.33 Lasso has become popular for producing interpretable models,4 while exhibiting the same stability properties as ridge regression. It uses the L 1 $$ {L}_1 $$ norm of a vector, the sum of the absolute values of its components, | |   θ   | | 1 = ∑ i = 1 n para |   w i   | $$ {\left\Vert\;\boldsymbol{\theta}\;\right\Vert}_1={\sum}_{i=1}^{n_{\mathrm{para}}} \mid {w}_i \mid $$ . Because of its similarities with a distance between city blocks, the L 1 $$ {L}_1 $$ norm is often referred to as the Manhattan distance or taxicab norm. Notably, the L 1 $$ {L}_1 $$ norm weighs all entries of the vector equally and is therefore less sensitive to outliers than the L 2 $$ {L}_2 $$ norm. Lasso supplements the regression (2) with a penalty term that consists of the L 1 $$ {L}_1 $$ norm multiplied by a penalty parameter α $$ \alpha $$ ,
L ( θ ; F ) = 1 n data ∑ i = 1 n data | | P ( F i , θ ) − P ^ i | | 2 + α | | θ | | 1 → min with | | θ | | 1 = ∑ i = 1 n para | w i | . $$ L\left(\boldsymbol{\theta}; \boldsymbol{F}\right)=\frac{1}{n_{\mathrm{data}}}\sum \limits_{i=1}^{n_{\mathrm{data}}}{\left\Vert \kern0.3em \boldsymbol{P}\Big({\boldsymbol{F}}_i,\boldsymbol{\theta} \Big)-{\hat{\boldsymbol{P}}}_i\kern0.3em \right\Vert}^2+\alpha \kern0.3em {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_1\to \min \kern2em \mathrm{with}\kern2em {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_1=\sum \limits_{i=1}^{n_{\mathrm{para}}}\mid \kern0.3em {w}_i\kern0.3em \mid . $$ ()
Its advantages are enabling feature selection and inducing sparsity by reducing some weights exactly to zero which effectively reduces model complexity and improves interpretability; mitigating overfitting by constraining the magnitude of the weights, which is especially important when data are limited or high-dimensional; and providing predictive insights by identifying the most relevant weights.34 Its disadvantages are introducing bias, which may result in underestimating certain coefficients and effects; and focusing on selective effects while discarding others, especially in nuanced multi-effect situations when the weights are closely correlated. Figure 1 shows that, for the special case of L 1 $$ {L}_1 $$ regularization, the regularization term adopts a non-strictly-convex pyramid shape that promotes sparsity by reducing some weights exactly to zero.
L   1/2 $$ {}_{\mathbf{1}/\mathbf{2}} $$ regularization or elastic net. L 1 / 2 $$ {L}_{1/2} $$ regularization, also known as elastic net, is a hybrid approach that seeks to combine the benefits of both L 1 $$ {L}_1 $$ and L 2 $$ {L}_2 $$ regularization.54 The elastic net supplements the regression (2) with two penalty terms in terms of the L 2 $$ {L}_2 $$ and L 1 $$ {L}_1 $$ norms multiplied by two independent penalty parameters α 2 $$ {\alpha}_2 $$ and α 1 $$ {\alpha}_1 $$ ,
L ( θ ; F ) = 1 n data ∑ i = 1 n data | | P ( F i , θ ) − P ^ i | | 2 + α 2 | | θ | | 2 2 + α 1 | | θ | | 1 → min with | | θ | | 2 2 = ∑ i = 1 n para | w i | 2 and | | θ | | 1 = ∑ i = 1 n para | w i | . $$ L\left(\boldsymbol{\theta}; \boldsymbol{F}\right)=\frac{1}{n_{\mathrm{data}}}\sum \limits_{i=1}^{n_{\mathrm{data}}}{\left\Vert \kern0.3em \boldsymbol{P}\Big({\boldsymbol{F}}_i,\boldsymbol{\theta} \Big)-{\hat{\boldsymbol{P}}}_i\kern0.3em \right\Vert}^2+{\alpha}_2\kern0.3em {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_2^2+{\alpha}_1\kern0.3em {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_1\to \min \kern0.5em \mathrm{with}\kern0.5em {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_2^2=\sum \limits_{i=1}^{n_{\mathrm{para}}}{\left|\kern0.3em {w}_i\kern0.3em \right|}^2\kern2.56804pt \hspace{0.3em}\mathrm{and}\kern2.56804pt \hspace{0.3em}{\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_1=\sum \limits_{i=1}^{n_{\mathrm{para}}}\mid \kern0.3em {w}_i\kern0.3em \mid . $$ ()

For α 1 = 0 $$ {\alpha}_1=0 $$ and α 2 = 0 $$ {\alpha}_2=0 $$ , it recovers the classical ridge regression and lasso as special cases. For α 1 > 0 $$ {\alpha}_1>0 $$ and α 2 > 0 $$ {\alpha}_2>0 $$ , L 1 / 2 $$ {L}_{1/2} $$ regularization shares many features with L p $$ {L}_p $$ regularization with 1 < p < 2 $$ 1<p<2 $$ and generates contours similar, but not identical to Figure 1, bottom left. However, in contrast to L p $$ {L}_p $$ regularization with 1 < p < 2 $$ 1<p<2 $$ ,30 L 1 / 2 $$ {L}_{1/2} $$ regularization not only promotes stability, but also induces sparsity while remaining convex.34 Its disadvantage is its added computational complexity. Since L 1 / 2 $$ {L}_{1/2} $$ regularization is not a special case of the L p $$ {L}_p $$ regularization family, we will not consider it further throughout this study.

L   0.5 $$ {}_{\mathbf{0.5}} $$ regularization. L p $$ {L}_p $$ regularization with powers 0 < p < 1 $$ 0<p<1 $$ has become a popular tool in subset selection since it promotes sparsity more aggressively than simple L 1 $$ {L}_1 $$ regularization. While L p $$ {L}_p $$ norms are traditionally defined for powers larger than one, p ≥ 1 $$ p\ge 1 $$ , the concept of applying powers smaller than one, 0 < p < 1 $$ 0<p<1 $$ , was introduced more than three decades ago in sparse regression of large systems.30 Notably, for powers smaller than one, the penalty term becomes non-convex and is no longer a norm in the classical sense. For the special case of p = 0 . 5 $$ p=0.5 $$ , the penalty term uses the sum of the square roots of the vector components, | |   θ   | | 0 . 5 = [   ∑ i = 1 n para | w i |   ] 2 $$ {\left\Vert\;\boldsymbol{\theta}\;\right\Vert}_{0.5}={\left[\;{\sum}_{i=1}^{n_{\mathrm{para}}}\sqrt{\mid {w}_i\mid}\;\right]}^2 $$ , and we can easily see that this construct no longer satisfies the triangle inequality. L 0 . 5 $$ {L}_{0.5} $$ regularization supplements the regression (2) with a penalty term that consists of this term, multiplied by a penalty parameter α $$ \alpha $$ ,
L ( θ ; F ) = 1 n data ∑ i = 1 n data | | P ( F i , θ ) − P ^ i | | 2 + α | | θ | | 0 . 5 0 . 5 → min with | | θ | | 0 . 5 0 . 5 = ∑ i = 1 n para | w i | . $$ L\left(\boldsymbol{\theta}; \boldsymbol{F}\right)=\frac{1}{n_{\mathrm{data}}}\sum \limits_{i=1}^{n_{\mathrm{data}}}{\left\Vert \kern0.3em \boldsymbol{P}\Big({\boldsymbol{F}}_i,\boldsymbol{\theta} \Big)-{\hat{\boldsymbol{P}}}_i\kern0.3em \right\Vert}^2+\alpha \kern0.3em {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_{0.5}^{0.5}\to \min \kern2em \mathrm{with}\kern2em {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_{0.5}^{0.5}=\sum \limits_{i=1}^{n_{\mathrm{para}}}\sqrt{\mid \kern0em {w}_i\kern0em \mid }. $$ ()
The advantages of L 0 . 5 $$ {L}_{0.5} $$ regularization, or more generally of L p $$ {L}_p $$ regularization with powers smaller than one, are enhanced sparsity potentially leading to more parsimonious models; and subset selection especially in high-dimensional datasets.55 Its disadvantages are its computational complexity induced by its non-convex nature; and its multiple local minima making subset selection complex and non-unique. Figure 1 shows that, for the special cases of L 0 . 75 $$ {L}_{0.75} $$ , L 0 . 5 $$ {L}_{0.5} $$ , and L 0 . 25 $$ {L}_{0.25} $$ regularization, the regularization term adopts an increasingly non-convex shape and promotes sparsity by more aggressively setting weights equal to zero.
L   0 $$ {}_{\mathbf{0}} $$ regularization or subset selection. L 0 $$ {L}_0 $$ regularization, is a form of subset selection that imposes a penalty on the number of non-zero parameters in a regression model. The origins of selecting a subset of relevant parameters date back to early efforts in regression modeling with the objective to discover parsimonious models with enhanced interpretation and prediction. However, formalizing this idea as an L 0 $$ {L}_0 $$ penalty method and connecting it rigorously to regularization has emerged more prominently with the advent of high-dimensional datasets.30 The L 0 $$ {L}_0 $$ norm is commonly referred to as sparse norm and is not a norm in a strict mathematical sense. It refers to the pseudo-norm, | |   θ   | | 0 = ∑ i = 1 n para I ( w i ≠ 0 ) $$ {\left\Vert\;\boldsymbol{\theta}\;\right\Vert}_0={\sum}_{i=1}^{n_{\mathrm{para}}}I\left({w}_i\ne 0\right) $$ , where I ( ∘ ) $$ I\left(\circ \right) $$ is the indicator function that is one if the condition inside the parenthesis is true and zero otherwise. As such, the L 0 $$ {L}_0 $$ norm counts the number of non-zero entries in a vector, which implies that this approach directly penalizes model complexity in terms of predictor inclusion. Notably, the L 0 $$ {L}_0 $$ norm is an explicit, discrete measure of sparsity. It is robust to outliers since it only counts the number of non-zero elements in the parameter vector and does not express preference for smaller or larger entries. L 0 $$ {L}_0 $$ regularization supplements the regression (2) with a penalty term that consists of the L 0 $$ {L}_0 $$ norm, multiplied by a penalty parameter α $$ \alpha $$ ,
L ( θ ; F ) = 1 n data ∑ i = 1 n data | | P ( F i , θ ) − P ^ i | | 2 + α | | θ | | 0 → min with | | θ | | 0 = ∑ i = 1 n para I ( w i ≠ 0 ) . $$ L\left(\boldsymbol{\theta}; \boldsymbol{F}\right)=\frac{1}{n_{\mathrm{data}}}\sum \limits_{i=1}^{n_{\mathrm{data}}}{\left\Vert \kern0.3em \boldsymbol{P}\Big({\boldsymbol{F}}_i,\boldsymbol{\theta} \Big)-{\hat{\boldsymbol{P}}}_i\kern0.3em \right\Vert}^2+\alpha \kern0.3em {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_0\to \min \kern2em \mathrm{with}\kern2em {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_0=\sum \limits_{i=1}^{n_{\mathrm{para}}}I\left({w}_i\ne 0\right). $$ ()
Its advantages are its conceptual simplicity by providing a direct mechanism for subset selection that directly penalizes non-zero parameters; reduced overfitting by promoting fewer non-zero parameters, in particular when data are limited; and enhanced model interpretability by focusing only on the relevant terms. Its disadvantages are its computational complexity that results from turning continuous model selection into an NP hard discrete combinatorial problem with 2 n $$ {2}^n $$ possible parameter combinations, making it computationally intractable for problems with large parameter sets; its non-convexity induced by the L 0 $$ {L}_0 $$ penalty term that leads to optimization challenges related to several local minima; and increased instability by discovering models for which slight changes in the data can result in an entirely different parameter set. In the contour plot of Figure 1, L 0 $$ {L}_0 $$ regularization would correspond to two discrete planes along the two parameter axes.

Predictability and interpretability. L p $$ {L}_p $$ regularization is an intricate balance between predictability and interpretability: For powers larger than one, p > 1 $$ p>1 $$ , L p $$ {L}_p $$ regularization can improve predictability, increase robustness, prevent overfitting, and enhance generalization to new data by penalizing outliers and reducing extreme coefficients. For powers equal to or smaller than one, p ≤ 1 $$ p\le 1 $$ , L p $$ {L}_p $$ regularization, can improve interpretability, promote simpler models, and identify the most influential predictors by encouraging sparsity and forcing some coefficients exactly to zero. Taken together, L p $$ {L}_p $$ regularization is a trade-off between interpretability and predictability, between simplicity and accuracy, and between bias and variance. Two hyperparameters, the power p $$ p $$ and the regularization strength α $$ \alpha $$ , allow us to fine-tuning of this balance. Throughout this manuscript, we will provide a library of systematic examples that illustrate the sensitivity of L p $$ {L}_p $$ regularization with respect to these two hyperparameters.

3 NEURAL NETWORKS

In this study, we adopt the concept of neural networks to perform regression in constitutive modeling with the objective to improve both predictability and interpretability. To demonstrate that our strategy generalizes to different types of neural networks, we compare two constitutive neural networks that have recently become popular in the context of automated model discovery.29, 50 Both networks are sparse neural networks by design, where sparsity is inspired by the underlying physics of hyperelasticity. In their input layer, they use characteristic features of the deformation gradient F $$ \boldsymbol{F} $$ to a priori satisfy the kinematic constraint of material objectivity, and acknowledge a characteristic isotropic and incompressible material behavior by satisfying the constraints of material symmetry and incompressibility.15 In their output layer, they learn a free energy function ψ $$ \psi $$ from which they derive the nominal stress P $$ \boldsymbol{P} $$ to a priori satisfy the dissipation inequality and, with it, thermodynamic consistency.10 In their hidden layers, both networks use a special set of custom-designed activation functions to a priori satisfy physical constraints3 and a particular network architecture to satisfy polyconvexity.9, 12

Invariant based and principal stretch based neural networks. We explore two types of neural networks that use different types of activation functions to represent two different classes of constitutive models: invariant and principal stretch based.56, 57 Both networks are generalizations of popular constitutive models that include widely used hyperelastic models as special cases.58-61 They are interpretable by design and their weights translate into physically meaningful parameters with physical units and a physical interpretation.17 Yet, there is a major difference between both models: the invariant model uses different functional forms for each activation function and results in a nonlinear regression problem, whereas the principal stretch model uses the same functional form with different but fixed exponents and results in a linear regression problem. This has critical implications on the convexity of the objective function, and with it, on the nature of the solution.

Data. We train our networks on both synthetic and real data from tension, compression, and shear tests. For the synthetic data, we generate stretch stress pairs for fixed parameters through forward simulation. Specifically, we calculate stresses over a wide range of tensile stretches, compressive stretches, and shear strains in ten equidistant increments, resulting in three data sets with eleven stretch-stress pairs each. For the real data, we extract stretch stress pairs from our previously published human brain experiments on 5 mm gray matter tissue cubes.62, 63 Specifically, we use the reported stresses averaged over n = 15 $$ n=15 $$ tensile, n = 17 $$ n=17 $$ compressive, and n = 35 $$ n=35 $$ shear experiments, in sixteen equidistant increments, resulting in three data sets with seventeen stretch-stress pairs each.29 All data are available on our GitHub repository, https://github.com/LivingMatterLab/CANN.

3.1 Invariant based neural network

Invariant based constitutive neural networks take the deformation gradient F $$ \boldsymbol{F} $$ as input and predict the free energy function ψ $$ \psi $$ as output from which we calculate the stress P = ∂ ψ / ∂ F $$ \boldsymbol{P}=\partial \psi /\partial \boldsymbol{F} $$ . From the deformation gradient, they extract a set of invariants, in our example I 1 $$ {I}_1 $$ and I 2 $$ {I}_2 $$ , and feed them into its two hidden layers.29, 64 The first layer generates the powers of the invariants, in our example the first and second ( ∘ ) $$ \left(\circ \right) $$ and ( ∘ ) 2 $$ {\left(\circ \right)}^2 $$ , and the second layer subjects these powers to specific functions, in our example to the identity and exponential ( ∘ ) $$ \left(\circ \right) $$ and exp ( ∘ ) $$ \exp \left(\circ \right) $$ . The free energy function ψ $$ \psi $$ is a sum of the resulting eight terms. Figure 2 illustrates the invariant based constitutive neural network with the eight functional building blocks highlighted in color, where the hot red colors relate to the first invariant and the cold blue colors to the second. During training, the network autonomously discovers the best model, out of 2 8 = 256 $$ {2}^8=256 $$ possible combinations of terms, and simultaneously learns its model parameters θ = { w i , j } $$ \boldsymbol{\theta} =\left\{\kern0.3em {w}_{i,j}\kern0.3em \right\} $$ . It minimizes the loss function (3), the difference between the stress predicted by the model P ( F i , θ ) $$ \boldsymbol{P}\left({\boldsymbol{F}}_i,\boldsymbol{\theta} \right) $$ and the experimentally measured stress P ^ i $$ {\hat{\boldsymbol{P}}}_i $$ , divided by the number of data points used for training n data $$ {n}_{\mathrm{data}} $$ ,
L ( θ ; F ) = 1 n data ∑ i = 1 n data | | P ( F i , θ ) − P ^ i | | 2 → min . $$ L\left(\boldsymbol{\theta}; \boldsymbol{F}\right)=\frac{1}{n_{\mathrm{data}}}\sum \limits_{i=1}^{n_{\mathrm{data}}}{\left\Vert \kern0.3em \boldsymbol{P}\Big({\boldsymbol{F}}_i,\boldsymbol{\theta} \Big)-{\hat{\boldsymbol{P}}}_i\kern0.3em \right\Vert}^2\to \min . $$ ()
To ensure thermodynamic consistency, the network does not learn the stress directly,16 but rather derives it from the free energy function.15 For our example in Figure 2, free energy function takes the following explicit representation,
ψ ( I 1 , I 2 ) = w 2 , 1 w 1 , 1 [ I 1 − 3 ] + w 2 , 2 [ exp ( w 1 , 2 [ I 1 − 3 ] ) − 1 ] + w 2 , 3 w 1 , 3 [ I 1 − 3 ] 2 + w 2 , 4 [ exp ( w 1 , 4 [ I 1 − 3 ] 2 ) − 1 ] + w 2 , 5 w 1 , 5 [ I 2 − 3 ] + w 2 , 6 [ exp ( w 1 , 6 [ I 2 − 3 ] ) − 1 ] + w 2 , 7 w 1 , 7 [ I 2 − 3 ] 2 + w 2 , 8 [ exp ( w 1 , 8 [ I 2 − 3 ] 2 ) − 1 ] , $$ {\displaystyle \begin{array}{ll}\hfill \psi \left({I}_1,{I}_2\right)& ={w}_{2,1}{w}_{1,1}\kern.2em \left[\kern0.1em {I}_1-3\kern0.1em \right]\kern0.38em +{w}_{2,2}\kern0.1em \left[\kern0.1em \exp \left(\kern0.1em {w}_{1,2}\kern0.1em \left[\kern0.1em {I}_1-3\kern0.1em \right]\kern0.4em \right)-1\kern0.1em \right]\\ {}\hfill & +{w}_{2,3}{w}_{1,3}\kern.2em {\left[\kern0.1em {I}_1-3\kern0.1em \right]}^2+{w}_{2,4}\kern0.1em \left[\kern0.1em \exp \left(\kern0.1em {w}_{1,4}\kern0.1em {\left[\kern0.1em {I}_1-3\kern0.1em \right]}^2\right)-1\kern0.1em \right]\\ {}\hfill & +{w}_{2,5}{w}_{1,5}\kern.2em \left[\kern0.1em {I}_2-3\kern0.1em \right]\kern0.38em +{w}_{2,6}\kern0.1em \left[\kern0.1em \exp \left(\kern0.1em {w}_{1,6}\kern0.1em \left[\kern0.1em {I}_2-3\kern0.1em \right]\kern0.4em \right)-1\kern0.1em \right]\\ {}\hfill & +{w}_{2,7}{w}_{1,7}\kern.2em {\left[\kern0.1em {I}_2-3\kern0.1em \right]}^2+{w}_{2,8}\kern0.1em \left[\kern0.1em \exp \left(\kern0.1em {w}_{1,8}\kern0.1em {\left[\kern0.1em {I}_2-3\kern0.1em \right]}^2\right)-1\kern0.1em \right],\end{array}} $$ ()
with the following derivatives with respect to the invariants I 1 $$ {I}_1 $$ and I 2 $$ {I}_2 $$ ,
∂ ψ ∂ I 1 = w 2 , 1 w 1 , 1 + w 2 , 2 w 1 , 2 exp ( w 1 , 2 [ I 1 − 3 ] ) + 2 [ I 1 − 3 ] [ w 2 , 3 w 1 , 3 + w 2 , 4 w 1 , 4 exp ( w 1 , 4 [ I 1 − 3 ] 2 ) ] ∂ ψ ∂ I 2 = w 2 , 5 w 1 , 5 + w 2 , 6 w 1 , 6 exp ( w 1 , 6 [ I 2 − 3 ] ) + 2 [ I 2 − 3 ] [ w 2 , 7 w 1 , 7 + w 2 , 8 w 1 , 8 exp ( w 1 , 8 [ I 2 − 3 ] 2 ) ] . $$ {\displaystyle \begin{array}{ll}\hfill \frac{\partial \psi }{\partial {I}_1}& ={w}_{2,1}{w}_{1,1}+{w}_{2,2}{w}_{1,2}\exp \left(\kern0.3em {w}_{1,2}\left[{I}_1-3\right]\right)+2\kern0.3em \left[{I}_1-3\right]\left[{w}_{2,3}{w}_{1,3}+{w}_{2,4}{w}_{1,4}\exp \left(\kern0.3em {w}_{1,4}{\left[{I}_1-3\right]}^2\right)\right]\\ {}\hfill \frac{\partial \psi }{\partial {I}_2}& ={w}_{2,5}{w}_{1,5}+{w}_{2,6}{w}_{1,6}\exp \left(\kern0.3em {w}_{1,6}\left[{I}_2-3\right]\right)+2\kern0.3em \left[{I}_2-3\right]\left[{w}_{2,7}{w}_{1,7}+{w}_{2,8}{w}_{1,8}\exp \left(\kern0.3em {w}_{1,8}{\left[{I}_2-3\right]}^2\right)\right].\end{array}} $$ ()
Using the second law of thermodynamics, we can derive the Piola stress, P = ∂ ψ / ∂ F $$ \boldsymbol{P}=\partial \psi /\partial \boldsymbol{F} $$ , as thermodynamically conjugate to the deformation gradient F $$ \boldsymbol{F} $$ ,56
P = ∂ ψ ∂ I 1 · ∂ I 1 ∂ F + ∂ ψ ∂ I 2 · ∂ I 2 ∂ F − p F − t , $$ \boldsymbol{P}=\frac{\partial \psi }{\partial {I}_1}\cdotp \frac{\partial {I}_1}{\partial \boldsymbol{F}}+\frac{\partial \psi }{\partial {I}_2}\cdotp \frac{\partial {I}_2}{\partial \boldsymbol{F}}-p\kern0.3em {\boldsymbol{F}}^{-\mathrm{t}}, $$ ()
where the term p   F − t $$ p\;{\boldsymbol{F}}^{-\mathrm{t}} $$ ensures perfect incompressibility in terms of the pressure p $$ p $$ that we determine from the boundary conditions. For the network free energy (12), the Piola stress is
P = [ w 2 , 1 w 1 , 1 + w 2 , 2 w 1 , 2 exp ( w 1 , 2 [ I 1 − 3 ] ) + 2 [ I 1 − 3 ] [ w 2 , 3 w 1 , 3 + w 2 , 4 w 1 , 4 exp ( w 1 , 4 [ I 1 − 3 ] 2 ) ] ∂ I 1 / ∂ F + [ w 2 , 5 w 1 , 5 + w 2 , 6 w 1 , 6 exp ( w 1 , 6 [ I 2 − 3 ] ) + 2 [ I 2 − 3 ] [ w 2 , 7 w 1 , 7 + w 2 , 8 w 1 , 8 exp ( w 1 , 8 [ I 2 − 3 ] 2 ) ] ∂ I 2 / ∂ F − p F − t . $$ {\displaystyle \begin{array}{llllllll}\boldsymbol{P}& =& \Big[& \kern-.5em {w}_{2,1}{w}_{1,1}& +& \kern-.5em {w}_{2,2}{w}_{1,2}& \exp \Big(\kern0.1em {w}_{1,2}& \left[{I}_1-3\right]\kern0.4em \Big)\\ {}& +& 2\kern0.3em \left[\kern0.3em {I}_1-3\kern0.3em \right]\Big[& \kern-.5em {w}_{2,3}{w}_{1,3}& +& \kern-.5em {w}_{2,4}{w}_{1,4}& \exp \Big(\kern0.1em {w}_{1,4}& {\left[{I}_1-3\right]}^2\left)\right]\kern0.2em \partial {I}_1/\partial \boldsymbol{F}\\ {}& +& \Big[& \kern-.5em {w}_{2,5}{w}_{1,5}& +& \kern-.5em {w}_{2,6}{w}_{1,6}& \exp \Big(\kern0.1em {w}_{1,6}& \left[{I}_2-3\right]\kern0.4em \Big)\\ {}& +& 2\kern0.3em \left[\kern0.3em {I}_2-3\kern0.3em \right]\Big[& \kern-.5em {w}_{2,7}{w}_{1,7}& +& \kern-.5em {w}_{2,8}{w}_{1,8}& \exp \Big(\kern0.1em {w}_{1,8}& {\left[{I}_2-3\right]}^2\left)\right]\kern0.2em \partial {I}_2/\partial \boldsymbol{F}-p\kern0.3em {\boldsymbol{F}}^{-\mathrm{t}}\kern0.3em .\end{array}} $$ ()
Notably, the Piola stress of the invariant based network (15) is a nonlinear function in the network weights w i , j $$ {w}_{i,j} $$ , which translates the loss function (11) into a nonlinear regression problem, with possibly multiple local minima. For this particular format, one of the first two weights of each row becomes redundant, and we can reduce the set of network parameters to twelve, θ = [ ( w 1 , 1 w 2 , 1 ) , w 1 , 2 , w 2 , 2 , ( w 1 , 3 w 2 , 3 ) , w 1 , 4 , w 2 , 4 ( w 1 , 5 w 2 , 5 ) , w 1 , 6 , w 2 , 6 , ( w 1 , 7 w 2 , 7 ) , w 1 , 8 , w 2 , 8 ] $$ \boldsymbol{\theta} =\left[\kern0.3em \left({w}_{1,1}{w}_{2,1}\right),{w}_{1,2},{w}_{2,2},\left({w}_{1,3}{w}_{2,3}\right),{w}_{1,4},{w}_{2,4}\left({w}_{1,5}{w}_{2,5}\right),{w}_{1,6},{w}_{2,6},\left({w}_{1,7}{w}_{2,7}\right),{w}_{1,8},{w}_{2,8}\kern0.3em \right] $$ . We train our invariant based network with tension, compression, and shear data and rewrite the loss function (11) in terms of two contributions that minimize the error between the normal and shear stresses predicted by the model, P 11 ( λ i ) $$ {P}_{11}\left({\lambda}_i\right) $$ and P 12 ( γ i ) $$ {P}_{12}\left({\gamma}_i\right) $$ , and the data, P ^ 11 , i $$ {\hat{P}}_{11,i} $$ and P ^ 12 , i $$ {\hat{P}}_{12,i} $$ , where n 11 $$ {n}_{11} $$ and n 12 $$ {n}_{12} $$ denote the different stretch and shear levels λ $$ \lambda $$ and γ $$ \gamma $$ ,
L ( θ ; λ , γ ) = 1 n 11 ∑ i = 1 n 11 | | P 11 ( λ i ) − P ^ 11 , i | | 2 + 1 n 12 ∑ i = 1 n 12 | | P 12 ( γ i ) − P ^ 12 , i | | 2 → min . $$ L\left(\boldsymbol{\theta}; \lambda, \gamma \right)=\frac{1}{n_{11}}\sum \limits_{i=1}^{n_{11}}{\left\Vert \kern0.3em {P}_{11}\left({\lambda}_i\right)-{\hat{P}}_{11,i}\kern0.3em \right\Vert}^2+\frac{1}{n_{12}}\sum \limits_{i=1}^{n_{12}}{\left\Vert \kern0.3em {P}_{12}\left({\gamma}_i\right)-{\hat{P}}_{12,i}\kern0.3em \right\Vert}^2\to \min . $$ ()
To explore the effect of scaling of the three individual stress terms, alternatively, we weigh all three experiments equally, and also train the network by minimizing the error between the normalized tensile, compressive, and shear stresses predicted by the model P ten ( λ i ) $$ {P}_{\mathrm{ten}}\left({\lambda}_i\right) $$ , P com ( λ i ) $$ {P}_{\mathrm{com}}\left({\lambda}_i\right) $$ , and P shr ( γ i ) $$ {P}_{\mathrm{shr}}\left({\gamma}_i\right) $$ , and the data P ^ ten , i $$ {\hat{P}}_{\mathrm{ten},i} $$ , P ^ com , i $$ {\hat{P}}_{\mathrm{com},i} $$ , and P ^ shr , i $$ {\hat{P}}_{\mathrm{shr},i} $$ , normalized by the maximum recorded tensile, compressive, and shear stresses, P ^ ten max $$ {\hat{P}}_{\mathrm{ten}}^{\mathrm{max}} $$ , P ^ com min $$ {\hat{P}}_{\mathrm{com}}^{\mathrm{min}} $$ , and P ^ shr max $$ {\hat{P}}_{\mathrm{shr}}^{\mathrm{max}} $$ , where n ten $$ {n}_{\mathrm{ten}} $$ , n com $$ {n}_{\mathrm{com}} $$ , and n shr $$ {n}_{\mathrm{shr}} $$ denote the different stretch and shear levels λ $$ \lambda $$ and γ $$ \gamma $$ ,
L ( θ ; λ , γ ) = 1 n ten ∑ i = 1 n ten P ten ( λ i ) − P ^ ten , i P ^ ten max 2 + 1 n com ∑ i = 1 n com P com ( λ i ) − P ^ com , i P ^ com min 2 + 1 n shr ∑ i = 1 n shr P shr ( γ i ) − P ^ shr , i P ^ shr max 2 → min . $$ L\left(\boldsymbol{\theta}; \lambda, \gamma \right)=\frac{1}{n_{\mathrm{ten}}}\sum \limits_{i=1}^{n_{\mathrm{ten}}}{\left|\left|\kern0.3em \frac{P_{\mathrm{ten}}\left({\lambda}_i\right)-{\hat{P}}_{\mathrm{ten},i}}{{\hat{P}}_{\mathrm{ten}}^{\mathrm{max}}}\right|\right|}^2+\frac{1}{n_{\mathrm{com}}}\sum \limits_{i=1}^{n_{\mathrm{com}}}{\left|\left|\kern0.3em \frac{P_{\mathrm{com}}\left({\lambda}_i\right)-{\hat{P}}_{\mathrm{com},i}}{{\hat{P}}_{\mathrm{com}}^{\mathrm{min}}}\right|\right|}^2+\frac{1}{n_{\mathrm{shr}}}\sum \limits_{i=1}^{n_{\mathrm{shr}}}{\left|\left|\kern0.3em \frac{P_{\mathrm{shr}}\left({\gamma}_i\right)-{\hat{P}}_{\mathrm{shr},i}}{{\hat{P}}_{\mathrm{shr}}^{\mathrm{max}}}\right|\right|}^2\to \min . $$ ()
Below, we briefly derive the explicit analytical expressions for the Piola stresses P 11 ( λ ) $$ {P}_{11}\left(\lambda \right) $$ in uniaxial tension and compression and P 12 ( γ ) $$ {P}_{12}\left(\gamma \right) $$ in simple shear, such that the tensile stress is P ten = P 11 $$ {P}_{\mathrm{ten}}={P}_{11} $$ for λ > 1 $$ \lambda >1 $$ , the compressive stress is P com = P 11 $$ {P}_{\mathrm{com}}={P}_{11} $$ for λ < 1 $$ \lambda <1 $$ , and the shear stress is P shr = P 12 $$ {P}_{\mathrm{shr}}={P}_{12} $$ for all γ $$ \gamma $$ .
Details are in the caption following the image
FIGURE 2
Open in figure viewerPowerPoint
Invariant based neural network for automated model discovery. The network takes the deformation gradient F $$ \boldsymbol{F} $$ as input and outputs the free energy function ψ $$ \psi $$ from which we calculate the stress P = ∂ ψ / ∂ F $$ \boldsymbol{P}=\partial \psi /\partial \boldsymbol{F} $$ . The network is invariant based, it first calculates the invariants I 1 $$ {I}_1 $$ and I 2 $$ {I}_2 $$ , and feeds them into its two hidden layers. The first layer generates the first and second powers ( ∘ ) $$ \left(\circ \right) $$ and ( ∘ ) 2 $$ {\left(\circ \right)}^2 $$ of the invariants and the second layer applies the identity and exponential function ( ∘ ) $$ \left(\circ \right) $$ and exp ( ∘ ) $$ \exp \left(\circ \right) $$ to these powers. The free energy function ψ $$ \psi $$ is a function of the eight color-coded terms. During training, the network discovers the best model, of 2 8 = 256 $$ {2}^8=256 $$ possible combinations of terms, to explain the experimental data P ^ $$ \hat{\boldsymbol{P}} $$ .
Uniaxial tension and compression. For the special case of uniaxial tension and compression in terms of the stretch λ $$ \lambda $$ , with λ 1 = λ $$ {\lambda}_1=\lambda $$ and λ 2 = λ − 1 / 2 $$ {\lambda}_2={\lambda}^{-1/2} $$ and λ 3 = λ − 1 / 2 $$ {\lambda}_3={\lambda}^{-1/2} $$ , the invariants take the following form,
I 1 = λ 2 + 2 λ and I 2 = 2 λ + 1 λ 2 and I 3 = 1 . $$ {I}_1={\lambda}^2+\frac{2}{\lambda}\kern1em \mathrm{and}\kern1em {I}_2=2\lambda +\frac{1}{\lambda^2}\kern1em \mathrm{and}\kern1em {I}_3=1. $$ ()
Using Equation (14) and the zero normal stress condition, P 22 = P 33 = 0 $$ {P}_{22}={P}_{33}=0 $$ , we obtain the following expression for the uniaxial stress stretch relation,
P 11 = 2 ∂ ψ ∂ I 1 + 1 λ ∂ ψ ∂ I 2 λ − 1 λ 2 , $$ {P}_{11}=2\kern0.3em \left[\frac{\partial \psi }{\partial {I}_1}+\frac{1}{\lambda}\frac{\partial \psi }{\partial {I}_2}\right]\left[\lambda -\frac{1}{\lambda^2}\right], $$ ()
which translates into the following explicit expression between our network stress P 11 $$ {P}_{11} $$ and the uniaxial stretch λ $$ \lambda $$ ,
P 11 = 2 [ [ w 2 , 1 w 1 , 1 + w 2 , 2 w 1 , 2 exp ( w 1 , 2 [ I 1 − 3 ] ) + 2 [ I 1 − 3 ] [ w 2 , 3 w 1 , 3 + w 2 , 4 w 1 , 4 exp ( w 1 , 4 [ I 1 − 3 ] 2 ) ] + 1 λ [ w 2 , 5 w 1 , 5 + w 2 , 6 w 1 , 6 exp ( w 1 , 6 [ I 2 − 3 ] ) + 2 [ I 2 − 3 ] [ w 2 , 7 w 1 , 7 + w 2 , 8 w 1 , 8 exp ( w 1 , 8 [ I 2 − 3 ] 2 ) ] ] λ − 1 λ 2 . $$ {\displaystyle \begin{array}{cc}{P}_{11}& =2\kern0.1em \left[\right[{w}_{2,1}{w}_{1,1}+{w}_{2,2}{w}_{1,2}\exp \left(\kern0.1em {w}_{1,2}\left[{I}_1-3\right]\right)+2\kern0.3em \left[{I}_1-3\right]\left[{w}_{2,3}{w}_{1,3}+{w}_{2,4}{w}_{1,4}\exp \left(\kern0.1em {w}_{1,4}{\left[{I}_1-3\right]}^2\right)\right]\\ {}& +\frac{1}{\lambda}\kern.3em \left[{w}_{2,5}{w}_{1,5}+{w}_{2,6}{w}_{1,6}\exp \left(\kern0.1em {w}_{1,6}\left[{I}_2-3\right]\right)+2\kern0.3em \left[{I}_2-3\right]\left[{w}_{2,7}{w}_{1,7}+{w}_{2,8}{w}_{1,8}\exp \left(\kern0.1em {w}_{1,8}{\left[{I}_2-3\right]}^2\right)\right]\right]\kern0.3em \left[\lambda -\frac{1}{\lambda^2}\right]\kern0.3em .\end{array}} $$ ()
Simple shear. For the special case of simple shear, in terms of the shear γ $$ \gamma $$ , with λ 1 = 1 2 [ + γ + 4 + γ 2 ] $$ {\lambda}_1=\frac{1}{2}\kern0.3em \left[\kern0.3em +\gamma +\sqrt{\kern0.3em 4+{\gamma}^2}\kern0.3em \right] $$ and λ 2 = 1 2 [ − γ + 4 + γ 2 ] $$ {\lambda}_2=\frac{1}{2}\kern0.3em \left[\kern0.3em -\gamma +\sqrt{\kern0.3em 4+{\gamma}^2}\kern0.3em \right] $$ and λ 3 = 1 $$ {\lambda}_3=1 $$ , the invariants take the following form,
I 1 = 3 + γ 2 and I 2 = 3 + γ 2 and I 3 = 1 . $$ {I}_1=3+{\gamma}^2\kern1em \mathrm{and}\kern1em {I}_2=3+{\gamma}^2\kern1em \mathrm{and}\kern1em {I}_3=1. $$ ()
Using Equation (14), we obtain the following shear stress stretch relation,
P 12 = 2 ∂ ψ ∂ I 1 + ∂ ψ ∂ I 2 γ , $$ {P}_{12}=2\kern0.3em \left[\frac{\partial \psi }{\partial {I}_1}+\frac{\partial \psi }{\partial {I}_2}\right]\gamma, $$ ()
which translates into the following explicit expression between our network shear stress P 12 $$ {P}_{12} $$ and the shear strain γ $$ \gamma $$ ,
P 12 = 2 [ w 2 , 1 w 1 , 1 + w 2 , 2 w 1 , 2 exp ( w 1 , 2 [ I 1 − 3 ] ) + 2 [ I 1 − 3 ] [ w 2 , 3 w 1 , 3 + w 2 , 4 w 1 , 4 exp ( w 1 , 4 [ I 1 − 3 ] 2 ) + w 2 , 5 w 1 , 5 + w 2 , 6 w 1 , 6 exp ( w 1 , 6 [ I 2 − 3 ] ) + 2 [ I 2 − 3 ] [ w 2 , 7 w 1 , 7 + w 2 , 8 w 1 , 8 exp ( w 1 , 8 [ I 2 − 3 ] 2 ) ] γ . $$ {\displaystyle \begin{array}{cc}{P}_{12}& =2\kern0.3em \left[{w}_{2,1}{w}_{1,1}+{w}_{2,2}{w}_{1,2}\exp \left(\kern0.3em {w}_{1,2}\left[{I}_1-3\right]\right)+2\kern0.3em \left[{I}_1-3\right]\right[{w}_{2,3}{w}_{1,3}+{w}_{2,4}{w}_{1,4}\exp \left(\kern0.3em {w}_{1,4}{\left[{I}_1-3\right]}^2\right)\\ {}& \kern1.25em +{w}_{2,5}{w}_{1,5}+{w}_{2,6}{w}_{1,6}\exp \left(\kern0.3em {w}_{1,6}\left[{I}_2-3\right]\right)+2\kern0.3em \left[{I}_2-3\right]\left[{w}_{2,7}{w}_{1,7}+{w}_{2,8}{w}_{1,8}\exp \left(\kern0.3em {w}_{1,8}{\left[{I}_2-3\right]}^2\right)\right]\kern0.3em \gamma \kern0.3em .\end{array}} $$ ()
Figure 3 illustrates the contours of the loss function L ( θ ; λ , γ ) $$ L\left(\boldsymbol{\theta}; \lambda, \gamma \right) $$ for all possible two-term models of the invariant based network in Figure 2. By combining any two terms of the model and setting all other weights equal to zero, we can generate 28 possible models. For these 28 combinations of two terms, we evaluate two versions of the loss function, non-normalized from Equation (16) and normalized from Equation (17), using the invariant based definitions of the normal stress (20) and shear stress (23). First, we generate synthetic data, P ^ 11 , i $$ {\hat{P}}_{11,i} $$ and P ^ 12 , i $$ {\hat{P}}_{12,i} $$ , for tensile stretches of λ = [ 1 . 0 , … , 2 . 0 ] $$ \lambda =\left[1.0,\dots, 2.0\right] $$ , compressive stretches of λ = [ 1 . 0 , … , 0 . 5 ] $$ \lambda =\left[1.0,\dots, 0.5\right] $$ , and shear strains of γ = [ 0 . 0 , … , 0 . 5 ] $$ \gamma =\left[0.0,\dots, 0.5\right] $$ , in ten equidistant increments each, assuming an exact solution with fixed weights of the first layer, w 1 , 1 = w 1 , 3 = w 1 , 5 = w 1 , 7 = 1 . 0 $$ {w}_{1,1}={w}_{1,3}={w}_{1,5}={w}_{1,7}=1.0 $$ and w 1 , 2 = w 1 , 4 = w 1 , 6 = w 1 , 8 = 0 . 25 $$ {w}_{1,2}={w}_{1,4}={w}_{1,6}={w}_{1,8}=0.25 $$ , and a pair of non-zero weights of the second layer, w 2 , i = 1 $$ {w}_{2,i}=1 $$ and w 2 , j = 1 $$ {w}_{2,j}=1 $$ , while fixing the remaining six weights of the second layer equal to zero. This results in 28 training data sets of eleven stretch-stress pairs each, for tension, compression, and shear. Second, we vary the two non-zero network weights in the ranges w i = [ 0 , … , 2 ] $$ {w}_i=\left[0,\dots, 2\right] $$ and w j = [ 0 , … , 2 ] $$ {w}_j=\left[0,\dots, 2\right] $$ . For each pair of weights, we evaluate the normal and shear stresses P 11 ( λ i ) $$ {P}_{11}\left({\lambda}_i\right) $$ and P 12 ( γ i ) $$ {P}_{12}\left({\gamma}_i\right) $$ using Equations (20) and (23), and extract the tensile, compressive, and shear stresses, P te n i ( λ ) $$ {P}_{\mathrm{te}{\mathrm{n}}_{\mathrm{i}}}\left(\lambda \right) $$ , P co m i ( λ ) $$ {P}_{\mathrm{co}{\mathrm{m}}_{\mathrm{i}}}\left(\lambda \right) $$ , and P sh r i ( γ ) $$ {P}_{\mathrm{sh}{\mathrm{r}}_{\mathrm{i}}}\left(\gamma \right) $$ . Third, we evaluate the non-normalized loss function (16) as the mean squared error between the model stresses P 11 ( λ i ) $$ {P}_{11}\left({\lambda}_i\right) $$ and P 12 ( γ i ) $$ {P}_{12}\left({\gamma}_i\right) $$ and the synthetically generated data stresses P ^ 11 , i $$ {\hat{P}}_{11,i} $$ and P ^ 12 , i $$ {\hat{P}}_{12,i} $$ , and plot its contours for each pair of weights w i $$ {w}_i $$ and w j $$ {w}_j $$ in the lower triangle. Next, we evaluate the normalized loss function (17) as the normalized mean squared error between the model stresses P ten ( λ i ) $$ {P}_{\mathrm{ten}}\left({\lambda}_i\right) $$ , P com ( λ i ) $$ {P}_{\mathrm{com}}\left({\lambda}_i\right) $$ , and P shr ( γ i ) $$ {P}_{\mathrm{shr}}\left({\gamma}_i\right) $$ , and the synthetically generated data stresses P ^ ten , i $$ {\hat{P}}_{\mathrm{ten},i} $$ , P ^ com , i $$ {\hat{P}}_{\mathrm{com},i} $$ , and P ^ shr , i $$ {\hat{P}}_{\mathrm{shr},i} $$ , and plot its contours for each pair of weights w i $$ {w}_i $$ and w j $$ {w}_j $$ in the upper triangle.
Details are in the caption following the image
FIGURE 3
Open in figure viewerPowerPoint
Loss functions of invariant based neural network. Contours of the loss function L ( θ ; λ , γ ) $$ L\left(\boldsymbol{\theta}; \lambda, \gamma \right) $$ for all 28 possible two-term models of the invariant based constitutive neural network in Figure 2. The loss function is evaluated across tensile stretches λ = [ 1 . 0 , … , 2 . 0 ] $$ \lambda =\left[1.0,\dots, 2.0\right] $$ , compressive stretches λ = [ 1 . 0 , … , 0 . 5 ] $$ \lambda =\left[1.0,\dots, 0.5\right] $$ , and shear strains γ = [ 0 . 0 , … , 0 . 5 ] $$ \gamma =\left[0.0,\dots, 0.5\right] $$ , with network weights in the ranges w i = [ 0 , … , 2 ] $$ {w}_i=\left[0,\dots, 2\right] $$ and w j = [ 0 , … , 2 ] $$ {w}_j=\left[0,\dots, 2\right] $$ . The minimum of the loss function indicates the exact solution w i = 1 $$ {w}_i=1 $$ and w j = 1 $$ {w}_j=1 $$ , represented through the white circle. The lower triangle illustrates the non-normalized loss function (16), the upper triangle illustrates the normalized loss function (17). All loss functions are convex, with contours varying from ellipsoids to valleys with long ridges, highlighting the collinearity of some w i $$ {w}_i $$ and w j $$ {w}_j $$ pairs.

Each loss function takes a minimum of L ( θ ; λ , γ ) = 0 $$ L\left(\boldsymbol{\theta}; \lambda, \gamma \right)=0 $$ for the exact solution, w i = 1 $$ {w}_i=1 $$ and w j = 1 $$ {w}_j=1 $$ , indicated through the white circles. From this minimum, both versions of the loss function, non-normalized and normalized, increase with both, decreasing and increasing weights w i $$ {w}_i $$ and w j $$ {w}_j $$ , and remain convex within the entire domain, for all 28 two-term models. Yet, the contours of the loss function vary significantly for different pairs of weights w i $$ {w}_i $$ and w j $$ {w}_j $$ , indicating its sensitivity with respect to the individual terms: some pairs of weights generate loss functions of ellipsoidal shape, for example, the {   w 2 , w 3   } $$ \left\{\;{w}_2,{w}_3\;\right\} $$ , {   w 2 , w 7   } $$ \left\{\;{w}_2,{w}_7\;\right\} $$ , {   w 3 , w 6   } $$ \left\{\;{w}_3,{w}_6\;\right\} $$ , {   w 6 , w 7   } $$ \left\{\;{w}_6,{w}_7\;\right\} $$ pairs in the normalized upper triangle, suggesting that in the studied stretch and shear range, these terms are non-collinear, and would represent a rich base for a potential constitutive model. Other pairs of weights generate loss functions with long ridges parallel to the parameter axes, for example, the {   w 2 , w 7   } $$ \left\{\;{w}_2,{w}_7\;\right\} $$ , {   w 2 , w 8   } $$ \left\{\;{w}_2,{w}_8\;\right\} $$ , {   w 6 , w 7   } $$ \left\{\;{w}_6,{w}_7\;\right\} $$ , {   w 6 , w 8   } $$ \left\{\;{w}_6,{w}_8\;\right\} $$ pairs in the non-normalized lower triangle, suggesting that these terms are almost collinear and not well suited as an independent base for a constitutive model. On average, the normalized pairs in the upper triangle seem to generate more convex loss functions than the non-normalized pairs in the lower triangle, suggesting that normalization helps to generate more convex loss functions, a richer functional base, and a more robust solution overall. Notably, the {   w 1 , w 5   } $$ \left\{\;{w}_1,{w}_5\;\right\} $$ model in the first row and fifth column and the {   w 5 , w 1   } $$ \left\{\;{w}_5,{w}_1\;\right\} $$ model in the fifth row and first column combine the linear terms in the first and second invariants, [   I 1 − 3   ] $$ \left[\;{I}_1-3\;\right] $$ and [   I 2 − 3   ] $$ \left[\;{I}_2-3\;\right] $$ , and represent the popular Mooney Rivlin model for rubber-like materials.59, 60 Overall, this simple example only illustrates the 28 two-term models out of a total set of all 256 possible models, only considers a limited stretch and shear range λ $$ \lambda $$ and γ $$ \gamma $$ , and only screens a narrow window of parameter ranges w i $$ {w}_i $$ . Even within these limitations, the contours of loss functions are rather difficult to interpret, making it difficult to comprehend the full potential of the entire network, even though it only consists of eight distinct terms.

3.2 Principal stretch based neural network

Principal stretch based constitutive neural networks take the deformation gradient F $$ \boldsymbol{F} $$ as input and predict the free energy function ψ $$ \psi $$ as output from which we calculate the stress P = ∂ ψ / ∂ F $$ \boldsymbol{P}=\partial \psi /\partial \boldsymbol{F} $$ . From the deformation gradient, they extract the principal stretches, λ 1 $$ {\lambda}_1 $$ , λ 2 $$ {\lambda}_2 $$ , and λ 3 $$ {\lambda}_3 $$ , and feed them into the hidden layer.50, 65 The hidden layer applies eight different exponents ( λ 1 n + λ 2 n + λ 3 n − 3 ) $$ \left({\lambda}_1^n+{\lambda}_2^n+{\lambda}_3^n-3\right) $$ to these stretches. The free energy function ψ $$ \psi $$ is a sum of the resulting eight terms. Figure 4 illustrates the principal stretch based constitutive neural network with the eight functional building blocks highlighted in color, where the dark red and green terms are identical to the dark red and green terms of the invariant based network in Figure 2, while the other six terms are different. During training, the network autonomously discovers the best model, out of 2 8 = 256 $$ {2}^8=256 $$ possible combinations of terms, and simultaneously learns its model parameters θ = { w i } $$ \boldsymbol{\theta} =\left\{\kern0.3em {w}_i\kern0.3em \right\} $$ . It minimizes the loss function (3), the difference between the stress predicted by the model P ( F i , θ ) $$ \boldsymbol{P}\left({\boldsymbol{F}}_i,\boldsymbol{\theta} \right) $$ and the experimentally measured stress P ^ i $$ {\hat{\boldsymbol{P}}}_i $$ , divided by the number of data points used for training n data $$ {n}_{\mathrm{data}} $$ ,
L ( θ ; F ) = 1 n data ∑ i = 1 n data | | P ( F i , θ ) − P ^ i | | 2 → min . $$ L\left(\boldsymbol{\theta}; \boldsymbol{F}\right)=\frac{1}{n_{\mathrm{data}}}\sum \limits_{i=1}^{n_{\mathrm{data}}}{\left\Vert \kern0.3em \boldsymbol{P}\Big({\boldsymbol{F}}_i,\boldsymbol{\theta} \Big)-{\hat{\boldsymbol{P}}}_i\kern0.3em \right\Vert}^2\to \min . $$ ()
The free energy of the principal stretch based model takes the following explicit representation,57, 66 ψ ( λ 1 , λ 2 , λ 3 ) = ∑ i = 1 n term w i   [   λ 1 α i + λ 2 α i + λ 3 α i − 3   ] $$ \psi \left({\lambda}_1,{\lambda}_2,{\lambda}_3\right)={\sum}_{i=1}^{n_{\mathrm{term}}}{w}_i\;\left[\;{\lambda}_1^{\alpha_i}+{\lambda}_2^{\alpha_i}+{\lambda}_3^{\alpha_i}-3\;\right] $$ , where the individual weights w i = μ i / α i $$ {w}_i={\mu}_i/{\alpha}_i $$ correspond to the shear moduli μ i $$ {\mu}_i $$ divided by the exponents α i $$ {\alpha}_i $$ . For the eight-term model in Figure 4, we fix these exponents to α = [ + 2 , + 4 , + 6 , + 8 , − 2 , − 4 , − 6 − 8 ] $$ \alpha =\left[+2,+4,+6,+8,-2,-4,-6-8\kern0.3em \right] $$ , such that the free energy becomes a sum of the following n term = 8 $$ {n}_{\mathrm{term}}=8 $$ terms,
ψ ( λ 1 , λ 2 , λ 3 ) = ∑ i = 1 3 w 1 [ λ i + 2 − 1 ] + w 2 [ λ i + 4 − 1 ] + w 3 [ λ i + 6 − 1 ] + w 4 [ λ i + 8 − 1 ] + w 5 [ λ i − 2 − 1 ] + w 6 [ λ i − 4 − 1 ] + w 7 [ λ i − 6 − 1 ] + w 8 [ λ i − 8 − 1 ] . $$ {\displaystyle \begin{array}{cc}\psi \left({\lambda}_1,{\lambda}_2,{\lambda}_3\right)=\sum \limits_{i=1}^3& \kern0.2em {w}_1\kern0.2em \left[\kern0.1em {\lambda}_i^{+2}-1\kern0.1em \right]+{w}_2\kern0.2em \left[\kern0.1em {\lambda}_i^{+4}-1\kern0.1em \right]+{w}_3\kern0.2em \left[\kern0.1em {\lambda}_i^{+6}-1\kern0.1em \right]+{w}_4\kern0.2em \left[\kern0.1em {\lambda}_i^{+8}-1\kern0.1em \right]\\ {}+& \kern0.2em {w}_5\kern0.2em \left[\kern0.1em {\lambda}_i^{-2}-1\kern0.1em \right]+{w}_6\kern0.2em \left[\kern0.1em {\lambda}_i^{-4}-1\kern0.1em \right]+{w}_7\kern0.2em \left[\kern0.1em {\lambda}_i^{-6}-1\kern0.1em \right]+{w}_8\kern0.2em \left[\kern0.1em {\lambda}_i^{-8}-1\kern0.1em \right].\end{array}} $$ ()
and its derivatives with respect to the principal stretches λ i $$ {\lambda}_i $$ takes the following form,
∂ ψ ∂ λ i = 2 w 1 λ i + 1 + 4 w 2 λ i + 3 + 6 w 3 λ i + 5 + 8 w 4 λ i + 7 − 2 w 5 λ i − 3 − 4 w 6 λ i − 5 − 6 w 7 λ i − 7 − 8 w 8 λ i − 9 . $$ \frac{\partial \psi }{\partial {\lambda}_i}=2\kern0.3em {w}_1{\lambda}_i^{+1}+4\kern0.3em {w}_2{\lambda}_i^{+3}+6\kern0.3em {w}_3{\lambda}_i^{+5}+8\kern0.3em {w}_4{\lambda}_i^{+7}-2\kern0.3em {w}_5{\lambda}_i^{-3}-4\kern0.3em {w}_6{\lambda}_i^{-5}-6\kern0.3em {w}_7{\lambda}_i^{-7}-8\kern0.3em {w}_8{\lambda}_i^{-9}. $$ ()
Using the second law of thermodynamics, we can derive the Piola stress, P = ∂ ψ / ∂ F $$ \boldsymbol{P}=\partial \psi /\partial \boldsymbol{F} $$ , as thermodynamically conjugate to the deformation gradient F $$ \boldsymbol{F} $$ ,57
P = ∑ i = 1 3 ∂ ψ λ i n i ⊗ N i − p F − t , $$ \boldsymbol{P}=\sum \limits_{i=1}^3\frac{\partial \psi }{\lambda_i}{\boldsymbol{n}}_i\otimes {\boldsymbol{N}}_i-p\kern0.3em {\boldsymbol{F}}^{-\mathrm{t}}, $$ ()
where N i $$ {\boldsymbol{N}}_i $$ and n i = F · N i $$ {\boldsymbol{n}}_i=\boldsymbol{F}\cdotp {\boldsymbol{N}}_i $$ are the eigenvectors in the undeformed and deformed configurations, and the term p   F − t $$ p\;{\boldsymbol{F}}^{-\mathrm{t}} $$ ensures perfect incompressibilty in terms of the pressure p $$ p $$ that we determine from the boundary conditions. For the network free energy (25), the Piola stress is
P = ∑ i = 1 3 [ 2 w 1 [ λ i + 1 − 1 ] + 4 w 2 [ λ i + 3 − 1 ] + 6 w 3 [ λ i + 5 − 1 ] + 8 w 4 [ λ i + 7 − 1 ] − 2 w 5 [ λ i − 3 − 1 ] − 4 w 6 [ λ i − 5 − 1 ] − 6 w 7 [ λ i − 7 − 1 ] − 8 w 8 [ λ i − 9 − 1 ] ] n i ⊗ N i − p F − t , $$ {\displaystyle \begin{array}{cc}\boldsymbol{P}& =\sum \limits_{i=1}^3\kern0.1em \Big[\kern0.1em 2\kern0.2em {w}_1\kern0.2em \left[\kern0.1em {\lambda}_i^{+1}-1\kern0.1em \right]+4\kern0.2em {w}_2\kern0.2em \left[\kern0.1em {\lambda}_i^{+3}-1\kern0.1em \right]+6\kern0.2em {w}_3\kern0.2em \left[\kern0.1em {\lambda}_i^{+5}-1\kern0.1em \right]+8\kern0.2em {w}_4\kern0.2em \left[\kern0.1em {\lambda}_i^{+7}-1\kern0.1em \right]\\ {}& \kern1.95em -2\kern0.2em {w}_5\kern0.2em \left[\kern0.1em {\lambda}_i^{-3}-1\kern0.1em \right]-4\kern0.2em {w}_6\kern0.2em \left[\kern0.1em {\lambda}_i^{-5}-1\kern0.1em \right]-6\kern0.2em {w}_7\kern0.2em \left[\kern0.1em {\lambda}_i^{-7}-1\kern0.1em \right]-8\kern0.2em {w}_8\kern0.2em \left[\kern0.1em {\lambda}_i^{-9}-1\kern0.1em \right]\kern0.1em \Big]\kern0.1em {\boldsymbol{n}}_i\otimes {\boldsymbol{N}}_i-p\kern0.1em {\boldsymbol{F}}^{-\mathrm{t}},\end{array}} $$ ()
parameterized in terms of eight network weights, θ = [ w 1 , w 2 , w 3 , w 4 , w 5 , w 6 , w 7 , w 8 ] $$ \boldsymbol{\theta} =\left[\kern0.3em {w}_1,{w}_2,{w}_3,{w}_4,{w}_5,{w}_6,{w}_7,{w}_8\kern0.3em \right] $$ . Notably, the Piola stress of the principal stretch based network (28) with fixed exponents is a linear function in the network weights w i $$ {w}_i $$ , which translates the loss function (24) into a linear regression problem, with a single unique global minimum.67 We train our principal stretch based network with tension, compression, and shear data and rewrite the loss function (24) in terms of two contributions that minimize the error between the tensile and compressive stresses predicted by the model P 11 ( λ i ) $$ {P}_{11}\left({\lambda}_i\right) $$ and the data P ^ 11 , i $$ {\hat{P}}_{11,i} $$ , and between the shear stresses predicted by the model P 12 ( γ i ) $$ {P}_{12}\left({\gamma}_i\right) $$ and the data P ^ 12 , i $$ {\hat{P}}_{12,i} $$ , where we include data from n 11 $$ {n}_{11} $$ different stretch levels λ $$ \lambda $$ and n 12 $$ {n}_{12} $$ different shear levels γ $$ \gamma $$ ,
L ( θ ; λ , γ ) = 1 n 11 ∑ i = 1 n 11 | | P 11 ( λ i ) − P ^ 11 , i | | 2 + 1 n 12 ∑ i = 1 n 12 | | P 12 ( γ i ) − P ^ 12 , i | | 2 → min . $$ L\left(\boldsymbol{\theta}; \lambda, \gamma \right)=\frac{1}{n_{11}}\sum \limits_{i=1}^{n_{11}}{\left\Vert \kern0.3em {P}_{11}\left({\lambda}_i\right)-{\hat{P}}_{11,i}\kern0.3em \right\Vert}^2+\frac{1}{n_{12}}\sum \limits_{i=1}^{n_{12}}{\left\Vert \kern0.3em {P}_{12}\left({\gamma}_i\right)-{\hat{P}}_{12,i}\kern0.3em \right\Vert}^2\to \min . $$ ()
For comparison, similar to the invariant based network in the Section 3.1, we also train the network by minimizing the error between the normalized tensile, compressive, and shear stresses predicted by the model P ten ( λ i ) $$ {P}_{\mathrm{ten}}\left({\lambda}_i\right) $$ , P com ( λ i ) $$ {P}_{\mathrm{com}}\left({\lambda}_i\right) $$ , and P shr ( γ i ) $$ {P}_{\mathrm{shr}}\left({\gamma}_i\right) $$ , and the data P ^ ten , i $$ {\hat{P}}_{\mathrm{ten},i} $$ , P ^ com , i $$ {\hat{P}}_{\mathrm{com},i} $$ , and P ^ shr , i $$ {\hat{P}}_{\mathrm{shr},i} $$ , normalized by the maximum recorded tensile, compressive, and shear stresses, P ^ ten max $$ {\hat{P}}_{\mathrm{ten}}^{\mathrm{max}} $$ , P ^ com min $$ {\hat{P}}_{\mathrm{com}}^{\mathrm{min}} $$ , and P ^ shr max $$ {\hat{P}}_{\mathrm{shr}}^{\mathrm{max}} $$ , where n ten $$ {n}_{\mathrm{ten}} $$ , n com $$ {n}_{\mathrm{com}} $$ , and n shr $$ {n}_{\mathrm{shr}} $$ denote the different stretch and shear levels λ $$ \lambda $$ and γ $$ \gamma $$ ,
L ( θ ; λ , γ ) = 1 n ten ∑ i = 1 n ten P ten ( λ i ) − P ^ ten , i P ^ ten max 2 + 1 n com ∑ i = 1 n com P com ( λ i ) − P ^ com , i P ^ com min 2 + 1 n shr ∑ i = 1 n shr P shr ( γ i ) − P ^ shr , i P ^ shr max 2 → min . $$ L\left(\boldsymbol{\theta}; \lambda, \gamma \right)=\frac{1}{n_{\mathrm{ten}}}\sum \limits_{i=1}^{n_{\mathrm{ten}}}{\left|\left|\kern0.3em \frac{P_{\mathrm{ten}}\left({\lambda}_i\right)-{\hat{P}}_{\mathrm{ten},i}}{{\hat{P}}_{\mathrm{ten}}^{\mathrm{max}}}\right|\right|}^2+\frac{1}{n_{\mathrm{com}}}\sum \limits_{i=1}^{n_{\mathrm{com}}}{\left|\left|\kern0.3em \frac{P_{\mathrm{com}}\left({\lambda}_i\right)-{\hat{P}}_{\mathrm{com},i}}{{\hat{P}}_{\mathrm{com}}^{\mathrm{min}}}\right|\right|}^2+\frac{1}{n_{\mathrm{shr}}}\sum \limits_{i=1}^{n_{\mathrm{shr}}}{\left|\left|\kern0.3em \frac{P_{\mathrm{shr}}\left({\gamma}_i\right)-{\hat{P}}_{\mathrm{shr},i}}{{\hat{P}}_{\mathrm{shr}}^{\mathrm{max}}}\right|\right|}^2\to \min . $$ ()
Below, we briefly derive the explicit analytical expressions for the Piola stresses P 11 ( λ ) $$ {P}_{11}\left(\lambda \right) $$ in uniaxial tension and compression and P 12 ( γ ) $$ {P}_{12}\left(\gamma \right) $$ in simple shear, such that the tensile stress is P ten = P 11 $$ {P}_{\mathrm{ten}}={P}_{11} $$ for λ > 1 $$ \lambda >1 $$ , the compressive stress is P com = P 11 $$ {P}_{\mathrm{com}}={P}_{11} $$ for λ < 1 $$ \lambda <1 $$ , and the shear stress is P shr = P 12 $$ {P}_{\mathrm{shr}}={P}_{12} $$ for all γ $$ \gamma $$ .
Details are in the caption following the image
FIGURE 4
Open in figure viewerPowerPoint
Principal stretch based neural network for automated model discovery. The network takes the deformation gradient F $$ \boldsymbol{F} $$ as input and outputs the free energy function ψ $$ \psi $$ from which we calculate the stress P = ∂ ψ / ∂ F $$ \boldsymbol{P}=\partial \psi /\partial \boldsymbol{F} $$ . The network is principal stretch based, it first calculates the principal stretches λ 1 $$ {\lambda}_1 $$ and λ 2 $$ {\lambda}_2 $$ and λ 3 $$ {\lambda}_3 $$ , and feeds them into its hidden layer. The hidden layer applies eight different exponents ( λ 1 n + λ 2 n + λ 3 n − 3 ) $$ \left({\lambda}_1^n+{\lambda}_2^n+{\lambda}_3^n-3\right) $$ to these principal stretches. The free energy function ψ $$ \psi $$ is a function of the eight color-coded terms. During training, the network discovers the best model, of 2 8 = 256 $$ {2}^8=256 $$ possible combinations of terms, to explain the experimental data P ^ $$ \hat{\boldsymbol{P}} $$ .
Uniaxial tension and compression. For the special case of uniaxial tension and compression in terms of the stretch λ $$ \lambda $$ , the principal stretches are
λ 1 = λ and λ 2 = λ − 1 / 2 and λ 3 = λ − 1 / 2 . $$ {\lambda}_1=\lambda \kern2em \mathrm{and}\kern2em {\lambda}_2={\lambda}^{-1/2}\kern2em \mathrm{and}\kern2em {\lambda}_3={\lambda}^{-1/2}. $$ ()
Using Equation (27) and the zero normal stress condition, P 22 = P 33 = 0 $$ {P}_{22}={P}_{33}=0 $$ , we obtain the following expression for the uniaxial stress stretch relation,
P 11 = 1 λ ∑ i = 1 n term α i w i [ λ α i − λ − α i / 2 ] , $$ {P}_{11}=\frac{1}{\lambda}\kern0.3em \sum \limits_{i=1}^{n_{\mathrm{term}}}{\alpha}_i\kern0.3em {w}_i\kern0.3em \left[\kern0.3em {\lambda}^{\alpha_i}-{\lambda}^{-{\alpha}_i/2}\kern0.3em \right], $$ ()
which translates into the following explicit expression between our network stress P 11 $$ {P}_{11} $$ and the uniaxial stretch λ $$ \lambda $$ ,
P 11 = 1 λ [ 2 w 1 [ λ + 2 − λ − 1 ] + 4 w 2 [ λ + 4 − λ − 2 ] + 6 w 3 [ λ + 6 − λ − 3 ] + 8 w 4 [ λ + 8 − λ − 4 ] − 2 w 5 [ λ − 2 − λ + 1 ] − 4 w 6 [ λ − 4 − λ + 2 ] − 6 w 7 [ λ − 6 − λ + 3 ] − 8 w 8 [ λ − 8 − λ + 4 ] ] . $$ {\displaystyle \begin{array}{cc}{P}_{11}& =\frac{1}{\lambda}\kern0.1em \Big[\kern0.1em 2\kern0.2em {w}_1\kern0.2em \left[\kern0.1em {\lambda}^{+2}-{\lambda}^{-1}\kern0.1em \right]+4\kern0.2em {w}_2\kern0.2em \left[\kern0.1em {\lambda}^{+4}-{\lambda}^{-2}\kern0.1em \right]+6\kern0.2em {w}_3\kern0.2em \left[\kern0.1em {\lambda}^{+6}-{\lambda}^{-3}\kern0.1em \right]+8\kern0.2em {w}_4\kern0.2em \left[\kern0.1em {\lambda}^{+8}-{\lambda}^{-4}\kern0.1em \right]\\ {}& \kern1.4em -2\kern0.2em {w}_5\kern0.2em \left[\kern0.1em {\lambda}^{-2}-{\lambda}^{+1}\kern0.1em \right]-4\kern0.2em {w}_6\kern0.2em \left[\kern0.1em {\lambda}^{-4}-{\lambda}^{+2}\kern0.1em \right]-6\kern0.2em {w}_7\kern0.2em \left[\kern0.1em {\lambda}^{-6}-{\lambda}^{+3}\kern0.1em \right]-8\kern0.2em {w}_8\kern0.2em \left[\kern0.1em {\lambda}^{-8}-{\lambda}^{+4}\kern0.1em \right]\kern0.1em \Big].\end{array}} $$ ()
Simple shear. For the special case of simple shear, in terms of the shear γ $$ \gamma $$ , we obtain the principal stretches,
λ 1 = γ + 4 + γ 2 2 and λ 2 = − γ + 4 + γ 2 2 = 1 λ 1 and λ 3 = 1 . $$ {\lambda}_1=\frac{\gamma +\sqrt{4+{\gamma}^2}}{2}\kern1em \mathrm{and}\kern1em {\lambda}_2=\frac{-\gamma +\sqrt{4+{\gamma}^2}}{2}=\frac{1}{\lambda_1}\kern1em \mathrm{and}\kern1em {\lambda}_3=1. $$ ()
Using Equation (27), we obtain the following expression for the shear stress stretch relation,
P 12 = 1 1 + λ 2 ∑ i = 1 n term α i w i [ λ α i + 1 − λ 1 − α i ] with λ = 1 2 γ + 4 + γ 2 = λ 1 = 1 λ 2 , $$ {P}_{12}=\frac{1}{1+{\lambda}^2}\kern0.3em \sum \limits_{i=1}^{n_{\mathrm{term}}}{\alpha}_i\kern0.3em {w}_i\kern0.3em \left[\kern0.3em {\lambda}^{\alpha_i+1}-{\lambda}^{1-{\alpha}_i}\kern0.3em \right]\kern2em \mathrm{with}\kern2em \lambda =\frac{1}{2}\kern0.3em \left[\kern0.3em \gamma +\sqrt{4+{\gamma}^2}\kern0.3em \right]={\lambda}_1=\frac{1}{\lambda_2}, $$ ()
which translates into the following explicit expression between our network shear stress P 12 $$ {P}_{12} $$ and shear strain γ $$ \gamma $$ ,
P 12 = 1 1 + λ 2 [ 2 w 1 [ λ + 3 − λ − 1 ] + 4 w 2 [ λ + 5 − λ − 3 ] + 6 w 3 [ λ + 7 − λ − 5 ] + 8 w 4 [ λ + 9 − λ − 7 ] − 2 w 5 [ λ − 1 − λ + 3 ] − 4 w 6 [ λ − 3 − λ + 5 ] − 6 w 7 [ λ − 5 − λ + 7 ] − 8 w 8 [ λ − 7 − λ + 9 ] ] . $$ {\displaystyle \begin{array}{cc}{P}_{12}& =\frac{1}{1+{\lambda}^2}\kern0.1em \Big[\kern0.1em 2\kern0.2em {w}_1\kern0.2em \left[\kern0.1em {\lambda}^{+3}-{\lambda}^{-1}\kern0.1em \right]+4\kern0.2em {w}_2\kern0.2em \left[\kern0.1em {\lambda}^{+5}-{\lambda}^{-3}\kern0.1em \right]+6\kern0.2em {w}_3\kern0.2em \left[\kern0.1em {\lambda}^{+7}-{\lambda}^{-5}\kern0.1em \right]+8\kern0.2em {w}_4\kern0.2em \left[\kern0.1em {\lambda}^{+9}-{\lambda}^{-7}\kern0.1em \right]\\ {}& \kern3.4em -2\kern0.2em {w}_5\kern0.2em \left[\kern0.1em {\lambda}^{-1}-{\lambda}^{+3}\kern0.1em \right]-4\kern0.2em {w}_6\kern0.2em \left[\kern0.1em {\lambda}^{-3}-{\lambda}^{+5}\kern0.1em \right]-6\kern0.2em {w}_7\kern0.2em \left[\kern0.1em {\lambda}^{-5}-{\lambda}^{+7}\kern0.1em \right]-8\kern0.2em {w}_8\kern0.2em \left[\kern0.1em {\lambda}^{-7}-{\lambda}^{+9}\kern0.1em \right]\kern0.1em \Big].\end{array}} $$ ()
Figure 5 illustrates the contours of the loss function L ( θ ; λ , γ ) $$ L\left(\boldsymbol{\theta}; \lambda, \gamma \right) $$ for all possible two-term models of the principal stretch based network in Figure 4. By combining any two terms of the model and setting all other second-layer weights equal to zero, we can generate 28 possible models. For these 28 combinations of two terms, we evaluate two versions of the loss function, non-normalized from Equation (29) and normalized from Equation (30), following the method described in Section 3.1, but now using the principal stretch based definitions of the normal stress (33) and shear stress (36). Similar to Section 3.1, we plot the non-normalized loss function in the lower triangle and the normalized loss function in the upper triangle. Again, by design, all loss functions take a minimum of L ( θ ; λ , γ ) = 0 $$ L\left(\boldsymbol{\theta}; \lambda, \gamma \right)=0 $$ for the exact solution, w i = 1 $$ {w}_i=1 $$ and w j = 1 $$ {w}_j=1 $$ , indicated through the white circles. From this minimum, both versions of the loss function, non-normalized and normalized, increase with both decreasing and increasing weights w i $$ {w}_i $$ and w j $$ {w}_j $$ , and remain convex within the entire domain, for all 28 two-term models. Notably, in contrast to the loss function contours of the invariant based model in Figure 3, the contours of the principal stretch based model in Figure 5 display less variation for different pairs of weights w i $$ {w}_i $$ and w j $$ {w}_j $$ : Only a few pairs of weights generate loss functions of ellipsoidal shape, for example, the {   w 3 , w 6   } $$ \left\{\;{w}_3,{w}_6\;\right\} $$ , {   w 4 , w 7   } $$ \left\{\;{w}_4,{w}_7\;\right\} $$ pairs in the non-normalized lower triangle, or the {   w 2 , w 6   } $$ \left\{\;{w}_2,{w}_6\;\right\} $$ , {   w 3 , w 7   } $$ \left\{\;{w}_3,{w}_7\;\right\} $$ , {   w 4 , w 8   } $$ \left\{\;{w}_4,{w}_8\;\right\} $$ pairs in the normalized upper and non-normalized lower triangles, suggesting that, in the studied stretch and shear range, only a few pairs of terms are non-collinear and would represent a solid base for a potential constitutive model. Most pairs of weights generate loss functions with long ridges parallel to the parameter axes, suggesting that many terms are almost collinear and not well suited as a functional base for a constitutive model. In contrast to the invariant based network, normalization does not seem to fix this issue, both the upper and lower triangle display this collinearity. Notably, the {   w 1 , w 5   } $$ \left\{\;{w}_1,{w}_5\;\right\} $$ model in the first row and fifth column and the {   w 5 , w 1   } $$ \left\{\;{w}_5,{w}_1\;\right\} $$ model in the fifth row and first column combine the positive and negative second powers of the principal stretches, [ λ 1 + 2 + λ 2 + 2 + λ 3 + 2 − 3 ] $$ \left[\kern0.3em {\lambda}_1^{+2}+{\lambda}_2^{+2}+{\lambda}_3^{+2}-3\kern0.3em \right] $$ and [ λ 1 − 2 + λ 2 − 2 + λ 3 − 2 − 3 ] $$ \left[\kern0.3em {\lambda}_1^{-2}+{\lambda}_2^{-2}+{\lambda}_3^{-2}-3\kern0.3em \right] $$ , and represent the popular Mooney Rivlin model,59, 60 which is identical to the {   w 1 , w 5   } $$ \left\{\;{w}_1,{w}_5\;\right\} $$ and {   w 5 , w 1   } $$ \left\{\;{w}_5,{w}_1\;\right\} $$ models of the invariant based model in Figure 3. Overall, while these contours are difficult to interpret, we can compare them directly to Figure 3 and realize that, within the studied stretch and shear range λ $$ \lambda $$ and γ $$ \gamma $$ , and parameter window w i $$ {w}_i $$ , the invariant based network seems to represent a much broader spectrum of functions than the principal stretch based network for which the functional base seems to be generally more narrow and almost collinear. We also note that the loss function is highly sensitive to normalization: For both networks, the normalized loss functions (17) and (30) tend to generate more convex shapes than the non-normalized loss functions (16) and (29), which is why we will focus on the maximum-stress normalized loss functions (17) and (30) in all following examples.
Details are in the caption following the image
FIGURE 5
Open in figure viewerPowerPoint
Loss functions of principal stretch based neural network. Contours of the loss function L ( θ ; λ , γ ) $$ L\left(\boldsymbol{\theta}; \lambda, \gamma \right) $$ for all 28 possible two-term models of the principal stretch based constitutive neural network in Figure 4. The loss function is evaluated across tensile stretches λ = [ 1 . 0 , … , 2 . 0 ] $$ \lambda =\left[1.0,\dots, 2.0\right] $$ , compressive stretches λ = [ 1 . 0 , … , 0 . 5 ] $$ \lambda =\left[1.0,\dots, 0.5\right] $$ , and shear strains γ = [ 0 . 0 , … , 0 . 5 ] $$ \gamma =\left[0.0,\dots, 0.5\right] $$ , with network weights in the ranges w i = [ 0 , … , 2 ] $$ {w}_i=\left[0,\dots, 2\right] $$ and w j = [ 0 , … , 2 ] $$ {w}_j=\left[0,\dots, 2\right] $$ . The minimum of the loss function indicates the exact solution w i = 1 $$ {w}_i=1 $$ and w j = 1 $$ {w}_j=1 $$ , represented through the white circle. The lower triangle illustrates the non-normalized loss function (29), the upper triangle illustrates the normalized loss function (30). All loss functions are convex, with contours varying from a few ellipsoids to many valleys with long ridges, highlighting the collinearity of many w i $$ {w}_i $$ and w j $$ {w}_j $$ pairs.

4 L p $$ {\boldsymbol{L}}_{\boldsymbol{p}} $$ REGULARIZED NEURAL NETWORKS

We now integrate the concepts of L p $$ {L}_p $$ regularization from Section 2 and constitutive neural network modeling from Section 3 and explore the resulting regression in view of predictability and interpretability. Specifically, we supplement the loss function of the constitutive neural network with a penalty term of L p $$ {L}_p $$ type,
L ( θ ; F ) = 1 n data ∑ i = 1 n data | | P ( F i , θ ) − P ^ i | | 2 + α | | θ | | p p → min with | | θ | | p p = ∑ i = 1 n para | w i | p . $$ L\left(\boldsymbol{\theta}; \boldsymbol{F}\right)=\frac{1}{n_{\mathrm{data}}}\sum \limits_{i=1}^{n_{\mathrm{data}}}{\left\Vert \kern0.3em \boldsymbol{P}\Big({\boldsymbol{F}}_i,\boldsymbol{\theta} \Big)-{\hat{\boldsymbol{P}}}_i\kern0.3em \right\Vert}^2+\alpha \kern0.3em {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_p^p\to \min \kern1em \mathrm{with}\kern1em {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_p^p=\sum \limits_{i=1}^{n_{\mathrm{para}}}{\left|\kern0.3em {w}_i\kern0.3em \right|}^p. $$ ()
The loss function minimizes the error between the model stress that we derive from the free energy of the neural network, P ( F i ; θ ) = ∂ ψ / ∂ F $$ \boldsymbol{P}\left({\boldsymbol{F}}_i;\boldsymbol{\theta} \right)=\partial \psi /\partial \boldsymbol{F} $$ , and the experimentally measured stress P ^ i $$ {\hat{\boldsymbol{P}}}_i $$ divided by the number of data points n data $$ {n}_{\mathrm{data}} $$ , penalized by the L p $$ {L}_p $$ norm, | |   θ   | | p p = ∑ i = 1 n para |   w i   | p $$ {\left\Vert\;\boldsymbol{\theta}\;\right\Vert}_p^p={\sum}_{i=1}^{n_{\mathrm{para}}}{\left|\;{w}_i\;\right|}^p $$ , of the parameter vector θ = { w i } $$ \boldsymbol{\theta} =\left\{\kern0.3em {w}_i\kern0.3em \right\} $$ made up of the network weights w i $$ {w}_i $$ , multiplied by the penalty parameter α ≥ 0 $$ \alpha \ge 0 $$ . Specifically, we use tension, compression, and shear data and specify the stress error as the normalized difference between the tensile, compressive, and shear stresses predicted by the neural network, P ten ( λ i ) $$ {P}_{\mathrm{ten}}\left({\lambda}_i\right) $$ , P com ( λ i ) $$ {P}_{\mathrm{com}}\left({\lambda}_i\right) $$ , and P shr ( γ i ) $$ {P}_{\mathrm{shr}}\left({\gamma}_i\right) $$ , and the data, P ^ ten , i $$ {\hat{P}}_{\mathrm{ten},i} $$ , P ^ com , i $$ {\hat{P}}_{\mathrm{com},i} $$ , and P ^ shr , i $$ {\hat{P}}_{\mathrm{shr},i} $$ , at n ten $$ {n}_{\mathrm{ten}} $$ , n com $$ {n}_{\mathrm{com}} $$ , and n shr $$ {n}_{\mathrm{shr}} $$ stretch and shear levels λ $$ \lambda $$ and γ $$ \gamma $$ ,
L ( θ ; λ , γ ) = 1 n ten ∑ i = 1 n ten P ten ( λ i ) − P ^ ten , i P ^ ten max 2 + 1 n com ∑ i = 1 n com P com ( λ i ) − P ^ com , i P ^ com min 2 + 1 n shr ∑ i = 1 n shr P shr ( γ i ) − P ^ shr , i P ^ shr max 2 + α | | θ | | p p → min . $$ L\left(\boldsymbol{\theta}; \lambda, \gamma \right)=\frac{1}{n_{\mathrm{ten}}}\sum \limits_{i=1}^{n_{\mathrm{ten}}}{\left|\left|\kern0.1em \frac{P_{\mathrm{ten}}\left({\lambda}_i\right)-{\hat{P}}_{\mathrm{ten},i}}{{\hat{P}}_{\mathrm{ten}}^{\mathrm{max}}}\right|\right|}^2+\frac{1}{n_{\mathrm{com}}}\sum \limits_{i=1}^{n_{\mathrm{com}}}{\left|\left|\kern0.1em \frac{P_{\mathrm{com}}\left({\lambda}_i\right)-{\hat{P}}_{\mathrm{com},i}}{{\hat{P}}_{\mathrm{com}}^{\mathrm{min}}}\right|\right|}^2+\frac{1}{n_{\mathrm{shr}}}\sum \limits_{i=1}^{n_{\mathrm{shr}}}{\left|\left|\kern0.1em \frac{P_{\mathrm{shr}}\left({\gamma}_i\right)-{\hat{P}}_{\mathrm{shr},i}}{{\hat{P}}_{\mathrm{shr}}^{\mathrm{max}}}\right|\right|}^2+\alpha \kern0.3em {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_p^p\to \min . $$ ()
In the following, we systematically explore the sensitivity of the loss function (38) with respect to the two hyperparameters of the L p $$ {L}_p $$ regularization, the power p $$ p $$ and the penalty parameter α $$ \alpha $$ . For illustrative purposes, we first focus on a simplified two-term model, the Mooney Rivlin model that is shared between both neural networks, before we explore both L p $$ {L}_p $$ regularized complete eight-term networks.
L   p $$ {}_{\boldsymbol{p}} $$ regularized Mooney Rivlin model. The Mooney Rivlin model59, 60 is a two-term constitutive model that is located right at the intersection of the invariant based neural network in Figure 2 and the principal stretch based network in Figure 4. Notably, it is the only model, for which both networks coincide. It uses the dark red term, [ I 1 − 3 ] = [ λ 1 + 2 + λ 2 + 2 + λ 3 + 2 − 3 ] $$ \left[\kern0.3em {I}_1-3\kern0.3em \right]=\left[\kern0.3em {\lambda}_1^{+2}+{\lambda}_2^{+2}+{\lambda}_3^{+2}-3\kern0.3em \right] $$ , and the green term, [ I 1 − 2 ] = [ λ 2 − 2 + λ 2 − 2 + λ 3 − 2 − 3 ] $$ \left[\kern0.3em {I}_1-2\kern0.3em \right]=\left[\kern0.3em {\lambda}_2^{-2}+{\lambda}_2^{-2}+{\lambda}_3^{-2}-3\kern0.3em \right] $$ , of both neural networks, and weighs them by the network weights, w 1 , 1   w 2 , 1 = w 1 $$ {w}_{1,1}\;{w}_{2,1}={w}_1 $$ and w 1 , 5   w 2 , 5 = w 5 $$ {w}_{1,5}\;{w}_{2,5}={w}_5 $$ , while all other network weights are identical to zero,
ψ ( I 1 , I 2 ) = w 1 , 1 w 2 , 1 [ I 1 − 3 ] + w 1 , 5 w 2 , 5 [ I 2 − 3 ] and ψ ( λ 1 , λ 2 , λ 3 ) = w 1 [ λ 1 + 2 + λ 2 + 2 + λ 3 + 2 − 3 ] + w 5 [ λ 1 − 2 + λ 2 − 2 + λ 3 − 2 − 3 ] . $$ \psi \left({I}_1,{I}_2\right)={w}_{1,1}\kern0.3em {w}_{2,1}\kern0.3em \left[\kern-0.1em {I}_1-3\kern0em \right]+{w}_{1,5}\kern0.2em {w}_{2,5}\kern0.3em \left[\kern-0.1em {I}_2-3\kern0em \right]\kern0.5em \mathrm{and}\kern0.5em \psi \left({\lambda}_1,{\lambda}_2,{\lambda}_3\right)={w}_1\kern0.3em \left[\kern0em {\lambda}_1^{+2}+{\lambda}_2^{+2}+{\lambda}_3^{+2}-3\kern0em \right]+{w}_5\kern0.3em \left[\kern0em {\lambda}_1^{-2}+{\lambda}_2^{-2}+{\lambda}_3^{-2}-3\kern0em \right]. $$ ()
This implies that the activation of any other weight will make the invariant and principal stretch based networks drift away from one another. The Mooney Rivlin model in Equation (39) includes the one-term dark red Neo Hooke model61 with [ I 1 − 3 ] = [ λ 1 + 2 + λ 2 + 2 + λ 3 + 2 − 3 ] $$ \left[\kern0.3em {I}_1-3\kern0.3em \right]=\left[\kern0.3em {\lambda}_1^{+2}+{\lambda}_2^{+2}+{\lambda}_3^{+2}-3\kern0.3em \right] $$ and the one-term green Blatz Ko model58 with [ I 1 − 2 ] = [ λ 1 − 2 + λ 2 − 2 + λ 3 − 2 − 3 ] $$ \left[\kern0.3em {I}_1-2\kern0.3em \right]=\left[\kern0.3em {\lambda}_1^{-2}+{\lambda}_2^{-2}+{\lambda}_3^{-2}-3\kern0.3em \right] $$ as special cases. For the Mooney Rivlin model, the L p $$ {L}_p $$ regularized loss function from Equation (38) specifies to
L ( w 1 , w 5 ; λ , γ ) = 1 n ten ∑ i = 1 n ten P ten ( λ i ) − P ^ ten , i P ^ ten max 2 + 1 n com ∑ i = 1 n com P com ( λ i ) − P ^ com , i P ^ com min 2 + 1 n shr ∑ i = 1 n shr P shr ( γ i ) − P ^ shr , i P ^ shr max 2 + α p [ w 1 p + w 5 p ] → min , $$ L\left({w}_1,{w}_5;\lambda, \gamma \right)=\frac{1}{n_{\mathrm{ten}}}\sum \limits_{i=1}^{n_{\mathrm{ten}}}{\left|\left|\frac{P_{\mathrm{ten}}\left({\lambda}_i\right)-{\hat{P}}_{\mathrm{ten},i}}{{\hat{P}}_{\mathrm{ten}}^{\mathrm{max}}}\right|\right|}^2+\frac{1}{n_{\mathrm{com}}}\sum \limits_{i=1}^{n_{\mathrm{com}}}{\left|\left|\frac{P_{\mathrm{com}}\left({\lambda}_i\right)-{\hat{P}}_{\mathrm{com},i}}{{\hat{P}}_{\mathrm{com}}^{\mathrm{min}}}\right|\right|}^2+\frac{1}{n_{\mathrm{shr}}}\sum \limits_{i=1}^{n_{\mathrm{shr}}}{\left|\left|\frac{P_{\mathrm{shr}}\left({\gamma}_i\right)-{\hat{P}}_{\mathrm{shr},i}}{{\hat{P}}_{\mathrm{shr}}^{\mathrm{max}}}\right|\right|}^2+{\alpha}_p\kern0.1em \left[\kern-0.15em {w}_1^p+{w}_5^p\right]\to \min, $$ ()
with the Mooney Rivlin stresses in tension, P ten = P 11 $$ {P}_{\mathrm{ten}}={P}_{11} $$ for λ > 1 $$ \lambda >1 $$ , and compression, P com = P 11 $$ {P}_{\mathrm{com}}={P}_{11} $$ for λ < 1 $$ \lambda <1 $$ , from Equations (20) and (33), and in shear, P shr = P 12 $$ {P}_{\mathrm{shr}}={P}_{12} $$ for all γ $$ \gamma $$ . from Equations (23) and (36),
P 11 = 2 [ λ − 1 / λ 2 ] [ w 1 + w 5 / λ ] and P 12 = 2 γ [ w 1 + w 5 ] . $$ {P}_{11}=2\kern0.3em \left[\lambda -1/{\lambda}^2\right]\left[{w}_1+{w}_5/\lambda \right]\kern2em \mathrm{and}\kern2em {P}_{12}=2\kern0.3em \gamma \kern0.3em \left[{w}_1+{w}_5\right]. $$ ()
Notably, the uniaxial stress P 11 $$ {P}_{11} $$ and shear stress P 12 $$ {P}_{12} $$ of the Mooney Rivlin model (41) are linear functions in the network weights w 1 $$ {w}_1 $$ and w 5 $$ {w}_5 $$ , which translates the neural network loss, ∑ i = 1 n ten | | [ P ten ( λ i ) − P ^ ten , i ] / P ten max | | 2 / n ten + ∑ i = 1 n com | | [ P com ( λ i ) − P ^ com , i ] / P com min | | 2 / n com + ∑ i = 1 n shr | | [ P shr ( λ i ) − P ^ shr , i ] / P shr max | | 2 / n shr $$ {\sum}_{i=1}^{n_{\mathrm{ten}}}{\left\Vert \left[{P}_{\mathrm{ten}}\left({\lambda}_i\right)-{\hat{P}}_{\mathrm{ten},i}\right]/{P}_{\mathrm{ten}}^{\mathrm{max}}\right\Vert}^2/{n}_{\mathrm{ten}}+{\sum}_{i=1}^{n_{\mathrm{com}}}{\left\Vert \left[{P}_{\mathrm{com}}\left({\lambda}_i\right)-{\hat{P}}_{\mathrm{com},i}\right]/{P}_{\mathrm{com}}^{\mathrm{min}}\right\Vert}^2/{n}_{\mathrm{com}}+{\sum}_{i=1}^{n_{\mathrm{shr}}}{\left\Vert \left[{P}_{\mathrm{shr}}\left({\lambda}_i\right)-{\hat{P}}_{\mathrm{shr},i}\right]/{P}_{\mathrm{shr}}^{\mathrm{max}}\right\Vert}^2/{n}_{\mathrm{shr}} $$ , of the loss function (40) into a linear regression problem, with a single unique global minimum.

Figure 6 illustrates the contours of the L p $$ {L}_p $$ regularized loss function L ( w 1 , w 5 ; λ , γ ) $$ L\left({w}_1,{w}_5;\lambda, \gamma \right) $$ for the two-term Mooney Rivlin model, with varying powers, p = [ 0 . 25 , 0 . 5 , 0 . 75 , 1 , 1 . 5 , 2 , 4 , 8 ] $$ p=\left[0.25,0.5,0.75,1,1.5,2,4,8\right] $$ , evaluated for the two parameters w 1 $$ {w}_1 $$ and w 5 $$ {w}_5 $$ , using synthetic data from tension, compression, and shear tests. The loss function consists of the neural network loss, ∑ i = 1 n ten | | [ P ten ( λ i ) − P ^ ten , i ] / P ten max | | 2 / n ten + ∑ i = 1 n com | | [ P com ( λ i ) − P ^ com , i ] / P com min | | 2 / n com + ∑ i = 1 n shr | | [ P shr ( λ i ) − P ^ shr , i ] / P shr max | | 2 / n shr $$ {\sum}_{i=1}^{n_{\mathrm{ten}}}{\left\Vert \left[{P}_{\mathrm{ten}}\left({\lambda}_i\right)-{\hat{P}}_{\mathrm{ten},i}\right]/{P}_{\mathrm{ten}}^{\mathrm{max}}\right\Vert}^2/{n}_{\mathrm{ten}}+{\sum}_{i=1}^{n_{\mathrm{com}}}{\left\Vert \left[{P}_{\mathrm{com}}\left({\lambda}_i\right)-{\hat{P}}_{\mathrm{com},i}\right]/{P}_{\mathrm{com}}^{\mathrm{min}}\right\Vert}^2/{n}_{\mathrm{com}}+{\sum}_{i=1}^{n_{\mathrm{shr}}}{\left\Vert \left[{P}_{\mathrm{shr}}\left({\lambda}_i\right)-{\hat{P}}_{\mathrm{shr},i}\right]/{P}_{\mathrm{shr}}^{\mathrm{max}}\right\Vert}^2/{n}_{\mathrm{shr}} $$ , illustrated in the first row and fifth column of Figures 3 and 5; supplemented by the L p $$ {L}_p $$ regularization, α p [ | w 1 | p + | w 5 | p ] $$ {\alpha}_p\kern0.3em \left[\kern0.3em {\left|\kern0.3em {w}_1\kern0.3em \right|}^p+{\left|\kern0.3em {w}_5\kern0.3em \right|}^p\kern0.3em \right] $$ , illustrated in Figure 1. For all eight graphs in Figure 6, we evaluate the loss function (40) using the Mooney Rivlin stresses (41). First, we generate synthetic data, P ^ ten $$ {\hat{P}}_{\mathrm{ten}} $$ , P ^ com $$ {\hat{P}}_{\mathrm{com}} $$ , P ^ shr $$ {\hat{P}}_{\mathrm{shr}} $$ for tensile stretches of λ = [ 1 . 0 , … , 2 . 0 ] $$ \lambda =\left[1.0,\dots, 2.0\right] $$ , compressive stretches of λ = [ 1 . 0 , … , 0 . 5 ] $$ \lambda =\left[1.0,\dots, 0.5\right] $$ , and shear strains of γ = [ 0 . 0 , … , 0 . 5 ] $$ \gamma =\left[0.0,\dots, 0.5\right] $$ , in ten equidistant increments each, assuming an exact solution with w 1 , 1   w 2 , 1 = w 1 = 1 $$ {w}_{1,1}\;{w}_{2,1}={w}_1=1 $$ and w 1 , 5   w 2 , 5 = w 5 = 1 $$ {w}_{1,5}\;{w}_{2,5}={w}_5=1 $$ , while fixing the remaining weights equal to zero. This results in the training data sets of eleven stretch-stress pairs for tension, compression, and shear. Second, we vary the two Mooney Rivlin network weights in the ranges w 1 = [ 0 , … , 2 ] $$ {w}_1=\left[0,\dots, 2\right] $$ and w 5 = [ 0 , … , 2 ] $$ {w}_5=\left[0,\dots, 2\right] $$ , and evaluate the tensile, compressive, and shear model stresses, P ten ( λ i ) $$ {P}_{\mathrm{ten}}\left({\lambda}_i\right) $$ , P com ( λ i ) $$ {P}_{\mathrm{com}}\left({\lambda}_i\right) $$ , P shr ( γ i ) $$ {P}_{\mathrm{shr}}\left({\gamma}_i\right) $$ using Equation (41). Third, we evaluate the loss function (40) as the normalized mean squared error between the model stresses P ten ( λ i ) $$ {P}_{\mathrm{ten}}\left({\lambda}_i\right) $$ , P com ( λ i ) $$ {P}_{\mathrm{com}}\left({\lambda}_i\right) $$ , P shr ( γ i ) $$ {P}_{\mathrm{shr}}\left({\gamma}_i\right) $$ and the synthetically generated data stresses P ^ ten $$ {\hat{P}}_{\mathrm{ten}} $$ , P ^ com $$ {\hat{P}}_{\mathrm{com}} $$ , P ^ shr $$ {\hat{P}}_{\mathrm{shr}} $$ , supplemented by the L p $$ {L}_p $$ regularization α p [ | w 1 | p + | w 5 | p ] $$ {\alpha}_p\kern0.3em \left[\kern0.3em {\left|\kern0.3em {w}_1\kern0.3em \right|}^p+{\left|\kern0.3em {w}_5\kern0.3em \right|}^p\kern0.3em \right] $$ for the eight different powers, p = [ 0 . 25 , 0 . 5 , 0 . 75 , 1 , 1 . 5 , 2 , 4 , 8 ] $$ p=\left[0.25,0.5,0.75,1,1.5,2,4,8\right] $$ . As these powers p $$ p $$ increase by two orders of magnitude, fixing the second hyperparameter α $$ \alpha $$ to one and the same value for all eight examples would increasingly emphasize the L p $$ {L}_p $$ regularization over minimizing the actual network loss, and generate increasingly biased results. Instead, for each power p $$ p $$ , we select the penalty parameter α $$ \alpha $$ such that the maximum value of the loss function within the screened parameter window, in the dark red upper right corner, at w 1 = 2 $$ {w}_1=2 $$ and w 5 = 2 $$ {w}_5=2 $$ , consists of equal contributions by the network term and the regularization term. This results in eight different penalty parameters, α = [ 4 . 69 , 3 . 94 , 3 . 31 , 2 , 79 , 1 . 97 , 1 , 39 , 0 . 35 , 0 . 02 ] $$ \alpha =\left[\kern0.3em 4.69,3.94,3.31,2,79,1.97,1,39,0.35,0.02\kern0.3em \right] $$ . For each set of hyperparameters { p , α } $$ \left\{p,\alpha \right\} $$ , we increase the penalty parameter in four increments, indicated through the four hyperplanes in each graph. Similar to the non-regularized loss functions of the invariant and principal stretch based networks in Figures 3 and 5, we highlight the minimum of the last of these four loss functions L ( w i , w j ; λ , γ ) $$ L\left({w}_i,{w}_j;\lambda, \gamma \right) $$ through a white sphere. Importantly, in contrast to the non-regularized loss functions in Figures 3 and 5, the regularized loss function in Figure 6 no longer has a minimum of L ( θ ; λ , γ ) = 0 $$ L\left(\boldsymbol{\theta}; \lambda, \gamma \right)=0 $$ at w 1 = 1 $$ {w}_1=1 $$ and w 5 = 1 $$ {w}_5=1 $$ . Instead, the minimum of the loss function and its location in the { w 1 , w 5 } $$ \left\{{w}_1,{w}_5\right\} $$ -space are now functions for the two hyperparameters p $$ p $$ and α $$ \alpha $$ . For the eight powers and penalty parameters we used in this example, the minima of the loss function at the location of the white sphere become min ( L ) = [   5 . 83 , 5 . 56 , 5 . 24 , 4 . 82 , 3 . 25 , 2 . 22 , 0 . 57 , 0 . 04   ] $$ \min (L)=\left[\;5.83,5.56,5.24,4.82,3.25,2.22,0.57,0.04\;\right] $$ , and their varying locations in the { w 1 , w 5 } $$ \left\{{w}_1,{w}_5\right\} $$ -space are indicated through the white spheres in Figure 6.

Details are in the caption following the image
FIGURE 6
Open in figure viewerPowerPoint
Loss functions of Lp regularized Mooney Rivlin model for synthetic data. Contours of the L p $$ {L}_p $$ regularized loss function, L ( w 1 , w 5 ; λ , γ ) $$ L\left({w}_1,{w}_5;\lambda, \gamma \right) $$ , for the two-term Mooney Rivlin model with varying powers, p = [ 0 . 25 , 0 . 5 , 0 . 75 , 1 , 1 . 5 , 2 , 4 , 8 ] $$ p=\left[0.25,0.5,0.75,1,1.5,2,4,8\right] $$ , evaluated for the two parameters, w 1 $$ {w}_1 $$ and w 5 $$ {w}_5 $$ , for synthetic data from tension, compression, and shear tests. For p ≤ 1 $$ p\le 1 $$ , top row, with the special case of L 1 $$ {L}_1 $$ regularization or lasso in the fourth column, L p $$ {L}_p $$ regularization promotes sparsity by training w 1 = 0 $$ {w}_1=0 $$ exactly to zero, but the loss function is no longer strictly convex and has multiple local minima. For p > 1 $$ p>1 $$ , bottom row, with the special case of L 2 $$ {L}_2 $$ regularization or ridge regression in the second column, L p $$ {L}_p $$ regularization promotes stability, retains both non-weights, w 1 > 0 $$ {w}_1>0 $$ and w 5 > 0 $$ {w}_5>0 $$ , and maintains a convex loss function with a single global minimum.

Figure 6 reveals several interesting features of the L p $$ {L}_p $$ regularized Mooney Rivlin model: most notably, the regularized loss function is highly sensitive to the power p $$ p $$ and varies significantly for p $$ p $$ below and above one, as we conclude from the different shapes in the first and second rows. For p ≤ 1 $$ p\le 1 $$ , in the top row, with the special case of L 1 $$ {L}_1 $$ regularization or lasso in the fourth column, L p $$ {L}_p $$ regularization promotes sparsity by training one of the weights exactly to zero, in this case w 5 = 0 $$ {w}_5=0 $$ , while the other weight remains positive, w 1 > 0 $$ {w}_1>0 $$ . Importantly, for p < 1 $$ p<1 $$ , the loss function is no longer convex and has two local minima, one at w 1 = 0 $$ {w}_1=0 $$ and one at w 5 = 0 $$ {w}_5=0 $$ . Notably, for a too small power, for example, for p < 0 . 25 $$ p<0.25 $$ , we observe a drastic regularization with sharp-contoured gradients towards the parameter planes, and the model loses robustness. For p > 1 $$ p>1 $$ , in the bottom row, with the special case of L 2 $$ {L}_2 $$ regularization or ridge regression in the second column, L p $$ {L}_p $$ regularization promotes stability and retains both non-zero weights, w 1 > 0 $$ {w}_1>0 $$ and w 5 > 0 $$ {w}_5>0 $$ . The loss function remains convex with a single global minimum. Increasing the penalty parameter α $$ \alpha $$ amplifies these effects and moves the regularized minimum further away from the non-regularized minimum. Taken together, while a regularization across a continuous spectrum of powers p $$ p $$ provides a lot of flexibility, the discovered weights w 1 $$ {w}_1 $$ and w 5 $$ {w}_5 $$ are highly sensitive to the selection of the two hyperparameters p $$ p $$ and α $$ \alpha $$ : while the power p $$ p $$ acts as a switch between sparsity and robustness, the penalty parameter α $$ \alpha $$ induces a trade-off between regularization and bias.

Figure 7 illustrates the contours of the L p $$ {L}_p $$ regularized loss function L ( w 1 , w 5 ; λ , γ ) $$ L\left({w}_1,{w}_5;\lambda, \gamma \right) $$ for the two-term Mooney Rivlin model, with varying powers, p = [ 0 . 25 , 0 . 5 , 0 . 75 , 1 , 1 . 5 , 2 , 4 , 8 ] $$ p=\left[0.25,0.5,0.75,1,1.5,2,4,8\right] $$ , and penalty parameters, α = [ 0 , 0 . 25 , 0 . 50 , 0 . 75 , 1 , 2 , 4 , ∞ ] $$ \alpha =\left[0,0.25,0.50,0.75,1,2,4,\infty \right] $$ , evaluated for the two parameters, w 1 $$ {w}_1 $$ and w 5 $$ {w}_5 $$ , using synthetic data from tension, compression, and shear tests. For all 64 contour plots, we evaluate the normalized loss function (40) following the method of Figure 6, but now by varying both hyperparameters, p $$ p $$ and α $$ \alpha $$ . Without regularization, left column, with α = 0 $$ \alpha =0 $$ , all eight contour plots are identical to the non-regularized Mooney Rivlin loss function in the first row and fifth column of Figures 3 and 5. Its minimum is identical to the exact solution, w 1 = 1 $$ {w}_1=1 $$ and w 5 = 1 $$ {w}_5=1 $$ , represented through the white circles. With infinite regularization, right column, with α = ∞ $$ \alpha =\infty $$ , all eight contour plots are a two-dimensional projection of the L p $$ {L}_p $$ regularization contours in Figure 1. For p ≤ 1 $$ p\le 1 $$ , in the four top rows, with increasing α $$ \alpha $$ , from left to right, the loss function gradually loses strict convexity, the minimum first moves towards w 1 ≥ 1 $$ {w}_1\ge 1 $$ and w 5 = 0 $$ {w}_5=0 $$ , and then towards w 1 → 0 $$ {w}_1\to 0 $$ and w 5 = 0 $$ {w}_5=0 $$ . For p > 1 $$ p>1 $$ , in the four bottom rows, with increasing α $$ \alpha $$ , from left to right, the loss function always remains convex, both weights always remain active, w 1 ≥ 0 $$ {w}_1\ge 0 $$ and w 5 ≥ 0 $$ {w}_5\ge 0 $$ , and move closer together as the minimum gradually moves towards zero, w 1 → 0 $$ {w}_1\to 0 $$ and w 5 → 0 $$ {w}_5\to 0 $$ .

Details are in the caption following the image
FIGURE 7
Open in figure viewerPowerPoint
Loss functions of Lp regularized Mooney Rivlin model for synthetic data. Contours of the L p $$ {L}_p $$ regularized loss function, L ( w 1 , w 5 ; λ , γ ) $$ L\left({w}_1,{w}_5;\lambda, \gamma \right) $$ , for the two-term Mooney Rivlin model with varying powers, p = [ 0 . 25 , 0 . 5 , 0 . 75 , 1 , 1 . 5 , 2 , 4 , 8 ] $$ p=\left[0.25,0.5,0.75,1,1.5,2,4,8\right] $$ , and penalty parameters α = [ 0 , 0 . 25 , 0 . 50 , 0 . 75 , 1 , 2 , 4 , ∞ ] $$ \alpha =\left[0,0.25,0.50,0.75,1,2,4,\infty \right] $$ , evaluated for the two parameters, w 1 $$ {w}_1 $$ and w 5 $$ {w}_5 $$ , for synthetic data from tension, compression, and shear tests. Without regularization, left column, with α = 0 $$ \alpha =0 $$ , the minimum of the loss function is identical to the exact solution w 1 = 1 $$ {w}_1=1 $$ and w 5 = 1 $$ {w}_5=1 $$ , represented through the white circle. With infinite regularization, right column, with α = ∞ $$ \alpha =\infty $$ , the loss function is identical to the L p $$ {L}_p $$ regularization term and the contours are identical to Figure 1. For p < 1 $$ p<1 $$ , with increasing α $$ \alpha $$ , the loss function gradually loses convexity, the minimum first moves towards w 1 ≥ 1 $$ {w}_1\ge 1 $$ and w 5 = 0 $$ {w}_5=0 $$ , and then towards w 1 → 0 $$ {w}_1\to 0 $$ and w 5 = 0 $$ {w}_5=0 $$ . For p > 1 $$ p>1 $$ , with increasing α $$ \alpha $$ , the loss function always remains convex, both weights always remain active, w 1 ≥ 0 $$ {w}_1\ge 0 $$ and w 5 ≥ 0 $$ {w}_5\ge 0 $$ , as the minimum moves gradually towards w 1 → 0 $$ {w}_1\to 0 $$ and w 5 → 0 $$ {w}_5\to 0 $$ .

Figure 7 confirms our observations from Figure 6 and provides additional insights into the L p $$ {L}_p $$ regularized Mooney Rivlin model: the regularized loss function is highly sensitive to both hyperparameters, p $$ p $$ and α $$ \alpha $$ . Decreasing the power to or below one, p ≤ 1 $$ p\le 1 $$ , increases interpretability by promoting sparsity as a subset of weights become exactly zero; smaller powers p $$ p $$ and larger penalty parameters α $$ \alpha $$ promote sparsity more drastically and generate increasingly less convex loss functions. Increasing the power above one, p > 1 $$ p>1 $$ , increases predictability by promoting robustness as the loss function becomes increasingly convex; larger powers p $$ p $$ and larger penalty parameters α $$ \alpha $$ promote robustness more drastically and generate increasingly more convex loss functions. These observations confirm the general notion that L p $$ {L}_p $$ regularization is an intricate balance between predictability and interpretability and between regularization and bias that requires a careful selection of the appropriate values for the hyperparameters p $$ p $$ and α $$ \alpha $$ .

Figure 8 illustrates the contours of the L p $$ {L}_p $$ regularized loss function L ( w 1 , w 5 ; λ , γ ) $$ L\left({w}_1,{w}_5;\lambda, \gamma \right) $$ for the two-term Mooney Rivlin model, with varying powers p $$ p $$ , evaluated for the two parameters w 1 $$ {w}_1 $$ and w 5 $$ {w}_5 $$ , but now using real data from tension, compression, and shear tests of human brain.62 For all eight graphs, we evaluate the loss function (40) as the normalized mean squared error between the model P ten ( λ i ) $$ {P}_{\mathrm{ten}}\left({\lambda}_i\right) $$ , P com ( λ i ) $$ {P}_{\mathrm{com}}\left({\lambda}_i\right) $$ , P shr ( γ i ) $$ {P}_{\mathrm{shr}}\left({\gamma}_i\right) $$ , and the data P ^ ten $$ {\hat{P}}_{\mathrm{ten}} $$ , P ^ com $$ {\hat{P}}_{\mathrm{com}} $$ , P ^ shr $$ {\hat{P}}_{\mathrm{shr}} $$ , for tensile stretches of λ = [ 1 . 0 , … , 1 . 1 ] $$ \lambda =\left[1.0,\dots, 1.1\right] $$ , compressive stretches of λ = [ 0 . 9 , … , 1 . 0 ] $$ \lambda =\left[0.9,\dots, 1.0\right] $$ , and shear strains of γ = [ 0 . 0 , … , 0 . 2 ] $$ \gamma =\left[0.0,\dots, 0.2\right] $$ , in 16 equidistant increments each,29 for varying Mooney Rivlin network weights in the ranges w 1 = [ 0 , … , 1 ] $$ {w}_1=\left[0,\dots, 1\right] $$ and w 5 = [ 0 , … , 1 ] $$ {w}_5=\left[0,\dots, 1\right] $$ , and apply L p $$ {L}_p $$ regularization, α p [ | w 1 | p + | w 5 | p ] $$ {\alpha}_p\kern0.3em \left[\kern0.3em {\left|\kern0.3em {w}_1\kern0.3em \right|}^p+{\left|\kern0.3em {w}_5\kern0.3em \right|}^p\kern0.3em \right] $$ , for eight different powers, p = [ 0 . 25 , 0 . 5 , 0 . 75 , 1 , 1 . 5 , 2 , 4 , 8 ] $$ p=\left[0.25,0.5,0.75,1,1.5,2,4,8\right] $$ . We select a penalty parameter α max = 0 . 6585 $$ {\alpha}^{\mathrm{max}}=0.6585 $$ , such that the maximum value of the loss function within the screened parameter window, in the dark red upper right corner, at w 1 = 1 $$ {w}_1=1 $$ and w 5 = 1 $$ {w}_5=1 $$ , consists of equal contributions by the network term and the regularization term. For each set of hyperparameters { p , α } $$ \left\{p,\alpha \right\} $$ , we increase the penalty parameter in four increments, α = [ 0 . 00 , 0 . 25 , 0 . 50 , 1 . 00 ] α max $$ \alpha =\left[0.00,0.25,0.50,1.00\right]{\alpha}^{\mathrm{max}} $$ , indicated through the four hyperplanes in each graph, and highlight the minimum of the last of these four loss functions L ( w i , w j ; λ , γ ) $$ L\left({w}_i,{w}_j;\lambda, \gamma \right) $$ through a white sphere. Similar to Figure 6 based on synthetic data, the minimum of the loss function and its location in the { w 1 , w 5 } $$ \left\{{w}_1,{w}_5\right\} $$ -space are functions for the two hyperparameters p $$ p $$ and α $$ \alpha $$ . For the plain non-regularized loss function, the minimum of the loss function is 0.0713 and its weights are w 1 = 0 . 00 $$ {w}_1=0.00 $$ and w 5 = 0 . 84 $$ {w}_5=0.84 $$ . For the eight powers, the minima of the loss function at the location of the white sphere are min ( L ) = [   0 . 67 , 0 . 63 , 0 . 56 , 0 . 50 , 0 . 34 , 0 . 24 , 0 . 11 , 0 . 08   ] $$ \min (L)=\left[\;0.67,0.63,0.56,0.50,0.34,0.24,0.11,0.08\;\right] $$ , and their varying locations in the { w 1 , w 5 } $$ \left\{{w}_1,{w}_5\right\} $$ -space are indicated through the white spheres in Figure 8.

Details are in the caption following the image
FIGURE 8
Open in figure viewerPowerPoint
Loss functions of Lp regularized Mooney Rivlin model for real data. Contours of the L p $$ {L}_p $$ regularized loss function, L ( w 1 , w 5 ; λ , γ ) $$ L\left({w}_1,{w}_5;\lambda, \gamma \right) $$ , for the two-term Mooney Rivlin model with varying powers, p = [ 0 . 25 , 0 . 5 , 0 . 75 , 1 , 1 . 5 , 2 , 4 , 8 ] $$ p=\left[0.25,0.5,0.75,1,1.5,2,4,8\right] $$ , evaluated for the two parameters, w 1 $$ {w}_1 $$ and w 5 $$ {w}_5 $$ , for real data from tension, compression, and shear tests of human brain. For p ≤ 1 $$ p\le 1 $$ , top row, with the special case of L 1 $$ {L}_1 $$ regularization or lasso in the fourth column, L p $$ {L}_p $$ regularization promotes sparsity by training w 1 = 0 $$ {w}_1=0 $$ exactly to zero, but the loss function is no longer strictly convex and has multiple local minima. For p > 1 $$ p>1 $$ , bottom row, with the special case of L 2 $$ {L}_2 $$ regularization or ridge regression in the second column, L p $$ {L}_p $$ regularization promotes stability, retains both non-weights, w 1 > 0 $$ {w}_1>0 $$ and w 5 > 0 $$ {w}_5>0 $$ , and maintains a convex loss function with a single global minimum.

Figure 8 reveals several interesting differences between the loss function for synthetic data in Figure 6 and for real data, in this case from human brain experiments, in Figure 8. Most importantly, for the synthetic data, we assumed an exact minimum at w 1 = 1 $$ {w}_1=1 $$ and w 5 = 1 $$ {w}_5=1 $$ , where the loss function is exactly zero for the non- L p $$ {L}_p $$ -regularized model, and takes the value of the regularization term α | | θ | | p p $$ \alpha \kern0.3em {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_p^p $$ otherwise. For the real data, we no longer know a priori where the exact minimum is and it is no longer exactly zero, since the Mooney Rivlin model is not exact for the real data. From screening the parameter plane, find the minimum loss at w 1 = 0 . 00 $$ {w}_1=0.00 $$ and w 5 = 0 . 84 $$ {w}_5=0.84 $$ . Strikingly, this suggests that the one-parameter Blatz Ko model58 with ψ = w 5 [ I 2 − 3 ] $$ \psi ={w}_5\kern0.3em \left[\kern0.3em {I}_2-3\kern0.3em \right] $$ and P = w 5 ∂ I 2 / ∂ P − p P − t $$ P={w}_5\kern0.3em \partial {I}_2/\partial \boldsymbol{P}-p\kern0.3em {\boldsymbol{P}}^{-t} $$ is better suited to describe the experimental data than the two-parameter Mooney Rivlin59, 60 model. However, we can clearly see the negative effect of over-regularization with too large penalty parameters α $$ \alpha $$ : for p < 1 $$ p<1 $$ , in the top row, the minimum of the loss function remains on the w 1 = 0 . 00 $$ {w}_1=0.00 $$ axis, but the Blatz Ko parameter is drastically reduced from its non-regularized value of w 5 = 0 . 84 $$ {w}_5=0.84 $$ to w 5 = 0 . 59 $$ {w}_5=0.59 $$ , w 5 = 0 . 49 $$ {w}_5=0.49 $$ , and even w 5 = 0 $$ {w}_5=0 $$ . For p ≥ 1 $$ p\ge 1 $$ , in the bottom row, the minimum of the loss function even moves away from the w 1 = 0 . 00 $$ {w}_1=0.00 $$ axis, and both parameters become activated at a similar magnitude between w 1 = [ 0 . 26 , … , 0 . 36 ] $$ {w}_1=\left[0.26,\dots, 0.36\right] $$ and w 5 = [ 0 . 27 , … , 0 . 47 ] $$ {w}_5=\left[0.27,\dots, 0.47\right] $$ . Taken together, the discovered weights w 1 $$ {w}_1 $$ and w 5 $$ {w}_5 $$ are highly sensitive to over-regularization for extreme ranges of the hyperparameters p $$ p $$ and α $$ \alpha $$ : extreme penalty parameters induce increased bias as the loss function increasingly focuses on minimizing the penalty term rather than the regularization problem itself.

4.1 L   p $$ {}_{\boldsymbol{p}} $$ regularized invariant based neural network

Similar to the previous example, we explore the effects of L p $$ {L}_p $$ regularization with respect to the two hyperparameters p $$ p $$ and α $$ \alpha $$ , but now for the full eight-term invariant based network,29 instead of the two-term Mooney Rivlin model,59, 60 and for training on real instead of synthetic data. We use tension, compression, and shear data from human brain tests,62 over a tensile range of λ = [ 1 . 0 , … , 1 . 1 ] $$ \lambda =\left[1.0,\dots, 1.1\right] $$ , a compressive range of λ = [ 0 . 9 , … , 1 . 0 ] $$ \lambda =\left[0.9,\dots, 1.0\right] $$ , and a shear range of γ = [ 0 . 0 , … , 0 . 2 ] $$ \gamma =\left[0.0,\dots, 0.2\right] $$ , sampled in 16 equidistant increments each, averaged over anywhere between n = 15 $$ n=15 $$ and n = 35 $$ n=35 $$ specimen.29 We train the invariant based neural network from Figure 2 in Section 3.1 and minimize the loss function from Equation (38) with the stress definitions (20) and (23) with three different powers, p = [ 0 . 5 , 1 . 0 , 2 . 0 ] $$ p=\left[0.5,1.0,2.0\right] $$ , and four different penalty parameters, α = [ 0 . 000 , 0 . 001 , 0 . 010 , 0 . 100 ] $$ \alpha =\left[0.000,0.001,0.010,0.100\right] $$ . We use the Adam optimizer, a robust adaptive algorithm for stochastic gradient-based first-order optimization.68

Figure 9 summarizes our four discovered models in terms of the nominal stress as a function of stretch or shear strain, with the penalty parameter α $$ \alpha $$ increasing from left to right. The circles represent the experimental data.62 The color-coded regions represent the stress contributions of the eight model terms according to Figure 2. The coefficients of determination R 2 $$ {R}^2 $$ quantify the goodness of fit. Overall, the L p $$ {L}_p $$ regularized invariant based network trains solidly and provides a good fit of the data. Without regularization, in the left column, the network discovers four non-zero terms, all in terms of the second invariant, [   I 2 − 3   ] $$ \left[\;{I}_2-3\;\right] $$ , indicated through cold green-to-blue colors,
ψ = w 5 [ I 2 − 3 ] + w 2 , 6 [ exp ( w 1 , 6 [ I 2 − 3 ] ) − 1 ] + w 7 [ I 2 − 3 ] 2 + w 2 , 8 [ exp ( w 1 , 8 [ I 2 − 3 ] 2 ) − 1 ] , $$ \psi ={w}_5\kern0.3em \left[\kern0.3em {I}_2-3\kern0.3em \right]+{w}_{2,6}\kern0.3em \left[\kern0.3em \exp \left(\kern0.3em {w}_{1,6}\kern0.3em \left[\kern0.3em {I}_2-3\kern0.3em \right]\right)-1\kern0.3em \right]+{w}_7\kern0.3em {\left[\kern0.3em {I}_2-3\kern0.3em \right]}^2+{w}_{2,8}\kern0.3em \left[\kern0.3em \exp \left(\kern0.3em {w}_{1,8}\kern0.3em {\left[\kern0.3em {I}_2-3\kern0.3em \right]}^2\right)-1\kern0.3em \right], $$
with stiffness-like parameters w 5 = 0 . 129 $$ {w}_5=0.129 $$  kPa, w 2 , 6 = 0 . 358 $$ {w}_{2,6}=0.358 $$  kPa, w 7 = 3 . 840 $$ {w}_7=3.840 $$  kPa, and w 2 , 8 = 1 . 406 $$ {w}_{2,8}=1.406 $$  kPa and exponential weights, w 1 , 6 = 1 . 152 $$ {w}_{1,6}=1.152 $$ and w 1 , 8 = 2 . 891 $$ {w}_{1,8}=2.891 $$ , and its stress takes the following form, P = [ w 5 + w 2 , 6 w 1 , 6 exp (   w 1 , 6 [   I 2 − 3   ] ) + 2   [   I 2 − 3   ] [ w 7 + w 2 , 8 w 1 , 8 exp (   w 1 , 8 [   I 2 − 3   ] 2 ) ] ] ∂ I 2 / ∂ F − p   F − t $$ \boldsymbol{P}=\left[{w}_5+{w}_{2,6}{w}_{1,6}\exp \left(\;{w}_{1,6}\left[\;{I}_2-3\;\right]\right)+2\;\left[\;{I}_2-3\;\right]\left[{w}_7+{w}_{2,8}{w}_{1,8}\exp \left(\;{w}_{1,8}{\left[\;{I}_2-3\;\right]}^2\right)\right]\right]\partial {I}_2/\partial \boldsymbol{F}-p\;{\boldsymbol{F}}^{-\mathrm{t}} $$ . As the penalty parameter increases, from left to right, the number of non-zero terms decreases. With a penalty parameter α = 0 . 010 $$ \alpha =0.010 $$ , in the third column, the network discovers three non-zero terms, all in terms of the second invariant, [   I 2 − 3   ] $$ \left[\;{I}_2-3\;\right] $$ , indicated through cold green-to-light-blue colors,
ψ = w 5 [ I 2 − 3 ] + w 2 , 6 [ exp ( w 1 , 6 [ I 2 − 3 ] ) − 1 ] + w 7 [ I 2 − 3 ] 2 , $$ \psi ={w}_5\kern0.3em \left[\kern0.3em {I}_2-3\kern0.3em \right]+{w}_{2,6}\kern0.3em \left[\kern0.3em \exp \left(\kern0.3em {w}_{1,6}\kern0.3em \left[\kern0.3em {I}_2-3\kern0.3em \right]\right)-1\kern0.3em \right]+{w}_7\kern0.3em {\left[\kern0.3em {I}_2-3\kern0.3em \right]}^2, $$
with stiffness-like parameters w 5 = 0 . 231 $$ {w}_5=0.231 $$  kPa, w 2 , 6 = 1 . 443 $$ {w}_{2,6}=1.443 $$  kPa, and w 7 = 7 . 364 $$ {w}_7=7.364 $$  kPa, and the exponential weight, w 1 , 6 = 5 . 102 $$ {w}_{1,6}=5.102 $$ , and its stress takes the following form, P = [ w 5 + w 2 , 6 w 1 , 6 exp (   w 1 , 6 [   I 2 − 3   ] ) + 2   [   I 2 − 3   ] w 7 ] ∂ I 2 / ∂ F − p   F − t $$ \boldsymbol{P}=\left[{w}_5+{w}_{2,6}{w}_{1,6}\exp \left(\;{w}_{1,6}\left[\;{I}_2-3\;\right]\right)+2\;\left[\;{I}_2-3\;\right]{w}_7\right]\partial {I}_2/\partial \boldsymbol{F}-p\;{\boldsymbol{F}}^{-\mathrm{t}} $$ . For the largest penalty parameter, in the right column, the network discovers a single non-zero term, the turquoise linear exponential term of the second invariant,
ψ = w 2 , 6 [ exp ( w 1 , 6 [ I 2 − 3 ] ) − 1 ] , $$ \psi ={w}_{2,6}\kern0.3em \left[\kern0.3em \exp \left(\kern0.3em {w}_{1,6}\kern0.3em \left[\kern0.3em {I}_2-3\kern0.3em \right]\right)-1\kern0.3em \right], $$
with the stiffness-like parameter w 2 , 6 = 0 . 2462 $$ {w}_{2,6}=0.2462 $$  kPa and the exponential weight w 1 , 6 = 2 . 9937 $$ {w}_{1,6}=2.9937 $$ , and its stress takes the following form, P = w 2 , 6 w 1 , 6 exp ( w 1 , 6 [ I 2 − 3 ] ) ∂ I 2 / ∂ F − p F − t $$ \boldsymbol{P}={w}_{2,6}\kern0.3em {w}_{1,6}\kern0.3em \exp \left(\kern0.3em {w}_{1,6}\kern0.3em \left[\kern0.3em {I}_2-3\kern0.3em \right]\right)\kern0.3em \partial {I}_2/\partial \boldsymbol{F}-p\kern0.3em {\boldsymbol{F}}^{-\mathrm{t}} $$ . While Figure 9 provides great visual insights into the performance of L 1 $$ {L}_1 $$ regularization with varying penalty parameters, it only represents a snapshot of model discovery in the eight-dimensional parameter space of the network. Subset selection and model discovery are not only sensitive to the initialization of the parameter vector θ = { w i } $$ \boldsymbol{\theta} =\left\{\kern0.3em {w}_i\kern0.3em \right\} $$ , but also to the stochastic nature of the Adam optimizer. This implies that different runs may produce different results. This raises the question how reproducible and robust the results in Figure 9 are for varying initial conditions and training runs.
Details are in the caption following the image
FIGURE 9
Open in figure viewerPowerPoint
Discovered models of L1 regularized invariant based network. Nominal stress as a function of stretch or shear strain for the invariant based neural network with L 1 $$ {L}_1 $$ regularization for varying penalty parameters α = [ 0 . 000 , 0 . 001 , 0 . 010 , 0 . 100 ] $$ \alpha =\left[\kern0.3em 0.000,0.001,0.010,0.100\kern0.3em \right] $$ , trained with human gray matter tension, compression, and shear data. Circles represent the experimental data. Color-coded regions represent the discovered model terms. Coefficients of determination R 2 $$ {R}^2 $$ indicate the goodness of fit.

Figure 10 summarizes the discovered weights for the invariant based network with L p $$ {L}_p $$ regularization for varying powers p = [   0 . 5 , 1 . 0 , 2 . 0 ] $$ p=\left[\;0.5,1.0,2.0\right] $$ and penalty parameters α = [ 0 . 000 , 0 . 001 , 0 . 010 , 0 . 100 ] $$ \alpha =\left[\kern0.3em 0.000,0.001,0.010,0.100\kern0.3em \right] $$ . For all twelve combinations of the two hyperparameters, we perform a total of n = 100 $$ n=100 $$ training runs each, with varying initial conditions for the network weights w i = {   w 1 , … , w 8   } $$ {w}_i=\left\{\;{w}_1,\dots, {w}_8\;\right\} $$ , such that each of the four models in Figure 9 is the result of one of the L 1 $$ {L}_1 $$ regularized training runs in the middle row. The colored boxes in Figure 10 indicate the relevance of the eight model terms, with their means and standard deviations. Interestingly, the L 0 . 5 $$ {L}_{0.5} $$ and L 0 . 1 $$ {L}_{0.1} $$ regularizations in the first and second rows perform qualitatively similarly: they both start with four dominant terms, all in terms of the second invariant. Except for a small number of outliers, they both converge to two dominant one-term models, the green [   I 2 − 3   ] $$ \left[\;{I}_2-3\;\right] $$ and the turquoise exp ( [   I 2 − 3   ] ) $$ \exp \left(\left[\;{I}_2-3\;\right]\right) $$ models, while all other weights train to zero. The fact that both networks alternate between these two terms is a result of the non-convex nature of the underlying nonlinear regression problem associated with the invariant based network and indicates the existence of multiple local minima. Instead, the L 2 $$ {L}_2 $$ regularization in the bottom row converges to a model that consistently trains the dark red [   I 1 − 3   ] $$ \left[\;{I}_1-3\;\right] $$ and red exp ( [   I 1 − 3   ] ) $$ \exp \left(\left[\;{I}_1-3\;\right]\right) $$ terms to zero, and maintains six non-zero terms, of which the yellow exp ( [   I 1 − 3   ] 2 ) $$ \exp \left({\left[\;{I}_1-3\;\right]}^2\right) $$ , turquoise exp ( [   I 2 − 3   ] ) $$ \exp \left(\left[\;{I}_2-3\;\right]\right) $$ , and dark blue exp ( [   I 2 − 3   ] 2 ) $$ \exp \left({\left[\;{I}_2-3\;\right]}^2\right) $$ terms are dominant.

Details are in the caption following the image
FIGURE 10
Open in figure viewerPowerPoint
Discovered models of Lp regularized invariant based network. Distribution of discovered weights for the invariant based neural network with L p $$ {L}_p $$ regularization for varying powers p = [   0 . 5 , 1 . 0 , 2 . 0 ] $$ p=\left[\;0.5,1.0,2.0\right] $$ and penalty parameters α = [ 0 . 000 , 0 . 001 , 0 . 010 , 0 . 100 ] $$ \alpha =\left[\kern0.3em 0.000,0.001,0.010,0.100\kern0.3em \right] $$ . Colored boxes indicate the relevance of the eight model terms, with means and standard deviations from n = 100 $$ n=100 $$ realizations with varying initializations of the network weights.

Figure 11 summarizes the convergence of the L p $$ {L}_p $$ regularized invariant based network in terms of the goodness of fit and number of terms, for varying powers p = [   0 . 5 , 1 . 0 , 2 . 0   ] $$ p=\left[\;0.5,1.0,2.0\;\right] $$ and penalty parameters α = [ 0 . 000 , 0 . 001 , 0 . 005 , 0 . 010 , 0 . 015 , 0 . 020 , 0 . 040 , 0 . 060 , 0 . 080 , 0 . 100 ] $$ \alpha =\left[\kern0.3em 0.000,0.001,0.005,0.010,0.015,0.020,0.040,0.060,0.080,0.100\kern0.3em \right] $$ . Red dots indicate the coefficient of determination R 2 $$ {R}^2 $$ , blue dots indicate the number of terms, with means and standard deviations from n = 10 $$ n=10 $$ realizations. A known shortcoming of the L p $$ {L}_p $$ regularization is that it introduces bias and moves the solution away from the minimum of the network loss towards the minimum of the regularization loss. This is particularly critical for our network in which all weights have a different meaning and potentially also a different magnitude. To quantify the effects of this potential limitation, Figure 11 compares the non-normalized regularization, | |   θ   | | p p = ∑ i = 1 n para |   w i   | p $$ {\left\Vert\;\boldsymbol{\theta}\;\right\Vert}_p^p={\sum}_{i=1}^{n_{\mathrm{para}}}{\left|\;{w}_i\;\right|}^p $$ , in terms of the weights w i $$ {w}_i $$ that we have used throughout this study against a normalized regularization, | |   θ   | | p p = ∑ i = 1 n para |   w i   /   w i , L 0   | p $$ {\left\Vert\;\boldsymbol{\theta}\;\right\Vert}_p^p={\sum}_{i=1}^{n_{\mathrm{para}}}{\left|\;{w}_i/{w}_{i,{L}_0}\;\right|}^p $$ , in terms of the L 0 $$ {L}_0 $$ -normalized weights w i / w i , L 0 $$ {w}_i\kern0.3em /\kern0.3em {w}_{i,{L}_0} $$ . Here, w i , L 0 $$ {w}_{i,{L}_0} $$ are the weights of the one-term models from the diagonal in Table 1.

Details are in the caption following the image
FIGURE 11
Open in figure viewerPowerPoint
Convergence of Lp regularized invariant based network. Goodness of fit and number of terms for the invariant based neural network with L p $$ {L}_p $$ regularization for varying powers p = [   0 . 5 , 1 . 0 , 2 . 0   ] $$ p=\left[\;0.5,1.0,2.0\;\right] $$ and penalty parameters α = [ 0 . 000 , 0 . 001 , 0 . 005 , 0 . 010 , 0 . 015 , 0 . 020 , 0 . 040 , 0 . 060 , 0 . 080 , 0 . 100 ] $$ \alpha =\left[\kern0.3em 0.000,0.001,0.005,0.010,0.015,0.020,0.040,0.060,0.080,0.100\kern0.3em \right] $$ . Top row uses a non-normalized regularization in terms of the weights w i $$ {w}_i $$ , bottom row uses a normalized regularization in terms of the L 0 $$ {L}_0 $$ -normalized weights w i / w i , L 0 $$ {w}_i\kern0.3em /\kern0.3em {w}_{i,{L}_0} $$ . Red dots indicate the coefficient of determination R 2 $$ {R}^2 $$ , blue dots indicate the number of terms, with means and standard deviations from n = 10 $$ n=10 $$ realizations.
TABLE 1. L   0 $$ {}_{\mathbf{0}} $$ regularized invariant based neural network.
w   1 $$ {}_{\mathbf{1}} $$ w   1,2 $$ {}_{\mathbf{1},\mathbf{2}} $$ , w   2,2 $$ {}_{\mathbf{2},\mathbf{2}} $$ w   3 $$ {}_{\mathbf{3}} $$ w   1,4 $$ {}_{\mathbf{1},\mathbf{4}} $$ , w   2,4 $$ {}_{\mathbf{2},\mathbf{4}} $$ w   5 $$ {}_{\mathbf{5}} $$ w   1,6 $$ {}_{\mathbf{1},\mathbf{6}} $$ , w   2,6 $$ {}_{\mathbf{2},\mathbf{6}} $$ w   7 $$ {}_{\mathbf{7}} $$ w   1,8 $$ {}_{\mathbf{1},\mathbf{8}} $$ , w   2,8 $$ {}_{\mathbf{2},\mathbf{8}} $$
w   1 $$ {}_{\mathbf{1}} $$ 0.796

0.237

0.918, 0.600

0.400

10.048

0.403

3.666, 2.718

0.000

0.840

0.000

0.957, 0.865

0.330

12.545

0.330

3.810, 3.286
Loss 0.092 + α $$ \alpha $$ 0.090 + 2 α $$ \alpha $$ 0.060 + 2 α $$ \alpha $$ 0.060 + 2 α $$ \alpha $$ 0.071 + 2 α $$ \alpha $$ 0.069 + 2 α $$ \alpha $$ 0.040 + 2 α $$ \alpha $$ 0.040 + 2 α $$ \alpha $$
w   1,2 $$ {}_{\mathbf{1},\mathbf{2}} $$ ,w   2,2 $$ {}_{\mathbf{2},\mathbf{2}} $$

0.918, 0.600

0.237

 1.076, 0.727

0.980, 0.410

9.811

1.219, 0.329

4.167, 2.342

0.369, 0.000

0.841

0.558, 0.000

3.186, 0.250

1.095, 0.294

12.547

0.822, 0.407

4.089, 2.993
Loss 0.090 + 2 α $$ \alpha $$ 0.089 + α $$ \alpha $$ 0.060 + 2 α $$ \alpha $$ 0.060 + 2 α $$ \alpha $$ 0.071 + 2 α $$ \alpha $$ 0.063 + 2 α $$ \alpha $$ 0.040 + 2 α $$ \alpha $$ 0.040 + 2 α $$ \alpha $$
w   3 $$ {}_{\mathbf{3}} $$

10.048

0.400

9.811

0.980, 0.410

 18.348

9.388

3.011, 2.977

8.151

0.507

8.173

0.876, 0.569

8.216

10.916

8.178

3.193, 3.422
Loss 0.060 + 2 α $$ \alpha $$ 0.060 + 2 α $$ \alpha $$ 0.086 + α $$ \alpha $$ 0.086 + 2 α $$ \alpha $$ 0.049 + 2 α $$ \alpha $$ 0.049 + 2 α $$ \alpha $$ 0.069 + 2 α $$ \alpha $$ 0.070
w   1,4 $$ {}_{\mathbf{1},\mathbf{4}} $$ , w   2,4 $$ {}_{\mathbf{2},\mathbf{4}} $$

3.666, 2.718

0.403

4.167, 2.342

1.219, 0.329

3.011, 2.977

9.388

 4.305, 4.244

3.134, 2.672

0.497

3.014, 2.669

1.266, 0.394

3.210, 2.559

10.896

2.973, 2.737

3.149, 3.477
Loss 0.060 + 2 α $$ \alpha $$ 0.060 + 2 α $$ \alpha $$ 0.086 + 2 α $$ \alpha $$ 0.086 + α $$ \alpha $$ 0.049 + 2 α $$ \alpha $$ 0.048 + 2 α $$ \alpha $$ 0.070 + 2 α $$ \alpha $$ 0.070 + 2 α $$ \alpha $$
w   5 $$ {}_{\mathbf{5}} $$

0.840

0.000

0.841

0.369, 0.000

0.507

8.151

0.497

3.134, 2.672

 0.840

0.234

0.747, 0.803

0.406

11.178

0.411

3.644, 3.030
Loss 0.071 + 2 α $$ \alpha $$ 0.071 + 2 α $$ \alpha $$ 0.049 + 2 α $$ \alpha $$ 0.049 + 2 α $$ \alpha $$ 0.071 + α $$ \boldsymbol{\alpha} $$ 0.070 + 2 α $$ \alpha $$ 0.033 + 2 α $$ \boldsymbol{\alpha} $$ 0.033 + 2 α $$ \boldsymbol{\alpha} $$
w   1,6 $$ {}_{\mathbf{1},\mathbf{6}} $$ ,w   2,6 $$ {}_{\mathbf{2},\mathbf{6}} $$

0.957, 0.865

0.000

3.186, 0.250

0.558, 0.000

0.876, 0.569

8.173

1.266, 0.394

3.014, 2.669

0.747, 0.803

0.234

 0.898, 0.923

1.022, 0.396

11.014

0.839, 0.484

3.427, 3.208
Loss 0.069 + 2 α $$ \alpha $$ 0.063 + 2 α $$ \alpha $$ 0.049 + 2 α $$ \alpha $$ 0.048 + 2 α $$ \alpha $$ 0.070 + 2 α $$ \alpha $$ 0.069 + α $$ \boldsymbol{\alpha} $$ 0.033 + 2 α $$ \boldsymbol{\alpha} $$ 0.033 + 2 α $$ \boldsymbol{\alpha} $$
w   7 $$ {}_{\mathbf{7}} $$

12.545

0.330

12.547

1.095, 0.294

10.916

8.216

10.896

3.210, 2.559

11.178

0.406

11.014

1.022, 0.396

 19.599

9.520

3.565, 2.834
Loss 0.040 + 2 α $$ \alpha $$ 0.040 + 2 α $$ \alpha $$ 0.069 + 2 α $$ \alpha $$ 0.070 + 2 α $$ \alpha $$ 0.033 + 2 α $$ \boldsymbol{\alpha} $$ 0.033 + 2 α $$ \boldsymbol{\alpha} $$ 0.059 + α $$ \boldsymbol{\alpha} $$ 0.059 + 2 α $$ \alpha $$
w   1,8 $$ {}_{\mathbf{1},\mathbf{8}} $$ , w   2,8 $$ {}_{\mathbf{2},\mathbf{8}} $$

3.810, 3.286

0.330

4.089, 2.993

0.822, 0.407

3.193, 3.422

8.178

3.149, 3.477

2.973, 2.737

3.644, 3.030

0.411

3.427, 3.208

0.839, 0.484

3.565, 2.834

9.520

 4.560, 4.280
Loss 0.040 + 2 α $$ \alpha $$ 0.040 + 2 α $$ \alpha $$ 0.070 + 2 α $$ \alpha $$ 0.070 + 2 α $$ \alpha $$ 0.033 + 2 α $$ \boldsymbol{\alpha} $$ 0.033 + 2 α $$ \boldsymbol{\alpha} $$ 0.059 + 2 α $$ \alpha $$ 0.060 + α $$ \boldsymbol{\alpha} $$
  • Note: Weights and remaining losses of the one- and two-term models of the L 0 $$ {L}_0 $$ regularized invariant based neural network. The diagonal summarizes the discovered one-term models penalized by α $$ \alpha $$ , the off-diagonal the two-term models penalized by 2 α $$ 2\alpha $$ . Best-in-class models are highlighted in bold.

Figure 11 confirms that regularization is a trade-off between error and complexity, or similarly, between the goodness of fit and the number of terms. While the L 0 . 5 $$ {L}_{0.5} $$ and L 1 $$ {L}_1 $$ regularizations behave qualitatively similarly and promote sparsity by reducing the number of non-zero terms to one, the L 2 $$ {L}_2 $$ promotes robustness by maintaining a large subset of six non-zero terms. The L 1 $$ {L}_1 $$ regularization is less aggressive than the L 0 . 5 $$ {L}_{0.5} $$ regularization and requires larger penalty parameters α $$ \alpha $$ to achieve a similar sparseness, which could induce a larger bias, away from minimum of the network loss towards the minimum of the regularization loss. Normalizing the L p $$ {L}_p $$ penalty term by using the L 0 $$ {L}_0 $$ -normalized weights w i / w i , L 0 $$ {w}_i\kern0.3em /\kern0.3em {w}_{i,{L}_0} $$ instead of the non-normalized weights w i $$ {w}_i $$ accelerates the positive effects of regularization, especially in the small-penalty-parameter regime, and could provide a viable solution to reduce regularization-induced bias. Ultimately, in the large-penalty-parameter regime, the non-normalized and normalized regularizations converge towards a similar goodness of fit and number of terms.

Taken together, our results confirm the general notion that L p $$ {L}_p $$ regularization increases interpretability for powers equal to or below one, p ≤ 1 $$ p\le 1 $$ , by promoting sparsity as a subset of weights train exactly to zero; and increases predictability for powers larger than one, p > 1 $$ p>1 $$ , by promoting robustness as a unique subset of weights emerges as dominant. Larger penalty parameters α $$ \alpha $$ amplify these trends at the price of an increased bias, which we can reduce, at least in part, by normalizing the network weights in the the penalty term.

L   0 $$ {}_{\mathbf{0}} $$ regularized invariant based neural network. For comparison, we explore the effects of L 0 $$ {L}_0 $$ regularization using the same tension, compression, and shear data from human brain tests as in the previous example.62 We train the invariant based neural network from Figure 2 in Section 3.1 and minimize the loss function from Equation (38) with the stress definitions (20) and (23), but now use a penalty term, α | | θ | | 0 $$ \alpha \kern0.3em {\left\Vert \kern0.3em \boldsymbol{\theta} \kern0.3em \right\Vert}_0 $$ , with the L 0 $$ {L}_0 $$ norm | |   θ   | | 0 = ∑ i = 1 n para I ( w i ≠ 0 ) $$ {\left\Vert\;\boldsymbol{\theta}\;\right\Vert}_0={\sum}_{i=1}^{n_{\mathrm{para}}}I\left({w}_i\ne 0\right) $$ , to penalize the total number of non-zero terms in the model. In essence, L 0 $$ {L}_0 $$ regularization turns network training into a discrete combinatorial problem with 2 8 − 1 = 255 $$ {2}^8-1=255 $$ possible models, 8 with a single term, 28 with two, 56 with three, 70 with four, 56 with five, 28 with six, 8 with seven, and 1 with all eight terms. For illustrative purposes, we focus on the eight one-term and 28 two-term models that follow by explicitly setting the other seven and six terms of the network to zero.

Table 1 summarizes the weights and remaining losses of the one- and two-term models of the L 0 $$ {L}_0 $$ regularized invariant based neural network. The diagonal summarizes the discovered one-term models, the off-diagonal the two-term models. The L 0 $$ {L}_0 $$ regularization penalizes the one-term models by α $$ \alpha $$ and the two-term models by 2 α $$ 2\alpha $$ . The boldface cells highlight the four best-in-class models of each category. Figures 12 and 13 illustrate the stress-stretch and stress-shear plots of these four one- and two-term models. Interestingly, all best-in-class models are models in terms of the second invariant I 2 $$ {I}_2 $$ indicated through the cold green-to-blue colors. None of the eight best models includes the first invariant I 1 $$ {I}_1 $$ indicated through the warm red-to-yellow colors. This finding contradicts the common practice of using primarily the first invariant, for example, in popular and widely used neo Hooke model. Strikingly, the classical Hooke model61 represented through the dark red term in both networks with ψ = w 1 [ I 1 − 3 ] $$ \psi ={w}_1\kern0.3em \left[\kern0.3em {I}_1-3\kern0.3em \right] $$ , a stiffness-like parameter w 1 = 0 . 7964 $$ {w}_1=0.7964 $$  kPa, a shear modulus μ = 2 w 1 = 1 . 5928 $$ \mu =2\kern0.3em {w}_1=1.5928 $$  kPa, and a remaining loss of 0 . 0918 + α $$ 0.0918+\alpha $$ has the largest remaining loss and performs the worst of all one-term models. Similarly, the Demiray model69 represented through the red term with ψ = w 2 , 2 [ exp ( w 1 , 2 [ I 1 − 3 ] ) − 1 ] $$ \psi ={w}_{2,2}\kern0.3em \left[\kern0.3em \exp \left(\kern0.3em {w}_{1,2}\kern0.3em \left[\kern0.3em {I}_1-3\kern0.3em \right]\kern0.3em \right)-1\kern0.3em \right] $$ , a stiffness-like parameter w 2 , 2 = 0 . 7265 $$ {w}_{2,2}=0.7265 $$  kPa, an exponent w 1 , 2 = 1 . 0763 $$ {w}_{1,2}=1.0763 $$ , and a remaining loss of 0 . 0894 + α $$ 0.0894+\alpha $$ , and the Holzapfel type model70 represented through the yellow term with ψ = w 2 , 4 [ exp ( w 1 , 4 [ I 1 − 3 ] 2 ) − 1 ] $$ \psi ={w}_{2,4}\kern0.3em \left[\kern0.3em \exp \left(\kern0.3em {w}_{1,4}\kern0.3em {\left[\kern0.3em {I}_1-3\kern0.3em \right]}^2\kern0.3em \right)-1\kern0.3em \right] $$ , a stiffness-like parameter w 2 , 4 = 4 . 2436 $$ {w}_{2,4}=4.2436 $$  kPa, an exponent w 1 , 4 = 4 . 3048 $$ {w}_{1,4}=4.3048 $$ , and a remaining loss of 0 . 0863 + α $$ 0.0863+\alpha $$ , also perform worse than all one-term second-invariant models. Yet, these results agree well with our previous observations that the second invariant is better suited to represent the behavior of brain tissue than the first invariant.29 The best-in-class one-term model with the lowest remaining loss is the model with the light blue quadratic term of the second invariant,
ψ = w 7 [ I 2 − 3 ] 2 , $$ \psi ={w}_7\kern0.3em {\left[\kern0.3em {I}_2-3\kern0.3em \right]}^2, $$
with the stiffness-like parameter w 7 = 19 . 5994 $$ {w}_7=19.5994 $$  kPa for which the stress takes the following form, P = 2 [ I 2 − 3 ] w 7 ∂ I 2 / ∂ F − p F − t $$ \boldsymbol{P}=2\kern0.3em \left[\kern0.3em {I}_2-3\kern0.3em \right]\kern0.3em {w}_7\kern0.3em \partial {I}_2/\partial \boldsymbol{F}-p\kern0.3em {\boldsymbol{F}}^{-\mathrm{t}} $$ . The best-in-class two-term model is the model with the turquoise linear exponential and the dark blue quadratic exponential terms of the second invariant,
ψ = w 2 , 6 [ exp ( w 1 , 6 [ I 2 − 3 ] ) − 1 ] + w 2 , 8 [ exp ( w 1 , 8 [ I 2 − 3 ] ) 2 − 1 ] , $$ \psi ={w}_{2,6}\kern0.3em \left[\kern0.3em \exp \left(\kern0.3em {w}_{1,6}\kern0.3em \left[\kern0.3em {I}_2-3\kern0.3em \right]\right)-1\kern0.3em \right]+{w}_{2,8}\kern0.3em \left[\kern0.3em \exp {\left(\kern0.3em {w}_{1,8}\kern0.3em \left[\kern0.3em {I}_2-3\kern0.3em \right]\right)}^2-1\kern0.3em \right], $$
with the stiffness-like parameters w 2 , 6 = 0 . 4835 $$ {w}_{2,6}=0.4835 $$  kPa and w 2 , 8 = 3 . 2080 $$ {w}_{2,8}=3.2080 $$  kPa, and the exponential weights w 1 , 6 = 0 . 8393 $$ {w}_{1,6}=0.8393 $$ and w 1 , 8 = 3 . 4273 $$ {w}_{1,8}=3.4273 $$ , for which the stress takes the following form, P = [ w 2 , 6 w 1 , 6 exp ( w 1 , 6 [ I 2 − 3 ] ) + 2 [ I 2 − 3 ] w 2 , 8 w 2 , 8 exp ( w 1 , 8 [ I 2 − 3 ] ) 2 ] ∂ I 2 / ∂ F − p F − t $$ \boldsymbol{P}=\left[\kern0.3em {w}_{2,6}\kern0.3em {w}_{1,6}\kern0.3em \exp \left(\kern0.3em {w}_{1,6}\kern0.3em \left[\kern0.3em {I}_2-3\kern0.3em \right]\right)+2\kern0.3em \left[\kern0.3em {I}_2-3\kern0.3em \right]\kern0.3em {w}_{2,8}\kern0.3em {w}_{2,8}\kern0.3em \exp {\left(\kern0.3em {w}_{1,8}\kern0.3em \left[\kern0.3em {I}_2-3\kern0.3em \right]\right)}^2\right]\kern0.3em \partial {I}_2/\partial \boldsymbol{F}-p\kern0.3em {\boldsymbol{F}}^{-\mathrm{t}} $$ . For this simple example, the remaining loss of the best one-term model is 0 . 0594 + α $$ 0.0594+\alpha $$ and the remaining loss of the best two-term model is 0 . 0328 + 2 α $$ 0.0328+2\alpha $$ . This implies that, for penalty parameters α < 0 . 0266 $$ \alpha <0.0266 $$ , the L 0 $$ {L}_0 $$ regularization would favor the two-term model, while for penalty parameters α ≥ 0 . 0266 $$ \alpha \ge 0.0266 $$ , the L 0 $$ {L}_0 $$ regularization would favor the one-term model. These simple considerations highlight the importance of the penalty parameter α $$ \alpha $$ , which explicitly acts as a discrete switch between the number of terms we want to include in our model.
Details are in the caption following the image
FIGURE 12
Open in figure viewerPowerPoint
Discovered best-in-class one-term models of L   0 $$ {}_{\mathbf{0}} $$ regularized invariant based network. Nominal stress as a function of stretch or shear strain for the invariant based constitutive neural network with L 0 $$ {L}_0 $$ regularization, trained with human gray matter tension, compression, and shear data. Circles represent the experimental data. Color-coded regions represent the discovered model terms. Coefficients of determination R 2 $$ {R}^2 $$ indicate the goodness of fit for each individual test; remaining loss L $$ L $$ indicates the quality of the overall fit.
Details are in the caption following the image
FIGURE 13
Open in figure viewerPowerPoint
Discovered best-in-class two-term models of L   0 $$ {}_{\mathbf{0}} $$ regularized invariant based network. Nominal stress as a function of stretch or shear strain for the invariant based constitutive neural network with L 0 $$ {L}_0 $$ regularization, trained with human gray matter tension, compression, and shear data. Circles represent the experimental data. Color-coded regions represent the discovered model terms. Coefficients of determination R 2 $$ {R}^2 $$ indicate the goodness of fit for each individual test; remaining loss L $$ L $$ indicates the quality of the overall fit.

Taken together, this example illustrates the discrete nature of the L 0 $$ {L}_0 $$ regularization as a discrete combinatorial problem that becomes increasingly expensive as the number of model terms increases. Our results emphasize the sensitivity of the L 0 $$ {L}_0 $$ regularization with respect to the penalty parameter α $$ \alpha $$ and highlight the trade-off between bias and variance: increasing the penalty parameter increases bias, reduces variance, and decreases model complexity as the total number of non-zero terms decreases towards one.

4.2 Lp regularized principal stretch based neural network

Similar to the previous example, we explore the effects of L p $$ {L}_p $$ regularization with respect to the two hyperparameters p $$ p $$ and α $$ \alpha $$ , but now for the full principal stretch based network with all eight terms,29 for training on tension, compression, and shear data from human brain tests.62 We train the principal based neural network from Figure 4 in Section 3.2 and minimize the loss function from Equation (38) with the stress definitions (33) and (36) with three different powers, p = [ 0 . 5 , 1 . 0 , 2 . 0 ] $$ p=\left[0.5,1.0,2.0\right] $$ , and four different penalty parameters, α = [ 0 . 000 , 0 . 001 , 0 . 010 , 0 . 100 ] $$ \alpha =\left[0.000,0.001,0.010,0.100\right] $$ using the Adam optimizer.68

Figure 14 summarizes our four discovered models in terms of the nominal stress as a function of stretch or shear strain, with the penalty parameter α $$ \alpha $$ increasing from left to right. The circles represent the experimental data.62 The color-coded regions represent the stress contributions of the eight model terms according to Figure 4. The coefficients of determination R 2 $$ {R}^2 $$ quantify the goodness of fit. Similar to the invariant based network in Section 4.1, the L p $$ {L}_p $$ regularized principal stretch based network trains solidly and provides a good fit of the data. Without regularization, in the left column, the network discovers all eight non-zero terms. As the penalty parameter increases, from left to right, the number of non-zero terms decreases. For the largest penalty parameter, in the right column, the network discovers a single dominant term, the dark blue [ λ i − 8 − 3 ] $$ \left[\kern0.3em {\lambda}_i^{-8}-3\kern0.3em \right] $$ term,
ψ = w 8 ∑ i = 3 3 [ λ i − 8 − 1 ] $$ \psi ={w}_8\kern0.3em \sum \limits_{i=3}^3\kern0.3em \left[\kern0.3em {\lambda}_i^{-8}-1\kern0.3em \right] $$
with a stiffness-like parameter w 8 = 0 . 0534 $$ {w}_8=0.0534 $$  kPa and a stress P = − 8 w 8 ∑ i = 3 3 [ λ i − 9 − 1 ] n i ⊗ N i − p F − t $$ \boldsymbol{P}=-8\kern0.3em {w}_8\kern0.3em {\sum}_{i=3}^3\kern0.3em \left[\kern0.3em {\lambda}_i^{-9}-1\kern0.3em \right]\kern0.3em {\boldsymbol{n}}_i\otimes {\boldsymbol{N}}_i-p\kern0.3em {\boldsymbol{F}}^{-\mathrm{t}} $$ .
Details are in the caption following the image
FIGURE 14
Open in figure viewerPowerPoint
Discovered models of L1 regularized principal stretch based network. Nominal stress as a function of stretch or shear strain for the principal stretch based neural network with L 1 $$ {L}_1 $$ regularization for varying penalty parameters α = [ 0 , 0 . 1 , 0 . 001 , 0 . 0001 ] $$ \alpha =\left[\kern0.3em 0,0.1,0.001,0.0001\kern0.3em \right] $$ , trained with human gray matter tension, compression, and shear data. Circles represent the experimental data. Color-coded regions represent the stress contributions of the eight model terms. Coefficients of determination R 2 $$ {R}^2 $$ indicate the goodness of fit.

Figure 15 summarizes the discovered weights for the principal stretch based network with L p $$ {L}_p $$ regularization for varying powers p = [   0 . 5 , 1 . 0 , 2 . 0 ] $$ p=\left[\;0.5,1.0,2.0\right] $$ and penalty parameters α = [ 0 . 000 , 0 . 001 , 0 . 010 , 0 . 100 ] $$ \alpha =\left[\kern0.3em 0.000,0.001,0.010,0.100\kern0.3em \right] $$ . For all twelve combinations of the two hyperparameters, we perform a total of n = 100 $$ n=100 $$ training runs each, with varying initial conditions for the network weights w i = {   w 1 , … , w 8   } $$ {w}_i=\left\{\;{w}_1,\dots, {w}_8\;\right\} $$ , such that each of the four models in Figure 14 is the result of one of the L 1 $$ {L}_1 $$ regularized training runs in the middle row. The colored boxes in Figure 15 indicate the relevance of the eight model terms, with their means and standard deviations. In contrast to the invariant based network in Section 4.1, the principal stretch based network consistently discovers similar terms across all three regularizations, with a clear preference for the dark blue [ λ i − 8 − 3 ] $$ \left[\kern0em {\lambda}_i^{-8}-3\kern0em \right] $$ term. The fact that all networks robustly discover similar terms is a result of the convex nature of the underlying linear regression problem associated with the principal stretch based network and indicates the existence of a single unique global minimum. However, the L 0 . 5 $$ {L}_{0.5} $$ and L 1 $$ {L}_1 $$ regularized networks gradually drop more non-zero terms as the penalty parameter increases, while the L 2 $$ {L}_2 $$ regularized network maintains all eight terms. As we had already anticipated from comparing Figures 3 and 5, the functional base of the principal stretch based network is more collinear than the base of the invariant based network, which result in a more gradual shift of the active weights, from w 1 $$ {w}_1 $$ towards w 8 $$ {w}_8 $$ , as the penalty parameter increases. The fact that all three regularizations converge to the boundary of our domain, the dark blue [ λ i − 8 − 3 ] $$ \left[\kern0.3em {\lambda}_i^{-8}-3\kern0.3em \right] $$ term with the minimum exponent of minus eight, suggests that the true best fit might lay outside the current parameter range, with even smaller exponents. This agrees well with previous studies50, 62 that have discovered one-term Ogden models with exponents of [ λ i − 18 − 3 ] $$ \left[\kern0em {\lambda}_i^{-18}-3\kern0em \right] $$ and [ λ i − 19 − 3 ] $$ \left[\kern0em {\lambda}_i^{-19}-3\kern0em \right] $$ .

Details are in the caption following the image
FIGURE 15
Open in figure viewerPowerPoint
Discovered models of Lp regularized principal stretch based network. Distribution of discovered weights for the principal stretch based neural network with L p $$ {L}_p $$ regularization for varying powers p = [   0 . 5 , 1 . 0 , 2 . 0 ] $$ p=\left[\;0.5,1.0,2.0\right] $$ and penalty parameters α = [ 0 . 000 , 0 . 001 , 0 . 010 , 0 . 100 ] $$ \alpha =\left[\kern0.3em 0.000,0.001,0.010,0.100\kern0.3em \right] $$ . Colored boxes indicate the relevance of the eight model terms, with means and standard deviations from n = 100 $$ n=100 $$ realizations with varying initializations of the network weights.

Taken together, this example illustrates that model discovery with L p $$ {L}_p $$ regularization generalizes well to different network types, irrespective of whether the terms are invariant or principal stretch based. Comparing both network types reveals that the method is sensitive to the nonlinear versus linear nature of the underlying regression problem: the nonlinear invariant based network alternates between different dominant terms associated with multiple local minima, while the linear principal stretch based network consistently discovers similar terms associated with a single unique global minimum.

5 CONCLUSION AND RECOMMENDATIONS

L p $$ {L}_p $$ regularization is a powerful technology to finetune the training process of a neural network. In automated model discovery, it provides the critical missing piece of the puzzle that enables a controlled down-selection of the discovered terms and focus on the most important features of the model while putting less emphasis on minor effects. By promoting sparsity of the parameter vector, L p $$ {L}_p $$ regularization inherently improves interpretability and provides valuable insights into the underlying nature of the data. Importantly, L p $$ {L}_p $$ regularization introduces two hyperparameters: the power p $$ p $$ by which it penalizes the individual model parameters, and the penalty parameter α $$ \alpha $$ by which it scales the relative importance of the regularization loss in comparison to the neural network loss. Both parameters enable a precise control of model discovery from data and it is crucial to understand their mathematical subtleties, computational implications, and physical effects. Here we reviewed the mathematics and computation of the most common representatives of the L p $$ {L}_p $$ family, and demonstrated their features in terms of two classes of constitutive neural networks, invariant and principal stretch based, trained with both, synthetic and real data. Training with synthetic data proved to be robust and stable, and generally provides excellent metrics for quality control since we know the exact solution. However, it remains a toy problem that fails to reveal the true usefulness in practical real-world applications. Training with real data was algorithmically robust, but challenging, since we know nothing about the exact solution. Our study uses neural networks as a tool for linear and nonlinear regression. We acknowledge that our results can be interpreted and well understood without resorting neural networks and generalize naturally to other regression techniques including symbolic regression, genetic programming, or system identification.

For conciseness, we have limited the scope of the present review: first, we only considered small networks with no more than eight terms, but point out that the automated model discovery generalizes well to isotropic networks with 12 terms,29 transversely isotropic networks with 16 terms,64 two-fiber family networks with 16 terms,11 and orthotropic networks with 32 terms. Second, we trained on all available data and did not investigate splitting the data into train and test sets, which we have done in our previous work.29, 50 Third, we did not explicitly study the effects of controlled noise, but point out that Figures 8-15 are all based on real experimental data with real natural noise. Fourth, we did not further explore hybrid top down approaches like SINDy,20 since the nonlinear nature of our optimization problem does not guarantee that we easily find the global minimum,47 from which we could initiate a sequential thresholded least squares down-selection; finally, we have not yet investigated the effects of L p $$ {L}_p $$ regularization on uncertainty quantification, something we are currently exploring in a separate Bayesian approach.

We would like to share the most important lessons we have learnt throughout this study:

Normalize first! We cannot overstate the importance of normalizing. Clearly, while normalizing is less of an issue in linear regression, it is critical in nonlinear regression. This holds for both the training data, illustrated in Figures 3 and 5, and the weights, illustrated in Figure 11. The loss function typically contains several terms of different magnitude that compete during minimization. It proves important to normalize by the number of data sets in each category, the magnitude of the tensile, compressive, and shear stresses, and the magnitude of the weights to balance the impact of the individual contributions.

L   0 $$ {}_{\mathbf{0}} $$ regularization is the most honest member of the Lp family: L 0 $$ {L}_0 $$ regularization or subset selection is honest, transparent, and unbiased. Its penalty parameter α $$ \alpha $$ acts as a direct switch to select the desired number of terms. It is the only member of the L p $$ {L}_p $$ family that explicitly controls the balance between the number of terms and the goodness of fit, illustrated in Figure 11. While L 0 $$ {L}_0 $$ regularization across the entire network translates into an expensive NP hard discrete combinatorial problem of the order of 2 n $$ {2}^n $$ , we recommend to begin any discovery by running an L 0 $$ {L}_0 $$ regularization for all possible one- and two-term models to determine the best-in-class models of each category and identify the dominant terms, similar to Figures 12 and 13 and Table 1. Importantly, in nonlinear regression, the best-in-class n $$ n $$ -term model may actually not be a subset of the best-in-class ( n + 1 ) $$ \left(n+1\right) $$ -term model, and successively removing terms like in iterative pruning51 or sequential thresholding least squares20 might not be a viable solution. Instead, running L 0 $$ {L}_0 $$ regularization for all possible one- and two-term models provides a quick first insight into the nature and hierarchy of the best-in-class models.22 From this initial first glimpse, we can proceed by successively adding terms. In addition, from the best-in-class one-term models, we can use the discovered weights w i , L 0 $$ {w}_{i,{L}_0} $$ to initialize the weights for higher order runs and to normalize the weights in the regularization term, α   | |   θ   | | p p = ∑ i n para |   w i / w i , L 0   | p $$ \alpha\;{\left\Vert\;\boldsymbol{\theta}\;\right\Vert}_p^p={\sum}_i^{n_{\mathrm{para}}}{\left|\;{w}_i/{w}_{i,{L}_0}\;\right|}^p $$ .

L   1 $$ {}_{\mathbf{1}} $$ regularization is powerful for subset selection, but needs large penalty parameters to be effective: L 1 $$ {L}_1 $$ regularization or lasso promotes sparsity by reducing a large subset of parameters exactly to zero. Notably, for all the examples in our study, this down-selection required quite large penalty parameters α $$ \alpha $$ —often on the order of one-to work effectively. This not only affects the magnitude of the discovered parameters, but often also the discovered model itself. For example, for α = 0 . 1 $$ \alpha =0.1 $$ , the n = 100 $$ n=100 $$ independent realizations of the L 1 $$ {L}_1 $$ regularization in Figure 10 alternate between the green and turquoise one-term models, while the unbiased plain L 0 $$ {L}_0 $$ regularization in Figure 12 and Table 1 ranks these two models clearly behind the blue and dark blue one-term models. To identify regularization-induced bias, we recommend to always compare the results of the L 1 $$ {L}_1 $$ regularization against the best-in-class low-term models of the L 0 $$ {L}_0 $$ regularization. This comparison is simple and inexpensive, and provides valuable insights into the magnitude of selection bias and the aggressiveness of the L 1 $$ {L}_1 $$ regularization.

L   0.5 $$ {}_{\mathbf{0.5}} $$ regularization promotes sparsity for small penalty parameters, but suffers from multiple local minima: L 0 . 5 $$ {L}_{0.5} $$ regularization addresses the shortcomings of the classical L 1 $$ {L}_1 $$ regularization by down-selecting more aggressively, requiring smaller penalty parameters, and introducing less bias. While L 0 . 5 $$ {L}_{0.5} $$ regularization works well in practice, it is computationally challenging. Its non-convexity introduces multiple local minima, indicated through the first rows in Figures 6-8, and through the green and turquoise one-term models in Figure 10, and the blue and dark blue one-term models in Figure 15. To avoid getting stuck in a local minimum, we highly recommend exploring different initialization strategies for the network weights. Specifically, we were able to robustly identify multiple local minima by initializing the weights with the L 0 $$ {L}_0 $$ regularized weights w i , L 0 $$ {w}_{i,{L}_0} $$ . Alternatively, we could gradually ramp up the effect of regularization by starting with a penalty parameter α = 0 $$ \alpha =0 $$ and smoothly increase it to a desired strength, essentially by moving from left to right in Figure 7. For quality control, we recommend comparing the remaining loss of each converged run against the remaining L 0 $$ {L}_0 $$ regularized baseline loss as reported in Table 1.

L   2 $$ {}_{\mathbf{2}} $$ regularization promotes stability, but is not suited for subset selection: L 2 $$ {L}_2 $$ regularization, L by design, is not suited to reduce a subset of terms exactly to zero. Instead, it maintains all terms as indicated in the bottom rows of Figures 10 and 15, each for n = 100 $$ n=100 $$ runs. From Figures 6-8 we conclude that, for increasing penalty parameters α $$ \alpha $$ , L 2 $$ {L}_2 $$ regularization reduces outliers by first bringing the weights closer together and then collectively reducing them toward zero. Clearly, L 2 $$ {L}_2 $$ regularization improves convexity, which makes model discovery more robust and more stable. However, it not only fails to down-select the number of terms, but also strongly biases the solution away from the minimum of the pure network loss towards the minimum of the regularization loss. We do not recommend using L 2 $$ {L}_2 $$ regularization, or any other member of the L p $$ {L}_p $$ regularization family with powers larger than one, p > 1 $$ p>1 $$ , to increase sparsity and improve interpretability in model discovery. Table 2 provides a side-by-side comparison of the L p $$ {L}_p $$ regularizations we explored throughout this study along with their advantages, disadvantages, and references.

TABLE 2. Lp regularization.
Algorithm Regularization Advantages Disadvantages
L   0 $$ {}_{\mathbf{0}} $$ Subset selection α | | θ | | 0 $$ \alpha \kern0.3em {\left\Vert \boldsymbol{\theta} \right\Vert}_0 $$ | | θ | | 0 = ∑ i I ( w i ≠ 0 ) $$ {\left\Vert \boldsymbol{\theta} \kern0.3em \right\Vert}_0={\sum}_iI\left({w}_i\ne 0\right) $$
  • •

    Penalizes number of non-zero terms

  • •

    Term count is inherently unbiased

  • •

    Conceptually simple and honest

  • •

    Promotes sparsity

  • •

    Improves interpretability

  • •

    Valuable insight for one or two terms
  • •

    Solves discrete combinatorial problem

  • •

    Results in NP hard problem

  • •

    Computationally expensive, 𝒪 ( 2 n )

  • •

    But manageable for one term, 𝒪 ( n )

  • •

    And manageable for two terms, 𝒪 ( n 2 )

L   0.5 $$ {}_{\mathbf{0.5}} $$ Compromise between L 0 $$ {L}_0 $$ and L 1 $$ {L}_1 $$ α | | θ | | 0 . 5 0 . 5 $$ \alpha \kern0.3em {\left\Vert \boldsymbol{\theta} \right\Vert}_{0.5}^{0.5} $$ | | θ | | 0 . 5 0 . 5 = ∑ i | w i | $$ {\left\Vert \boldsymbol{\theta} \kern0.3em \right\Vert}_{0.5}^{0.5}={\sum}_i\sqrt{\mid \kern0.1em {w}_i\kern0.1em \mid } $$
  • •

    Improved efficiency compared to L 0 $$ {L}_0 $$

  • •

    Improved sparsity compared to L 1 $$ {L}_1 $$

  • •

    Reduces some parameters exactly to zero

  • •

    Works even for smaller α $$ \alpha $$ and less bias
  • •

    Non-convex, multiple local minima

  • •

    Increased computational complexity
L   1 $$ {}_{\mathbf{1}} $$ Lasso least absolute shrinkage and selection operator α | | θ | | 1 $$ \alpha \kern0.3em {\left\Vert \boldsymbol{\theta} \right\Vert}_1 $$ | | θ | | 1 = ∑ i | w i | $$ {\left\Vert \boldsymbol{\theta} \kern0.3em \right\Vert}_1={\sum}_i\mid \kern0.1em {w}_i\kern0.1em \mid $$
  • •

    Weighs all components equally

  • •

    Less sensitive to outliers than L 2 $$ {L}_2 $$

  • •

    Reduces some parameters exactly to zero

  • •

    Promotes sparsity

  • •

    Improves interpretability
  • •

    Not strictly convex, local minima

  • •

    Emphasizes selective effects

  • •

    Introduces bias, inaccurate for large α $$ \alpha $$

L   1/2 $$ {}_{\mathbf{1}/\mathbf{2}} $$ Elastic net compromise between L1 and L2 α 1 | | θ | | 1 + α 2 | | θ | | 2 2 $$ {\alpha}_1\kern0.3em {\left\Vert \boldsymbol{\theta} \right\Vert}_1+{\alpha}_2\kern0.3em {\left\Vert \boldsymbol{\theta} \right\Vert}_2^2 $$ | | θ | | 1 = ∑ i | w i | $$ {\left\Vert \boldsymbol{\theta} \kern0.3em \right\Vert}_1={\sum}_i\mid \kern0.1em {w}_i\kern0.1em \mid $$ | | θ | | 2 2 = ∑ i | w i | 2 $$ {\left\Vert \boldsymbol{\theta} \kern0.3em \right\Vert}_2^2={\sum}_i{\left|\kern0.1em {w}_i\kern0.1em \right|}^2 $$
  • •

    Improved stability compared to L 1 $$ {L}_1 $$

  • •

    Improved sparsity compared to L 2 $$ {L}_2 $$

  • •

    Increased computational complexity
L   2 $$ {}_{\mathbf{2}} $$ Ridge regression α 2 | | θ | | 2 2 $$ {\alpha}_2\kern0.3em {\left\Vert \boldsymbol{\theta} \right\Vert}_2^2 $$ | | θ | | 2 2 = ∑ i | w i | 2 $$ {\left\Vert \boldsymbol{\theta} \kern0.3em \right\Vert}_2^2={\sum}_i{\left|\kern0.1em {w}_i\kern0.10em \right|}^2 $$
  • •

    Uses components squared

  • •

    Reduces outliers, improves predictability

  • •

    Increases robustness

  • •

    Promotes stability
  • •

    Introduces bias

  • •

    Moves parameters towards each other

  • •

    Reduces but maintains all parameters

  • •

    Does not promote sparsity
  • Note: Comparison of special cases, advantages, disadvantages, and references.

Densifying instead of sparsifying: The closure problem is a common challenge in both fluid and solid mechanics. It refers to the difficulty of fully specifying the constitutive equations that relate stresses and strains and characterize the material behavior. In fluid mechanics, the closure problem is closely related to turbulence modeling, where it approximates intricate interactions between different scales, and can be well represented through polynomials.20 In solid mechanics, the closure problem characterizes complex material behaviors at the microscopic scale, and is traditionally often represented through a combination of polynomials,25 exponentials,69, 70 logarithms,71 and powers.57, 66 In the context of model discovery, assuming perfect data, polynomial models translate into a convex linear optimization problem with a single unique global minimum, while exponential, logarithmic, or power models translate into a non-convex nonlinear optimization problem with possibly multiple local minima. For convex discovery problems with a unique global minimum, inducing sparsity has been well established through a top down approach in which we first calculate a dense parameter vector at the global minimum, and then sparsify the parameter vector by sequentially thresholding and removing the least relevant terms.20, 25, 27, 47 For non-convex discovery problems with multiple local minima, this approach is infeasible since different initial conditions may result in different solutions with non-unique parameter vectors.67 Instead of trying to sparsify a dense parameter vector, we recommend to gradually densify the parameter vector from scratch. This bottom up approach iteratively solves the discrete combinatorial problem and densifies the parameter vector by sequentially adding the most relevant terms.72 Importantly, instead of solving the NP hard discrete combinatorial problem associated with a complete L 0 $$ {L}_0 $$ regularization that screens all possible combinations of terms at 𝒪 ( 2 n ) , we recommend to gradually add terms, starting with the best-in-class one-term model at 𝒪 ( n ) , adding a second term at 𝒪 ( n ) , and repeating addition until the incremental improvement of the overall loss function meets a user-defined convergence criterion. At most, this algorithm involves 𝒪 ( n 2 ) evaluations of the loss function to land on a fully populated dense parameter vector. Importantly, for non-convex model discovery problems, this algorithm—while cost effective and well-rationalized—is not guaranteed to converge to the global minimum. Instead of successively adding up to n $$ n $$ terms, for practical purposes, it is often sufficient to limit the number of desirable terms to one, two, three or four, and identify the best-in-class model of each class, which requires a discrete comparison of (   8 ! / ( n ! ( 8 − n ) ! )   ) $$ \left(\;8!/\left(n!\left(8-n\right)!\right)\;\right) $$ discrete models, in our case 8, 28, 56, or 70.22 Out of all possible discovery algorithms, this is the most honest, unbiased, and transparent approach.

Taken together, our study suggests that L p $$ {L}_p $$ regularized constitutive neural networks are a powerful technology for automated model discovery that allows us to identify interpretable constitutive models from data. We anticipate that our results generalize to L p $$ {L}_p $$ regularization for model discovery with other techniques such as symbolic regression or system identification, and, more broadly, to model discovery in other fields such as biology, chemistry, or medicine. The ability to discover new knowledge from data could have tremendous applications in generative material design where it could shape the path to manipulate matter, alter properties of existing materials, and discover new materials with targeted properties.

AUTHOR CONTRIBUTIONS

JAMC: Method development; simulation; data analysis; result interpretation; manuscript writing. SRSP: Method development; simulation; data analysis; result interpretation; manuscript writing. KL: Study design; method development; simulation; data analysis; result interpretation; manuscript writing. EK: Study design; method development; simulation; data analysis; result interpretation; manuscript writing.

ACKNOWLEDGMENTS

This work was supported by the National Science Foundation Graduate Research Fellowship to Jeremy McCulloch and Skyler St. Pierre, by the DAAD Fellowship to Kevin Linka, and by the NSF CMMI Award 2320933 Automated Model Discovery for Soft Matter to Ellen Kuhl.

    CONFLICT OF INTEREST STATEMENT

    The authors declare no potential conflict of interests.

    DATA AVAILABILITY STATEMENT

    The data that support the findings of this study are openly available in the Living Matter Lab GitHub repository at https://github.com/LivingMatterLab/CANN, References 29 and 50.

      The full text of this article hosted at iucr.org is unavailable due to technical difficulties.