Combining machine learning and domain decomposition methods for the solution of partial differential equations—A review

2.1 General domain decomposition and model problems

2.2 Machine learning approach

As input data for the neural network, we use samples, that is, point-wise evaluations of the coefficient function or the material distribution within the two subdomains adjacent to an edge E_ij; see Figure 4. As output data for the neural network, we save the classification whether the corresponding eigenvalue problem is necessary for a robust algorithm (class 1) or not (class 0); see also Figure 4. In particular, this means that we solve the corresponding eigenvalue problem for each configuration within our training and validation data set and save the related classification based on the number of selected constraints as the ground truth for the DNN; see also [30, section 3.2]. Let us remark that we also suggested a classification with three different classes in 27, 30, 31 for adaptive FETI-DP. In that case, class 1 is again split into eigenvalue problems resulting in exactly a single constraint (class 1a) and eigenvalue problems resulting in more than one constraint (class 1b). This can be useful if, for example, an appropriate approximation for the first adaptive constraint is known which can be computed at a low cost. In 27, 30, 31, we always use a frugal constraint 29 instead of solving an eigenvalue problem for each edge from class 1a. This further reduces the setup cost due to a higher percentage of eigenvalue problems saved compared with the fully adaptive approach. For simplicity, we will not discuss the three-class classification approach in the present review article in detail. As we can also observe from Figure 4, the described classification problem is, in principle, an image recognition problem. However, whereas in image classification the resolution of an image needs to be the same for all image samples, we use an approach that is independent of the finite element distribution. This is achieved by computing a sampling grid which is especially independent of the resolution of the underlying finite element grid. For all the sampling points in the computed sampling grid, we evaluate the coefficient function and use these evaluations as input data for the neural network. The only assumption that we make is that the sampling grid is fine enough to resolve all geometric details of the coefficient function within each subdomain. Let us remark that, for rectangular subdomains, the sampling grid can be chosen such that it completely covers both subdomains adjacent to an edge. For irregular decompositions obtained by, for example, METIS, small areas might not be covered by the sampling grid, but typically these areas are further away from the edge. This usually does not affect the performance of the algorithm. It also motivates the use of smaller sampling grids on areas solely around the edge to save computational cost; see also 31. Let us note that for the classification of an edge E_ij we solely use coefficient values within the neighboring subdomains ${\mathrm{\Omega }}_i$ and ${\mathrm{\Omega }}_j$.

Training data. To arrange a representative training data set consisting of a number of different coefficient distributions on two neighboring subdomains, two different strategies have been tested so far. First, in 30, we have used a set of carefully designed training data, that is, specific coefficient distributions. Later on, in 27, 31, we also denote this set by smart data in contrast to training data sets obtained by a randomized approach. For the smart data, the basic coefficient distributions shown in Figure 5 are used as a basis to create the full set of training data for the numerical experiments in 30. For this specific purpose all of these nine coefficient distributions are mirrored, rotated, and shifted within a subdomain. Second, in 27, we have used randomized coefficient distributions which in principle can be designed without any a priori knowledge. Nevertheless, some structure has to be enforced to obtain satisfactory results. For the first part of this random data training set, we randomly generate the coefficient for each pixel, consisting of two triangular finite elements, independently and only control the ratio of high and low coefficient values. Here, we use 30%, 20%, 10%, and 5% of high coefficient values. For the second part, we also control the distribution of the coefficients to a certain degree by randomly generating either horizontal or vertical stripes of a maximum length of four or eight pixels, respectively; see Figure 6. Additionally, we generate new coefficient distributions by superimposing pairs of horizontal and vertical coefficient distributions.

Sampling on slabs. We now describe how we reduce the number of sampling points used as input data for the neural network. In 30, the computed sampling grid covers both neighboring subdomains of an edge entirely—at least in case of a regular domain decomposition. Let us remark that in case of irregular domain decompositions, our sampling strategy might miss small areas further away from the edge; see, for example, [30, fig. 4]. However, this does not affect the performance of our algorithm. Although the preparation of the training data as well as the training of the neural network can be performed in an offline-phase, we try to generate the training data as efficient and fast as possible. For all sampling points, we need to determine the corresponding finite element as well as to evaluate the coefficient function for the respective finite element. Therefore, there is clearly potential to save resources and compute time in the training as well as in the evaluation phase by reducing the number of sampling points used as input data for the neural network. In general, the coefficient variations close to the edge are the most relevant, that is, the most critical for the condition number bound of FETI-DP. Therefore, to reduce the total number of sampling points in the sampling grid, reducing the density of the grid points with increasing distance to the edge is a natural approach. More drastically, one could exclusively consider sampling points in a neighborhood of the edge, that is, on slabs next to the edge. We consider the latter approach here; see also Figure 4 for an illustration of the sampling points inside slabs.

2.3 Adaptive FETI-DP

The description of standard as well as adaptive FETI-DP is essentially taken from 30. For more details on standard FETI-DP see, for example, 46, 49, 78 and for a detailed description of adaptive FETI-DP see 43, 44, 59.

2.4 Adaptive GDSW

In this section, we will briefly introduce the standard GDSW preconditioner as well as the AGDSW preconditioner. The presentation follows the original work introducing the AGDSW coarse space for overlapping Schwarz methods 26. As for the adaptive FETI-DP method and in contrast to 26, we will only consider the two-dimensional case here.

2.4.3 AGDSW coarse basis functions

In order to extend the GDSW preconditioner to be robust for arbitrary coefficient distributions, we construct edge basis functions based on local generalized eigenvalue problems. In particular, the coarse basis functions are constructed as discrete harmonic extensions of the corresponding edge eigenmodes.

Let E_ij be an edge and ${\mathrm{\Omega }}_{ij}$ be the union of its adjacent subdomains ${\mathrm{\Omega }}_i$ and ${\mathrm{\Omega }}_j$. Additionally, we define the following extension-by-zero operator from E_ij to ${\mathrm{\Omega }}_{ij}$:

$$\begin{array}{ll}{z}_{E_{ij}\to {\mathrm{\Omega }}_{ij}}:{\stackrel{\circ }{V}}^h(E_{ij})\to V_0^h\left({\mathrm{\Omega }}_{ij}\right),v\mapsto z_{E_{ij}\to {\mathrm{\Omega }}_{ij}}(v):=\left\{\begin{array}{ll}v& \text{at all interior nodes of}{E}_{ij},\\ 0& \text{at all other nodes in}\overline{{\mathrm{\Omega }}_{ij}};\end{array}\right.& \end{array}$$

see Figure 7 (right) for a graphical representation. Here,

$$\begin{array}{llll}{V}_0^h\left({\mathrm{\Omega }}_{ij}\right)& :=\left\{v{|}_{{\mathrm{\Omega }}_{ij}}:v\in V^h(\mathrm{\Omega }),v=0\text{in}\mathrm{\Omega }\setminus {\mathrm{\Omega }}_{ij}\right\},& & \end{array}$$

and ${\stackrel{\circ }{V}}^h(E_{ij}):=\{v{|}_{E_{ij}\setminus \partial E_{ij}}:v\in V^h(\mathrm{\Omega })\}$ is defined via the interior nodes of E_ij. By ${\mathscr{H}}_{E_{ij}\to {\mathrm{\Omega }}_{ij}}$, we denote the discrete harmonic extension w.r.t. $a_{\mathrm{\Omega }}(·,·)$ from E_ij to ${\mathrm{\Omega }}_{ij}$. Specifically, let $V_{0,E_{ij}}^h({\mathrm{\Omega }}_l):=\{w{|}_{{\mathrm{\Omega }}_l}:w\in V^h(\mathrm{\Omega }),w=0\text{on}{E}_{ij}\}$. Then, for ${\tau }_{E_{ij}}\in {\stackrel{\circ }{V}}^h(E_{ij})$, the extension $v_{E_{ij}}:={\mathscr{H}}_{E_{ij}\to {\mathrm{\Omega }}_{ij}}({\tau }_{E_{ij}})$ is given by the solution of

$$\begin{array}{ll}\begin{array}{rc}{a}_{{\mathrm{\Omega }}_l}(v_{E_{ij}},v)=0& \forall v\in V_{0,E_{ij}}^h({\mathrm{\Omega }}_l),l=i,j,\\ v_{E_{ij}}={\tau }_{E_{ij}}& \text{on}{E}_{ij}.\end{array}& \end{array}$$ ()

Here, we only prescribe values on E_ij, whereas the whole boundary of $\partial {\mathrm{\Omega }}_{ij}$ is considered as Neumann boundary in Equation (9); cf. Figure 7 (left). In order to compute these extensions with Neumann boundary conditions, we need the local Neumann matrices K⁽ⁱ⁾.

Using the two extension operators $z_{E_{ij}\to {\mathrm{\Omega }}_{ij}}$ and ${\mathscr{H}}_{E_{ij}\to {\mathrm{\Omega }}_{ij}}$ from E_ij to ${\mathrm{\Omega }}_{ij}$, we solve the following generalized eigenvalue problem on each edge E_ij: find ${\tau }_{\ast ,E_{ij}}\in V_0^h(E_{ij}):=\{v{|}_{E_{ij}}:v\in V^h(\mathrm{\Omega }),v=0\text{on}\partial E_{ij}\}$ such that

$$\begin{array}{ll}{a}_{{\mathrm{\Omega }}_{ij}}({\mathscr{H}}_{E_{ij}\to {\mathrm{\Omega }}_{ij}}({\tau }_{\ast ,E_{ij}}),{\mathscr{H}}_{E_{ij}\to {\mathrm{\Omega }}_{ij}}(\theta ))={\lambda }_{\ast ,E_{ij}}{a}_{{\mathrm{\Omega }}_{ij}}(z_{E_{ij}\to {\mathrm{\Omega }}_{E_{ij}}}({\tau }_{\ast ,E_{ij}}),z_{E_{ij}\to {\mathrm{\Omega }}_{E_{ij}}}(\theta ))\forall \theta \in V_0^h\left(E_{ij}\right).& \end{array}$$ ()

Let the eigenvalues be sorted in non descending order, that is, ${\lambda }_{1,E_{ij}}\le {\lambda }_{2,E_{ij}}\le \cdots \le {\lambda }_{m,E_{ij}}$, and the eigenmodes accordingly, where $m=\dim \left(V_0^h\left(E_{ij}\right)\right)$. Furthermore, let the eigenmodes ${\tau }_{\ast ,E_{ij}}$ be normalized in the sense that $a_{{\mathrm{\Omega }}_{ij}}(z_{E_{ij}\to {\mathrm{\Omega }}_{E_{ij}}}({\tau }_{k,E_{ij}}),z_{E_{ij}\to {\mathrm{\Omega }}_{E_{ij}}}({\tau }_{j,E_{ij}}))={\delta }_{kj}$, where ${\delta }_{kj}$ is the Kronecker delta symbol.

To construct the AGDSW coarse space, we select all eigenmodes ${\tau }_{\ast ,E_{ij}}$ with eigenvalues below a certain threshold, that is, ${\lambda }_{\ast ,e}\le to{l}_{ℰ}$. Then, the eigenmodes are first extended by zero to the whole interface $\mathrm{\Gamma }$ and then to the interior using energy-minimizing extensions:

$$v_{\ast ,E_{ij}}:={\mathscr{H}}_{\mathrm{\Gamma }\to \mathrm{\Omega }}(z_{E_{ij}\to \mathrm{\Gamma }}({\tau }_{\ast ,E_{ij}})).$$ ()

We obtain the AGDSW coarse space

$$V_{\text{AGDSW}}^{to{l}_{ℰ}}:=\underset{v\in 𝒱}{⨁}\text{span}\left\{{\varphi }_v\right\}\oplus \left(\underset{e\in ℰ}{⨁}\text{span}\left\{v_{k,e}:{\lambda }_{k,e}\le to{l}_{ℰ}\right\}\right),$$

where the ${\varphi }_v$ are the coarse basis functions corresponding to vertices in the classical GDSW coarse space. Note that the left hand side of the eigenvalue problem (10) is singular, and its kernel contains the constant function on the edge. Thus, the classical GDSW coarse space is always contained in the AGDSW coarse space.

We note that the computation of the AGDSW coarse space, based on local eigenvalue problems associated with edges, is similar to the adaptive FETI-DP approach. This is sufficient to apply the same machine learning strategy to predict the location of AGDSW coarse basis functions, despite strong differences in the specific formulation of the eigenvalue problem. Nevertheless, there is one important difference when applying the machine learning approach: as already mentioned, in AGDSW, the first coarse basis function is always necessary, but it can be computed without actually solving the eigenvalue problem. Hence, an eigenvalue problem will only be marked as necessary in our machine learning based AGDSW if more than one coarse basis function corresponds to an eigenvalue lower than the chosen tolerance. This is different to ML-FETI-DP.

2.5 Numerical results for FETI-DP

In order to be competitive, the ML-FETI-DP algorithm, that is, the combination of adaptive FETI-DP and ML as described above, has to have two important properties or, in other words, there are two main objectives in the design of the ML-FETI-DP and the underlying DNN. First, ML-FETI-DP has to be robust and has to have a similar condition number as adaptive FETI-DP. Second, a lower setup cost than for the fully adaptive approach is desirable, that is, a large part of the local eigenvalue problems has to be omitted. Unfortunately, both objectives are sometimes contradictory and a good trade-off has to be found in the design of the classification network which is used within ML-FETI-DP. As already mentioned, a cheap setup can be reached by a strong reduction in the number of eigenvalue problems but also by reducing the number of sampling points in the sampling grid.

In contrast, to obtain a small condition number and a robust algorithm, it is essential that the DNN delivers no false negative edges, that is, does not falsely classify edges to class 0. To set up the neural network properly and to obtain satisfactory results, many technical details have to be considered. Before we discuss all those details of different training data sets and the reduction of sampling points, we will first concentrate on a single parameter which illustrates the balancing act between robustness and efficiency of the method. The simplest adjustment of the DNN is the ML threshold $\tau$ for the decision boundary between critical edges (class 1) and uncritical edges (class 0), which can be varied between zero and one. Choosing a high value for $\tau$ implies that we set a high priority on a low setup cost as edges are more likely classified as uncritical. In that case, the risk of having false negative edges is very high. In contrast, choosing a small $\tau$ implies that we set a high priority on a robust ML-FETI-DP algorithm and solve some unnecessary eigenvalue problems in the setup phase—one for each false positive edge. We usually prefer the latter choice. Choosing an appropriate ML threshold, the ML-FETI-DP algorithm can be as robust as adaptive FETI-DP with a reduced setup cost; see Table 2 for a first comparison with standard FETI-DP and adaptive FETI-DP. Here, $\tau =0.45$ proved to be a good choice while choosing a threshold of $\tau =0.5$ deteriorates the robustness by adding false negative edges; see Figure 8. We further provide the receiver operating characteristic (ROC) curve and a precision-recall plot for the optimized neural network model in Figure 9. Here, the threshold $\tau$ is varied between zero and one and the considered thresholds $\tau \in \{0.45,0.5\}$ are indicated as circles.

In the remainder of this section we will discuss all the technical details which underlie the excellent performance of ML-FETI-DP shown in Tables 1 and 2. First, we show comparison results for the ML-FETI-DP algorithm using different sets of training data, as described in Section 2.2, for both stationary diffusion and linear elasticity problems. Second, we summarize the obtained numerical results when reducing the sampling effort, that is, computing a reduced number of sampling points for input data of the neural network. As realistic test problems for our algorithm, we use 10 different and randomly chosen subsections of the microsection of a dual-phase steel in Figure 10 (right); see also Figures 10 (left) and 1 for examples of a specific subsection used as the coefficient distribution in our computations. Let us note that these specific coefficient distributions are explicitly not included in the training and validation data. Additionally, we provide the resulting accuracy values and percentages of false negative and false positive edges for the different training and validation data sets to prove that all trained neural networks provide a reliable model. All computations in this section have been performed using the machine learning implementations in TensorFlow 1 and Scikit-learn 66 as well as our Matlab implementation of the adaptive FETI-DP method.

Different training data sets. We now summarize the numerical results from 27, 30, where the performance of the proposed machine learning based adaptive FETI-DP algorithm was tested for different choices of training and validation data sets to train our neural network. As described in detail in 27, we use a set of 4500 smart data configurations (denoted by “S”) and sets of 4500 and 9000 random data configurations (denoted by “R1” and “R2,” respectively) each individually as well as a combination of 4500 smart and 4500 random data configurations, which will be denoted by “SR.” Let us note that we did not observe a significant improvement for a larger number of 18 000 random data configurations.

As already mentioned, we deliberately investigated the performance of the trained DNNs for the different training data sets for different coefficient distributions, which were explicitly not included in the training data. In particular, we applied the respective trained networks to 10 different randomly chosen subsections of a microsection as shown in Figure 10 (right). In all presented computations, we consider ${\rho }_1=1e6$ in the black part of the microsection and ${\rho }_2=1$ elsewhere in case of a stationary diffusion problem and E₁ = 1e6, E₂ = 1, and a constant $\nu =0.3$ in case of a linear elasticity problem, respectively. For the discretization of this model problem we use a regular decomposition of the domain $\mathrm{\Omega }$ into 64 square subdomains and a subdomain size of H/h = 64. For the selection of the adaptive coarse constraints we use a tolerance of TOL = 100. For the test data, that is, the microsection subsections, we will only compute the local eigenvalue problems on edges which are classified as critical (class 1) by the neural network. On all uncritical edges (class 0), we do not enforce any constraints. For the respective ML classification, we use a ML threshold $\tau$ of 0.5 and 0.45 for the classification for the decision boundary between critical and uncritical edges. Let us note that a lower threshold $\tau$ decreases the false negative rate of the predictions and thus increases the robustness of our algorithm; cf. also the explanations on the balancing act of our algorithm at the beginning of this section. For the 10 microsections and the stationary diffusion problem (see Table 3) all four different training data sets result in a robust algorithm when using an ML threshold $\tau =0.45$. For all these approaches, we obtain no false negative edges, which are critical for the convergence of the algorithm. However, the use of 4500 and 9000 random data (see R1 and R2) results in a higher number of false positive edges compared to the sole use of 4500 smart data, resulting in a larger number of computed eigenvalue problems. For linear elasticity and the 10 microsections, see Table 4, the results are fairly comparable. Again, the use of the smart data training set provides the best trade-off between robustness and computation cost since we obtain no false negative edges and only 4.8 false positive edges on average. Additionally, also using 9000 random data (R2) as well as the combined training data set (SR) provides no false negative edges when choosing the ML threshold $\tau =0.45$. However, for both cases we have a slightly increased number of false positive edges compared to S which results in a slightly higher computational effort. On the other hand, the advantage of the randomized training data lies in the random generation of the coefficient distributions which is possible without any specific problem-related knowledge. Let us also remark that setting up a smart data set in three dimensions is very complicated and we thus prefer a randomly generated set in this case. Given the relatively low share of additional false positive edges compared to the number of saved eigenvalue problems, we can conclude that we can achieve comparable results using the randomized training data when generating a sufficient amount of coefficient configurations.

Reducing the computational effort. In addition to the performance of different training data sets, we have also investigated the effect of using a reduced number of sampling points as input data for the neural network by sampling in slabs of different width; see also Figure 4. As the smart data as well as an increased number of randomized training data showed comparable results in the previous experiments, we solely have used the smart data for this investigation. Moreover, we focus on the ML threshold $\tau =0.45$ which led to the most robust results for the full sampling approach.

We show comparison results for the original approach with full sampling as introduced in 30 and for further, different sampling approaches on slabs. In particular, for subdomains of width H, we consider the cases of sampling in one half and one quarter, that is, H/2 and H/4, as well as the extreme case when sampling only inside minimal slabs of the width of one finite element, that is, h. As already mentioned, our machine learning problem is, in principle, an image recognition task. Thus, the latter sampling approaches correspond to the idea of using only a fraction of pixels of the original image as input data for the neural network. We have focused on linear elasticity problems for the remainder of this paragraph.

As we can observe from Table 5, using again the microsection problem in Figure 10 (left) as a test problem, both sampling in slabs of width H/2 and H/4 results in a robust algorithm in terms of the condition number and the iteration count when using the ML threshold $\tau =0.45$. For both approaches, we obtain no false negative edges which are critical for the convergence of the algorithm. However, using fewer sampling points results in a higher absolute number of false positives edges and therefore in a larger number of computed eigenvalue problems with a higher computational effort. When applying the extreme case of sampling only in slabs of width h, we do not obtain a robust algorithm for the microsection problem since the convergence behavior clearly deteriorates. Hence, it is questionable if the latter sampling approach does provide a reasonable machine learning model. The weak performance of the latter sampling approach can also be observed for the training data; see the following discussion on the training and validation data. With respect to our model problems we can summarize that reducing the size of the sampling grid in order to reduce the effort in the training and evaluation of the neural network still leads to a robust algorithm. However, we can also observe that the slab width for the sampling cannot be chosen too small and a sufficient number of finite elements close to the edge have to be covered by the sampling.

Finally, we present results for the whole set of training data using cross-validation and a fixed ratio of 20% as validation data to test the generalization properties of our neural networks. Please note that due to different heterogeneity of the various training data, the accuracies in Table 6 are not directly comparable with each other. However, the results in Table 6 serve as a sanity check to prove that the specific trained model is able to generate appropriate predictions. Let us further note that we always have to generate a separate training data set for stationary diffusion and linear elasticity problems, respectively, since the number of necessary adaptive constraints can differ for the two aforementioned model problems. However, as the performance for the different sets of training data is comparable for both types of model problems, we exclusively show the results for the training data for linear elasticity problems in Table 6. As we can observe from the results in Table 6 training the neural network with the set of smart data as well as with a combination of the smart and randomized coefficient distributions leads to the highest accuracy values, that is, the highest numbers of true positive and true negative edges in relation to the total number of edges. However, also training the neural network with solely the randomly generated coefficient functions leads to an appropriate model while we can observe the tendency that a higher number of training data is needed to obtain comparable accuracy values. Moreover, when using the smart training data, both sampling with a width of H/2 and H/4 leads to accuracy values in the classification that are only slightly lower than for the full sampling approach. In particular, we obtain slightly increased false positive values, corresponding to an increased computational effort in terms of the solution of eigenvalue problems. For the extreme case of sampling only in slabs of minimal width h, thus, using the minimal possible width in terms of finite elements, the accuracy value drops from 88.4% to 71.7% for the two-class model for the threshold $\tau =0.45$. Thus, also for the training data, it is questionable whether this extreme case does still provide a reliable machine learning model. This is also supported by the weak performance of the respective trained neural network for the test problem in form of the microsection; see also Table 5.

2.6 Numerical results for GDSW

In this section, for the first time, we will apply our machine learning framework to an overlapping adaptive DDM, the AGDSW method; see also Section 2.4. The AGDSW algorithm is also based on the solution of localized eigenvalue problems on edges in 2D and on faces and edges in 3D. Thus, we can extend our machine learning techniques to the classification of critical edges for the GDSW coarse space. In particular, the presented results have not yet been published in previous works and are an extension of the already existing publications 27, 30, 31 for the FETI-DP algorithm. As the classification of critical edges can be different for AGDSW than for adaptive FETI-DP, we have generated a separate training data set for the GDSW approach. As for FETI-DP, we generated 4500 randomized coefficient distributions using the techniques described in Figure 6 to build the training and validation data set. We denote this training data set by R1'. For each pair of neighboring subdomains sharing an edge we solved the respective eigenvalue problem and saved the classification whether at least one adaptive constraint is necessary for the respective edge or not for the robustness of the algorithm. In particular, we used the overlap $\delta =1$ and the tolerance $to{l}_{ℰ}=0.01$ for the generation of the training data. Let us briefly comment on the classification of edges for the GDSW approach. As already mentioned in Section 2.4.3, in AGDSW, the first coarse basis function is always necessary for robustness. It corresponds to the constant function on the edge and can thus be computed without actually solving the eigenvalue problem. Therefore, for ML-AGDSW all critical edges where more than the single constant constraint is necessary are classified as class 1. Let us note that this is different to class 1 for ML-FETI-DP.

To prove that the described machine learning framework can successfully be applied to other domain decomposition approaches than the FETI-DP method, we show comparison results for standard GDSW, using exclusively one constant constraint per edge, the AGDSW, and our new ML-AGDSW algorithm. Since these are the first results obtained for GDSW, here, we focus on stationary diffusion problems and regular domain decompositions. As a test problem for our ML-AGDSW method we use again subsections of the microsection problem in Figure 10. However, we explicitly use different decompositions of the domain $\mathrm{\Omega }$, that is, different discretizations as for the FETI-DP method in Section 2.5. For all presented computations we set $\rho =1e6$ in the black parts of the microsections and $\rho =1$ elsewhere.

As a first test problem, we use a decomposition of the microsection in Figure 10 (left) into 4 × 4 subdomains and 6272 finite elements per subdomain. The obtained results for the different GDSW coarse spaces are presented in Table 7. As we can observe from Table 7, using the standard GDSW coarse space clearly fails to provide a robust algorithm since 297 iterations are needed until convergence and the condition number bounds has the magnitude of the coefficient contrast. Thus, using exclusively constant constraints for each edge is not sufficient for this model problem and additional coarse constraints are necessary. Using our ML-AGDSW approach and the ML threshold $\tau =0.45$ we are able to achieve nearly the same performance as the AGDSW. Even though we obtain a single false negative edge the respective condition number estimate is clearly independent of the coefficient contrast and the iteration number is sufficient. As a second test problem we use a domain decomposition with 8 × 8 subdomains and again 6272 finite elements per subdomain for the microsection problem in Figure 10 (left); see Table 8. Here, using the ML threshold $\tau =0.45$ we obtain no false negative edges and four false positive edges. As a result we observe almost the same convergence properties and condition number estimate as for the AGDSW while we only have to solve 27 instead of 112 eigenvalue problems on edges. Let us note that we have used the same ML thresholds $\tau \in \{0.45,0.5\}$ for the decision boundary between the two classes of edges as for the ML-FETI-DP approach. As the presented results are the first obtained results for the ML-AGDSW approach, we have not yet fully optimized the decision threshold $\tau$ for the ML-AGDSW approach. This will be done in future research on the ML-AGDSW method. Finally, we provide the results, that is, the accuracy values and the percentages of false positive and false negative edges for the training and validation data in Table 9. These results serve as a sanity check that our trained neural network provides a reliable model. Let us remark that we have slightly modified the network architecture compared to the networks used for ML-FETI-DP to achieve comparable results on the training and validation data.

2.7 Ongoing and future research

The techniques described above can also be adapted such that they can be applied in three dimensions; see 28. In 28, we describe the central difficulties and the necessary extensions and show numerical results for ML-FETI-DP in three dimensions. Let us briefly describe the main differences from the two-dimensional case. First, the essential eigenvalue problems are no longer related to edges, but to faces between two neighboring subdomains. Second, we only considered randomly generated training data, since generating a smart, that is, manually selected set of coefficient distributions containing all the necessary phenomena, is a complicated task in three dimensions. Third, building an appropriate grid of sampling points with a consistent ordering of the sampling points is more difficult in three dimensions when considering irregular domain decompositions obtained by, for example, the graph partitioning software METIS. To obtain such an ordering, we first project each face onto a two-dimensional plane, optimize the triangulation of the projection, define a consistently ordered sampling grid on the optimized projection, and finally project this sampling grid back to three dimensions. Using these modifications, the ML-FETI-DP approach can successfully be used in three dimensions as well; see 28 for details.

In addition to the three-dimensional ML-FETI-DP approach, there are various possibilities for future research considering the combination of ML and adaptive DDMs. First, it is our goal to include ML-FETI-DP and ML-AGDSW into our parallel software packages based on PETSc and, respectively, Trilinos. This is important to prove that both approaches can actually save computation time compared to the adaptive methods in a parallel computation. Second, a three-dimensional version of ML-AGDSW is planned 32. Third, we plan to investigate the opportunity to predict the adaptive coarse constraints or coarse basis functions directly using a machine learning approach, that is, a method without the need for solving any eigenvalue problems in the online stage.

3.1 Physics-informed neural networks

3.2 The Deep Ritz method

3.3 Deep domain decomposition methods—using PINNs and Deep Ritz as subdomain solvers

Having defined a solver for the boundary value problem in Equation (12) by training the neural network $𝒩$ with certain collocation points and the loss function given in Equation (13) or Equation (19), respectively, it can also be used to solve the subdomain problems in DDMs. This approach has been applied to a parallel overlapping Schwarz method using PINNs 54 and the Deep Ritz method 52. Both resulting algorithms are denoted as Deep Domain Decomposition method and abbreviated as DeepDDM in 53 and D3M in 51. To better distinguish them, we will make use of the acronyms DeepDDM and D3M in the following. Although a neural network $𝒩$ in this subsection can be either defined using PINNs or the Deep Ritz method, to avoid a proliferation of notation, we do not distinguish here between them in the notation.

Algorithm 1. Brief sketch of the DeepDDM and D3M algorithms; see [54, Algorithm 2] and [52, Algorithms 1 and 2], respectively, for details

Let us finally summarize some important findings and remarks on the training of the introduced physics-constrained NNs. First, when using a stochastic gradient descent method and dividing the total set of collocation points into mini-batches, it is important for the performance that all boundary and interface collocation points or at least a sufficient share of them are redundantly part of each mini-batch. Only the interior collocation points can be divided in mini-batches as usual in the training of DNNs. This is based on the observation that for solving a general PDE, the boundary information is quite essential. For a further analysis of the respective numbers of collocations points, we refer to 52, 54. Second, both DeepDDM and D3M inherit the parallelization capacities from the underlying DDM since the physics-constrained NNs local to the subdomains can be trained in parallel. The only synchronization point is the exchange of information on the overlap.

3.4 Ongoing and future research

There are several opportunities to extend the aforementioned deep domain decomposition methods; some of these approaches are already subject to ongoing research.

Within each iteration in Algorithm 1, the local PINNs on the subdomains are trained from scratch with new boundary data in the collocation points on the interface. However, as this overlapping Schwarz method is a fixed point iteration, it may be a reasonable assumption that the local solutions in two successive iterations are rather similar. Therefore, in the sense of transfer learning, the trained neural network from the previous iteration can be used as a good initial guess for the training of the current neural network on the same subdomain.

In order to improve the numerical scalability and accuracy of the one-level algorithm, a coarse correction can be added, resulting in a two-level deep domain decomposition methods. Recently, this approach has been presented by Hyea Hyun Kim at the 26th International Domain Decomposition Conference 38. In particular, an additional PINN on the whole domain $\mathrm{\Omega }$ has been added to enable fast transport of global information in the Schwarz iteration. In order to keep the load between the local and the global models in balance, the complexity of the coarse PINN has to be chosen accordingly, for example, the same as for the local PINNs.

Moreover, instead of an overlapping Schwarz DDM, nonoverlapping DDMs can be employed, for instance, based on Dirichlet-Neumann, Neumann-Neumann, or Dirichlet-Dirichlet (FETI) methods. In addition to Dirichlet data, all these approaches require the transfer of Neumann boundary data on the interface. Therefore, the normal derivatives of the local neural networks have to be evaluated and trained, respectively, on the interface. The normal derivatives can be easily computed for neural networks using the backpropagation algorithm. A related approach for solving a boundary value problem using a partitioned deep learning algorithm based on direct substructuring has also recently been presented by Hailong Sheng at the 26th International Domain Decomposition Conference 72. In particular, the problem is partitioned into local Dirichlet problems and a Schur complement system on the interface. This approach could also be extended to a multilevel direct substructuring or nested dissection approach, respectively; cf. 73.