Comptes Rendus Mécanique

. The present work aims at analyzing issues related to the data manifold dimensionality. The interest of the study is twofold: (i) ﬁrst, when too many measurable variables are considered, manifold learning is expectedtoextractuselessvariables;(ii)second,andmoreimportant,thesametechnique,manifoldlearning, could be utilized for identifying the necessity of employing latent extra variables able to recover single-valued outputs. Both aspects are discussed in the modeling of materials and structural systems by using unsupervised manifold learning strategies.


Introduction
Recently, data-driven description of materials has been gaining popularity.Many complex material behaviors resisting traditional modeling procedures, or that are too complex from a microstructural viewpoint, are approached by using data-based descriptions.Different approaches are being considered.Among them include those based exclusively on measured data, others that extract the manifolds related to data, and others that attempt to enforce thermodynamic and thermomechanical consistency.The interested reader can refer to [1][2][3][4][5][6][7] and the numerous references therein.This work focuses on techniques based on the use of manifolds and their associated manifold learning procedures for extracting them from the available data.In general, data involve many dimensions.Consider first a sequence of three-dimensional (3D) fields defined in a domain Ω ⊂ R 3 partitioned into D voxels.Each of these fields contains many data, one datum at each voxel.Each field can be represented as a point in a vector space of dimension D (the number of voxels), where we can presume each of the D coordinate axes as reporting the value that the field of interest takes in the associated voxel.Thus, each field becomes a point in that high-dimensional space of dimension D, R D .If important correlations exist among the different fields, these points are expected to be distributed on a low-dimensional subspace embedded in the D-dimensional space.Techniques aiming at extracting these reduced subspaces, the so-called slow manifolds, sketched in Figure 1, are key tools for manipulating data and extracting their hidden information.
Thus, data define in general slow manifolds embedded in very large vector spaces due to the significant hidden correlations among them.The number of uncorrelated explicative dimensions usually becomes much smaller than the a priori assumed dimension of the space for accommodating the data.The extraction of these slow manifolds can be successfully accomplished by using linear and nonlinear dimensionality reduction techniques such as principal component analysis (PCA) in the linear case and its nonlinear counterparts ( PCA, kernel-based PCA [k-PCA], LLE, tSNE, etc.) [8][9][10][11][12].
These techniques can be applied to several physical systems.Moreover, when the slow manifold is determined, the solution at any point on it can be computed very accurately from a simple interpolation of neighboring data on the manifold [13], enabling almost real-time predictions and the associated real-time decision-making.
However, extracting knowledge from data associated with an existing but hidden model requires the following: • identifying the manifold intrinsic dimension, • discovering hidden parameters, • discarding useless parameters, and • discovering the models originating the data.These questions are addressed, illustrated, and discussed in the present work in a purely methodological manner, aiming at illustrating the key concepts that could open numerous future possibilities in the field of mechanics of materials, processes, structures, and systems.
In our previous works, we addressed problems involving thousands of dimensions [13][14][15][16], proving that despite the apparent richness in many cases, the embedding regards a relatively low-dimensional space.However, in most of the problems that we have treated until now, their complexity prevented fine analyses of their solutions.The present paper, which is purely methodological, considers simple problems defined in low dimensions, with known solutions being easily visualizable, for facilitating an analysis and discussion.Of course, and as proved in the works just referred to, all the methodologies apply to multidimensional settings.

Unsupervised manifold learning
Let us consider a vector y ∈ R D containing experimental or synthetic data from measurements or numerical simulation.These results are often referred to as snapshots.If they are obtained by numerical simulation, they consist of nodal values of the essential variable.Therefore, these variables will be somehow correlated and, notably, there will be a linear transformation W defining the vector ξ ∈ R d , with d < D, which contains the still unknown latent variables such that y = Wξ.
The D × d transformation matrix W, which satisfies the orthogonality condition W T W = I d , is the main ingredient of the PCA and can be computed as detailed in Appendix A from the covariance matrix associated with a number (M) of snapshots y 1 , . . ., y M , which constitute the columns of matrix Y.
While PCA works with the covariance matrix (i.e., YY T ), multidimensional scaling (MDS) works with the Gram matrix containing scalar products (i.e., S = Y T Y) as described in Appendix A.
On the other hand, the k-PCA is based on the fact that data not linearly separable in D dimensions could be linearly separated if previously projected to a space in Q > D dimensions.However, the true advantage arises from the fact that it is not necessary to write down the analytical expression of that mapping as described in Appendix A.

An illustrative structural mechanics case study
Consider first a hypothetical mechanical system consisting of a prismatic beam whose three dimensions, height, width, and length, are denoted, respectively, by h, b, and L, all of them being measurable quantities.In what follows, we consider a particular output P that constitutes also a measurable quantity (buckling critical load, etc.) assumed related to those parameters from an existing but actually hidden model even if in what follows we will consider hypothetical, and most of the time, unphysical models.Thus, we consider a set of data composed of M measures y i = {h i , b i , L i , P i }, i = 1, . . ., M, with h i , b i , and L i being randomly chosen from a uniform probability distribution in their respective intervals of existence, I h , I b , and I L , respectively, defined from the following: Without any other pre-existing knowledge, one expects the output depending on the three geometrical parameters (i.e., P = P (h, b, L)).In what follows, we consider three different scenarios.

Output depending on a single parameter
In this section, we assume a quite simple model that relates the output to a single parameter, where α = 1000 in the numerical tests carried out.We perform M = 1000 measures, where  The measures constitute a set of M = 1000 points in R 4 on which the k-PCA is applied by using the Gaussian kernel where β = 10.
Figure 2 depicts the highest eigenvalues among M resulting from the k-PCA, those lying between the highest value λ 1 and 10 −6 λ 1 .
The slow manifold associated with ξ is represented by selecting the first three reduced coordinates (ξ 1 , ξ 2 , ξ 3 ) as shown in Figure 3, where its one-dimensional (1D) intrinsic dimension is noted.This result was expected from the considered model expressed by (3).The points on the slow manifold are colored depending on the values of h, b, L, and P , evidencing that b constitutes the latent variable and that the output P scales (visually) linearly with it.
The process of coloring the data points in the embedded manifold deserves some additional comments due to the fact that this is used in all the analyses reported in the present paper.Manifold learning techniques look for a low-dimensional manifold defined by data points.As soon as the slow manifold is extracted, the different data points can be mapped on it.This visualization is only possible when the number of dimensions allows a simple graphical representation (as is the case for the problems addressed in the present paper).Then, these points can be colored depending on the value of the different initial coordinates, and one expects that if there is a correlation (direct or inverse and linear or nonlinear) between the initial and reduced coordinates, the colors must exhibit a certain grading.
Even if this analysis seems quite dependent on the low dimensionality of the embedding, in higher dimensional embeddings, the analysis can be performed by using local statistics.Thus, by considering a data point in the slow manifold and its closest neighbors, a local statistical analysis can be easily performed with the standard deviation indicating the dispersion of the data (equivalent to the local dispersion of colors).
One could be slightly surprised by the nonlinearity that the manifold exhibits despite the linearity of model ( 3).This nonlinearity is an artifact of the nonlinear kernel (4) used.As the model is linear, one could expect the ability of the PCA to address the problem at hand.For this purpose, it suffices transforming the kernel into its linear counterpart, giving rise to the PCA κ(y i , y j ) = y i • y j .
(5)  In this case, as expected, a single nonzero eigenvalue results, and consequently the dimension of the reduced space becomes one (i.e., ξ = ξ).The associated manifold is depicted in Figure 4. Now, we consider a slightly different model, again depending on a single variable but in a nonlinear manner, according to  where again α = 1000.Figure 5 depicts the 1D slow manifold, where points are colored according to the values of the variables h and P .Here, even if the direct relation can be noted, its nonlinearity is much less evident to visualize.Finally, we consider the model where α = 1000.Figure 6 depicts the 1D slow manifold, where points are colored according to the values of the variables L and P to emphasize the inverse relation between them.

Output depending on two parameters
In this section, we consider a model involving two of the three variables, in particular, where α = 1000.Figure 7 depicts the two-dimensional (2D) slow manifold, where points are colored according to the values of variables h, b, L, and P .Here, the direct and inverse effects of h and L with respect to P can be noted as well as the fact that parameter b seems, and in fact is, useless.

Identifying hidden variables
The previous case studies revealed the ability to extract the intrinsic dimensionality of the slow manifold as well as the possibility to identify useless parameters.The present case addresses a very different situation in which the model involves the three variables in the discussed case, h, b, and L. However, only two of them were measured, namely h and L with the output P , with b remaining inaccessible.
Thus, we have where α = 1000.Therefore, the M = 1000 collected data y i , i = 1, . . ., M, reads as Figure 8 depicts the reduced points (ξ 1 , ξ 2 , ξ 3 ), which as can be seen are distributed in a domain ω ⊂ R 3 .However, no dimensionality reduction is noted, and the embedding remains 3D.A direct consequence is that many values of the output P exist for the same values on the measured inputs h and L, related to the different values of b, which affect the output P .However, as b is not measured, its value is not considered in the data points.
Such a multivalued output does not represent any conceptual difficulty.It indicates that even if both variables participate in the output, there may be others that were not considered or those considered may be useless for explaining the output.To conclude about the pertinence of these variables with respect to the considered output, we consider coloring the reduced data depending on the h and L values (refer to Figure 9).We compare them with the one where the color scales with the output P reported in Figure 8.
Thus, one could conclude that both variables h and L are relevant for explaining the output P .If we assume that the output P should be univocally explained from a small number of variables, clearly only two variables (here h and L) are not sufficient.One extra dimension suffices for recovering a single-valued output, that is, considering the reduced points in four dimensions R 4 .
As visualizing things in four dimensions is quite a difficult task, for the sake of clarity in the exposition, in what follows, we propose addressing a simpler model involving lower dimensional spaces.
We consider the simpler model where α = 1000.However, the M = 1000 collected data y i , i = 1, . . ., M, only deal with h and P , (i.e., Figure 10 depicts the dataset y i = {h i , P i }, where it can be noted that many values of the output P are found for the same value of the variable h. For univocally expressing the output P , we consider again the k-PCA.We compute the reduced dataset in a 3D space by considering the first three coordinates (ξ 1 , ξ 2 , ξ 3 ), while coloring these  points with respect to h, to prove that h represents an explanatory variable, or with respect to P (refer to Figure 11).Figure 12 depicts the same manifold but now colored by using the hidden variable b, which proves that it contributes to explaining the output P and constitutes the hidden variable.Even if we just proved that a latent variable exists, and that it corresponds to b, as in practice we ignore this fact, we never measure the quantity b.Furthermore, it is even possible that we ignore its existence; the proposed procedure is only informative but not constructive.
Obviously, there is no unique choice.The same behavior is obtained by coloring the reduced dataset with respect to any function b p h q , where p, q ∈ R. Thus, any measurable variable ensuring such a uniform color grading could be used as the latent variable for constructing the model.There are an infinite number of possibilities but with certain constraints; in the present case, it must involve b.
To conclude this section, we address a similar but more complex situation.We consider now the richer model ( 9) with α = 1000.The M = 1000 collected data y i , i = 1, . . ., M, only deal with the input variable h and the output P (i.e., y i = {h i , P i } ∈ R 2 ), where two variables, b and L, involved in model ( 9) remain hidden.Figure 13 depicts the dataset y i = {h i , P i }, where again it can be noted that many values of the output P are found for the same value of the variable h.
For univocally expressing the output P , we consider again the k-PCA.We compute the reduced dataset in a 3D space by considering again the first three coordinates (ξ 1 , ξ 2 , ξ 3 ) while coloring  these points with respect to h to prove that h represents an explanatory variable of the output P as Figure 14 proves.Figure 15 clearly reveals that by coloring the reduced points with respect to the two variables b and L taken solely, they do not represent the unique hidden variable able to explain using h the output P .The hidden latent variable according to model ( 9) should combine b and h. Figure 16 proves that the combined parameter b/L 2 perfectly works when compared with the manifold colored with respect to the output P .
It is important to note that the latent variable must involve the term b/L 2 up to any power and eventually a multiple of any power of h.

Discussion
The previous numerical experiments allow drawing a conclusion on the ability of unsupervised manifold learning techniques for identifying useless data and the existence of latent variables.
The procedure is based on the ability of embedding high-dimensional data in a lowdimensional space, where the dimension represents an approximation of the inherent dimensionality of the data.When the expected dimensionality reduction does not occur and the k-PCA reveals univocally that an extra dimension is required to accommodate the data, this indicates the existence of a hidden latent variable.
However, when the dimensionality reduction applies, by analyzing locally the manifold (any embedded data point with its closest neighbors) with respect to every original coordinate attached to each embedded data point, one can conclude that correlations exist between the initial and reduced coordinates (color grading or low statistical variance).When the dispersion increases to a large extent, one can infer that the initial coordinate analyzed is not directly correlated (even if it could be correlated in a more complex or combined manner) as discussed.
To prove the generality of these conclusions, Section 4 addresses a more complex scenario.It deals with the constitutive modeling of materials, particularly those whose behavior depends on the deformation history of the material.

Simple mechanical behaviors
In this section, we consider some simple 1D mechanical behaviors as that depicted in Figure 17.
Figure 18 shows a set of M = 1500 strain-stress pairs y i = {ε i , σ i }, i = 1, . . ., M, related to an hypothetical 1D elastic-plastic behavior with linear hardening.The depicted points represent the final mechanical state (strain-stress) of M loading-unloading random trajectories.It can be noted that with only the information of possible mechanical states, given a strain value, there are many probable values of permissible stresses and vice versa.
There is nothing physically inconsistent in having multivalued fields, but if one is looking for single-valued stress, then an extra latent variable must be introduced.
If, as in Section 3, we apply the k-PCA on the dataset y i , i = 1, . . ., M, as expected, the 2D manifold appears embedded in the R 3 space as depicted in Figure 19.This indicates that three mechanical variables, stress and strain completed by a latent extra variable p, are mutually related (e.g., σ = σ(ε, p)), giving rise to a 2D manifold embedded in R 3 .
As previously discussed, many latent variables could be certainly considered for obtaining for example a single-valued stress value.Historically, the plastic strain ε p , illustrated in Figure 20, was widely considered as a latent variable.Coloring the reduced points ξ i depicted in Figure 19 according to the plastic strain results in the colored manifold shown in Figure 21.This validates the choice of the plastic strain as a latent variable.

Toward alternative representations of the mechanical state
An alternative representation of the mechanical state, avoiding the choice of nonevident latent variables, consists in assuming that the stress at time t depends on the whole strain history, that is, on ε(τ), τ ≤ t .
When memory effects are neglected, the mechanical state can be described at time t from the strain and stress increments and the present state of stress-strain.That is, ∀t i = i∆t , y i = {ε i , σ i , ∆σ i /∆ε i }, where ∆ε i has a given magnitude and two possible signs (positive in loading and negative in unloading).Different scenarios can be analyzed: • Linear elastic regime without plastic strain.In this case, as expected, the slow manifold constituted by the reduced states ξ i related to the mechanical states y i , depicted in Figure 22, in the 3D space (ξ 1 , ξ 2 , ξ 3 ) becomes 1D.• Nonlinear elastic behavior (with the tangent modulus E T = E ε 2 ) without plastic strain.
In this case, the results are similar to those just discussed.In Figure 23, the nonlinear behavior can be noted.• Linear elastic regime with nonzero plastic strain (perfect plasticity-no hardening).In the present case, in the mechanical state within the elastic domain, we recover from y i a reduced slow manifold of dimension two as expected, which is depicted in Figure 24.• Linear elastic regime with nonzero plastic strain (perfect plasticity-no hardening) with activated damage.In the present case, with respect to the previously discussed scenario, we presume that the material degrades with the magnitude of the plastic strain and the tangent modulus decreases accordingly.As Figure 25 depicts, this case is quite similar to the previous scenario.However, when coloring with respect to the tangent modulus, the expected uniform degraded map is noted.• The present case study considers, in addition to the points inside the elastic domain, another set of mechanical states on the elastic domain boundary with a different   variable that is the product of the plastic strain with another extra variable.The latter, which could represent strain-rate sensitivity, is assumed to take arbitrary values here due to the fact that we are more interested in methodological aspects than in physical considerations.The considered mechanical states are depicted in Figure 28.When applying the k-PCA to the set of mechanical states y i = {ε i , σ i , ∆σ i /∆ε i }, the resulting manifold remains 3D.As expected, no dimensionality reduction is accomplished as Figure 29 reveals, where many values of the elastic tangent modulus can be found for the same values of the stress and strain.To investigate the nature of this behavior, we depict in Figure 30 the elastic tangent modulus versus the plastic strain, where as expected it can be noted that the former does not depend exclusively on the latter.By applying the k-PCA to the data shown in Figure 30, which are nonseparable in two dimensions, one expects  to separate them by embedding in a 3D space as previously discussed and as Figure 31 proves.Finally, Figure 32 presents the slow manifold from Figure 31 but now colored with respect to the plastic strain or the extra latent variable.

Conclusion
The present work introduced, tested, and discussed issues related to manifold dimensionality with two major purposes: (i) first, when too many measurable variables are employed, manifold learning is able to discard the useless variables; (ii) second and more important, the same technique can be employed for discovering the necessity of employing and then measuring an extra latent variable that is able to recover and ensure single-valued outputs.Both issues were analyzed and discussed in two case studies, one with respect to structural mechanics and the other with respect to path-dependent material constitutive behaviors.Indeed, the physical interpretation of the discovered latent variable could imply the introduction of other measurable variables.This topic should be deeply analyzed.
The main interest in discovering these manifolds is that for a new accessible mechanical state, the output can be inferred by a simple interpolation from its neighbors on the manifold.The main aim of the present work is the construction and analysis of these manifolds.Future works, currently in progress, will focus on their use in performing data-driven simulations.
where V contains the orthonormal eigenvectors; Λ is a diagonal matrix containing the eigenvalues sorted in descending order.
Substituting ( 16) into (15), we arrive at This equality holds when the d columns of W are taken to be collinear with d columns of V. We then preserve the eigenvectors associated with the d nonzero eigenvalues, which gives We therefore conclude that the eigenvalues in Λ represent the variance of the latent variables (diagonal entries of C ξξ ).

A.2. Multidimensional scaling
The PCA works with the covariance matrix of the experimental results, YY T .However, the MDS works with the Gram matrix containing scalar products (i.e., S = Y T Y) [8].
The MDS preserves pairwise scalar products: Computing the eigenvalues of S, we arrive at which in turn gives A.

Kernel-based principal component analysis
The k-PCA is based on the fact that data not linearly separable in D dimensions could be linearly separated if they are previously projected to a space in Q > D dimensions.However, the true advantage arises from the fact that it is not necessary to write down the analytical expression of that mapping.
The symmetric matrix Φ = Z T Z, with Z containing the snapshots z i ∈ R Q , i = 1, . . ., M, associated with y i ∈ R D , has to be decomposed into eigenvalues and eigenvectors.The procedure for centering data z i is carried out in an implicit way.The eigenvector decomposition reads as giving rise to The difficulties of operating in a high-dimensional space of dimension, in general, Q D, and the mapping unavailability are circumvented by introducing the kernel functional κ (also known as the kernel trick).This allows computing scalar products in R Q while operating in R D by applying the Mercer theorem.This theorem establishes that if κ(u, v) (where u ∈ R D and v ∈ R D ) is continuous, symmetric, and positive definite, then it defines an inner product in the mapped space R Q .Many different kernels exist; some of them are reported in [8].

Figure 3 .
Figure 3. Slow manifold in the 3D space defined by the first reduced coordinates (ξ 1 , ξ 2 , ξ 3 ).Each point is colored according to the value of the coordinate h (top left), b (top right), L (bottom left), or P (bottom right).

Figure 4 .
Figure 4. Slow manifold ξ when considering the PCA linear dimensionality reduction.Each point is colored according to the value of the variable b.

Figure 5 .
Figure 5. Slow manifold in the 3D space defined by the first three reduced coordinates (ξ 1 , ξ 2 , ξ 3 ).Each point is colored according to the value of the coordinate h (left) or P (right).

Figure 6 .
Figure 6.Slow manifold in the 3D space defined by the first three reduced coordinates (ξ 1 , ξ 2 , ξ 3 ).Each point is colored according to the value of the coordinate L (left) or P (right).

Figure 7 .
Figure 7. Slow manifold in the 3D space defined by the first three reduced coordinates (ξ 1 , ξ 2 , ξ 3 ).Each point is colored according to the value of the coordinate h (top left), b (top right), L (bottom left), or P (bottom right).

Figure 8 .
Figure 8. Reduced representation ξ i ∈ R 3 of the data y i ∈ R 3 , where the reduced points are colored according to the value of the output P .

Figure 9 .
Figure 9. Reduced points colored according to the values of the coordinates h (left) and L (right).

Figure 11 .
Figure 11.Reduced points colored according to the values of the coordinates h (left) and P (right).

Figure 12 .
Figure 12.Reduced points colored according to the value of the output b.

Figure 14 .
Figure 14.Reduced points colored according to the value of the coordinate h.

Figure 15 .
Figure 15.Reduced points colored according to the values of the variables b (left) and L (right).

Figure 16 .
Figure 16.Reduced points colored according to the values of the variables b/L 2 (left) and P (right).

Figure 18 .
Figure 18.Strain-stress mechanical states related to elastic-plastic behavior with linear hardening.

Figure 20 .
Figure 20.Plastic strain definition (left) and mechanical states y i colored according to the plastic strain (right).

Figure 21 .
Figure 21.Manifold colored according to the plastic strain.

Figure 22 .
Figure 22.Slow manifold related to linear elastic behavior and colored according to the tangent modulus (i.e., ∆σ i /∆ε i ).

Figure 25 .
Figure 25.Slow manifold related to damageable elastic-plastic behavior operating within the elastic domain colored according to the tangent modulus.

Figure 26 .
Figure 26.Slow manifold related to elastic-plastic behavior operating within the elastic domain and on the elastic domain boundary colored according to the tangent modulus.

Figure 27 .
Figure 27.Slow manifold related to elastic-plastic behavior operating within the elastic domain and on the elastic domain boundary, colored according to the tangent modulus, when damage is activated.

Figure 28 .
Figure28.Mechanical states within the elastic domain, colored according to the tangent modulus, the last depending on the product of two latent variables, the plastic strain, and another arbitrarily chosen variable.

Figure 29 .
Figure 29.Slow manifold related to elastic-plastic behavior operating within the elastic domain, colored according to the tangent modulus, with the last scaling with the product of the plastic strain and an extra latent variable.

Figure 32 .
Figure 32.Slow manifold of the elastic tangent modulus colored according to the plastic strain (left) and the extra latent variable (right).