%~Mouliné par MaN_auto v.0.27.3 2023-05-22 09:44:30
\documentclass[CRMECA, Unicode, XML, screen, Thematic]{cedram}



\ItHasTeXPublished

\newcommand{\bbR}{\mathbb{R}}
\newcommand{\bbK}{\mathbb{K}}
\newcommand{\bbL}{\mathbb{L}}
\newcommand{\bbM}{\mathbb{M}}
\newcommand{\bbB}{\mathbb{B}}
\newcommand{\ttR}{\mathtt{R}}
\newcommand{\ttN}{\mathtt{N}}
\newcommand{\ttK}{\mathtt{K}}
\newcommand{\ttD}{\mathtt{D}}
\newcommand{\ttLR}{\mathtt{LR}}
\newcommand{\ttspan}{\mathtt{span}}
\newcommand{\ttQ}{\mathtt{Q}}
\newcommand{\ttM}{\mathtt{M}}
\newcommand{\bfU}{\mathbf{U}}
\newcommand{\bfF}{\mathbf{F}}
\newcommand{\bfZ}{\mathbf{Z}}
\newcommand{\bfR}{\mathbf{R}}
\newcommand{\bfM}{\mathbf{M}}
\newcommand{\bfV}{\mathbf{V}}
\newcommand{\bfzo}{\mathbf{0}}
\newcommand{\clT}{\mathcal{T}}
\newcommand{\clI}{\mathcal{I}}
\newcommand{\clR}{\mathcal{R}}
\newcommand{\clS}{\mathcal{S}}
\newcommand{\clH}{\mathcal{H}}
\newcommand{\clC}{\mathcal{C}}
\newcommand{\clL}{\mathcal{L}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\graphicspath{{./figures/}}

\newcommand*{\mk}{\mkern -1mu}
\newcommand*{\Mk}{\mkern -2mu}
\newcommand*{\mK}{\mkern 1mu}
\newcommand*{\MK}{\mkern 2mu}

%\hypersetup{urlcolor=purple, linkcolor=blue, citecolor=red}

\let\oldtilde\tilde
\renewcommand*{\tilde}[1]{\mathchoice{\widetilde{#1}}{\widetilde{#1}}{\oldtilde{#1}}{\oldtilde{#1}}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\RubricEF{The French `Année de la Mécanique': some views on recent advances in solid and fluid mechanics}{L'année de la mécanique : quelques points de vue sur des avancées récentes en mécanique des solides et des fluides}

\title{Conciliating accuracy and efficiency to empower engineering based on performance: a short journey}

\keywords{Physics-based modeling, Machine learning, Artificial Intelligence, Data-driven modeling, Model Order Reduction, POD, PGD, Virtual, Digital and Hybrid Twins}

\author{\firstname{Francisco} \lastname{Chinesta}\IsCorresp}
\address{PIMM lab, Arts et Metiers Institute of Technology, 151 Boulevard de H\^opital, 75013 Paris, France}
\address{CNRS@CREATE LTD, 1 Create Way, 08-01 CREATE Tower, Singapore 138602}
\email[F. Chinesta]{francisco.chinesta@ensam.eu}
\thanks{F. C. is supported by the CREATE-ID ESI-ENSAM research chair. E. C. is supported by ESI Group through the Simulated Reality project, UZ-2019-0060. 
This work has been partially funded by the Spanish Ministry of Science and Innovation, AEI /10.13039/501100011033, through Grant number PID2020-113463RB-C31 and the Regional Government of Aragon and the European Social Fund, group T24-20R.}

\author{\firstname{Elias} \lastname{Cueto}}
\address{Aragon Institute of Engineering Research, Universidad de Zaragoza, Maria de Luna s/n, 50018 Zaragoza, Spain}
\email[E. Cueto]{ecueto@unizar.es}




\begin{abstract}
This paper revisits the different arts of engineering. The \emph{art of modeling} for describing the behavior of complex systems from the solution of partial differential equations that are expected to govern their responses. Then, the \emph{art of simulation} concerns the ability of solving these complex mathematical objects expected to describe the physical reality as accurately as possible (accuracy with respect to the exact solution of the models) and as fast as possible. Finally, the \emph{art of decision making} needs to ensure accurate and fast predictions for efficient diagnosis and prognosis. For that purpose physics-informed digital twins (also known as Hybrid Twins) will be employed, allying real-time physics (where complex models are solved by using advanced model order reduction techniques) and physics-informed data-driven models for filling the gap between the reality and the physics-based model predictions. The use of physics-aware data-driven models in tandem with physics-based reduced order models allows us to predict very fast without compromising accuracy. This is compulsory for diagnosis and prognosis purposes.
\end{abstract}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\dateposted{2023-06-22}
\begin{document}
\maketitle

\section{The whole picture}

At the beginning was science. The so-called modern science, for some historians of science Galileo's heritage, would have to wait a bit more. From the very beginning, the human being created technology to master the natural environment and take profit of it for constructing tools that facilitated the everyday life. Later, was the time to surpass the status quo, to break limits, to make non-natural things---like flying for instance---and engineering became a major protagonist, looking for improving performances. Thus, the dream of Leonardo was flying, the wings were there for making flying possible, or simpler. In many cases the application preceded the scientific understanding.


As Theodore Von Karman said, scientists study the world as it is, engineers create the world that never has been. The first industrial revolution arose from steam engines and transformed industry. Later on, electricity gave rise to the second revolution, the one enabling producing more and faster, and this one not only transformed industry but also the society.

Alessandro Volta was invited two centuries ago at the French academy of sciences in Paris to present his battery. He exhibited the action of applying the electrodes of his battery to a frog. All the attendants were very impressed by the frog jumps. After that exciting session, the legend says that Napoleon asked: Alessandro, do you think that one day, this thing that you call electricity, could serve to something else that making jumps to the frogs? Few generations later, can you imagine one single day of your \emph{everyday} life without electricity?

Later, electronics entered the scene and, with it, automation. Welcome to the third industrial revolution that enabled not only making faster, but making better. Despite all these advances, engineering remained product-based, despite of the fact that the society is looking for performance. The product is a simple way to access to this performance. When we buy an electric drill, we are in fact trying to buy a good quality hole. However, engineering remained product-oriented. Why?

Certainty, due to the fact that even if physics-based models are very rich (in terms of physics), it was difficult to both (i) address the product in its environment, usually too large, too complex, too uncertain, too fluctuating\dots; and (ii) solve them under stringent real-time constraints, needed for optimal decision in operation, the ultimate goal of engineers and engineering.

\section{The three arts of the engineering}

The just referred challenges were addressed by empowering three arts of engineering.

\subsection{The art of modeling}

Science begun by empirical observation, something that is often called the first paradigm of science. This is science in the times of Tycho Brahe. But science soon embraced---in fact, with Brahe's disciple, Kepler---a new paradigm: that of scientific theory. The \emph{art of modeling} concerned all the physics throughout all the description scales of materials, processes and structures, a rich and holistic physics-based approach. This \emph{art} allowed us to make accurate predictions concerning materials, manufacturing processes and structural analysis, even in extreme conditions and with the computational facilities available 45 years ago. These were in some cases smaller than the one we have in the smartphone placed in our pocket.

Despite of the encountered computing limitations it was possible for instance, in 1985, to simulate a crash test (involving fast-dynamics, multi-contact, plastic deformation, damage and rupture, the worst among all the imaginable behaviors), as sketched in Figure~\ref{crash}. This was a major accomplishment in computer simulation that can be found in the Computer History Museum at San Jose (California) \cite{crash}.

\begin{figure}[h!]
\centering
\includegraphics[width=0.85\linewidth]{Crash.png}
\caption{Pioneering crash analyses. Picture courtesy of ESI Group.}
\label{crash}
\end{figure}

However, these accomplishments remained limited by the available computational resources as well as by the characteristic time of calculation and response. Replacing product management by the management of performance needs for faster predictions, for faster and more optimal designs, for faster and better decisions.

\subsection{The art of simulation}

Thus, the \emph{art of modeling} was enriched with a second art, the \emph{art of simulation}---the third paradigm of science---that can be summarized in a single sentence: the ability of applying the best numerical technique to a given problem, ensuring the best accuracy and efficiency. The \emph{art of simulation} is also a long history, more than 50 years old, that began when computers irrupted and replaced the human analogical calculation by its radically new, and much faster, digital counterpart. But the new engineering based on performance, at the beginning of the third millennium, needs to proceed faster, in real-time. Sometimes even faster.

It was at the end of the XX century, that new technologies were proposed and implemented, that without breaking down its physics-based rich descriptions, allowed to speed-up their predictive capacities, i.e., the solution of the complex models that such rich engineering descriptions imply. New capabilities were possible by involving the use of Model Order Reduction (MOR).

To introduce the main idea behind MOR, we consider the picture of a very touristic monument on Paris, the one depicted in Figure~\ref{panth}. When showing this image to our Parisian colleagues, asking them about what monument it is, the response is immediate: it is the \emph{Pantheon}. Even if we are very happy with that response, our next question is, after removing the picture from their eyes: how many columns are placed in its front? There, the responses are quite diverse, 5 for some of the interrogated people, 6 for others, some indicate seven\dots The surprise arrives when our colleagues are informed on the exact number of columns: just 22. Every response was very far from the exact value.


\begin{figure}[h!]
\centering
\includegraphics[width=0.65\linewidth]{panth.jpg}
\caption{A highly visited monument in Paris.}\label{panth}
\end{figure}


What does this experiment prove? It proves the fact that we can be sure on something even when we absolutely ignore the details. Thus, MOR could be defined as the mathematical technique able to extract essential features while ignoring accessorial details. Obviously, this definition entails the dependence on the definition of essential and accessorial. For a tourist or a computational algorithm for recognizing monuments from pictures, the essential is recognizing the monument, whereas for the enterprise in charge of the columns maintenance the number of columns seems crucial! Thus, models can and should be reduced with respect to a given goal, and the validity of reduced models depend on the circumstances in which they were learned.

For describing MOR in a more proper way, we consider a field of interest $u(x,t)$, with the quantity $u(\bullet)$ being a scalar, a vector or a tensor field, that depends on the physical space, represented by the space coordinates $x \in \Omega \subset \bbR^\ttD$ ($\ttD=3$ when operating in the three-dimensional space) and time $t \in \clT \subset \bbR$.

For computing numerically such a rich field, the first step consists in reducing the number of unknowns---a step called discretization---that approximates the solution from the value of the field at different locations $x_i, \ i=1,\, \ldots,\,\ttN$, (the so-called nodes in finite elements notation) and at different time instants $t_m, \ m=1,\,\ldots,\,\ttM$, according to the general form
\begin{equation}\label{ap}
u(x,t_m) \approx \sum \limits_{i=1}^\ttN U_i^m N_i(x),
\end{equation}
where when considering an interpolative approximation, i.e., $ N_i(x_j)=\delta_{ij}$, with $\delta$ the Kronecker delta, $U_i^m=u(x_i,t_m)$. The previous expression represents the usual approximation employed in the Finite Elements Method (FEM), where the approximation functions $N_i(x)$ are called {\em shape functions}.

For computing the $\ttN$ unknown values of the field at each time instant $t_m$, an equivalent number of equations is required. For that, physics offers in general the model to which the evolution (in space and time) of the field under consideration is subjected, represented in its most general form as
\begin{equation}
\clL(u(x,t);\mu)= F(x,t;\mu), \;\text{in }\; \Omega_\mu,
\end{equation}
where $\clL(\bullet)$ represents a generic linear or nonlinear differential operator, $F(\bullet)$ is the so-called forcing term, and $\mu$ a set of parameters. Some of these parameters could affect the domain in which the problem is defined $\Omega_\mu$ (the so-called geometrical parameters), some of them could affect the physical model itself and others the forcing term. Thus, it is expected that the solution will depend on the choice made for the different parameters grouped in vector $\mu$, i.e., $u(x,t;\mu)$.

In general, time derivatives involved in the differential operator are approximated by using adequate finite difference schemes, taking profit of the causality that allows computing the present from the past. Thus, the general physics-based model, at time $t_m$, reads
\begin{equation}
\tilde{\clL}(u(x,t_m);\mu)= \tilde{F}(x,t_m;\mu), \; \text{in } \; \Omega_\mu, \ t=t_m,
\end{equation}
that involves the solution at the present time, i.e. $u(x,t_m;\mu)$ that by using the approximation~\eqref{ap} consists of $\ttN$ unknowns, the $\ttN$ nodal values of the field at the present time $t_m$ and for the considered set of parameters $\mu$, i.e. $U_i^m(\mu)$, $i=1, \,\ldots,\,\ttN$. In the previous equation, $\tilde{\bullet}$ represents the semi-discretized differential form.

Now, if we define the model residual at present time $t_m$ and for the given choice of the parameters $\mu$, as
\begin{equation}
\clR_\mu (u(x,t_m)) = \tilde{\clL}(u(x,t_m);\mu) - \tilde{F}(x,t_m;\mu),
\end{equation}
to enforce its nullity, the weighted residual formulation proceeds by enforcing, for any test function $u^\ast(x,t_m)$ in an appropriate functional space, a vanishing integral
\begin{equation}\label{wr}
\int_{\Omega_\mu} u^\ast(x,t_m) \ \clR_\mu(u(x,t_m)) \ dx = 0.
\end{equation}

Within the Galerkin setting, the test functions are approximated from
\begin{equation}\label{tp}
u^\ast(x,t_m) \approx \sum \limits_{i=1}^\ttN U_i^\ast N_i(x),
\end{equation}
that introduced into the weighted residual formulation Eq.~\eqref{wr} and taking into account the arbitrariness of the $\ttN$ coefficients $U_i^\ast$ allows (after some technical manipulations: integration by parts, domain partition into elements, numerical integration, enforcement of essential boundary conditions\dots) to the algebraic system
\begin{equation}\label{as}
\bbK \bfU^m = \bfF,
\end{equation}
where matrix $\bbK$ could depend on the solution itself $\bfU^m$, in the case of nonlinear modes, on time, and also on the parameters choice $\mu$. The forcing term $\bfF$ will depend, in turn, on present and/or past time instants, as well as on the parameters choice. All these dependences are not explicitly indicated for the sake of notational simplicity.

Even if the reduction accomplished is unimaginable, a \emph{double infinity} of points and time instants, condensed into $\ttN \times \ttM$ discrete values (nodes and time instants), sometimes the complexity of the involved physics and the associated problem solutions, needs for several millions of nodes where then solution must be computed every mili-second, or even micro-second, as it is the case when analyzing the crash test before referred.

Solving too many algebraic systems, each one containing millions of rows and columns, is feasible with the nowadays computational resources. However, fulfilling the stringent real-time constraints while keeping reasonable the computational resources (if simulation is expected be used in small and medium enterprises, as a consequence of democratizing simulation) is far to be obvious. Despite of the impressive reduction just mentioned, that leaded to the algebraic system~\eqref{as}, its repeated solution continues being, very often, too expensive. MOR could alleviate these difficulties.

Imagine for a while, and without loss of generality, a steady-state linear problem, whose discretized form reads
\begin{equation}\label{ss}
\bbK \bfU = \bfF.
\end{equation}
This discrete model is able to compute the response $\bfU$ for any choice of the forcing term $\bfF$, that is, $\bfU^1$ results from the forcing $\bfF^1=(1,0,\,\ldots,\, 0)^T$, $\bfU^2$ from $\bfF^2=(0,1,0,\, \ldots,\, 0)^T$ and so on, until the response $\bfU^\ttN$ related to $\bfF^\ttN=(0,0,\,\ldots,\, 1)^T$. Because of the linearity, superposition applies and the response calculation to any forcing spanned into the canonical basis just considered, is straightforward.

However, in general, only a region of this impressively large domain $\bbR^\ttN$ of all the possible responses is, and will be, explored by a system during its life. Even the human being, each one of us, and despite the impressive freedom that we have, we explore only a small region of the domain containing all the possibilities open to us.

Thus, one could expect the response $\bfU$ expressible in a smaller approximation basis, instead of the one of dimension $\ttN$ just considered. If a reduced approximation basis exists, i.e., all the expected solutions $\bfU$ could be approximated by using a reduced basis of size $\ttR$, i.e., $\bfU = \ttspan \{ \bfR_1, \,\ldots,\,\bfR_\ttR \}$, with $\ttR \ll \ttN$, then by putting the different vectors composing the reduced basis into the columns of matrix $\bbB$, i.e. $\bbB = (\bfR_1\,\ldots\,\bfR_\ttR)$, one can write $\bfU = \bbB \bfV$, where $\bfV$ is a $\ttR \times 1$ vector, which component $V_k$ represents the contribution of the approximation function $\bfR_k$ for representing $\bfU$.

Thus, the algebraic system~\eqref{ss} can be rewritten in its reduced form
\begin{equation}\label{ss}
\bbK \bbB \bfV = \bfF,
\end{equation}
that,projected into the reduced basis (Galerkin projection), becomes
\begin{equation}\label{rss}
\bbB^T \bbK \bbB \bfV = \bbB^T \bfF,
\end{equation}
which defines an algebraic system of size $\ttR \times \ttR$.

The only remaining issue is the procedure to be employed for discovering that reduced basis able to span the problem solution. The construction of this basis is based on an appropriate learning procedure. For that purpose, manifold learning can be successfully applied for extracting the so-called slow manifold where the problem solution lives. Principal Component Analysis (PCA) performs well in the case of linear manifolds, whereas nonlinear manifold learning performs very well when the solution manifold exhibits a noticeable nonlinearity~\cite{LEE07,ROW00,MAA08}.

When the reduced basis is extracted by using PCA, the projected algebraic system, Eq. \eqref{rss} represents the so-called Proper Orthogonal Decomposition (POD) based model order reduction, widely employed in many domains of science and engineering~\cite{CHI15}. The main issues remain the ones related to the reduced basis generality and adaptation, the error quantification, the sampling strategy for extracting the reduced basis (some improvements were proposed in the context of the so-called Reduced Basis method~\cite{CHI15,VER1,VER2,MAD1,MAD2}), addressing multi-parametric models and the efficient treatment of nonlinear models that needs advanced techniques (hyper-reduction~\cite{RYC05,CHA17a}, Empirical Interpolation Methods (EIM) \cite{BAR} and its discrete counterpart~\cite{CHA10}, cross approximations~\cite{AGU19}, among many other alternatives).

Until now, and despite of the fact of grouping all these techniques under the denomination of \emph{model order reduction}, the subjacent physical models were never reduced. All them come from the considered physics, with all their richness encapsulated into the matrix $\bbK$, resulting from a standard discretization. The only thing that was reduced was the basis in which the response $\bfU$ is expressed, as a consequence of the reduced nature of the forcing that remains constrained into a certain manifold $\omega$ (with intrinsic dimension $\ttR$ in the discussed case) of the whole space $\bbR^\ttN$.

An alternative procedure, more aligned with the model order reduction rationale, consists of calculating action/reaction pairs, i.e., $\left (\bfU^k,\bfF^k\right)$, $k=1, \ldots, \ttK$, and looking for the lowest rank matrix $\bbK^{\ttLR}$, i.e., the most reduced model, minimizing, by using a certain $p$-norm, the cost $\clC (\bbK^{\ttLR})$~\cite{REI19}:
\begin{equation}
\clC\left(\bbK^\ttLR \right) = \sum_{k=1}^\ttK \left\| \bbK^\ttLR \bfU^k - \bfF^k\right\|_p,
\end{equation}
as employed for discovering dynamical systems within the so-called Dynamic Mode Decomposition (DMD) \cite{SCH10,WIL15}.

In~\cite{REI19} the just reduced model extractor was extended for addressing nonlinear models, in~\cite{REI21} for operating within a reduced basis and also for extracting the reduced transfer function that approximates the inverse of model. Moreover, in~\cite{SAN21}, the procedure was adapted for constructing stable time integrators. We will come back later to the construction of models from data by using machine learning (ML) techniques. Before that, we will address the parametric dimension.

In the parametric case, POD-based techniques continues to perform well, as soon as the reduced basis is extracted from a training that covers as much as possible the parametric domain. Then, the solution can be projected into the reduced basis and model parameters inferred online by using performant data assimilation techniques, for instance Bayesian or Kalman filters.

Proper Orthogonal Decomposition with Interpolation, PODI, \cite{LY01} has been largely employed. When using the PODI, the coefficients affecting each of the vectors composing the reduced basis, are assumed to depend on the parameters, whose quantification is performed by using standard regressions. As a main limitation of this technique we can cite the number of parameters that complexities the construction of the regression.

For improving efficiency, several reduced bases are calculated at different locations of the parametric space, instead of a unique reduced basis in the whole parametric space. When different reduced bases are computed for different choices of the parameters, they must be interpolated for covering the whole parametric space before using it for approximating the solution and proceeding from the weighted residual form. Interpolation is a tricky issue and some efficient procedures were proposed for performing that interpolation, as for example the employ of Grassmannian manifolds~\cite{AMS08}.

Another way of computing parametric solutions of parametrized partial differential equations consists in applying the so-called Proper Generalized Decomposition, PGD, based on the separated representation of the solution. This was originally proposed for defining non-incremental transient solutions~\cite{LAD99}. In the parametric setting, the PGD proceeds by assuming a fully separated representation of the problem solution, where parameters are assumed extra-coordinates~\cite{CHI13}. Thus, if for the sake of simplicity we assume a model depending on a single parameter $\mu$, the parametric solution approximation $u^\ttQ(x,t,\mu)$ reads
\begin{equation}\label{sf}
u^\ttQ(x,t,\mu) = \sum_{i=1}^\ttQ X_i(x) T_i(t) M_i(\mu),
\end{equation}
with $x \in \Omega$, $t \in \clT$ and $\mu \in \clI$.

To compute the different unknown functions involved in the separated representation, Eq.~\eqref{sf}, the usual weighted residual form is extended according to
\begin{equation}\label{xwf}
\int_{\Omega \times \clT \times \clI} u^\ast(x,t,\mu) \clR(u(x,t,\mu)) \ dx \ dt \ d\mu = 0.
\end{equation}

Within a Galerkin framework, when looking to the $q^{\rm th}$ functional product, the trial and test functions to be employed within the integral form, Eq.~\eqref{xwf}, read respectively
\begin{align}
u^q(x,t,\mu) &= \sum_{i=1}^{q-1} X_i(x) T_i(t) M_i(\mu) + X_q(x) T_q(t) M_q(\mu) \equiv u^{q-1}(x,t,\mu) + X_q(x) T_q(t) M_q(\mu),
\\
\intertext{and}
u^\ast(x,t,\mu) &= X^\ast(x) T_q(t) M_q(\mu) + X_q(x) T^\ast(t) M_q(\mu) + X_q(x) T_q(t) M^\ast(\mu).
\end{align}

The separated representation constructor, deeply described in~\cite{CHI14}, proceeds by using an alternate directions, fixed point algorithm that computes the unknown function at the enrichment iteration $q$: $X_q(x)$ from $T_q(t)$ and $M_q(\mu)$ taken at the previous iteration, $T_q(t)$ from $X_q(x)$ and $M_q(\mu)$, and finally $M_q(\mu)$ from $X_q(x)$ and $T_q(t)$. The iteration continues until reaching the fixed point, and then the next function product, $q+1$ is considered.

The main difficulties in applying the just described procedure are the necessity of performing an affine decomposition of the problem residual involved in Eq.~\eqref{xwf} for ensuring the efficiency of the sequential calculation of each function that involves the corresponding problem coordinate. Such affine decomposition is not direct, mainly in the case of nonlinear models~\cite{CHI13,AGU19}. The other difficulty is related to the procedure intrusiveness that makes difficult its use in tandem with usual commercial software.

The main interest of computing such parametric solutions $u(x,t,\mu)$ is the ability of deriving real-time engineering, by computing any solution related to any parameters choice in almost real-time, fact that makes possible simulation, optimization, inverse analysis, simulation-based control and uncertainty propagation, all them under the stringent real-time constraints.

To overcome the difficulties related to the complex nonlinear problems and the procedure intrusiveness, one option consists in constructing metamodels (also known as surrogates, \linebreak response surfaces, virtual charts or vademecums). The construction of these metamodels can be performed by considering a simple workflow: (i) defining a sampling of the parametric space; (ii) computing a high fidelity solution for each parameter choice; (iii) using an adequate regression for extending the solution known at the points in the sampling everywhere in the parametric space.

There are many alternatives for performing the first and third tasks, trying to conciliate:\linebreak (i) active learning; making use of a sparse and very reduced data set, considering the best goal-oriented sampling, aimed at roughly linearly scaling with the dimension of the parameters space; (ii) the high fidelity solutions post-compression by extracting first and using then (by projection) reduced bases; (iii) rich enough approximation bases while avoiding overfitting, based on the use of sparse regularizations for enforcing parsimony (elastic-net, ridge, lasso, \dots) \cite{BRU16}; (iv) using orthogonal basis for evaluating sensibilities in a direct manner; and (v) efficiently addressing the high-dimensional spaces induced by the multi-parametric models, where the use of separated representations are specially suitable~\cite{TIB96,BOR19,IBA18,SAN21b}.

Other alternatives for performing regressions consist of using state-of-the-art techniques, like Support Vector Regression, SVR, \cite{AWA15}, Decision Trees~\cite{KIR02} or their Random Forest counterpart~\cite{BRE01}, Deep Neural Networks~\cite{SCH15,GOO16}. These last are very efficient when a large enough amount of data is available.

The use of those parametric solutions (or their parametric transfer functions counterpart) makes it possible to link system components and connect components with their environment, to constitute the so-called \emph{Augmented Virtual Prototype}, AVP, a nominal virtual replica of a real system. AVP allowed to address rich descriptions of physical systems in almost real-time. But are those descriptions and the associated predictions in agreement with observations and measurements on the physical system?

In some cases, it is. In many others, however, this is not the case and significant deviations appear, limiting predictive capabilities. An epistemic \emph{ignorance} seems to persist in our conceptualization and subsequent description of the physical reality. Models are models, but sometimes reality seems to contain something else, the so-called \emph{ignorance} (the part of the reality that our models ignore), and that, as discussed below, constitutes an immense source of opportunities.\linebreak As Stephen Hawking once said: \emph{The greatest obstacle is not ignorance---it is the illusion of knowledge}.


\subsection{Towards the art of decision making in complex systems of systems}

Advanced procedures are urgently needed, not only for performing faster, but for performing well. The \emph{art of decision making} irrupted almost everywhere, facilitated by our nowadays global interconnected word, where the Internet of Things, IoT, is fully deployed. This is at the heart of the fourth industrial revolution, with \emph{data} as one of its protagonists, but not the only one. Smart cities and nations, with the human in the loop, are attracting interest of scientists, engineers, politicians and decision makers. This fourth industrial revolution has its counterpart in science: we speak of the fourth paradigm of science, where large scientific infrastructures produce huge volumes of data that are readily incorporated into the scientific pipeline.

Complex systems of systems, containing many interconnected components, are concerned by uncertainty and variability that limit the performances of physics-based models. On the other hand, data, even when available, rarely cover the system extension (and scales) in space and time, and then, diagnosis and prognosis based exclusively on data encounter their own limitations.\linebreak The hybrid paradigm, combining both, the existing knowledge and the collected data representing the considered reality, seems a valuable route for conciliating accuracy and efficiency.

As commented, data becomes a major protagonist. However, the word \emph{data}, simple at a first view, encompass a rich essence. First, there is a vast typology: list, images, time series, fields, graphs\dots\ Second, extracting the intrinsic data dimensionality is also a tricky issue, needing for efficient linear and nonlinear dimensionality reduction procedures~\cite{LEE07}, also known as manifold learning techniques, able to discriminate useless and useful. In recent times, some authors speak even of the fifth paradigm of science, in which data are produced by simulation in order to feed scientific discovery and theories.

But, how to realize that a model is not accurate enough? The simplest way: by making measurements (collecting data) and comparing these measurements with the predictions based on the existing knowledge, provided by the state-of-the-art models, that can be very efficiently solved as discussed above.

When these measurements, even adequately assimilated into the models (model calibration), does not result in a calibrated model able to describe with high fidelity the observed reality, something seems missing in the reality representation. This should be modeled to improve designs and decisions.

One possibility is offered by Artificial Intelligence, AI. Machine Learning, a major protagonist of AI, is able to create predictive models from available or collected data, with an additional added value, the fact of providing that prediction in almost real-time.

However, creating models from scratch, simply because the existing models based on physics were not accurate enough, is not the best choice. Creating a model based on data from scratch needs a lot of data, and in engineering and technology, data is synonym of cost. Sometimes data collection also implies to consider an ethical dimension, to fulfill existing regulations, or must address technical difficulties, related for example to the sensors placement, data transfer\dots

In our works we advocated by the alliance between both (i) the analogical world of knowledge and physics-based models, and (ii) the more recent digital world of data, manipulated by more and more powerful (accurate, frugal and explainable---certifiable---) techniques of Artificial\linebreak Intelligence.

This new alliance is materialized in the so-called Digital or Hybrid Twins, in which physics-based models are enriched to decrease their intrinsic ignorance, in a pragmatic way, from data representing the deviation between predictions and measurements. The hybrid paradigm not only allows reducing the amount of needed data (now it is only expected to describe the gap between the reality and the physics-based predictions), but also the ability of explaining the part of the model based on the existing physics or knowledge, and then facilitating the design or decision certification.

In order to reduce the amount of data to model the observable phenomena (when models do not exist or are too inaccurate) or for enriching the existing physics-based models within the hybrid paradigm previous introduced, physics-aware (also known as physics-informed) Artificial Intelligence seems to be the most appealing route.

The so-called Physics Informed Neural Networks, PINN, \cite{RAI17,RAI17b} consider the approximation of the unknown function $u(x,t)$ as a regression problem defined on an adapted neural network. Then, as soon as the physics is assumed fully known and adequately described by a partial differential equation, the derivatives involved in the differential operator can be applied on the NN and the residual nullity is enforced from the NN loss function.

Sometimes, not everything concerning the physics is fully known. If we consider a hyper-elastic material, the best option consists in learning the free energy by constructing a regression linked to the state variables, in such a way that its derivatives leads to the constitutive equation. Then, the free energy is learned to be consistent with the collected data on the structural component, under the equilibrium constraints.

Similar procedures apply in the so-called Structure Preserving NN (also known as Thermodynamic Informed NN) where the free energy and the dissipation potential are computed in such a way that energy balance and entropy production are ensured~\cite{GON18b,GON19,HER20,HER20b,MOY19,MOY20}.

In those works, inspired from the GENERIC framework~\cite{GRM}, the state $\bfZ$ evolution, $\dot{\bfZ}$, reads
\begin{equation}
\dot{\bfZ} = \bbL \nabla_Z \clH + \bbM \nabla_Z \clS,
\end{equation}
where the first term of the right-hand member represents the reversible evolution (Hamiltonian contribution) whereas the second one represents the dissipative contribution, with $\clH$ and $\clS$ the energy and the entropy respectively.

Learning matrices $\bbL$ (known to be skew-symmetric) and $\bbM$, symmetric and positive semi-definite, as well as both potentials $\clH$ and $\clS$ (subjected to some constraints: the Jacobi identities as well as the consistency conditions $\bbL \nabla_Z \clS = \bfzo$ and $\bfM \nabla_Z \clH = \bfzo$) is performed from the existing data concerning the state time evolution $\bfZ(t_i)$, $i=1,2,\,\ldots$ The learnt model has very interesting properties, as are the ones related to energy conservation and positive dissipation, giving rise to stable and accurate time integrators.

The main issue found when learning such thermodynamic-aware models is not the regression implementation, but the choice of the variables in the state vector, an issue discussed in~\cite{IBA20}.\linebreak In very small systems the state variables are easily identified, however in large (continuous) systems such a choice is far of being trivial. In that case, many options exist. One among them consists of performing a dimensionality reduction. The use of most of manifold learning dimensionality reduction has as main drawback the unavailability of performing the inverse mapping for coming back from the reduced space to the departure one.

In the general nonlinear case, an appealing alternative consists in the use of NN-based (sparse) autoencoders~\cite{HIN94}, where encoding and decoding is learned, while models operate (and are learnt) in a transparent way in the internal layer of reduced dimension where active reduced coordinates (the so-called latent variables or latent space) act. Autoencoders allow to learn the best model representation, that in many cases, other than reducing the dimension of the model, makes possible reducing the model complexity (and therefore its nonlinearity), enabling the use of simpler and cheaper regression techniques operating at the level of the latent space, where sometimes linear regressions suffice.

In the same way that data allows enriching the existing knowledge (i.e., models based on physics), the existing knowledge and physics-based models allow to replace the big data by a smart data paradigm, by simply responding three main questions: (i) what data to collect,\linebreak (ii) when and (iii) where. For instance, for having an idea on the temperature in Paris we can use two thermometers, one placed inside our apartment, the other in the balcony, and take the temperatures at mid-night, early in the morning and at noon. Two thermometers and 3 measures, to have a reasonable approximation of the temperature evolution each day in Paris. However, for taking that decision about the measurements we used our knowledge. We thus avoid using one million thermometers distributed all along Paris and collecting the temperature at each one, each millisecond. This is expensive and useless. This example illustrates the difference between big-data and smart-data. Smarter seems definitively better!

Reconciliation and alliance between physics-based and data-driven models, leads to the so-called Hybrid Twin, HT~\cite{CHI20}. It constitutes a digital replica of a material, process, structure, component, systems or systems of systems, able to replace or substitute the real system for anticipating future responses, to access to its intimate state, while retaining the following features, qualities and functionalities: (i) accuracy guaranteed by the hybrid approach; (ii) frugal, being based on the smart-data paradigm; (iii) holistic throughout all the involved physics and description scales; (iv) providing real-time responses by invoking advanced model order reduction techniques; (v) explicable and certifiable; (vi) adaptable; (vii) reliable; (viii) resilient; (ix) informed -IoT; (x) systemic to address the system within its environment; and (xi) usable.

The HT of a component, of a system, or of a system of systems can and must interact with the human being. The last continues to be, even today, the main contributor in many domains where imagination, intuition, complex abstraction and interpretability are needed, as well as qualitative reasoning, including the emotional dimension.

It is well known that people remember very differently what they learnt after two weeks: 10\% to 20\% of what they heard or read, 90\% of what they did (acted upon) in a real or virtual experience. Thus, we must experience physics, we must be protagonists and not only observers.

Data produced by the HT must be contextualized to be profitable, interpretable and exploitable by humans. New human-centric facilities irrupted and were combined with the hybrid paradigm. Among them the immersive virtual reality, VR, augmented reality, AR, \cite{BAD19} in multi-physics evolving settings, and hybrid reality, HR, in which the human interacts with the virtual world (a digital replica of the reality) in the form of holograms that can be experienced visually, from the touch, and even by integrating other sensorial sensations: smell or taste. Recording emotions could lead to better adapt a product to each specific final user, e.g. adapt the car to the driver, adapt the text-book to the reader, adapt a cosmetic cream to the customer\dots

This human-centric HT constitutes the major protagonist of an incipient Augmented Intelligence, that does not aim at replacing the human, but enriching him to face the ultimate art, the \emph{art of decision-making}. The one that is not based on a simple reptilian pattern recognition: I am hungry, I eat\dots necessary, but primitive. Here we are speaking about the art of decision making, that has another more elaborated dimension making use of the two sides of the brain, the rational and the creative.

\section{Final remarks}

We recently experienced the impact of COVID-19 pandemic, everywhere, and adapted to it in an active and constructive manner. The world is definitively changing, evolving, and things will be definitively different even when pandemic will definitively disappear. During this pandemic, many people worked at home, some activities seems being nowadays resilient enough, but other remain much less, as for example production machines and chains.

Without any doubt, a resilient, human centric world, allying real and virtual, cold digital data and warmer emotions, all them harmoniously entangled, will constitute the new revolution. It will be not only an industrial one. This time, more than ever, it will have a huge human and societal dimension.

Engineering is succeeding in this new framework, sculpting a new world, with major protagonists: HT (combining real-time physics and real-time physics-informed AI), blockchain, quantum computing, multi-sensorial and emotional human-centric\dots empowering human creativity and imagination, enabling moving further, better and faster\dots It is not a far dream, it is becoming a reality, that results from an intimate combination of the three just revisited arts of engineering: the art of modeling, the art of simulation and the art of decision making.

\section*{Conflicts of interest}

The authors declare no competing financial interest.

\section*{Dedication}

The manuscript was written through contributions of all authors. All authors have given approval to the final version of the manuscript.


\section*{Acknowledgments}

Authors are supported by their respective ESI Group research contracts, whose support are gratefully acknowledged.

This research is part of the programme DesCartes and is supported by the National Research Foundation, Prime Minister Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme.





\bibliographystyle{crunsrt}
\bibliography{crmeca20220811}
\end{document}
