CC-BY 4.0
1. Introduction
Analysis of stone composition allows establishing dedicated treatments that can eliminate stones with a reduced probability of relapses. To this end, non-enhanced computed tomography (CT) is currently the first step of an etiological approach by a urologist. Zhang et al. showed that dual-source dual-energy CT can predict common stone composition for both pure and mixed stones [1]. However, the limitations of such an approach mainly lie in (1) a substantial radiation exposure, (2) an accuracy of the results directly dependent on the extraction quality of the piece of stone, (3) the fact that this approach does not give access to the morphological aspect of urinary stones, which may be the key for an accurate etiological diagnosis [2, 3].
Alternatively, a morpho-constitutional examination of urinary stones becomes a method of choice [2, 3, 4, 5]. It is now well established that such examinations play an essential role in the etiological diagnosis, as explained in the international morpho-constitutional classification of urinary stones [5]. This analysis can be performed during a post-operative session by a biologist and consists of collecting morpho-constitutional information based on both microscopic morphology, using binocular stereomicroscope, and infrared spectroscopy recognition (FTIR). Alternatively, Endoscopic Stone Recognition (ESR), which is conducted by a urologist, can also conveniently provide essential morphological observations of stones in-situ during pre- and intra-operative sessions [6, 7].
In this article, the principles of ESR are reviewed; then, various LASER technologies designed for kidney stone fragmentation are presented. ESR, which can be supported by Artificial Intelligence (AI), may find a widespread use in a clinical practice. The requirements for such a generalized use are described.
2. Basic principles of ESR
ESR-based analysis is, to a certain extent, similar to that applied by a biologist in a dedicated laboratory: detailed knowledge on the classification of the different types of stone surface, section and nuclei is a prerequisite; this expertise encompasses the use of well-established stone descriptions obtained with microscopy images ex-vivo [2] and endoscopic images in-situ [6, 7] classified with respect to the nomenclature established by Corrales et al. [5]. A relatively fast learning curve was reported when urologists following the ESR training encountered the most frequent stone morphologies, such as: stones with a single crystalline component [8], calcium oxalate monohydrate (COM) (also referred to as type I, subdivided in subtypes Ia, Ib, Ic, Id or Ie—subscripts in the Latin alphabet differentiate morphological subtypes, each of them being associated with a specific etiology), calcium oxalate dihydrate (COD, morphological type II subdivided as IIa and IIb subtypes) and uric acid (UA, morphological type III subdivided as IIIa and IIIb subtypes) [9]. However, a much steeper learning curve was needed when calcium phosphates or mixed stone morphologies are involved [7, 8]; this issue may limit ESR translation to daily clinical practice [9].
3. LASER devices for kidney stone fragmentation
Since the beginning of the 2000’s, the LASER Holmium-Yag technology is accessible to any urologist for urinary-stone treatments [10]. Thanks to recent technological innovations, LASER spraying of all types of stones is now feasible and two main interventional strategies were implemented:
- “Pop corning” mode: using moderate LASER frequencies ranging from 10 to 15 Hz, the urologist can either (i) split a stone in two parts or (ii) fragment it into small pieces. Stone fragments of a size of 250–500 μm can be collected, thus allowing for post-operative examinations of the microscopic morphology [10, 11, 12, 13, 14, 15].
- “Dusting” mode: using higher LASER frequencies ranging from 20 to 320 Hz, the operator can conveniently spray the stone [13]. Since 2017, this is achievable using Holmium-Yag devices by means of the “Moses Effect” [12] and, more recently, using Thulium Fiber LASER (TFL) devices. There, micro fragments below 250 μm are created, 20 μm being generally the wanted size [14].
Both the Holmium-Yag using “Moses Effect” and TFL devices thus appear effective technological revolutions able to quickly operate in interventional theaters [12, 13, 16]. This technological breakthrough has a clearly obvious therapeutic objective: to eliminate the presence of any stone fragments at the end of the interventional procedure. In addition, therapeutic approaches have been rated for decades by both urologists and manufacturers according to their capability to eliminate stone fragments.
However, LASER fragmentation of stones, when achieving “dusting” modes [11, 12, 13, 14, 15], has two consequences that hamper the etiological approach in lithiasis:
- High frequency LASER stone fragmentation irreversibly hampers the morphological analysis of the targeted stones [14, 15]. FTIR dust examinations of the stone powder, with a fragment size below 250 μm, may not provide sufficient information for the lithogenic stage.
- IR spectra can be modified when the LASER stone fragmentation is achieved in dusting mode with or without high-frequency TFL or “Moses Effect” [14, 15], which hampers FTIR dust examinations. Keller et al. observed IR-conversion of several crystalline elements in-vitro during “Moses Effect”, in particular IR changes from COD towards COM, IR changes from carbapatite towards amorphous carbonated calcium phosphate, MAP in newberyite and BR in monetite as well as IR changes from brushite towards carbapatite (see Figure 1 and Table 1) [14, 15].
Constitutional analysis by Fourier transform infrared spectroscopy. Comparison between residual fragments and stone dust revealed spectra changes for several constitutional stone types. (A) Conversion from COD towards COM in dust after Moses lithotripsy. (B) The conversion from COD towards COM in dust was more pronounced after Moses lithotripsy compared to conventional Holmium lithotripsy. (C) Changes towards an amorphous phase in CA dust after Moses lithotripsy with flattening and displacement of the 1035 cm−1 band. (D) A spectral change of CA dust was only found after Moses lithotripsy, but not after conventional Holmium lithotripsy. (E) Changes towards a differing and amorphous crystalline phase in MAP dust after Moses lithotripsy. (F) A spectral change of MAP dust was only found after Moses lithotripsy, but not after conventional Holmium lithotripsy. (G) Changes from BR towards CA after Moses lithotripsy. (H) Spectral changes were found in CA dust both after Moses and conventional Holmium lithotripsy. Figure and caption reproduced with permission from Keller et al. [15].
Observations from morpho-constitutional analysis of stone dust after TFL lithotripsy
| Stone type | Preserved characteristics from initital stones? | Morphological analysis (SEM) | Constitutional analysis (FTIR) |
|---|---|---|---|
| COM | Yes | Preserved lamellar organization | Preserved major bands at 1617, 1315 and 781 cm−1 |
| COD | Partially | Almost complete loss of bipyramidal organization | Displacement of major bands from 1645 and 1324 cm−1 to 1624 and 1318 cm−1, respectively, as well as appearance of a prominent band at 781 cm−1 indicating a conversion from COD to COM |
| UA | No | Altered organization with appearance of a needle-like organization | Major bands at 3603, 1613, 1531 and 1003 cm−1, indicating a conversion from UA to sodium hydrogen urate monohydrate |
| CA | Yes | Preserved organization with spherical particles within more smooth blocks | Preserved major bands at 1035, 604 and 563 cm−1 |
| STR | No | Loss of the encased monodirectional needle organization resulting in a pile of randomly configured needles that are partially attached to each other | Loss of the ammonium bands at 1470 and 1435 cm−1. Very large and displaced major band at 1014 cm−1 indicating a conversion from STR to newberyite |
| BR | Partially | Partially preserved longitudinal beguette-like organization | Displacement of major bands from 1136 and 1063 cm−1 to 1134 and 1066 cm−1, respectively, as well as the appearance of major bands at 1002 and 564 cm−1, as well as additional bands at 1406 and 892 cm−1, indicating a convrsion from BR to monetite |
| CYS | Partially | Preserved layered organization, with an almost complete loss of the hexagonal organization | Loss of the 1656 cm−1 band, unusually prominent band at 1622 cm−1 and large, prominent and displaced band at 1337 cm−1, indicating a profound change in crystalline organization |
Table and caption reproduced with permission from Keller et al. 2020 [14].
SEM: scanning electron microscopy, FTIR: fourier transform infrared spectroscopy, COM: calcuum oxalate monohydrate, COD: calcium oxalate dihydrate, UA: uric acid, CA: carbapatite, STR: struvite, BR: brushite, CYS: cystine.
The so-called “stone free rate” therapeutic objective [10, 11, 12, 13, 14, 15, 17] is therefore now achievable. However, it must be underlined that the morphological type and related etiology are better indicators of stone recurrence than stone composition only. Hence, the absence of intra-operative examination (in other terms: no morphological data) may preclude the robustness of the etiological approach in lithiasis and relapse prevention [5].
Fortunately, it has been reported that visual examinations of stone can be obtained using ESR before and during LASER destruction [7, 8]. While recent (chronological) lithogenic events are observable on the stone surface, older events are located on a stone section and are accessible during LASER fragmentation. The stone nucleus, which is the oldest part, can thereby provide information about the initial lithogenic context. Recently, Estrade et al. demonstrated that these examinations can be readily supplemented by observations of the stone section and nucleus if the urologist carefully splits the stone in two parts using LASER [8].
The traceability of the morphology of the surface, the section and the nucleus of a urinary stone can be recorded, thus allowing a retrospective expertise, if necessary, by a biologist, an expert urologist or machine learning. Therefore, intra-operative ESR and post-operative infrared (FTIR) examinations of LASER-fragmented stones are crucial to preserve an etiological approach in lithiasis.
4. Towards automatic intra-operative ESR
Black et al. recently have shown that artificial intelligence (AI) applied to various types of microscopic images of stones ex-vivo is a promising tool for automatic ESR (AESR) [18]. While Serrat et al. fed texture and color features of stones into a random forest classifier [19], Black et al. obtained much improved scores using a deep Convolutional Neural Network (CNN) [18]. However, both approaches used ex-vivo stone fragments placed into a controlled environment. Hence, images were not disturbed by motion blur, specular reflections or scene illumination variations, as it is encountered in common practice during an intra-operative endoscopic imaging session. Martinez et al. showed the potential of AESR approaches using in-situ images of pure stones acquired in a clinical setting with ureteroscopes [20]. More recently, from intra-operative endoscopic images, first AESR clinical results evidencing morphological elements that constituted both pure and mixed urinary stones in-situ have been reported [21]. As an illustration, Table 2 reports typical preliminary AESR results obtained with six morphology classes (four pure stones: Ia/COM, Ia active, IIb/COD and IIIb/UA, and two mixed stones divided into two morphologies: Ia/COM + IIb/COD and Ia/COM + IIIb/UA) from images obtained before LASER fragmentation. In this study, a urologist (VE, 20 years of experience) prospectively examined intra-operative endoscopic digital images of stones acquired between January 2018 and November 2020 (single center) using a flexible digital ureterorenoscope (Olympus URF-V CCD sensor). Images from 436 urinary stones were collected (pure stones: Ia/COM = 221, Ia active = 32, IIb/COD = 66, IIIb/UA = 31; mixed stones: Ia/COM + IIb/COD = 74, Ia/COM + IIIb/UA = 12). A deep convolutional neural network (CNN), ResNet-152-V2 [22], was trained to predict stone types from collected images. For the statistical analysis, stones were randomly divided into training (70%) and testing (30%) sets (stratified split/10 trials) [23]. In average, the accuracy was higher than 86% for both pure and mixed stones with the experimental setup used [21].
Diagnostic performance of a CNN classifier (ResNet-152-V2) to predict the surface of pure stones COM, COD and AU with respective morphologies (Ia, Ia active, IIb and IIIb) and mixed stones COM + COD and COM + AU with morphologies (Ia + IIb and Ia + IIIb) from collected images
| Kidney type | Accuracy (%) | AUROC | Sensitivity (%) | Specificity (%) | PPV (%) | NPV (%) | FPR (%) | FNR (%) |
|---|---|---|---|---|---|---|---|---|
| Ia | 86 ± 2 | 0.86 ± 0.02 | 89 ± 3 | 84 ± 5 | 85 ± 4 | 88 ± 3 | 16 ± 5 | 11 ± 3 |
| Ia active | 93 ± 1 | 0.75 ± 0.06 | 53 ± 13 | 97 ± 1 | 64 ± 8 | 96 ± 1 | 3 ± 1 | 47 ± 13 |
| IIb | 93 ± 1 | 0.86 ± 0.04 | 77 ± 9 | 96 ± 1 | 74 ± 6 | 96 ± 1 | 4 ± 1 | 23 ± 9 |
| IIIb | 98 ± 1 | 0.95 ± 0.04 | 92 ± 7 | 98 ± 1 | 81 ± 11 | 99 ± 1 | 2 ± 1 | 8 ± 7 |
| Ia + IIb | 89 ± 3 | 0.79 ± 0.04 | 63 ± 8 | 94 ± 3 | 70 ± 9 | 92 ± 1 | 6 ± 3 | 37 ± 8 |
| Ia + IIIb | 98 ± 1 | 0.66 ± 0.16 | 33 ± 3 | 100 ± 1 | 52 ± 47 | 98 ± 1 | 0 ± 1 | 67 ± 31 |
AUROC: area under the ROC (Receiver Operating Characteristic) curve; PPV: positive predictive value; NPV: negative predictive value; FPR: false predictive rate; FNR: false negative rate.
5. Requirements of intra-operative AESR in the etiological approach
ESR can be supported by a computer-assisted approach, which may deliver reproducible results and minimizes operator dependency. In our opinion, it is imperative to build-up a digital endoscopic image database annotated according to the criteria published in 2020 [7]. However, several issues still need to be addressed for a wide AESR integration into a clinical routine.
First, it is difficult to obtain a gold standard database of annotated stone images, which is a necessary prerequisite for the training of an IA network [24, 25]. On the one hand, infrared spectroscopy can be used as a gold standard, but its accuracy relies on the equipment used, the reference spectra and the qualification of the operator. On the other hand, one can use ESR as a gold standard, but any subjectivity or potential selection bias of the endo-urologist may be present in the annotated dataset. We believe that a concordance study between endoscopic and microscopy examinations, as described in [7], may provide a database of confirmed annotated ESR image of stones corresponding to specific aetiologies or lithogenic mechanisms.
Second, AI generally relies on a large amount of labeled training images to provide accurate stone predictions [22, 23]; however, a sufficient amount of images is hardly achievable in the context of rare diseases. This is problematic on the clinical level because such pathologies are precisely those in need for a rapid and reliable management. Therefore, AI algorithms must be improved in order to further reduce the amount of training data, thereby allowing AESR of rare stone diseases with a satisfactory reliability. Several paths of investigation may be taken to reach this goal; for instance, training databases may be conveniently supplemented by high-quality ex-vivo images of complete and fractioned stones [7, 21]. In addition, an interesting aspect to explore is the potential benefit of AI classification algorithms upon clustering of different stone types, but also the impact of possible overlaps between these clusters. Ultimately, the development of specific image filtering algorithms and the automatic segmentation of relevant image regions of interest should allow reducing the amount of training samples needed, and may give access to rather reliable AESR workflows for specific stone types.
Representative automatic ESR results obtained before LASER fragmentation (surface image). Examples of both correctly (left panel) and misclassified images (right panel; type reported on the far left is not recognized by the network) are shown. In-situ surface images (left image of each panel) are reported for each stone composition. Ia/COM, IIb/COD and IIIb/UA pure morphologies are reported in the first three rows. For each mixed stone (last two rows), a mixture of the corresponding pure morphologies is visible. Activation maps (right image of each panel) show areas where the network concentrates its attention. Figure and caption reproduced with permission from Estrade et al. [21].
Representative automatic ESR results obtained after LASER fragmentation (section images). Examples of both correctly (left panel) and misclassified images (right panel: type reported on the far left is not recognized by the network) are shown. In-situ section images (left image of each panel) are reported for each stone composition. Ia/COM, IIb/COD and IIIb/UA pure morphologies are reported in the first three rows. For each mixed stone (last two rows), a mixture of the corresponding pure morphologies is visible. Activation maps (right image of each panel) show areas where the network concentrates its attention. Figure and caption reproduced with permission from Estrade et al. [21].
Third, it must be underlined that a general problem with the deep neural network approach lies in the difficulty to interpret and understand what the model has learned. Regularly called “black box” algorithms, deep neural networks are often questioned because of their lack of explicitness; computer-assisted predictions of stone morphologies may thus be hardly explainable. However, a general interest in AI techniques designed to address such a key point is growing in order to devise robust validation procedures. In particular, recent AI studies underlined the benefit of generating maps to properly understand where the deep learning algorithm of interest was “looking” in the endoscopic image to make its decision [26]. In that regard, recent efforts must be reported in the identification of the learned patterns by AI models. During endoscopy, it is essential to highlight the most characteristic locations inferred by the network in the endoscopic image of interest. Figures 2 and 3, reproduced from [21], illustrate the usefulness of providing deep CNN-algorithm based “attention” maps to explain predictions computed by deep CNNs. In these figures, “attention” maps are overlaid onto the endoscopic image in order to visually assess whether CNN-derived hallmarks corroborate with clinically relevant stone features. In the presented results, hot (red) spots were found on relevant urological features in 98% of the correctly classified (“True positive”) images (see images in left columns in Figures 2 and 3). Conversely, hot spots were observable outside the stone in 33% and 25% of misclassified surface and section images, respectively (see for example Figures 2d, 2j or 3b). Such a tool may be a great asset for urologists to intra-operatively understand the predictions made by the AI model.
Nevertheless, it must be stressed that the infrared (SPIR) examination of stone powder by a dedicated laboratory remains essential to perform in order the morpho-constitutional stone analysis to provide additional and complementary information [2, 3, 4, 5, 6, 7]. In our opinion, the combination of intra-operative ESR and post-operative infrared (FTIR) examinations of laser-fragmented stones may further improve the etiological approach by urologists in lithiasis [21]. However, it must be reported that, after LASER stone fragmentation, only a few stone fragments may be available for a subsequent IR stone analysis. It may thus be difficult to attribute the collected fragments to the different areas in the stone that can be observed within an endoscopic session. Collecting non- or partially-fragmented stones may be preferable since outer stone surface and different layers of inner stone (after cutting or crushing) can be recorded. In such a case, corresponding portion of the stone may therefore be easily sampled and sent for IR analysis.
6. Conclusions
LASER spraying of all types of urinary stones became common practice. Similarly, morpho-constitutional examination of urinary stones becomes a method of choice for the effective management of patients suffering from kidney stones and regarding the prevention of stone recurrence [2, 3, 4, 5]. While urologists following the ESR training experience a relatively fast learning curve when they encounter the most frequent stone morphologies, they need more practice when calcium phosphate or mixed stones morphologies are involved. Trained on confirmed high-quality endoscopic images annotated according to the criteria published by Estrade and colleagues [7], AI proves to be a good candidate for automatic ESR of the morphological elements composing urinary stones. Recent works demonstrated that AI is a promising tool to identify both pure and mixed kidney stone compositions from endoscopic images acquired intra-operatively [20, 21]. ESR performed before complete LASER-induced spraying may thereby be beneficial to maintain an etiological approach in lithiasis. The combination of automatic intra-operative ESR and post-operative infrared examinations of laser-fragmented stones may improve the etiological approach in lithiasis [7, 21].
In our opinion, it is imperative to extend ESR learning in the curriculum of urologists and further improve current AI developments; hence, we believe that such objectives must be highly prioritized by urological and scientific communities.
Abbreviations
| AESR | Automated Endoscopic Stone Recognition |
| AI | Artificial Intelligence |
| AUROC | Area Under the ROC curve |
| BR | Brushite |
| CA | Carbapatite |
| CNN | Convolutional Neural Network |
| COD | Calcium Oxalate Dihydrate |
| COM | Calcium Oxalate Monohydrate |
| CT | Computed Tomography |
| CYS | Cystine |
| DSDECT | Dual-Source Dual-Energy Computed Tomography |
| ESR | Endoscopic Stone Recognition |
| FNR | False Negative Rate |
| FPR | False Predictive Rate |
| FTIR | Fourier Transform InfraRed spectroscopy |
| Grad-CAM | Gradient-weighted Class Activation Mapping |
| Ho-Yag | Holmium-Yag LASER |
| IR | Infra Red |
| LASER | Light Amplification by Stimulated Emission of Radiation |
| MAP | Magnesium Ammonium Phosphate (struvite) |
| NPV | Negative Predictive Value |
| PPV | Positive Predictive value |
| ROC | Receiver Operating Characteristic |
| STR | Struvite |
| TFL | Thulium Fiber LASER |
| UA | Uric Acid |
| VE | Vincent Estrade |
Conflicts of interest
Authors have no conflict of interest to declare.
Acknowledgments
Experiments presented in this paper were carried out using the PlaFRIM experimental test-bed and supported by Inria, CNRS (LABRI and IMB), Université de Bordeaux, Bordeaux INP and Conseil Régional d’Aquitaine (see https://www.plafrim.fr/). The authors gratefully acknowledge the support of NVIDIA Corporation (Santa Clara, CA, USA) after their donation of a TITAN X GPU used in this research.