Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Geospatial Artificial Intelligence: Potentials of Machine Learning for 3D Point Clouds and Geospatial Digital Twins

Geospatial Artificial Intelligence: Potentials of Machine Learning for 3D Point Clouds and... Artificial intelligence (AI) is changing fundamentally the way how IT solutions are implemented and operated across all application domains, including the geospatial domain. This contribution outlines AI-based techniques for 3D point clouds and geospatial digital twins as generic components of geospatial AI. First, we briefly reflect on the term “AI” and outline technol- ogy developments needed to apply AI to IT solutions, seen from a software engineering perspective. Next, we characterize 3D point clouds as key category of geodata and their role for creating the basis for geospatial digital twins; we explain the feasibility of machine learning (ML) and deep learning (DL) approaches for 3D point clouds. In particular, we argue that 3D point clouds can be seen as a corpus with similar properties as natural language corpora and formulate a “Naturalness Hypothesis” for 3D point clouds. In the main part, we introduce a workflow for interpreting 3D point clouds based on ML/ DL approaches that derive domain-specific and application-specific semantics for 3D point clouds without having to create explicit spatial 3D models or explicit rule sets. Finally, examples are shown how ML/DL enables us to efficiently build and maintain base data for geospatial digital twins such as virtual 3D city models, indoor models, or building information models. Keywords Geospatial artificial intelligence · Machine learning · Deep learning · 3D point clouds · Geospatial digital twins · 3D city models Zusammenfassung Georäumliche Künstliche Intelligenz: Potentiale des Maschinellen Lernens für 3D-Punktwolken und georäumliche digitale Zwillinge. Künstliche Intelligenz (KI) verändert grundlegend die Art und Weise, wie IT-Lösungen in allen Anwendungs- bereichen, einschließlich dem Geoinformationsbereich, implementiert und betrieben werden. In diesem Beitrag stellen wir KI-basierte Techniken für 3D-Punktwolken als einen Baustein der georäumlichen KI vor. Zunächst werden kurz der Begriff ,,KI” und die technologischen Entwicklungen skizziert, die für die Anwendung von KI auf IT-Lösungen aus der Sicht der Softwaretechnik erforderlich sind. Als nächstes charakterisieren wir 3D-Punktwolken als Schlüsselkategorie von Geodaten und ihre Rolle für den Aufbau von räumlichen digitalen Zwillingen; wir erläutern die Machbarkeit der Ansätze für Maschinelles Lernen (ML) und Deep Learning (DL) in Bezug auf 3D-Punktwolken. Insbesondere argumentieren wir, dass 3D-Punktwolken als Korpus mit ähnlichen Eigenschaften wie natürlichsprachliche Korpusse gesehen werden können und formulieren eine ,,Natürlichkeitshypothese” für 3D-Punktwolken. Im Hauptteil stellen wir einen Worko fl w zur Interpretation von 3D-Punktwolken auf der Grundlage von ML/DL-Ansätzen vor, die eine domänenspezifische und anwendungsspezifische Semantik für 3D-Punktwolken ableiten, ohne explizite räumliche 3D-Modelle oder explizite Regelsätze erstellen zu müssen. Abschließend wird an Beispielen gezeigt, wie ML/DL es ermöglichen, Basisdaten für räumliche digitale Zwillinge, wie z.B. für virtuelle 3D-Stadtmodelle, Innenraummodelle oder Gebäudeinformationsmodelle, effizient aufzubauen und zu pflegen. 1 Introduction * Jürgen Döllner Artificial intelligence (AI) is changing the way IT solutions doellner@hpi.uni-potsdam.de are designed, built and operated. AI is not limited to spe- Hasso Plattner Institute, Digital Engineering Faculty, cific application areas—it is finding its way into almost all University of Potsdam, Prof.-Dr.-Helmert-Str. 2-3, industries and domains. As a collection of general-purpose 14482 Potsdam, Germany Vol.:(0123456789) 1 3 16 PFG (2020) 88:15–24 technologies, AI has significant impact because it “can trans- generalize and learn from past experiences, and to be intel- form opportunities not only for economic growth, but also ligent using learning, thinking, problem solving, perception for corporate profitability” (Purdy and Daugherty 2017). and language. For geospatial domains, fundamental questions include There is generally no sharp border between AI and non- how AI can be specifically applied to or has to be specifi- AI technology. Consider, for example, an autopilot steering cally created for spatial data. Janowicz et al. (2020) give an an aircraft: in its beginning, it was perceived as AI, while overview of spatially explicit AI, which “utilizes advance- today it has become a common operating technical compo- ments in techniques and data cultures to support the crea- nent. That is to say, “when AI reaches mainstream usage it tion of more intelligent geographic information as well as is frequently no longer considered as such” (Haenlein and methods, systems, and services for a variety of downstream Kaplan 2019). In that sense, the term “AI” is mostly used to tasks”. This arising scientific discipline, called geospatial label such technology that goes beyond current technology artificial intelligence (GeoAI), which “combines innova - boundaries. In this contribution, we refer to AI in a geospa- tions in spatial science, artificial intelligence methods in tial context and focus on potentials of ML and DL for 3D machine learning (e.g. deep learning), data mining, and point clouds. high-performance computing to extract knowledge from spatial big data” (Vopham et al. 2018), will in particular 1.2 AI‑Based IT improve existing and create new technologies for geospatial information systems (GIS). From a software engineering perspective, several technolo- The relevance of AI for geospatial domains has been real- gies and disciplines (Fig. 1) are required for the implementa- ized many years ago already regarding, for example, expert tion of AI-based IT solutions: systems and knowledge-based systems (Openshaw and Openshaw 1997), geographically problem solving (Smith • Big data management Many AI-based techniques require 1984), or analysing social sensing data (Wang et al. 2018a). big data to be applied effectively. For example, ML and In this paper, we concentrate on AI-based approaches for DL need training data, which is typically distilled from a specific category of 3D geodata, 3D point clouds, which big data. Big data generally are characterized by a num- are fundamental in photogrammetry, remote sensing and ber of key traits (Kitchin and McArdle 2016), including: computer vision (Weinmann et al. 2015) and have manifold – large data amounts (volume), applications for building geospatial digital twins. – rapid data capturing or generation (velocity), – different data types and structures (variety), 1.1 The Term “Artificial Intelligence” – manifold relations among data sets (complexity), – high inherent data uncertainty (veracity). The notion “AI” implies a number of conceptual difficul- ties, such as the definition of “natural”, “human” or “gen- Big data have turned out to be a key driver for digital eral-purpose” intelligence. Put simply, the “most common transformation processes as summarized in the famous misconception about artificial intelligence begins with the statement “Data is the new oil. Data is just like crude. common misconception about natural intelligence. This mis- conception is that intelligence is a single dimension” (Kelly 2017). In the general public, AI often triggers associations and expectations such as simulating or overcoming human intelligence. If AI is pragmatically seen as technological progress, then “AI is going to amplify human intelligence not replace it, the same way any tool amplifies our abili - ties” (Lecun 2017). One of the AI applications that exem- plified these controversies was ELIZA. This famous first chatbot, built by Josef Weizenbaum in 1966 (Weizenbaum 1966), was a speech-based simulation of a psychologist’s interaction with a patient, which “demonstrated the kind of risk potential what was enclosed within such technological developments” (Palatini 2014). This topic is discussed fur- ther by Copeland (1993), who analyses which challenges and obstacles AI needs be to solved before “thinking” machines could be constructed. As he states, the key to AI is the abil- Fig. 1 Disciplines and technologies used to implement AI-based IT ity of computers to think rationally, to discover meaning, to 1 3 PFG (2020) 88:15–24 17 It’s valuable, but if unrefined it cannot really be used” clustering, auto-encoding, and others” (Hatcher and Yu (Humby 2006), which also is subject to a controversy 2018). As a specific form of representation learning, (Marr 2018). which in turn is a specific form of ML, DL is based on A broad range of approaches exist to capture, synthe- artificial neural networks (ANNs) such as convolutional size and simulate data about our geospatial reality. In that neural networks (CNNs) (Goodfellow et al. 2016). In the respect, geospatial data generally represent big data and end, DL represents “essentially a statistical technique for the “oil” for the geospatial digital economy. classifying patterns, based on sample data, using neural Analytics Analytics refers to analytical reasoning and networks with multiple layers” (Marcus 2018). aims at providing concepts, methods, techniques, and It builds representations expressed in terms of simpler tools to efficiently collect, organize, and analyse big data. representations, i.e. we can build complex concepts out Its objectives include to examine data, to draw conclu- of simpler concepts. “It has turned out to be very good at sions, to get insights, to acquire knowledge, and to sup- discovering intricate structures in high-dimensional data port decision making. In that respect, analytics “can be and is therefore applicable to many domains of science, viewed as a sub-process in the overall process of ’insight business and government” (LeCun et al. 2015), but is extraction’ from big data” (Gandomi and Haider 2015). also faced by a number of limitations such as being “data For all variants, such as descriptive, predictive, and hungry” and low support for transfer or hierarchical data prescriptive analytics, ML/DL approaches support “the (Marcus 2018). analysis and learning of massive amounts of unsuper- AI accelerators The growth of AI-based applications vised data, making it a valuable tool for big data analytics comes along with commodity, cost-efficient graphics where raw data are largely unlabeled and uncategorized” processing units (GPUs) that are evolving to become (Najafabadi et al. 2015). high-performance accelerators for data-parallel comput- Machine learning “Machine learning is programming ing. Further, there is a growing number of purpose-built computers to optimize a performance criterion using systems for DL, for example, tensor processing units example data or past experience”, which is required, in (TPUs), along with fully integrated hardware and soft- particular, “where we cannot write directly a computer ware [e.g. NVidia DGX/HGX; TensorFlow (Abadi et al. program to solve a given problem” (Alpaydin 2014). That 2016)]. They are known as AI accelerators, which differ is to say, IT solutions do not rely on explicit or procedural with respect to hardware costs, training performance, problem solving strategies but are based on processing computing power, and energy consumption (Wang et al. and analysing patterns and inference. For that reason, ML 2019) but foster and simplify AI-based software develop- offers a different programming paradigm for the imple - ment and operation (Reuther et al. 2019). mentation of geospatial IT solutions. ML techniques can be generally classified into super - To implement AI-based IT, components also include frame- vised ML, unsupervised ML, and reinforcement learn- works and systems for computer vision (e.g. image analysis, ing. Supervised ML, for example, acquires knowledge image understanding), speech and text analysis (e.g. text- by building mathematical models based on training data, to-speech, speech-to-text), knowledge representation and which are used to predict labels for input data. The under- discovery, reasoning systems, etc. lying models “may be predictive to make predictions in the future, or descriptive to gain knowledge from data, or both” (Alpaydin 2014):1.3 Feature Space “The promise and power of machine learning rest on its ability to generalize from examples and to handle ML/DL-based solutions describe the input data by means of noise” (Allamanis et al. 2018). For this purpose, ML feature vectors, i.e. by n-dimensional vectors whose numeri- offers a high degree of robustness regarding the input cal components describe selected aspects of a phenomenon data set. A fundamental risk, however, lies in “overtrain- to be observed, that is, the feature space equals a high- ing” the model. If overtrained, it will perform very well dimensional vector space. Dimensionality reduction tech- on the training data, but will poorly generalize to new niques allow us to transform “high-dimensional data into a data. The model becomes incapable of generalizing, i.e. meaningful representation of reduced dimensionality” (Van it is overfitting the training data. For that reason, “prop- Der Maaten et al. 2009) and, this way, to efficiently manage, erly controlling or regularizing the training is key to out- process, and visualize feature spaces. of-sample generalization” (Zhang et al. 2018). One of the key concepts required for geospatial ML and Deep learning As Hatcher and Yu point out, DL applies DL consists in finding adequate feature spaces for geospa- “multi-neuron, multi-layer neural networks to perform tial entities. A single point of the 3D point cloud does not learning tasks, including regression, classification, allow for constructing meaningful feature vectors. To do so, 1 3 18 PFG (2020) 88:15–24 the local neighbourhood of the point must be analysed by selecting, for example, the k nearest points or points within a radius r (Weinmann et al. 2015). Next, geometric, topo- logic or any other high-level features are computed from the neighbourhood region that constitute components of the fea- ture vector. “Typical recurring features include computing planarity, linearity, scatter, surface variance, vertical range, point colour, eigenentropy and omnivariance” (Griffiths and Boehm 2019). Fig. 2 Example of a high-density 3D point cloud of an indoor envi- ronment 2 3D Point Clouds and predicting behaviour and processes related to the cor- As universal 3D representations, 3D point clouds “can rep- responding physical entities. resent almost any type of physical object, site, landscape, geographic region, or infrastructure—at all scales and with any precision” as Richter (2018) states, who discusses algo- 2.2 Characteristics rithms and data structures for out-of-core processing, ana- lysing, and classifying of 3D point clouds. To acquire 3D A 3D point cloud represents a set of 3D points in a given point clouds, various technologies can be applied includ- coordinate system and can be characterized by: ing airborne or terrestrial laser scanning, mobile mapping, RGB-D cameras (Zollhöfer et al. 2018), image matching, or Uniform representation—unstructured, unordered set multi-beam echo sounding. of 3D points (e.g. in an Euclidian space); Discrete representation—discrete samples of shapes without restrictions regarding topology or geometry; 2.1 Geospatial Digital Twins Irregularity—expose irregular spatial distribution and varying spatial density; 3D point clouds are ubiquitous for geospatial applications Incompleteness—due to the discrete sampling, repre- such as for environmental monitoring, disaster management, sentations are incomplete by nature; urban planning, building information models, or self-driving Ambiguity—the semantics (e.g. surface type, object vehicles. More precisely, 3D point clouds are commonly type) of a single point generally cannot be determined used as base data for reconstructing 3D models (e.g. digital without considering its neighbourhood; terrain models, virtual 3D city models, building informa- • Per-point attributes—each point can be attributed by tion models), but can also be understood as point-based 3D additional per-point data such as colour or surface nor- models, for example, in the case when 3D point clouds are mal; dense (Fig. 2). Massiveness—depending on the density of the captur- In particular, they represent key components of geospa- ing technology, 3D point clouds may consist of mil- tial digital twins, that is, digital replica of spatial entities lions or billions of points. and phenomena. Digital twins, in general, are composed of three parts, “which are the physical entities in the phys- The key trait of 3D point clouds, a fundamental category ical world, the virtual models in the virtual world, and of 3D geospatial data, consists in the absence of any struc- the connected data that tie the two worlds” (Qi and Tao tural, hierarchical, or semantics-related information—a 3D 2018). While the connection between both can be handled point cloud is a simple, unordered set of 3D points. “The by sensors, the virtual models have to be derived from lack of topology and connectivity, however, is strength the physical counterparts. For example, 3D point clouds and weakness at the same time” (Gross and Pfister 2007). are used to derive 3D indoor models, which are essential There is, therefore, a strong demand for solutions that components for real-time building information models allow us to enrich 3D point clouds with information. together with sensor networks and IoT devices (Khajavi et  al. 2019); they also “represent a generic approach to 2.3 3D Point Cloud Time Series capture, model, analyse, and visualize digital twins used by operators in Industry 4.0 application scenarios” (Posada For a growing number of applications, 3D point clouds are et  al. 2018). In that regard, geospatial digital twins are captured and processed with high frequency. For example, if means for monitoring, visualizing, exploring, optimizing, 1 3 PFG (2020) 88:15–24 19 a surveillance system captures its target environment every consists in finding appropriate training and test data. A cru- second, it results in a stream of 3D point clouds . cial issue represents robustness of ML techniques as ML If 3D point clouds are captured or generated at different models “are vulnerable to adversarial examples formed by points in time having overlapping geospatial regions, these applying small carefully chosen perturbations to inputs that sets are inherently related. By 3D point cloud time series, cause unexpected classification errors” (Rozsa et al. 2016). we refer to a collection of 3D point clouds taken at different points in time for a common geospatial region. The collec- 2.5 Naturalness Hypothesis tion of 3D point clouds represents, in a sense, a 4D point cloud. The “Naturalness Hypothesis” (Allamanis et  al. 2018), 3D point cloud time series have a high degree of redun- which is investigated in natural language recognition and dancy, which needs to be exploited to achieve efficient man - software analytics, helps understanding further why ML agement, processing, compression and storage, for example, and DL approaches provide effective instruments for ana- separating static from dynamic structures. Redundancy can lysing and interpreting 3D point clouds. In general, one also be used to improve accuracy and robustness of 3D point key approach to ML and DL is to find out whether a given cloud interpretations and related predictions. problem domain corresponds to or has similar statistical properties as large natural language corpora (Jurafsky and 2.4 Feasibility of ML/DL‑Based Approaches Martin 2000). Here, ML/DL-based approaches have shown extraordinary success in natural language recognition, natu- Kanevski et al. investigate the general applicability of ML ral language translation, question answering, text mining, to geospatial data and conclude “the key feature of the ML text comprehension, etc. The most important finding in these models/algorithms is that they learn from data and can be areas is that objects (e.g. spoken or written texts) are less used in cases when the modelled phenomenon is not very diverse than they initially seem: most human expressions well described, which is the case in many applications (“utterances”) are much simpler, much more repetitive, and of geospatial data” (Kanevski et al. 2009). The complete much more predictable than the expressiveness of the lan- absence of structure, order and semantics as well as the guage body suggests, whereby “these utterances can be very inherent irregularity, incompleteness, and ambiguity explain usefully modelled using modern statistical methods” (Hindle why 3D point clouds are not very well described and difficult et al. 2012). This phenomenon can be understood with meas- candidates for procedural and algorithmic programming. ures of perplexity and cross-entropy (de Boer et al. 2005). Their characteristics, however, allow us to effectively apply 3D point clouds seen as a form of natural communication ML/DL to 3D point clouds: as well as geospatial environments are ultimately repetitive regardless of the endless variations they may exhibit. In a Big data: 3D point clouds are spatial big data that can be sense, 3D point clouds are just “spatial utterances” that can cost-efficiently generated for almost all types of spatial be modelled using statistical methods. 3D point clouds, environments—big data are a prerequisite for ML/DL- thereby, constitute 3D point cloud corpora to which ML based approaches; technology can be applied taking advantage of the statisti- Fuzziness: 3D point clouds show inherent fuzziness and cal distributional properties estimated over representative noise as they are sampling shapes by means of discrete point cloud corpora. representations—ML/DL are particularly handling well Following the schema for an argumentation originally set fuzzy and noise data; up for software engineering (Hindle et al. 2012), we formu- Semantics: Depending on the concrete application late an ML/DL-centric naturalness hypothesis for 3D point domain, semantic concepts can be defined and corre- clouds as follows: sponding training data can be configured to distill the required semantics. 3D point clouds, in theory, are complex, expressive and powerful, but the 3D point clouds actually cap- In the case of 3D point clouds, ML/DL support comput- tured or generated in geospatial domains are far less ing domain-specific and application-specific information, complex, far less expressive and repetitive. Their pre- typically by point classification, point cloud segmentation, dictable statistical properties can be captured in sta- object identification, and shape reconstruction. Compared tistical language models and leveraged for geospatial to traditional procedural-like, heuristic, or empiric-based data analysis. algorithms, ML/DL-based techniques generally have signifi- General ML and DL approaches, however, need to be cantly less implementation complexity as the implementa- adapted to the characteristics of 3D point clouds. “Most tion relies on general-purpose AI frameworks having a high critically, standard deep neural network models require input robustness and high rate of innovation. The customization 1 3 20 PFG (2020) 88:15–24 data with regular structure, while point clouds are funda- ranging from object classification, part segmentation, to mentally irregular: point positions are continuously distrib- scene semantic parsing” (Qi et al. 2016a). uted in the space and any permutation of their ordering does Applications or services that require spatial information not change the spatial distribution” (Wang et al. 2018b). encoded in 3D point clouds specify the exact set of features to be extracted and the spatial extent to be searched. The available feature types depend on how the ML and DL sub- 3 ML/DL‑Based Point Cloud Interpretation systems have been trained before. The analysis processes the request by triggering the evaluation to obtain the results. 3D point clouds provide cost-efficient raw data for creating At no point in time, 3D point clouds are pre-processed or the basis for geospatial digital twins at all scales but they are pre-evaluated nor do they require any intermediate repre- purely geometric data without any structural or semantics sentations, i.e. the interpretation works on-demand on the information about the objects they represent. Motivated by raw point cloud data. the naturalness hypothesis, ML and DL can be applied to In Fig.  3 the classical geoprocessing workflow (a) is analyse and interpret that data as well as to provide powerful compared to the workflow enabled by 3D point cloud inter - techniques if it comes to discrete irregular, incomplete, and pretation (b). While (a) is based on generating more and ambiguous data of a given corpus—exactly what character- more detailed and semantically well-defined representa- izes 3D point clouds. tions, the workflow (b) operates on raw data, extracting the demanded features using corresponding training data. The 3.1 Interpretation Concept ML/DL engine that implements that workflow needs core functionality such as: The ML/DL-based processing of 3D point clouds is based on the concept of interpretation known from programming Point classification: According to defined point cat- languages. In that scheme, however, analytics and semantics egories (e.g vegetation, built structures, water, streets) derivation do not require steps that “compile” raw data into labels are computed and attached as per-point attributes higher-level representations. To process data, for example, together with the probability for this category assign- the PointNet neural network “directly consumes point clouds ment. For example, Roveri et al. “automatically transform and well respects the permutation invariance of points in the the 3D unordered input data into a set of useful 2D depth input” and provides a “unified architecture for applications Fig. 3 3D point cloud processing: classical workflow a based on 3D reconstruction, 3D modelling, and object derivation; b ML/DL workflow based on 3D point cloud interpretation 1 3 PFG (2020) 88:15–24 21 images, and classify them by exploiting well-performing image classification CNNs” (Roveri et al. 2018). Point cloud segmentation: Segmentation as a core opera- tion for 3D point clouds helps reducing fragmentation and subdividing large point clouds. Typically, it is based on identifying 3D geometry features such as edges, pla- nar facets, or corners. ML and DL, in contrast, allow us to take advantage of semantic cues and affordances found in 3D point clouds. For example, we can segment “local geometric structures by constructing a local neighbour- Fig. 4 Example of an analysis of underground structures based on hood graph and applying convolution-like operations on 3D point clouds captured by radar combined with four trajectories the edges connecting neighbouring pairs of points, in the of ground-penetrating radar data. The point cloud is coloured with a height gradient. Dataset from the city of Essen, Germany spirit of graph neural networks” (Wang et al. 2018b). Shape recognition: Shapes are essential for understand- ing 3D environments. To recognize them, a combined Service-based computing: The approach is scalable as 2D–3D approach (Stojanovic et al. 2019b) consists of generating 2D renderings from 3D point clouds that are it can be fully mapped to a service-oriented architec- ture and scalable hardware (e.g. GPU clusters), built by evaluated by image analysis. For this purpose, CNNs can combine “information from multiple views of a 3D shape lower-level and higher-level services and mashups. On-demand computation: Downstream services allow into a single and compact shape descriptor offering even better recognition performance” (Su et al. 2015) com- for on-demand computation. For many classifications, the intepretation can be executed even in real time (e.g. pared to approaches that operate directly on raw 3D point clouds. Large general-purpose repositories of 3D objects, object detection out of point clouds for surveillance pur- poses). in addition, provide a solid training data base. • • Object classification: Applications generally require Raw data processing: Storage and handling of mas- sive 3D point clouds, including time-variant ones, can object-based information to be extracted from 3D point clouds, for example, signs and poles of the street space. be optimized independently as the interpretation only requires fast spatial access to point cloud contents. Based on classified and segmented 3D point clouds, CNNs based upon volumetric representations or CNNs Storage efficiency: There are no pre-selected or pre-built 3D models or intermediate representations. The approach based upon multi-view representations are commonly applied to this end; Qi et al. (2016b) give an overview of therefore works well for massive or time-varying 3D point clouds. In particular, the original precision of the the space of methods available. raw data is never reduced as raw data are fed directly into the ML/DL processes. The non-uniform sampling density typically found in 3D point clouds represents a key challenge for ML/DL-based 3.2 Examples learning. Qi et al. (2017) propose a hierarchical CNN that operates on nested partitions of an input point set because In a joint research project, we are developing a robust, high- “features learned in dense data may not generalize to sparsely sampled regions. Consequently, models trained performance engine for experimental ML/DL-based geo- spatial analytics. It provides features to store, manage, and for sparse point cloud may not recognize fine-grained local structures”. visualize massive 4D point clouds. In Fig. 4, 3D point cloud interpretation has been used to ML/DL-based interpretation enables us to implement generic analysis components for 3D point clouds. As no extract the underground infrastructure entities in the street space from mobile mapping data (Wolf et al. 2019). The intermediate representations are required, analysis results are only created once they are requested and they are only visualization shows the extracted tubes and also has detected street elements such as manhole covers. In Fig. 5, 3D point computed for the specific region the application has defined. Among the advantages of this approach are: cloud interpretation has identified and classified points according to different categories of street space furniture. Configurability: The ML/DL training data together with Figure 6 shows how time series of 3D point clouds, for example taken during a mobile scan, can be interpreted to feature vector definitions allow for many label types to be predicted. For it, the generic, domain-independent extract relevant objects such as street signs, vehicles, veg- etation, etc. mechanism offers a high degree of configurability. 1 3 22 PFG (2020) 88:15–24 In Fig. 7, a composite classification is illustrated: the bike and the person riding the bike are identified and then can be combined as ’person-riding-a-bike’. High-level abstractions can be built in a post-processing step or as part of the ML/ DL processes. 4 AI for Geospatial Digital Twins Fig. 6 Example of object classification (based on PointNet) within a dynamic scenario given by a time series of 3D point clouds A key demand in digital transformation processes represent digital twins in the sense of digital representations and rep- originality, while using only a moderate degree of explicit lica that reflect key traits, behaviour and states of a living or non-living physical entity (El Saddik 2018). The con- modelling for extracted features (Fig. 8). This, on the one hand, simplifies management and storage, in particular, if struction of base data for geospatial digital twins based on explicitly defined 3D model schemata is a labour-intensive it comes to time series. On the other hand, it helps iden- tifying and classifying ambiguous or fuzzy entities as and error-prone process, for example, virtual 3D city models with high level of detail such as CityGML LOD3 or LOD4 needed to, for example robustly and automatically build geospatial digital twins. (Löwner et al. 2016) as 3D reconstruction processes as well as the modelling schemata must deal with inaccurate, incom- plete data and generally cannot deal with special cases that are not provided in the modelling scheme. Whether we apply 5 Conclusions strong mathematics or fine-tuned heuristics, a reconstructed 3D model almost always lacks details and it can hardly mir- AI is radically changing programming paradigms and soft- ware solutions in all application domains. In the geospatial ror weakly sampled, unusual, or fuzzy entities. 3D point clouds represent raw data of geospatial entities domain, the data characteristics are particularly suitable for ML/DL approaches as geodata fits into the concept of a in a well-defined, consistent, and simple way, in particu- lar, for spatial environments such as indoor spaces (Sto- “linguistic corpus” as sketched in the context of the natural- ness hypothesis. ML/DL-based analysis and extraction of janovic et al. 2019a), building information models, and cities. ML/DL-based interpretation can both efficiently and features out of 3D point clouds, for example, can be used to derive application-specific, domain-specific and task- effectively, analyse and organize 3D point clouds without being restricted by explicitly defined modelling schemata. specific semantics. Above all, ML/DL-based interpretation of 3D point Above all, it flexibly generates semantics on-demand and on-the-fly, that is, it helps “healing” one of the biggest clouds enables us to transcend explicit geospatial model- ling and, therefore, to overcome complex, heuristics-based weaknesses of 3D point clouds—the lack of structure and semantics. There is virtually no limitation for the specific reconstructions and model-based abstractions. In that regard, types of 3D objects, structures, or phenomena that can be identified and extracted by ML/DL-based 3D point cloud interpretation. In addition, ML/DL-based interpretation operates on the raw geospatial data, i.e. it retains a high degree of Fig. 5 Example of point classification for a typical street space sce- nario. Ground, buildings, vehicles, pedestrians, and different street Fig. 7 Example of a composite classification (data by courtesy of furniture objects are classified with a PointNet-based approach and Stadt Hamburg). Bicycle and cyclist are segregated by DL-based are visualized by different colours classification 1 3 PFG (2020) 88:15–24 23 References Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Cor- rado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow IJ, Harp A, Irving G, Isard M (2016) TensorFlow: large-scale machine learning on heterogeneous distributed systems. CoRR arXiv :1603.04467 Allamanis M, Barr E, Devanbu P, Sutton C (2018) A survey of machine learning for big code and naturalness. ACM Comput Surv 51(4):81:1–81:37 Alpaydin E (2014) Introduction to machine learning, 3rd edn. MIT Press, Adaptive Computation and Machine Learning Fig. 8 The higher the degree of explicit modelling, the less of the Copeland J (1993) Artificial Intelligence: a philosophical introduction. original raw data is preserved as more and more assumptions and Wiley-Blackwell, USA abstraction are introduced. Hence, data details disappear and less gen- de Boer P, Kroese D, Shie M, Rubinstein R (2005) A tutorial on the eral-purpose models result cross-entropy method. Ann Operations Res 134(1):19–67 El Saddik A (2018) Digital twins: the convergence of multimedia tech- nologies. IEEE MultiMedia 25(2):87–92 AI technology can be used to simplify and accelerate work- Gandomi A, Haider M (2015) Beyond the hype: big data concepts, flows for geodata processing and geoinformation systems. methods, and analytics. Int J Inf Manag 35(2):137–144 Of course, crucial ML/DL-related challenges result from Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press. the demand for effective training data and efficient feature http://www.deepl earni ngboo k.org Griffiths D, Boehm J (2019) A review on deep learning techniques representations. for 3D sensed data classification. CoRR arXiv :abs/1907.04444 , Last but not least, ML/DL-based solutions offer simpli - Gross M, Pfister H (2007) Point-based graphics. Morgan Kaufmann fications in the dimension of software engineering. Large Publishers Inc, USA parts of today’s implementations (often historically grown Haenlein M, Kaplan A (2019) A brief history of artificial intelligence: on the past, present, and future of artificial intelligence. Calif with large amounts of so called technical debts) will be par- Manag Rev 61(4):5–14 tically replaced by ML/DL “black box” subsystems, which Hatcher W, Yu W (2018) A survey of deep learning: platforms, applica- have far less software management and software develop- tions and emerging research trends. IEEE Access 6:24411–24432. ment complexities. In particular, most heuristics-based, https ://doi.org/10.1109/ACCES S.2018.28306 61 Hindle A, Barr E, Su Z, Gabel M, Devanbu P (2012) On the Natural- explicitly programmed analysis routines, which tend to be ness of Software. In: Proceedings of the 34th international confer- difficult to parameterize and to configure, can be migrated ence on software engineering, IEEE Press, ICSE ’12, pp 837–847 this way. As a consequence, ML/DL-based approaches, in Humby C (2006) http://www.humby anddu nn.com the long run, have the potential to “eat up” many of today’s Janowicz K, Gao S, McKenzie G, Hu Y, Bhaduri B (2020) GeoAI: spatially explicit artificial intelligence techniques for geographic explicitly programmed GIS implementations. knowledge discovery and beyond. Int J Geogr Inf Sci. https://doi. org/10.1080/13658 816.2019.16845 00 Acknowledgements Open Access funding provided by Projekt DEAL. Jurafsky D, Martin J (2000) Speech and language processing: an intro- We thank Benjamin Hagedorn, Johannes Wolf, Rico Richter, and duction to natural language processing, computational linguistics, Vladeta Stojanovic for their contributions to HPI’s geospatial ML/DL and speech recognition, 1st edn. Prentice Hall PTR, USA research. We also thank the GraphicsVision.AI association for their Kanevski M, Foresti L, Kaiser C, Pozdnoukhov A, Timonin V, Tuia D support by the Málaga research retreat and pointcloudtechnology. (2009) Machine learning models for geospatial data. Handbook com for providing us the PunctumTube 3D point cloud platform. This of theoretical and quantitative geography. University of Lausanne, research work was partially supported by the German Federal Ministry Lausanne, pp 175–227 of Education and Research (BMBF) as part of the research grands for Kelly K (2017) The AI Cargo Cult: the myth of a superhuman AI. Tech. HPI AI Lab, PunctumTube and GeoPortfolio. rep., Backchannel. https://www .wired.com/2017/04/t he-myth-of- a-super human -ai Open Access This article is licensed under a Creative Commons Attri- Khajavi SH, Hossein Motlagh N, Jaribion A, Werner LC, Holmström bution 4.0 International License, which permits use, sharing, adapta- J (2019) Digital twin: vision, benefits, boundaries, and crea- tion, distribution and reproduction in any medium or format, as long tion for buildings. IEEE Access 7:147406–147419. https ://doi. as you give appropriate credit to the original author(s) and the source, org/10.1109/ACCES S.2019.29465 15 provide a link to the Creative Commons licence, and indicate if changes Kitchin R, McArdle G (2016) What makes big data, big data? Explor- were made. The images or other third party material in this article are ing the ontological characteristics of 26 datasets. Big Data Soc. included in the article’s Creative Commons licence, unless indicated https ://doi.org/10.1177/20539 51716 63113 0 otherwise in a credit line to the material. If material is not included in Lecun Y (2017) AI is going to amplify human intelligence not replace the article’s Creative Commons licence and your intended use is not it. FAZ Netzwirtschaft. https ://www.faz.net/-gqm-8yrxk permitted by statutory regulation or exceeds the permitted use, you will LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436– need to obtain permission directly from the copyright holder. To view a 444. https ://doi.org/10.1038/natur e1453 9 copy of this licence, visit http://creativ ecommons .or g/licenses/b y/4.0/. Löwner MO, Gröger G, Benner J, Biljecki F, Nagel C (2016) Proposal for a new LOD and multi-representation concept for CityGML. In: ISPRS annals of the photogrammetry, remote sensing and spatial information sciences, vol 4, pp 3–12 1 3 24 PFG (2020) 88:15–24 Marcus G (2018) Deep learning: a critical appraisal. CoRR arXiv Smith T (1984) Artificial intelligence and its applicability to geographi- :abs/1801.00631 cal problem solving. Prof Geogrh 36(2):147–158 Marr B (2018) Here’s why data is not the new oil. Forbes. www.forbe Stojanovic V, Trapp M, Döllner J, Richter R (2019a) Classification of s.com indoor point clouds using multiviews. In: The 24th international Najafabadi MM, Villanustre F, Khoshgoftaar TM et al (2015) Deep conference on 3D web technology, Web3D, Los Angeles, July learning applications and challenges in big data analytics. J Big 26-28, 2019, pp 1–9 Data 2:1. https ://doi.org/10.1186/s4053 7-014-0007-7 Stojanovic V, Trapp M, Richter R, Döllner J (2019b) Generation of Openshaw S, Openshaw C (1997) Artificial intelligence in geography. approximate 2D and 3D floor plans from 3D point clouds. In: Pro- Wiley, USA ceedings of the 14th international joint conference on computer Palatini K (2014) Joseph Weizenbaum, responsibility and humanoid vision, imaging and computer graphics theory and applications, robots. In: Funk M, Irrgang B (eds) Robotics in Germany and VISIGRAPP 2019, Vol 1: GRAPP, pp 177–184 Japan. Philosophical and technical perspectives. Peter Lang, USA, Su H, Maji S, Kalogerakis E, Learned-Miller E (2015) Multi-view pp 163–169 convolutional neural networks for 3D shape recognition. In: Pro- Posada J, Zorrilla M, Dominguez A, Simoes B, Eisert P, Stricker D, ceedings of the 2015 IEEE international conference on computer Rambach J, Döllner J, Guevara M (2018) Graphics and media vision (ICCV), IEEE Computer Society, ICCV ’15, pp 945–953 technologies for operators in industry 4.0. IEEE Comput Graph Van Der Maaten L, Postma E, Van den Herik J (2009) Dimensional- Appl 38(5):119–132 ity reduction: a comparative review. J Mach Learn Res 10:66–71 Purdy M, Daugherty P (2017) How AI boosts industry profits and inno- Vopham T, Hart J, Laden F, Chiang Y (2018) Emerging trends in vation. http://www.accen ture.com geospatial artificial intelligence (geoAI): potential applications Qi C, Su H, Mo K, Guibas L (2016a) PointNet: deep learning on for environmental epidemiology. Environ Health. https ://doi. point sets for 3D classification and segmentation. CoRR arXiv org/10.1186/s1294 0-018-0386-x :abs/1612.00593 Wang D, Szymanski B, Abdelzaher T, Ji H, Kaplan L (2018a) The age Qi C, Su H, Nießner M, Dai A, Yan M, Guibas L (2016b) Volumetric of social sensing. CoRR arXiv :abs/1801.09116 and multi-view CNNs for object classification on 3D data. CoRR Wang Y, Sun Y, Liu Z, Sarma S, Bronstein M, Solomon J (2018b) arXiv :1604.03265 Dynamic graph CNN for learning on point clouds. CoRR arXiv Qi Q, Tao F (2018) Digital twin and big data towards smart manufac- :1801.07829 turing and industry 4.0: 360 degree comparison. IEEE Access Wang Y, Wang Q, Shi S, He X, Tang Z, Zhao K, Chu X (2019) Bench- 6:3585–3593 marking the performance and power of AI accelerators for AI Qi V, Yi L, Su H, Guibas L (2017) PointNet++: Deep hierarchical training. arXiv :1909.06842 feature learning on point sets in a metric space. CoRR arXiv Weinmann M, Schmidt A, Mallet C, Hinz S, Rottensteiner F, Jutzi B :1706.02413 (2015) Contextual classification of point cloud data by exploiting Reuther A, Michaleas P, Jones M, Gadepally V, Samsi S, Kepner J individual 3D neighborhoods. ISPRS Ann Photogramm Remote (2019) Survey and benchmarking of machine learning accelera- Sens Spatial Inf Sci II–3/W4:271–278 tors. In: 2019 IEEE high performance extreme computing confer- Weizenbaum J (1966) ELIZA—a computer program for the study ence (HPEC), pp 1–9 of natural language communication between man and machine. Richter R (2018) Concepts and techniques for processing and rendering Commun ACM 9(1):36–45 of massive 3D point clouds. PhD thesis, University of Potsdam, Wolf J, Richter R, Döllner J (2019) Techniques for automated classi- Faculty of Digital Engineering, Hasso Plattner Institute fication and segregation of mobile mapping 3D point clouds. In: Roveri R, Rahmann L, Öztireli C, Gross M (2018) A network archi- Proceedings of the 14th international joint conference on com- tecture for point cloud classification via automatic depth images puter vision, imaging and computer graphics theory and applica- generation. In: 2018 IEEE conference on computer vision and tions, VISIGRAPP 2019, vol 1: GRAPP, pp 201–208 pattern recognition, CVPR 2018, Salt Lake City, UT, USA, June Zhang C, Vinyals O, Munos R, Bengio S (2018) A study on overfitting 18–22, 2018, pp 4176–4184 in deep reinforcement learning. CoRR arXiv :1804.06893 Rozsa A, Günther M, Boult T (2016) Are accuracy and robustness cor- Zollhöfer M, Stotko P, Görlitz A, Theobalt C, Niessner M, Klein R, related. In: 2016 15th IEEE international conference on machine Kolb A (2018) State of the art on 3D reconstruction with RGB-D learning and applications (ICMLA), pp 227–232 cameras. Comput Graph Forum 37:625–652 1 3 http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png PFG – Journal of Photogrammetry, Remote Sensing and Geoinformation Science Springer Journals

Geospatial Artificial Intelligence: Potentials of Machine Learning for 3D Point Clouds and Geospatial Digital Twins

Loading next page...
 
/lp/springer-journals/geospatial-artificial-intelligence-potentials-of-machine-learning-for-MGSHFCJfHC
Publisher
Springer Journals
Copyright
Copyright © The Author(s) 2020
ISSN
2512-2789
eISSN
2512-2819
DOI
10.1007/s41064-020-00102-3
Publisher site
See Article on Publisher Site

Abstract

Artificial intelligence (AI) is changing fundamentally the way how IT solutions are implemented and operated across all application domains, including the geospatial domain. This contribution outlines AI-based techniques for 3D point clouds and geospatial digital twins as generic components of geospatial AI. First, we briefly reflect on the term “AI” and outline technol- ogy developments needed to apply AI to IT solutions, seen from a software engineering perspective. Next, we characterize 3D point clouds as key category of geodata and their role for creating the basis for geospatial digital twins; we explain the feasibility of machine learning (ML) and deep learning (DL) approaches for 3D point clouds. In particular, we argue that 3D point clouds can be seen as a corpus with similar properties as natural language corpora and formulate a “Naturalness Hypothesis” for 3D point clouds. In the main part, we introduce a workflow for interpreting 3D point clouds based on ML/ DL approaches that derive domain-specific and application-specific semantics for 3D point clouds without having to create explicit spatial 3D models or explicit rule sets. Finally, examples are shown how ML/DL enables us to efficiently build and maintain base data for geospatial digital twins such as virtual 3D city models, indoor models, or building information models. Keywords Geospatial artificial intelligence · Machine learning · Deep learning · 3D point clouds · Geospatial digital twins · 3D city models Zusammenfassung Georäumliche Künstliche Intelligenz: Potentiale des Maschinellen Lernens für 3D-Punktwolken und georäumliche digitale Zwillinge. Künstliche Intelligenz (KI) verändert grundlegend die Art und Weise, wie IT-Lösungen in allen Anwendungs- bereichen, einschließlich dem Geoinformationsbereich, implementiert und betrieben werden. In diesem Beitrag stellen wir KI-basierte Techniken für 3D-Punktwolken als einen Baustein der georäumlichen KI vor. Zunächst werden kurz der Begriff ,,KI” und die technologischen Entwicklungen skizziert, die für die Anwendung von KI auf IT-Lösungen aus der Sicht der Softwaretechnik erforderlich sind. Als nächstes charakterisieren wir 3D-Punktwolken als Schlüsselkategorie von Geodaten und ihre Rolle für den Aufbau von räumlichen digitalen Zwillingen; wir erläutern die Machbarkeit der Ansätze für Maschinelles Lernen (ML) und Deep Learning (DL) in Bezug auf 3D-Punktwolken. Insbesondere argumentieren wir, dass 3D-Punktwolken als Korpus mit ähnlichen Eigenschaften wie natürlichsprachliche Korpusse gesehen werden können und formulieren eine ,,Natürlichkeitshypothese” für 3D-Punktwolken. Im Hauptteil stellen wir einen Worko fl w zur Interpretation von 3D-Punktwolken auf der Grundlage von ML/DL-Ansätzen vor, die eine domänenspezifische und anwendungsspezifische Semantik für 3D-Punktwolken ableiten, ohne explizite räumliche 3D-Modelle oder explizite Regelsätze erstellen zu müssen. Abschließend wird an Beispielen gezeigt, wie ML/DL es ermöglichen, Basisdaten für räumliche digitale Zwillinge, wie z.B. für virtuelle 3D-Stadtmodelle, Innenraummodelle oder Gebäudeinformationsmodelle, effizient aufzubauen und zu pflegen. 1 Introduction * Jürgen Döllner Artificial intelligence (AI) is changing the way IT solutions doellner@hpi.uni-potsdam.de are designed, built and operated. AI is not limited to spe- Hasso Plattner Institute, Digital Engineering Faculty, cific application areas—it is finding its way into almost all University of Potsdam, Prof.-Dr.-Helmert-Str. 2-3, industries and domains. As a collection of general-purpose 14482 Potsdam, Germany Vol.:(0123456789) 1 3 16 PFG (2020) 88:15–24 technologies, AI has significant impact because it “can trans- generalize and learn from past experiences, and to be intel- form opportunities not only for economic growth, but also ligent using learning, thinking, problem solving, perception for corporate profitability” (Purdy and Daugherty 2017). and language. For geospatial domains, fundamental questions include There is generally no sharp border between AI and non- how AI can be specifically applied to or has to be specifi- AI technology. Consider, for example, an autopilot steering cally created for spatial data. Janowicz et al. (2020) give an an aircraft: in its beginning, it was perceived as AI, while overview of spatially explicit AI, which “utilizes advance- today it has become a common operating technical compo- ments in techniques and data cultures to support the crea- nent. That is to say, “when AI reaches mainstream usage it tion of more intelligent geographic information as well as is frequently no longer considered as such” (Haenlein and methods, systems, and services for a variety of downstream Kaplan 2019). In that sense, the term “AI” is mostly used to tasks”. This arising scientific discipline, called geospatial label such technology that goes beyond current technology artificial intelligence (GeoAI), which “combines innova - boundaries. In this contribution, we refer to AI in a geospa- tions in spatial science, artificial intelligence methods in tial context and focus on potentials of ML and DL for 3D machine learning (e.g. deep learning), data mining, and point clouds. high-performance computing to extract knowledge from spatial big data” (Vopham et al. 2018), will in particular 1.2 AI‑Based IT improve existing and create new technologies for geospatial information systems (GIS). From a software engineering perspective, several technolo- The relevance of AI for geospatial domains has been real- gies and disciplines (Fig. 1) are required for the implementa- ized many years ago already regarding, for example, expert tion of AI-based IT solutions: systems and knowledge-based systems (Openshaw and Openshaw 1997), geographically problem solving (Smith • Big data management Many AI-based techniques require 1984), or analysing social sensing data (Wang et al. 2018a). big data to be applied effectively. For example, ML and In this paper, we concentrate on AI-based approaches for DL need training data, which is typically distilled from a specific category of 3D geodata, 3D point clouds, which big data. Big data generally are characterized by a num- are fundamental in photogrammetry, remote sensing and ber of key traits (Kitchin and McArdle 2016), including: computer vision (Weinmann et al. 2015) and have manifold – large data amounts (volume), applications for building geospatial digital twins. – rapid data capturing or generation (velocity), – different data types and structures (variety), 1.1 The Term “Artificial Intelligence” – manifold relations among data sets (complexity), – high inherent data uncertainty (veracity). The notion “AI” implies a number of conceptual difficul- ties, such as the definition of “natural”, “human” or “gen- Big data have turned out to be a key driver for digital eral-purpose” intelligence. Put simply, the “most common transformation processes as summarized in the famous misconception about artificial intelligence begins with the statement “Data is the new oil. Data is just like crude. common misconception about natural intelligence. This mis- conception is that intelligence is a single dimension” (Kelly 2017). In the general public, AI often triggers associations and expectations such as simulating or overcoming human intelligence. If AI is pragmatically seen as technological progress, then “AI is going to amplify human intelligence not replace it, the same way any tool amplifies our abili - ties” (Lecun 2017). One of the AI applications that exem- plified these controversies was ELIZA. This famous first chatbot, built by Josef Weizenbaum in 1966 (Weizenbaum 1966), was a speech-based simulation of a psychologist’s interaction with a patient, which “demonstrated the kind of risk potential what was enclosed within such technological developments” (Palatini 2014). This topic is discussed fur- ther by Copeland (1993), who analyses which challenges and obstacles AI needs be to solved before “thinking” machines could be constructed. As he states, the key to AI is the abil- Fig. 1 Disciplines and technologies used to implement AI-based IT ity of computers to think rationally, to discover meaning, to 1 3 PFG (2020) 88:15–24 17 It’s valuable, but if unrefined it cannot really be used” clustering, auto-encoding, and others” (Hatcher and Yu (Humby 2006), which also is subject to a controversy 2018). As a specific form of representation learning, (Marr 2018). which in turn is a specific form of ML, DL is based on A broad range of approaches exist to capture, synthe- artificial neural networks (ANNs) such as convolutional size and simulate data about our geospatial reality. In that neural networks (CNNs) (Goodfellow et al. 2016). In the respect, geospatial data generally represent big data and end, DL represents “essentially a statistical technique for the “oil” for the geospatial digital economy. classifying patterns, based on sample data, using neural Analytics Analytics refers to analytical reasoning and networks with multiple layers” (Marcus 2018). aims at providing concepts, methods, techniques, and It builds representations expressed in terms of simpler tools to efficiently collect, organize, and analyse big data. representations, i.e. we can build complex concepts out Its objectives include to examine data, to draw conclu- of simpler concepts. “It has turned out to be very good at sions, to get insights, to acquire knowledge, and to sup- discovering intricate structures in high-dimensional data port decision making. In that respect, analytics “can be and is therefore applicable to many domains of science, viewed as a sub-process in the overall process of ’insight business and government” (LeCun et al. 2015), but is extraction’ from big data” (Gandomi and Haider 2015). also faced by a number of limitations such as being “data For all variants, such as descriptive, predictive, and hungry” and low support for transfer or hierarchical data prescriptive analytics, ML/DL approaches support “the (Marcus 2018). analysis and learning of massive amounts of unsuper- AI accelerators The growth of AI-based applications vised data, making it a valuable tool for big data analytics comes along with commodity, cost-efficient graphics where raw data are largely unlabeled and uncategorized” processing units (GPUs) that are evolving to become (Najafabadi et al. 2015). high-performance accelerators for data-parallel comput- Machine learning “Machine learning is programming ing. Further, there is a growing number of purpose-built computers to optimize a performance criterion using systems for DL, for example, tensor processing units example data or past experience”, which is required, in (TPUs), along with fully integrated hardware and soft- particular, “where we cannot write directly a computer ware [e.g. NVidia DGX/HGX; TensorFlow (Abadi et al. program to solve a given problem” (Alpaydin 2014). That 2016)]. They are known as AI accelerators, which differ is to say, IT solutions do not rely on explicit or procedural with respect to hardware costs, training performance, problem solving strategies but are based on processing computing power, and energy consumption (Wang et al. and analysing patterns and inference. For that reason, ML 2019) but foster and simplify AI-based software develop- offers a different programming paradigm for the imple - ment and operation (Reuther et al. 2019). mentation of geospatial IT solutions. ML techniques can be generally classified into super - To implement AI-based IT, components also include frame- vised ML, unsupervised ML, and reinforcement learn- works and systems for computer vision (e.g. image analysis, ing. Supervised ML, for example, acquires knowledge image understanding), speech and text analysis (e.g. text- by building mathematical models based on training data, to-speech, speech-to-text), knowledge representation and which are used to predict labels for input data. The under- discovery, reasoning systems, etc. lying models “may be predictive to make predictions in the future, or descriptive to gain knowledge from data, or both” (Alpaydin 2014):1.3 Feature Space “The promise and power of machine learning rest on its ability to generalize from examples and to handle ML/DL-based solutions describe the input data by means of noise” (Allamanis et al. 2018). For this purpose, ML feature vectors, i.e. by n-dimensional vectors whose numeri- offers a high degree of robustness regarding the input cal components describe selected aspects of a phenomenon data set. A fundamental risk, however, lies in “overtrain- to be observed, that is, the feature space equals a high- ing” the model. If overtrained, it will perform very well dimensional vector space. Dimensionality reduction tech- on the training data, but will poorly generalize to new niques allow us to transform “high-dimensional data into a data. The model becomes incapable of generalizing, i.e. meaningful representation of reduced dimensionality” (Van it is overfitting the training data. For that reason, “prop- Der Maaten et al. 2009) and, this way, to efficiently manage, erly controlling or regularizing the training is key to out- process, and visualize feature spaces. of-sample generalization” (Zhang et al. 2018). One of the key concepts required for geospatial ML and Deep learning As Hatcher and Yu point out, DL applies DL consists in finding adequate feature spaces for geospa- “multi-neuron, multi-layer neural networks to perform tial entities. A single point of the 3D point cloud does not learning tasks, including regression, classification, allow for constructing meaningful feature vectors. To do so, 1 3 18 PFG (2020) 88:15–24 the local neighbourhood of the point must be analysed by selecting, for example, the k nearest points or points within a radius r (Weinmann et al. 2015). Next, geometric, topo- logic or any other high-level features are computed from the neighbourhood region that constitute components of the fea- ture vector. “Typical recurring features include computing planarity, linearity, scatter, surface variance, vertical range, point colour, eigenentropy and omnivariance” (Griffiths and Boehm 2019). Fig. 2 Example of a high-density 3D point cloud of an indoor envi- ronment 2 3D Point Clouds and predicting behaviour and processes related to the cor- As universal 3D representations, 3D point clouds “can rep- responding physical entities. resent almost any type of physical object, site, landscape, geographic region, or infrastructure—at all scales and with any precision” as Richter (2018) states, who discusses algo- 2.2 Characteristics rithms and data structures for out-of-core processing, ana- lysing, and classifying of 3D point clouds. To acquire 3D A 3D point cloud represents a set of 3D points in a given point clouds, various technologies can be applied includ- coordinate system and can be characterized by: ing airborne or terrestrial laser scanning, mobile mapping, RGB-D cameras (Zollhöfer et al. 2018), image matching, or Uniform representation—unstructured, unordered set multi-beam echo sounding. of 3D points (e.g. in an Euclidian space); Discrete representation—discrete samples of shapes without restrictions regarding topology or geometry; 2.1 Geospatial Digital Twins Irregularity—expose irregular spatial distribution and varying spatial density; 3D point clouds are ubiquitous for geospatial applications Incompleteness—due to the discrete sampling, repre- such as for environmental monitoring, disaster management, sentations are incomplete by nature; urban planning, building information models, or self-driving Ambiguity—the semantics (e.g. surface type, object vehicles. More precisely, 3D point clouds are commonly type) of a single point generally cannot be determined used as base data for reconstructing 3D models (e.g. digital without considering its neighbourhood; terrain models, virtual 3D city models, building informa- • Per-point attributes—each point can be attributed by tion models), but can also be understood as point-based 3D additional per-point data such as colour or surface nor- models, for example, in the case when 3D point clouds are mal; dense (Fig. 2). Massiveness—depending on the density of the captur- In particular, they represent key components of geospa- ing technology, 3D point clouds may consist of mil- tial digital twins, that is, digital replica of spatial entities lions or billions of points. and phenomena. Digital twins, in general, are composed of three parts, “which are the physical entities in the phys- The key trait of 3D point clouds, a fundamental category ical world, the virtual models in the virtual world, and of 3D geospatial data, consists in the absence of any struc- the connected data that tie the two worlds” (Qi and Tao tural, hierarchical, or semantics-related information—a 3D 2018). While the connection between both can be handled point cloud is a simple, unordered set of 3D points. “The by sensors, the virtual models have to be derived from lack of topology and connectivity, however, is strength the physical counterparts. For example, 3D point clouds and weakness at the same time” (Gross and Pfister 2007). are used to derive 3D indoor models, which are essential There is, therefore, a strong demand for solutions that components for real-time building information models allow us to enrich 3D point clouds with information. together with sensor networks and IoT devices (Khajavi et  al. 2019); they also “represent a generic approach to 2.3 3D Point Cloud Time Series capture, model, analyse, and visualize digital twins used by operators in Industry 4.0 application scenarios” (Posada For a growing number of applications, 3D point clouds are et  al. 2018). In that regard, geospatial digital twins are captured and processed with high frequency. For example, if means for monitoring, visualizing, exploring, optimizing, 1 3 PFG (2020) 88:15–24 19 a surveillance system captures its target environment every consists in finding appropriate training and test data. A cru- second, it results in a stream of 3D point clouds . cial issue represents robustness of ML techniques as ML If 3D point clouds are captured or generated at different models “are vulnerable to adversarial examples formed by points in time having overlapping geospatial regions, these applying small carefully chosen perturbations to inputs that sets are inherently related. By 3D point cloud time series, cause unexpected classification errors” (Rozsa et al. 2016). we refer to a collection of 3D point clouds taken at different points in time for a common geospatial region. The collec- 2.5 Naturalness Hypothesis tion of 3D point clouds represents, in a sense, a 4D point cloud. The “Naturalness Hypothesis” (Allamanis et  al. 2018), 3D point cloud time series have a high degree of redun- which is investigated in natural language recognition and dancy, which needs to be exploited to achieve efficient man - software analytics, helps understanding further why ML agement, processing, compression and storage, for example, and DL approaches provide effective instruments for ana- separating static from dynamic structures. Redundancy can lysing and interpreting 3D point clouds. In general, one also be used to improve accuracy and robustness of 3D point key approach to ML and DL is to find out whether a given cloud interpretations and related predictions. problem domain corresponds to or has similar statistical properties as large natural language corpora (Jurafsky and 2.4 Feasibility of ML/DL‑Based Approaches Martin 2000). Here, ML/DL-based approaches have shown extraordinary success in natural language recognition, natu- Kanevski et al. investigate the general applicability of ML ral language translation, question answering, text mining, to geospatial data and conclude “the key feature of the ML text comprehension, etc. The most important finding in these models/algorithms is that they learn from data and can be areas is that objects (e.g. spoken or written texts) are less used in cases when the modelled phenomenon is not very diverse than they initially seem: most human expressions well described, which is the case in many applications (“utterances”) are much simpler, much more repetitive, and of geospatial data” (Kanevski et al. 2009). The complete much more predictable than the expressiveness of the lan- absence of structure, order and semantics as well as the guage body suggests, whereby “these utterances can be very inherent irregularity, incompleteness, and ambiguity explain usefully modelled using modern statistical methods” (Hindle why 3D point clouds are not very well described and difficult et al. 2012). This phenomenon can be understood with meas- candidates for procedural and algorithmic programming. ures of perplexity and cross-entropy (de Boer et al. 2005). Their characteristics, however, allow us to effectively apply 3D point clouds seen as a form of natural communication ML/DL to 3D point clouds: as well as geospatial environments are ultimately repetitive regardless of the endless variations they may exhibit. In a Big data: 3D point clouds are spatial big data that can be sense, 3D point clouds are just “spatial utterances” that can cost-efficiently generated for almost all types of spatial be modelled using statistical methods. 3D point clouds, environments—big data are a prerequisite for ML/DL- thereby, constitute 3D point cloud corpora to which ML based approaches; technology can be applied taking advantage of the statisti- Fuzziness: 3D point clouds show inherent fuzziness and cal distributional properties estimated over representative noise as they are sampling shapes by means of discrete point cloud corpora. representations—ML/DL are particularly handling well Following the schema for an argumentation originally set fuzzy and noise data; up for software engineering (Hindle et al. 2012), we formu- Semantics: Depending on the concrete application late an ML/DL-centric naturalness hypothesis for 3D point domain, semantic concepts can be defined and corre- clouds as follows: sponding training data can be configured to distill the required semantics. 3D point clouds, in theory, are complex, expressive and powerful, but the 3D point clouds actually cap- In the case of 3D point clouds, ML/DL support comput- tured or generated in geospatial domains are far less ing domain-specific and application-specific information, complex, far less expressive and repetitive. Their pre- typically by point classification, point cloud segmentation, dictable statistical properties can be captured in sta- object identification, and shape reconstruction. Compared tistical language models and leveraged for geospatial to traditional procedural-like, heuristic, or empiric-based data analysis. algorithms, ML/DL-based techniques generally have signifi- General ML and DL approaches, however, need to be cantly less implementation complexity as the implementa- adapted to the characteristics of 3D point clouds. “Most tion relies on general-purpose AI frameworks having a high critically, standard deep neural network models require input robustness and high rate of innovation. The customization 1 3 20 PFG (2020) 88:15–24 data with regular structure, while point clouds are funda- ranging from object classification, part segmentation, to mentally irregular: point positions are continuously distrib- scene semantic parsing” (Qi et al. 2016a). uted in the space and any permutation of their ordering does Applications or services that require spatial information not change the spatial distribution” (Wang et al. 2018b). encoded in 3D point clouds specify the exact set of features to be extracted and the spatial extent to be searched. The available feature types depend on how the ML and DL sub- 3 ML/DL‑Based Point Cloud Interpretation systems have been trained before. The analysis processes the request by triggering the evaluation to obtain the results. 3D point clouds provide cost-efficient raw data for creating At no point in time, 3D point clouds are pre-processed or the basis for geospatial digital twins at all scales but they are pre-evaluated nor do they require any intermediate repre- purely geometric data without any structural or semantics sentations, i.e. the interpretation works on-demand on the information about the objects they represent. Motivated by raw point cloud data. the naturalness hypothesis, ML and DL can be applied to In Fig.  3 the classical geoprocessing workflow (a) is analyse and interpret that data as well as to provide powerful compared to the workflow enabled by 3D point cloud inter - techniques if it comes to discrete irregular, incomplete, and pretation (b). While (a) is based on generating more and ambiguous data of a given corpus—exactly what character- more detailed and semantically well-defined representa- izes 3D point clouds. tions, the workflow (b) operates on raw data, extracting the demanded features using corresponding training data. The 3.1 Interpretation Concept ML/DL engine that implements that workflow needs core functionality such as: The ML/DL-based processing of 3D point clouds is based on the concept of interpretation known from programming Point classification: According to defined point cat- languages. In that scheme, however, analytics and semantics egories (e.g vegetation, built structures, water, streets) derivation do not require steps that “compile” raw data into labels are computed and attached as per-point attributes higher-level representations. To process data, for example, together with the probability for this category assign- the PointNet neural network “directly consumes point clouds ment. For example, Roveri et al. “automatically transform and well respects the permutation invariance of points in the the 3D unordered input data into a set of useful 2D depth input” and provides a “unified architecture for applications Fig. 3 3D point cloud processing: classical workflow a based on 3D reconstruction, 3D modelling, and object derivation; b ML/DL workflow based on 3D point cloud interpretation 1 3 PFG (2020) 88:15–24 21 images, and classify them by exploiting well-performing image classification CNNs” (Roveri et al. 2018). Point cloud segmentation: Segmentation as a core opera- tion for 3D point clouds helps reducing fragmentation and subdividing large point clouds. Typically, it is based on identifying 3D geometry features such as edges, pla- nar facets, or corners. ML and DL, in contrast, allow us to take advantage of semantic cues and affordances found in 3D point clouds. For example, we can segment “local geometric structures by constructing a local neighbour- Fig. 4 Example of an analysis of underground structures based on hood graph and applying convolution-like operations on 3D point clouds captured by radar combined with four trajectories the edges connecting neighbouring pairs of points, in the of ground-penetrating radar data. The point cloud is coloured with a height gradient. Dataset from the city of Essen, Germany spirit of graph neural networks” (Wang et al. 2018b). Shape recognition: Shapes are essential for understand- ing 3D environments. To recognize them, a combined Service-based computing: The approach is scalable as 2D–3D approach (Stojanovic et al. 2019b) consists of generating 2D renderings from 3D point clouds that are it can be fully mapped to a service-oriented architec- ture and scalable hardware (e.g. GPU clusters), built by evaluated by image analysis. For this purpose, CNNs can combine “information from multiple views of a 3D shape lower-level and higher-level services and mashups. On-demand computation: Downstream services allow into a single and compact shape descriptor offering even better recognition performance” (Su et al. 2015) com- for on-demand computation. For many classifications, the intepretation can be executed even in real time (e.g. pared to approaches that operate directly on raw 3D point clouds. Large general-purpose repositories of 3D objects, object detection out of point clouds for surveillance pur- poses). in addition, provide a solid training data base. • • Object classification: Applications generally require Raw data processing: Storage and handling of mas- sive 3D point clouds, including time-variant ones, can object-based information to be extracted from 3D point clouds, for example, signs and poles of the street space. be optimized independently as the interpretation only requires fast spatial access to point cloud contents. Based on classified and segmented 3D point clouds, CNNs based upon volumetric representations or CNNs Storage efficiency: There are no pre-selected or pre-built 3D models or intermediate representations. The approach based upon multi-view representations are commonly applied to this end; Qi et al. (2016b) give an overview of therefore works well for massive or time-varying 3D point clouds. In particular, the original precision of the the space of methods available. raw data is never reduced as raw data are fed directly into the ML/DL processes. The non-uniform sampling density typically found in 3D point clouds represents a key challenge for ML/DL-based 3.2 Examples learning. Qi et al. (2017) propose a hierarchical CNN that operates on nested partitions of an input point set because In a joint research project, we are developing a robust, high- “features learned in dense data may not generalize to sparsely sampled regions. Consequently, models trained performance engine for experimental ML/DL-based geo- spatial analytics. It provides features to store, manage, and for sparse point cloud may not recognize fine-grained local structures”. visualize massive 4D point clouds. In Fig. 4, 3D point cloud interpretation has been used to ML/DL-based interpretation enables us to implement generic analysis components for 3D point clouds. As no extract the underground infrastructure entities in the street space from mobile mapping data (Wolf et al. 2019). The intermediate representations are required, analysis results are only created once they are requested and they are only visualization shows the extracted tubes and also has detected street elements such as manhole covers. In Fig. 5, 3D point computed for the specific region the application has defined. Among the advantages of this approach are: cloud interpretation has identified and classified points according to different categories of street space furniture. Configurability: The ML/DL training data together with Figure 6 shows how time series of 3D point clouds, for example taken during a mobile scan, can be interpreted to feature vector definitions allow for many label types to be predicted. For it, the generic, domain-independent extract relevant objects such as street signs, vehicles, veg- etation, etc. mechanism offers a high degree of configurability. 1 3 22 PFG (2020) 88:15–24 In Fig. 7, a composite classification is illustrated: the bike and the person riding the bike are identified and then can be combined as ’person-riding-a-bike’. High-level abstractions can be built in a post-processing step or as part of the ML/ DL processes. 4 AI for Geospatial Digital Twins Fig. 6 Example of object classification (based on PointNet) within a dynamic scenario given by a time series of 3D point clouds A key demand in digital transformation processes represent digital twins in the sense of digital representations and rep- originality, while using only a moderate degree of explicit lica that reflect key traits, behaviour and states of a living or non-living physical entity (El Saddik 2018). The con- modelling for extracted features (Fig. 8). This, on the one hand, simplifies management and storage, in particular, if struction of base data for geospatial digital twins based on explicitly defined 3D model schemata is a labour-intensive it comes to time series. On the other hand, it helps iden- tifying and classifying ambiguous or fuzzy entities as and error-prone process, for example, virtual 3D city models with high level of detail such as CityGML LOD3 or LOD4 needed to, for example robustly and automatically build geospatial digital twins. (Löwner et al. 2016) as 3D reconstruction processes as well as the modelling schemata must deal with inaccurate, incom- plete data and generally cannot deal with special cases that are not provided in the modelling scheme. Whether we apply 5 Conclusions strong mathematics or fine-tuned heuristics, a reconstructed 3D model almost always lacks details and it can hardly mir- AI is radically changing programming paradigms and soft- ware solutions in all application domains. In the geospatial ror weakly sampled, unusual, or fuzzy entities. 3D point clouds represent raw data of geospatial entities domain, the data characteristics are particularly suitable for ML/DL approaches as geodata fits into the concept of a in a well-defined, consistent, and simple way, in particu- lar, for spatial environments such as indoor spaces (Sto- “linguistic corpus” as sketched in the context of the natural- ness hypothesis. ML/DL-based analysis and extraction of janovic et al. 2019a), building information models, and cities. ML/DL-based interpretation can both efficiently and features out of 3D point clouds, for example, can be used to derive application-specific, domain-specific and task- effectively, analyse and organize 3D point clouds without being restricted by explicitly defined modelling schemata. specific semantics. Above all, ML/DL-based interpretation of 3D point Above all, it flexibly generates semantics on-demand and on-the-fly, that is, it helps “healing” one of the biggest clouds enables us to transcend explicit geospatial model- ling and, therefore, to overcome complex, heuristics-based weaknesses of 3D point clouds—the lack of structure and semantics. There is virtually no limitation for the specific reconstructions and model-based abstractions. In that regard, types of 3D objects, structures, or phenomena that can be identified and extracted by ML/DL-based 3D point cloud interpretation. In addition, ML/DL-based interpretation operates on the raw geospatial data, i.e. it retains a high degree of Fig. 5 Example of point classification for a typical street space sce- nario. Ground, buildings, vehicles, pedestrians, and different street Fig. 7 Example of a composite classification (data by courtesy of furniture objects are classified with a PointNet-based approach and Stadt Hamburg). Bicycle and cyclist are segregated by DL-based are visualized by different colours classification 1 3 PFG (2020) 88:15–24 23 References Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Cor- rado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow IJ, Harp A, Irving G, Isard M (2016) TensorFlow: large-scale machine learning on heterogeneous distributed systems. CoRR arXiv :1603.04467 Allamanis M, Barr E, Devanbu P, Sutton C (2018) A survey of machine learning for big code and naturalness. ACM Comput Surv 51(4):81:1–81:37 Alpaydin E (2014) Introduction to machine learning, 3rd edn. MIT Press, Adaptive Computation and Machine Learning Fig. 8 The higher the degree of explicit modelling, the less of the Copeland J (1993) Artificial Intelligence: a philosophical introduction. original raw data is preserved as more and more assumptions and Wiley-Blackwell, USA abstraction are introduced. Hence, data details disappear and less gen- de Boer P, Kroese D, Shie M, Rubinstein R (2005) A tutorial on the eral-purpose models result cross-entropy method. Ann Operations Res 134(1):19–67 El Saddik A (2018) Digital twins: the convergence of multimedia tech- nologies. IEEE MultiMedia 25(2):87–92 AI technology can be used to simplify and accelerate work- Gandomi A, Haider M (2015) Beyond the hype: big data concepts, flows for geodata processing and geoinformation systems. methods, and analytics. Int J Inf Manag 35(2):137–144 Of course, crucial ML/DL-related challenges result from Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press. the demand for effective training data and efficient feature http://www.deepl earni ngboo k.org Griffiths D, Boehm J (2019) A review on deep learning techniques representations. for 3D sensed data classification. CoRR arXiv :abs/1907.04444 , Last but not least, ML/DL-based solutions offer simpli - Gross M, Pfister H (2007) Point-based graphics. Morgan Kaufmann fications in the dimension of software engineering. Large Publishers Inc, USA parts of today’s implementations (often historically grown Haenlein M, Kaplan A (2019) A brief history of artificial intelligence: on the past, present, and future of artificial intelligence. Calif with large amounts of so called technical debts) will be par- Manag Rev 61(4):5–14 tically replaced by ML/DL “black box” subsystems, which Hatcher W, Yu W (2018) A survey of deep learning: platforms, applica- have far less software management and software develop- tions and emerging research trends. IEEE Access 6:24411–24432. ment complexities. In particular, most heuristics-based, https ://doi.org/10.1109/ACCES S.2018.28306 61 Hindle A, Barr E, Su Z, Gabel M, Devanbu P (2012) On the Natural- explicitly programmed analysis routines, which tend to be ness of Software. In: Proceedings of the 34th international confer- difficult to parameterize and to configure, can be migrated ence on software engineering, IEEE Press, ICSE ’12, pp 837–847 this way. As a consequence, ML/DL-based approaches, in Humby C (2006) http://www.humby anddu nn.com the long run, have the potential to “eat up” many of today’s Janowicz K, Gao S, McKenzie G, Hu Y, Bhaduri B (2020) GeoAI: spatially explicit artificial intelligence techniques for geographic explicitly programmed GIS implementations. knowledge discovery and beyond. Int J Geogr Inf Sci. https://doi. org/10.1080/13658 816.2019.16845 00 Acknowledgements Open Access funding provided by Projekt DEAL. Jurafsky D, Martin J (2000) Speech and language processing: an intro- We thank Benjamin Hagedorn, Johannes Wolf, Rico Richter, and duction to natural language processing, computational linguistics, Vladeta Stojanovic for their contributions to HPI’s geospatial ML/DL and speech recognition, 1st edn. Prentice Hall PTR, USA research. We also thank the GraphicsVision.AI association for their Kanevski M, Foresti L, Kaiser C, Pozdnoukhov A, Timonin V, Tuia D support by the Málaga research retreat and pointcloudtechnology. (2009) Machine learning models for geospatial data. Handbook com for providing us the PunctumTube 3D point cloud platform. This of theoretical and quantitative geography. University of Lausanne, research work was partially supported by the German Federal Ministry Lausanne, pp 175–227 of Education and Research (BMBF) as part of the research grands for Kelly K (2017) The AI Cargo Cult: the myth of a superhuman AI. Tech. HPI AI Lab, PunctumTube and GeoPortfolio. rep., Backchannel. https://www .wired.com/2017/04/t he-myth-of- a-super human -ai Open Access This article is licensed under a Creative Commons Attri- Khajavi SH, Hossein Motlagh N, Jaribion A, Werner LC, Holmström bution 4.0 International License, which permits use, sharing, adapta- J (2019) Digital twin: vision, benefits, boundaries, and crea- tion, distribution and reproduction in any medium or format, as long tion for buildings. IEEE Access 7:147406–147419. https ://doi. as you give appropriate credit to the original author(s) and the source, org/10.1109/ACCES S.2019.29465 15 provide a link to the Creative Commons licence, and indicate if changes Kitchin R, McArdle G (2016) What makes big data, big data? Explor- were made. The images or other third party material in this article are ing the ontological characteristics of 26 datasets. Big Data Soc. included in the article’s Creative Commons licence, unless indicated https ://doi.org/10.1177/20539 51716 63113 0 otherwise in a credit line to the material. If material is not included in Lecun Y (2017) AI is going to amplify human intelligence not replace the article’s Creative Commons licence and your intended use is not it. FAZ Netzwirtschaft. https ://www.faz.net/-gqm-8yrxk permitted by statutory regulation or exceeds the permitted use, you will LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436– need to obtain permission directly from the copyright holder. To view a 444. https ://doi.org/10.1038/natur e1453 9 copy of this licence, visit http://creativ ecommons .or g/licenses/b y/4.0/. Löwner MO, Gröger G, Benner J, Biljecki F, Nagel C (2016) Proposal for a new LOD and multi-representation concept for CityGML. In: ISPRS annals of the photogrammetry, remote sensing and spatial information sciences, vol 4, pp 3–12 1 3 24 PFG (2020) 88:15–24 Marcus G (2018) Deep learning: a critical appraisal. CoRR arXiv Smith T (1984) Artificial intelligence and its applicability to geographi- :abs/1801.00631 cal problem solving. Prof Geogrh 36(2):147–158 Marr B (2018) Here’s why data is not the new oil. Forbes. www.forbe Stojanovic V, Trapp M, Döllner J, Richter R (2019a) Classification of s.com indoor point clouds using multiviews. In: The 24th international Najafabadi MM, Villanustre F, Khoshgoftaar TM et al (2015) Deep conference on 3D web technology, Web3D, Los Angeles, July learning applications and challenges in big data analytics. J Big 26-28, 2019, pp 1–9 Data 2:1. https ://doi.org/10.1186/s4053 7-014-0007-7 Stojanovic V, Trapp M, Richter R, Döllner J (2019b) Generation of Openshaw S, Openshaw C (1997) Artificial intelligence in geography. approximate 2D and 3D floor plans from 3D point clouds. In: Pro- Wiley, USA ceedings of the 14th international joint conference on computer Palatini K (2014) Joseph Weizenbaum, responsibility and humanoid vision, imaging and computer graphics theory and applications, robots. In: Funk M, Irrgang B (eds) Robotics in Germany and VISIGRAPP 2019, Vol 1: GRAPP, pp 177–184 Japan. Philosophical and technical perspectives. Peter Lang, USA, Su H, Maji S, Kalogerakis E, Learned-Miller E (2015) Multi-view pp 163–169 convolutional neural networks for 3D shape recognition. In: Pro- Posada J, Zorrilla M, Dominguez A, Simoes B, Eisert P, Stricker D, ceedings of the 2015 IEEE international conference on computer Rambach J, Döllner J, Guevara M (2018) Graphics and media vision (ICCV), IEEE Computer Society, ICCV ’15, pp 945–953 technologies for operators in industry 4.0. IEEE Comput Graph Van Der Maaten L, Postma E, Van den Herik J (2009) Dimensional- Appl 38(5):119–132 ity reduction: a comparative review. J Mach Learn Res 10:66–71 Purdy M, Daugherty P (2017) How AI boosts industry profits and inno- Vopham T, Hart J, Laden F, Chiang Y (2018) Emerging trends in vation. http://www.accen ture.com geospatial artificial intelligence (geoAI): potential applications Qi C, Su H, Mo K, Guibas L (2016a) PointNet: deep learning on for environmental epidemiology. Environ Health. https ://doi. point sets for 3D classification and segmentation. CoRR arXiv org/10.1186/s1294 0-018-0386-x :abs/1612.00593 Wang D, Szymanski B, Abdelzaher T, Ji H, Kaplan L (2018a) The age Qi C, Su H, Nießner M, Dai A, Yan M, Guibas L (2016b) Volumetric of social sensing. CoRR arXiv :abs/1801.09116 and multi-view CNNs for object classification on 3D data. CoRR Wang Y, Sun Y, Liu Z, Sarma S, Bronstein M, Solomon J (2018b) arXiv :1604.03265 Dynamic graph CNN for learning on point clouds. CoRR arXiv Qi Q, Tao F (2018) Digital twin and big data towards smart manufac- :1801.07829 turing and industry 4.0: 360 degree comparison. IEEE Access Wang Y, Wang Q, Shi S, He X, Tang Z, Zhao K, Chu X (2019) Bench- 6:3585–3593 marking the performance and power of AI accelerators for AI Qi V, Yi L, Su H, Guibas L (2017) PointNet++: Deep hierarchical training. arXiv :1909.06842 feature learning on point sets in a metric space. CoRR arXiv Weinmann M, Schmidt A, Mallet C, Hinz S, Rottensteiner F, Jutzi B :1706.02413 (2015) Contextual classification of point cloud data by exploiting Reuther A, Michaleas P, Jones M, Gadepally V, Samsi S, Kepner J individual 3D neighborhoods. ISPRS Ann Photogramm Remote (2019) Survey and benchmarking of machine learning accelera- Sens Spatial Inf Sci II–3/W4:271–278 tors. In: 2019 IEEE high performance extreme computing confer- Weizenbaum J (1966) ELIZA—a computer program for the study ence (HPEC), pp 1–9 of natural language communication between man and machine. Richter R (2018) Concepts and techniques for processing and rendering Commun ACM 9(1):36–45 of massive 3D point clouds. PhD thesis, University of Potsdam, Wolf J, Richter R, Döllner J (2019) Techniques for automated classi- Faculty of Digital Engineering, Hasso Plattner Institute fication and segregation of mobile mapping 3D point clouds. In: Roveri R, Rahmann L, Öztireli C, Gross M (2018) A network archi- Proceedings of the 14th international joint conference on com- tecture for point cloud classification via automatic depth images puter vision, imaging and computer graphics theory and applica- generation. In: 2018 IEEE conference on computer vision and tions, VISIGRAPP 2019, vol 1: GRAPP, pp 201–208 pattern recognition, CVPR 2018, Salt Lake City, UT, USA, June Zhang C, Vinyals O, Munos R, Bengio S (2018) A study on overfitting 18–22, 2018, pp 4176–4184 in deep reinforcement learning. CoRR arXiv :1804.06893 Rozsa A, Günther M, Boult T (2016) Are accuracy and robustness cor- Zollhöfer M, Stotko P, Görlitz A, Theobalt C, Niessner M, Klein R, related. In: 2016 15th IEEE international conference on machine Kolb A (2018) State of the art on 3D reconstruction with RGB-D learning and applications (ICMLA), pp 227–232 cameras. Comput Graph Forum 37:625–652 1 3

Journal

PFG – Journal of Photogrammetry, Remote Sensing and Geoinformation ScienceSpringer Journals

Published: Feb 26, 2020

There are no references for this article.