Deep Generative Models in Engineering Design: A Review

Lyle Regenwetter; Amin Heyrani Nobari; Faez Ahmed

doi:10.1115/1.4053859

Deep Generative Models in Engineering Design: A Review

Regenwetter, Lyle;Heyrani Nobari, Amin;Ahmed, Faez 2022-02-16 00:00:00 Deep Generative Models in Engineering Design: A Review Lyle Regenwetter Amin Heyrani Nobari Dept. of Mechanical Engineering Dept. of Mechanical Engineering Massachusetts Institute of Technology Massachusetts Institute of Technology Cambridge, MA 02139 Cambridge, MA 02139 Email: [email protected] Email: [email protected] Faez Ahmed Dept. of Mechanical Engineering Massachusetts Institute of Technology Cambridge, MA 02139 Email: [email protected] Automated design synthesis has the potential to revolu- 1 Introduction tionize the modern engineering design process and improve The human design process is a ubiquitous element of access to highly optimized and customized products across modern society, playing a critical role in the technologies countless industries. Successfully adapting generative Ma- producing the food we eat, the products we use, and the chine Learning to design engineering may enable such au- spaces in which we live. Accelerating the design process tomated design synthesis and is a research subject of great through automation can reduce cost and increase industrial importance. We present a review and analysis of Deep productivity, which would be immensely desirable for global Generative Machine Learning models in engineering de- productivity and prosperity. Integrating AI into the design sign. Deep Generative Models (DGMs) typically leverage process can alleviate dependence on human experts and rev- deep networks to learn from an input dataset and synthesize olutionize user customizability, providing specialized prod- new designs. Recently, DGMs such as feedforward Neural ucts for individual users without the prohibitive cost of man- Networks (NNs), Generative Adversarial Networks (GANs), ual design. Driven by the widespread potential to advance Variational Autoencoders (VAEs), and certain Deep Rein- global equity and prosperity through design automation, forcement Learning (DRL) frameworks have shown promis- methods such as “generative design” have recently emerged ing results in design applications like structural optimiza- alongside advanced computing and automation technologies. tion, materials design, and shape synthesis. The prevalence “Generative design” is the process in which algorithms of DGMs in engineering design has skyrocketed since 2016. directly synthesize designs either via explicit programming Anticipating continued growth, we conduct a review of re- or implicit learning. Early generative design methods leaned cent advances to beneﬁt researchers interested in DGMs for heavily on explicit programming of human design exper- design. We structure our review as an exposition of the tise through manually-deﬁned design representation methods algorithms, datasets, representation methods, and applica- like grammars [1]. While practical for explicitly encoding tions commonly used in the current literature. In particu- design constraints and objectives, these rule-based frame- lar, we discuss key works that have introduced new tech- works ignored opportunities for implicit leaning on infor- niques and methods in DGMs, successfully applied DGMs mation and knowledge encoded in the vast expanse of exist- to a design-related domain, or directly supported the devel- ing designs. As the availability of computational resources opment of DGMs through datasets or auxiliary methods. We increased over the past decade, data-intensive methods like further identify key challenges and limitations currently seen deep learning opened doors to successfully automate com- in DGMs across design ﬁelds, such as design creativity, han- plex human tasks such as image processing and natural lan- dling constraints and objectives, and modeling both form guage processing. and functional performance simultaneously. In our discus- In deep learning, data is propagated through sequential sion, we identify possible solution pathways as key areas on layers to learn progressively higher-level meaning, an ar- which to target future work. chitecture generally known as an Artiﬁcial Neural Network (ANN) or just Neural Network (NN) [2]. Most of the deep learning-based approaches pioneered during the 2010s lever- 1 Copyright © by ASME arXiv:2110.10863v4 [cs.LG] 16 Mar 2022 aged extensive quantities of data to avoid explicit feature and creativity into DGMs. Despite these advancements, engineering. This trend is mirrored in engineering design DGMs for engineering design are still in their infancy and with algorithms learning data distributions instead of requir- will require further efforts to effectively overcome these fun- ing them to be predeﬁned. Among these algorithms are Deep damental challenges. Generative Models (DGMs) — deep learning models that Our primary goal in this work is to help build cohe- can approximate complicated, high-dimensional probability sion between the countless active researchers in the design distributions using a large dataset. In this review paper, we ﬁeld working with DGMs and furthermore provide a start- speciﬁcally deﬁne “Deep Generative Models” as algorithms ing point for researchers entering the ﬁeld. In particular, we that are capable of generating new samples using deep learn- seek to provide researchers with a reference guide in plan- ing. Generative Adversarial Networks (GANs) and Varia- ning projects in the data-driven generative design space. To tional Autoencoders (VAEs) are two classes of DGMs that this end, we provide an overview of common methods and have demonstrated compelling synthesis of images, text, and tools (Sec. 2), a discussion of different data parameterization tabular data in numerous domains. Considering that images, methods (Sec. 3), a review of potentially relevant research text, and tabular data are all common representation methods across various design domains (Sec. 5), an overview of rele- for design, one might assume that DGMs should be capable vant datasets (Sec. 6), and an analysis of common challenges of synthesizing full designs as well with relative ease. How- in the ﬁeld (Sec. 7). Figure 1 provides an overview of the ever, several unique properties of the generative design task standard process to apply DGMs in engineering design. pose particular challenges for DGMs. Many of these chal- lenges are so fundamental that the future success of DGMs in engineering design is largely contingent on the ability to 2 Overview of Deep Generative Models overcome them. We list four of these challenges below: Deep Generative Machine Learning approaches share the goal of high-quality synthesis but signiﬁcantly vary in 1. Modeling design performance: Real-world functional methodology. In practice, we identify four common ap- performance is critical in many engineering design proaches to generate designs: direct generation using deep tasks. Developing performance-aware DGMs capable neural networks (DNN), adversarial generation with Gener- of synthesizing designs for a given set of target require- ative Adversarial Networks (GAN), generation from embed- ments (a process termed as inverse design) is a challeng- ding vectors using Variational Autoencoders (VAE), and se- ing task that is exacerbated by the computational cost of quential generation using Reinforcement learning (RL). In numerical simulation and the even greater difﬁculty of the design community, we observe that GANs, VAEs, and real-world evaluation. RL are most commonly used for design synthesis. While 2. Data sparsity: Compared to other research ﬁelds like DNNs as well as extensions like recurrent neural networks Computer Vision, which have massive publicly avail- (RNNs) are occasionally used for direct design synthesis, able datasets, the availability of large, well-annotated, they are more frequently used for non-generative tasks. In public datasets in engineering is severely lacking. Fur- this section, we brieﬂy discuss the background and method- thermore, even when data is available, the distribution ology of GANs, VAEs, and RL. of the data often does not cover the design space evenly, with much sparsity often observed in the data. 3. The creativity gap: In conventional DGM applications, 2.1 Generative Adversarial Networks the overarching goal is to mimic the training data and Originally introduced in 2014, Generative Adversarial emulate existing designs. In engineering design, emula- Networks (GANs) [3] found initial success with convincing tion of existing products is often undesirable. Designers image synthesis performance [4, 5, 6, 7]. We provide an in- typically aim to introduce products with novel features troduction of GANs but refer the reader to [8] for a detailed to target new market segments. overview. A generative adversarial network [3] consists of 4. Usability and feasibility: For synthesized designs to be two models — a generator and a discriminator. The genera- physically fabricated, they must be physically feasible. tor G maps an arbitrary noise distribution to the data distribu- Furthermore, designs must be encoded in a data repre- tion, in our case the distribution of designs, and can thus gen- sentation that contains enough parametric detail to be erate new data; simultaneously, the discriminator D learns to converted into a representation usable for fabrication. distinguish between real and generated data. Both models are usually built with deep neural networks. As D improves, Over the past few years, the design community has made G also improves as it learns to generate data that fools D. substantial progress in using generative machine learning (ML) models to create new designs. DGMs have been ap- plied to a broad range of design tasks such as structural opti- Common challenges in training GAN models: GANs mization, materials design, and shape synthesis. Over time, are often considered difﬁcult to train, suffering from train- researchers have introduced increasingly advanced methods, ing instability stemming from several sources [9, 10]. One which have begun to address some of the above challenges. common issue in GAN training occurs when the discrimi- For example, many works have proposed approaches to in- nator overpowers the generator, easily distinguishing gener- corporate design performance and optimization into DGM ated samples, and causing the gradient to the generator to training. Other works have explored incorporating novelty vanish, effectively halting the generator’s training. This is- 2 Copyright © by ASME Fig. 1: This ﬁgure outlines the typical components of design synthesis problems using Deep Generative Models. Some design problems are more suitable for speciﬁc design representation methods, which also inﬂuences the type of deep generative model architectures required. sue has been addressed by many researchers. For example, GAN conditioning: In the design domain, we often have the WGAN replaces the discriminator with a critic, modiﬁes design constraints, requirements, and objectives that any the GAN’s loss function to estimate the Wasserstein (Earth generated design should satisfy. To use DGMs for such prob- Mover’s) distance between the original data and generated lems, these requirements should be imposed on them. For data distributions, and modiﬁes the training process [11, 12]. example, we may seek to train a DGM to generate bikes, but depending on our user, we may want to constrain it to gener- Another problem that GANs face is the issue of “mode ate only roadbikes or mountain bikes without retraining for collapse,” where the generator fails to encompass all modes each generation task. Model conditioning is one method to in the data distribution or even generates only a handful of do this. Several proposed approaches add conditioning to unique samples that are capable of fooling the discrimina- the GAN using a condition vector which is intended to be tor. To overcome these issues, researchers have developed interpretable. Typically GANs are discretely conditioned by novel algorithmic techniques [13, 14, 15] to reward diversity feeding the condition vector into both the generator and dis- in samples. 3 Copyright © by ASME criminator, in a conﬁguration known as a Conditional GAN actor after taking actions, based on the effects of said actions (cGAN) [16]. Instead of feeding the condition vector into on the environment. In this scenario, the actor’s goal is to the discriminator, an auxiliary network and cross entropy maximize the rewards it receives by making decisions (i.e., loss can instead be used to reconstruct the condition vec- taking actions) such that the total reward is maximized. From tor from the generated samples in a conﬁguration known as this point of view, reinforcement learning can be thought of an Information Maximizing GAN (InfoGAN) [17]. Con- as an approach similar to optimization, where an objective ditioning is also essential in design applications where in- (maximizing the reward) is being optimized. verse design is being done on performance metrics, which One of the ﬁrst attempts at introducing deep learning to often exist in continuous spaces (e.g., stiffness, lift coef- the reinforcement learning approach was done in 2013 by ﬁcient, drag coefﬁcient, density, etc.). Researchers have Mnih et al., when they introduced deep learning to a rein- come up with continuous conditioning solutions for GANs forcement learning process known as Q-Learning [26]. Q- such as the Regressional GAN [18], continuous conditional Learning refers to learning the state-action value function GAN (CcGAN) [19] and performance conditioned diverse or Q-function, which is a progressively updated estimate of GAN (PcDGAN) [20]. the expected reward to be received from taking a particular action in a particular state. Mnih et al., attempted to learn the Q-function using convolutional neural networks (CNN). 2.2 Variational Autoencoders Many Deep RL techniques have been introduced since this Introduced in 2013, Variational Autoencoders found sig- ﬁrst work by Mnih et al.. Further exploration of the details niﬁcant success in many machine learning applications. Au- of these approaches is left to the reader. toencoders are unsupervised embedding algorithms consist- In practice, when applying RL to design applications, ing of an encoder that maps an input design into a (typi- the design process is usually broken down into a sequential cally) lower-dimensional latent space and a decoder that re- process of building a design or altering existing designs in constructs the design as accurately as possible from the la- steps (i.e., actions taken to alter or expand the current state tent space. The encoder and decoder are conventionally im- of a design being generated) and the reward is measured by plemented using deep neural networks. To generate new the quality or performance of the resulting design (i.e., the samples, latent vectors are sampled from the latent space environment). While RL requires no dataset, this advantage and fed through the decoder. Typically, the distribution of is balanced by dependence on meaningful and reliable re- the real data mapped to the latent space of an autoencoder ward signals, which may often require a high-ﬁdelity simu- is sparse, meaning that sampling a realistic latent vector lation environment. One major beneﬁt of RL over GANs and is difﬁcult. This limitation is addressed with the introduc- VAEs is the fact that the reward function can be set based on tion of the Variational Autoencoder (VAE), ﬁrst proposed by any objective which does not need to be differentiable. In Kingma et al. [21]. The Variational Autoencoder adds in a contrast, any objective added to the loss function of a GAN probabilistic sampling in the latent space that regularizes the or VAE must be differentiable since GANs and VAEs are latent distribution. In practice, the VAEs’s encoder outputs n trained using the gradient-based optimization [15]. means and n variances, from which n-dimensional latent vec- tors are sampled before decoding. To maintain a predictable latent space distribution, The VAE adds a Kullback-Liebler 3 Overview of Design Representation Methods (KL) divergence [22] loss between the distribution of the la- In this section, we discuss common design representa- tent space and a standard Gaussian. Interested readers are tion methods seen in DGMs for engineering design which encouraged to refer to the literature [23] for a more detailed are visualized in Section 2 of Figure 1. We include a deﬁni- overview. tion and discuss the pros and cons of each method. Conditional VAEs: Just as we do for GANs, we may also seek to condition VAE training on design constraints or user 3.1 Images preferences. The VAE has a natural advantage over the GAN Design data often comes in the form of images (e.g. in that its latent space is typically already structured. Since microstructure scans) or can be represented in image form this structure may be fairly weak and difﬁcult to interpret, ex- (e.g. Topology Optimization). An image consists of a rect- plicitly conditioning VAEs may still be desirable. The Con- angular grid of pixels, each of which contains a color pa- ditional VAE (cVAE) [24] extends on the conventional VAE rameter. They are commonly represented by third-order ten- by adding a conditioning vector as an input to both the en- sors (height widt h channels). Common color schemes coder and decoder and helps achieve this goal. are black-and-white (boolean color channel), grayscale (in- teger color channel), and color (3-4 integer color chan- 2.3 Reinforcement Learning nels). Pros: The image is an information-rich represen- Reinforcement Learning fundamentally differs from the tation and can capture many details of a design. The use other DGMs discussed in that it learns without a dataset in an of convolution/convolution-transpose ﬁlters in deep learn- unsupervised fashion through a large set of trial and error in- ing provides a convenient tool for learning/generation of teractions between an actor and an environment [25]. This is both high-level and low-level features as well as upsam- typically done through some reward signal being sent to the pling/downsampling. Many cutting-edge ML techniques are 4 Copyright © by ASME pioneered in the computer vision domain and are often di- 3.4 Meshes rectly applicable to images. Cons: Representing designs us- Meshes are a common method to represent objects in 3D ing pixels means that the generated design images can be space. Triangular meshes are by far the most commonly used infeasible for downstream tasks. Accurately fabricating de- form. Triangular meshes are the native representation used in signs based on images can be difﬁcult or impossible. Even many computer graphics algorithms and software, as well as performance evaluation using conventional simulation tools many Finite Element tools. Pros: Meshes can be directly vi- like FEA or CFD can require an intermediate conversion sualized and simulated in many FEA or CFD tools, enabling from an image to a 3D model. The poor usability of im- easy pipelines for performance evaluation using numerical ages is exacerbated by the prevalence of artifacts in many methods. A mesh can be considered a specialized type of applications (hanging pixels, disconnected geometry, etc.). graph and can leverage graph operators like graph convolu- Artifacts are especially common when training on (typically) tional operators. Cons: In contrast to other representations small datasets in the design domain since training is often ter- like voxelizations and point clouds, meshes are more chal- minated early due to over-ﬁtting concerns. All in all, images lenging to directly generate using Machine Learning meth- can be considered surrogate representations of engineering ods, despite recent advances in algorithms that directly gen- designs and may lack domain knowledge and information on erate meshes [29, 30, 31]. the physical realization of the design. Therefore, DGMs us- ing images as representations often have a gap between the 3.5 Signed Distance Functions generated images and the actual design they are representing. The Signed Distance Function/Field (SDF) is a repre- sentation method that consists of a (typically 3D) functional map from a coordinate point to an SDF value. The magni- 3.2 Voxelizations tude of this value indicates the distance to the nearest point Voxels are 3D grid points that are effectively the 3D on the surface of the object and the sign indicates whether the equivalent of pixels. As such, voxelizations share many char- point is inside or outside the object. SDFs themselves can acteristics with images. In practice, voxels aren’t conducive be represented in many ways, for example, as a rasterized to ‘color’ parameterization and are typically represented as grid in which each ‘voxel’ contains a continuous numerical booleans (space vs. object). This effectively makes them value denoting the SDF value at that point. Pros: SDFs can third-order tensors with dimension (height widt h de pt h). serve as a convenient intermediate parameterization for many Pros: Voxelizations support 3D convolution which can learn learning tasks. Cons: Like point clouds or voxels, SDFs are high-level and low-level features in 3D. Cons: Compared to difﬁcult to use in downstream tasks without ﬁrst converting images and other representations, the curse of dimensional- to BRep or polygonal representations. ity is especially pronounced with voxels, with the number of parameters scaling with the cube of spatial resolution. 3.6 Parameterizations Voxelizations share the same issues as images. Their us- We use the term “parametric” data to encompass any de- ability is limited in downstream tasks and artifacts are very sign representation consisting of a collection of design pa- prevalent. Like images, voxelizations serve as surrogate rep- rameters where any spatial or temporal signiﬁcance of pa- resentations (often representing CAD models which origi- rameters is unknown. Most parametric data can be orga- nate from parametric representations or 3D shapes which nized in tabular form with each row being a collection of originate from meshes). Like many other representations parameters representing a single design and each column de- such as point clouds and Signed Distance Fields, voxeliza- scribing a design parameter. Tabular design data often con- tions often require conversion before they can be used in sists of a collection of mixed-datatype parameters where re- downstream tasks. For example, they are often converted lations between these parameters may be unclear or nonex- to Boundary Representation (BRep) or polygonal represen- istent. Since parametric data may come in many varieties, tations, which are often the native parameterizations of ren- the pros and cons discussed may not apply to every case. dering and graphics software, Finite Element Analysis, and Pros: Quality parametric data is typically very information- Computational Fluid Dynamics simulation. dense (i.e. requiring fewer parameters to encode the same level of geometric detail). Whereas spatially-organized rep- resentations such as pixels or voxels encode designs with 3.3 Point Clouds uniform information density, parametric data can contain Point Clouds are simple collections of points, often in more detail in design-critical areas without the need for up- 3D space, which are deﬁned to be within some object. Pros: sampling the entire representation. This information density Point Clouds can represent arbitrarily complex geometry often comes with a lower dimensionality which can make with a ﬁnite number of points, though ﬁdelity may vary. optimization of parametrically represented designs signiﬁ- Point Clouds are often the native output of 3D scanning soft- cantly easier. Parametric data may also be more support- ware, making them relatively easy to create [27]. Cons: Like ive of downstream tasks, especially if design parameters are Voxelizations and Signed Distance Functions, Point Clouds human-interpretable. For example, a detailed enough design often require conversion to BRep or polygonal representa- parameterization may allow generated designs to be directly tions such as meshes [28] for downstream tasks. fabricated using conventional (non-additive) manufacturing 5 Copyright © by ASME techniques. Human-interpretable parameterizations can also and their success in molecular graph generation [50, 51], give human designers a tractable method to interact with gen- there is less usage of graph-based DGMs in the design erative methods to allow for human-in-loop design. Finally, community, possibly due to the lack of graph-based design design parameters can sometimes be directly linked to the datasets. latent space of a generative method, as demonstrated in nu- merous works [32, 33], creating a pipeline to directly condi- tion design generation on high-level design goals. Linking 4 Literature Review Methodology design parameters with latent variables has several potential Sec. 5 discusses speciﬁc works that apply Deep Genera- advantages, such as enabling more effective optimization or tive Models to engineering design or make advancements to inverse design using generative methods. Cons: Learning existing generative ML methods in the context of engineer- parametric data can be particularly challenging. Parametric ing design. We consider works based on a predeﬁned scope, data commonly uses mixed datatypes and inherits the train- with each work we discuss meeting the following selection ing challenges of the constituent components. Multimodal criteria. Note that these criteria only apply to the engineer- distributions, skewed categories, non-Gaussian distributions, ing design papers presented and that the works we cite to add data sparsity, and poor data scaling additionally make the ap- context to or substantiate the discussion of fundamentals, ap- plication of DGMs and training very difﬁcult. Since methods plications, and datasets need not adhere to these rules. that are robust to all of the mentioned challenges are difﬁcult 1. We limit our consideration speciﬁcally to papers involv- to come by, successfully applying existing methods to the ing Deep Generative Models, focusing on Variational parametric data domain can be hard. Finally, since paramet- Autoencoders, Generative Adversarial Networks, and ric data may be nontrivial to convert to 3D models, generated Reinforcement Learning in particular. Works must uti- parametric designs may be challenging to evaluate using nu- lize deep learning. Works only considering design opti- merical simulations or through qualitative visualization. mization are excluded. 2. We only consider work speciﬁcally relevant to engineer- 3.7 Grammars ing design and not other domains (such as computer sci- Grammars are representation methods consisting of ence). variables, terminal symbols, nonterminal symbols, and a set 3. We consider only work published between Jan. 2014 and of rules. Rules describe how non-terminal symbols can ex- Sep. 2021, when we conclude our review, as many piv- pand into other terminal and nonterminal symbols. The most otal works in deep learning (CNNs, VAEs, GANs) were prevalent grammars in engineering design are graph and spa- introduced in this period. tial grammars [1], and they have been applied in a wide To identify works, we speciﬁcally searched for “Generative variety of applications, such as the design of satellites and Adversarial Network,” “Variational Autoencoder,” and “Re- electro-mechanical systems. Grammars can be especially inforcement Learning” in Google Scholar. We initially con- useful to dictate feasible assembly hierarchies of design com- ﬁned our search to a set of known design venues, speciﬁ- ponents as in [34]. Pros: Grammars can explicitly con- cally the Journal of Mechanical Design, the Proceedings of strain design spaces to feasible or desirable regions by nature the International Design Engineering Technical Conferences, of their construction, thereby encoding domain knowledge. Computer-Aided Design Journal, International Conference Cons: Grammars are challenging to implicitly learn and of- on Engineering Design, and Artiﬁcial Intelligence for Engi- ten must be manually deﬁned. Grammars can also restrict the neering Design, Analysis, and Manufacturing Journal. This exploration of the design space. A survey on grammar-based helped us identify an initial set of seed papers. From all design synthesis approaches is provided in [1]. search results identiﬁed through these methods, 41 papers were deemed to be relevant and included. The relevance was 3.8 Graphs decided by independent assessment by two raters, who are Graphs are a highly ﬂexible representation method con- also authors of this paper. For papers where there was a sisting of nodes and edges, which can be directed or undi- disagreement, all authors discussed them and mutually de- rected. Graph-based representations have proven success- cided on their classiﬁcation. Next, papers cited in these seed ful for many different aspects of design generation and opti- papers were considered and added to this paper. 22 papers mization and provided avenues for describing complex sys- from other venues were deemed to be relevant and included tems efﬁciently. Pros: Graphs are highly adaptable and are as well. Of the 63 papers, 48 were published between Jan. capable of representing many different kinds of complex sys- 2019 and Aug. 2021, while a mere 15 were published be- tems and designs [35, 36, 37, 38]. Graphs also provide repre- tween Jan. 2016 and Dec. 2018, indicating strong growth in sentations for design processes and modeling complex inter- the ﬁeld in recent years. actions in systems [39,40] which may enable methods for au- tomating the design of systems or modeling inter-part depen- dencies. Graph neural networks (GNNs) [41, 42, 43, 44, 45] 5 Application Domains in Engineering Design provide an excellent tool for machine learning on graphs. Engineering design encompasses a wide variety of ap- Cons: Despite the developments of graph neural networks plications, ranging from designing aircraft models to small- (GNNs) in the computer science community [46, 47, 48, 49] scale metamaterials. To structure different types of applica- 6 Copyright © by ASME Table 1: Characterization of application studies by domain and architecture. Some works use multiple architectures or ﬁt into multiple application domains and are listed more than once. We additionally classify works by datatype used: Image , c v m s g h p Point Cloud , Voxelelization , Mesh , Signed Distance Field , Grammar , Graph , and Parametric (other parameterization) Topology Optimization Materials 2D Shape Synthesis 3D shape synthesis Other Domains i vi p i p i i Plain NN [52] [53] [54] [55] [56] [57] [58] i i i i i vi v i vi c GAN [59] [60] [61] [62] [63] [64] [65] [66] [67] [68] [69] i i i i p p p ph GAN+Conditioning [70] [71] [72] [73] [74] [75] [76] [33] i p v i New GAN-based [72] [15] [77] [78] i i i i i i v p p p AE/VAE [79] [80] [81] [82] [32] [83] [84] [85] , [86] , [87] i † i p i cVAE [88] [89] [85] [90] i s i p New VAE-based [91] [92] [93] p p i p g i p p p RL-based [94] [95] [96] [34] [97] [98] [99] v i i i i i i i p v p g i p i p g Other Method [100] [101] [102] [103] [67] [104] [105] [106] [95] [100] [107] [34] [108] [109] [110] i i vi i i i +Style Transfer [79] [64] [65] [80] [57] [104] i i s +Genetic Alg. [82] [105] [92] i i p +Bayesian Opt. [64] [83] [76] *Model-based reconstruction using parameters in a lower-dimensional space † Technically not images, but use an image-like parameterization structure tions using DGMs, we grouped them into a few categories of TO algorithms to rapidly converge, saving computational application areas, which are discussed in this section and are cost. We consider a baseline for DGMs in TO, in which reported in Table 1. an existing generative architecture is applied to a dataset of TO-generated designs and the network is trained to gener- ate samples mimicking the training data. A few papers fall 5.1 Deep Generative Models in Topology Optimization within this category of methods: Rawat and Chen [60] train a WGAN on a dataset generated through TO and train an auxiliary network to additionally predict performance met- rics. Sharpe and Seepersad [70] expand on this baseline with a cGAN conditioned on volume fraction and load location. Guo et al. [79] train a VAE with an additional style trans- Fig. 2: Sample topologies generated by 3D Topology Opti- fer [119,120] loss on a TO-generated dataset for heat transfer mization and propose iterative strategies for targeted design optimiza- Topology Optimization (TO) is a research ﬁeld with a tion using the VAE’s latent space. Style transfer is discussed long history of research and methods. TO searches a de- further in Sec. 5.2. sign space to ﬁnd an ideal spatial distribution of material to optimize some predeﬁned objective. Common areas of ap- Iterative DGM training, ﬁltering, and human guidance plication include solid mechanics [111, 112], ﬂuid dynam- in TO: Oh et al. [62, 61] propose a method that expands ics [113, 114], additive manufacturing [115, 116], and heat on the baseline generative-network-ﬁtting by iteratively syn- transfer [117, 118]. While Topology Optimization is con- thesizing new designs, optimizing them in TO, then drop- sidered a method for generative design, standard TO does ping designs that are too similar from the full collection of not leverage deep learning and is not considered a DGM. In designs. The authors use a modiﬁed Boundary Equilibrium recent years, however, several papers have proposed meth- GAN (BEGAN) [121], which extends on the WGAN and be- ods to use TO and DGMs together, in many cases to address gin training on a collection of existing TO-generated topolo- the computational cost of TO on large amounts of data. De- gies. However, unlike the previously discussed generative- spite differences between architectures, we found that many network-ﬁtting approaches used in literature [60, 70, 79], the DGMs for TO share certain characteristics. In particular, retraining and re-optimization allows the framework to gen- due to the predominant use of voxelized or pixelized rep- erate, optimize, and explore new areas of the design space. resentations, many hybrid TO-Generative ML models use The proposed method is applied to the problem of wheel de- methods originally developed for computer vision appli- sign, with the key motivation being to ﬁnd a trade-off be- cations, such as convolutional neural networks and super- tween the aesthetics of real designs and the structural per- resolution [59, 53]. formance of designs found through TO. The authors present strong empirical results on this wheel design problem. Supervised generation of optimized topologies: To avoid the computational cost of Topology Optimization, DGMs The issue of gaps in the design space of topolo- have been used to predict ﬁnal optimized topologies di- gies is also addressed by Fujita et al. [101], who pro- rectly. This approach can also be used as an initialization pose an approach leveraging a Variational Deep Embedding technique for conventional TO, which allows conventional (VaDE) [122]. An initial dataset is generated using TO. The 7 Copyright © by ASME proposed method sequentially identiﬁes voids in the design culating optimal solutions (using TO) for proposed solutions space using the VaDE, decodes designs from the void, opti- that are in greatest violation of these conditions. mizes them using TO, then adds them to the training dataset. Although the above works have proposed methods with Other DGM approaches in topology generation: Sos- automated retraining steps, human input can also be injected novik and Oseledets [52] propose to use gradient informa- into the training process. For example, Valdez et al. [73] tion from a topology distribution to inform estimates of a propose a cGAN-based human-in-loop topology design gen- ﬁnal topology through a DNN. Their DNN takes both the eration framework in which the designer iteratively selects density distribution and gradient of this density distribution design clusters to gradually hone in on preferable designs. from some intermediate step of a TO process to generate an estimate of the ﬁnal topology without waiting for TO con- DGMs for TO that utilize super-resolution: Since one vergence. The authors demonstrate binary accuracy of 98% of the key limitations of Topology Optimization is computa- when predicting the ﬁnal topology after only ﬁve iterations tional cost, which scales with the resolution, a major focus of TO, when the full TO process would take 100 iterations to in DGMs for TO has focused on attaining high-resolution complete. TO-like results. A common approach uses super-resolution, In a different kind of approach, Keshavarz- which is a technique used in computer vision to convert low- zadeh et al. [54], address the generalization problem resolution images to high-resolution ones. Several works of training for different scales and different domains. expand on the baseline DGM-in-TO framework by learn- They parametrically represent shapes in both 2D and 3D ing from a low-resolution TO-generated dataset, then per- using their proposed “Disjunctive Normal Shape Model” forming super-resolution on synthesized topologies, such (DNSM) [54]. Using this DNSM, they create a platform as Yu et al. [72], who add a cGAN for upscaling. A for resolution-independent shape reconstruction. They then similar approach proposed by Li et al. [59] uses a Super- train an NN model to generate optimal topologies given Resolution GAN (SRGAN) [123] to generate high-resolution boundary conditions and other problem-speciﬁc information optimal topologies for heat transfer problems after generat- in the DNSM space, which then can be reconstructed at any ing low-dimensional topologies on a different GAN. Other resolution using the DNSM. They demonstrate that their researchers have proposed transfer learning for the super- DNSM method overcomes the super-resolution problem resolution task instead of using GANs or VAEs for super- across a variety of problems and domains. resolution. For example, Behzadi et al. [53], train a feedfor- ward NN-based model to predict optimized topologies di- 5.2 Microstructure, Nanostructure, and Metamaterials rectly without any discriminator and using the MSE loss. Once they have trained their model on the low-resolution data, they lock the model weights and add a few layers to the model to increase the output resolution and only train said layers to obtain high-resolution samples. This avoids the need for a large quantity of high-resolution data or the extra Fig. 3: Deep Generative Models are often employed to gen- time required to train a high-resolution model from scratch. erate 3D Metamaterial Unit cells Many papers that have applied super-resolution techniques demonstrate that after initial training, their framework con- Many design applications not only require the need to sistently generates near-optimal topologies many times faster design the topology or shape of the artifact, but also the ma- than classic TO [72, 59]. terial properties within it. Inverse design of materials is one of the key elements of Computational Materials Science. The Improving DGM-based topology generation using physi- most common approach to developing inverse materials de- cal properties: Several papers have succeeded in improv- sign frameworks is the development of Process-Structure- ing the baseline performance of DGMs for topology gen- Property (PSP) links, i.e. understanding how a particular eration through the use of physical properties of the de- material processing approach impacts its microstructure and sign domain. For example, Nie et al. [71] propose to adapt how the corresponding microstructure impacts its physical the cGAN architecture Pix2Pix [124] to generate synthetic properties [128]. Designing material microstructures for di- topologies based on spatial ﬁelds of various physical param- rect use is difﬁcult, as there must then be some fabrication eters (displacement, strain energy density, and Von Mises process to generate a material with the target microstructure. stress) as input to the generator. Topologies generated by A common goal in computational materials science is mi- TO are taken as ground truth for training. The authors crostructure image “reconstruction,” in other words, generat- also propose a new generator architecture combining the ing microstructure images that exhibit certain characteristics. Squeeze and Excitation ResNet [125, 126] with U-Net [127]. Bostanabad et al. [128] give an overview of the Microstruc- Cang et al. [55] use a neural network to generate optimized ture Characterization and Reconstruction (MCR) ﬁeld. Re- topologies from loading conditions. They propose an ap- construction can accelerate downstream tasks like augment- proach to rapidly evaluate the deviation of proposed solu- ing the training data of networks that attempt to model PSP tions from the problem’s optimality conditions. They then links. Better modeling PSP links can in turn create more ac- progressively augment their dataset during training by recal- curate generative material design pipelines. Many classes of 8 Copyright © by ASME DGMs have been applied to this reconstruction task, includ- Liu et al. [68]. ing GANs [64], VAEs [80] and Convolutional Deep Belief Networks (CDBNs) [129] [103, 102, 106]. Other studies at- Photonics and phononics: Within microstructure design, tempt to bridge the gap between microstructure and proper- the design of materials with photonic or phononic proper- ties in generative tasks using trained black-box surrogates, ties is an area of interest in numerous industries including such as Tan et al.’s work [63]. Most research focuses on 2D sensing, communications, and display technology. The work microstructure images, though several also consider 3D vox- by Yang et al. [64] is an example of a subﬁeld of genera- elizations [66, 68]. tive microstructure design targeting the development of mi- Since reconstruction often involves mimicking exist- crostructures with particular photonic or phononic proper- ing microstructures, several papers have applied style trans- ties. Molesky et al. [131] also provide a review of inverse fer [119, 120] to the problem as part of a DGM architecture. design in nanophotonics. Style transfer is in essence a loss between two images that In the photonics and phononics subﬁeld, performance attempts to capture the difference in “style” between the im- evaluations such as the one used in [64] are frequently ages. It is typically calculated by comparing the interme- used to incorporate performance into DGMs. For exam- diate layers of an auxiliary style convolutional neural net- ple, the technique of optimization using learned mappings work. Cang et al. [80], for example, propose a VAE with from the latent space of a trained autoencoder to the property style transfer which is targeted at applications where only a space has been employed in several papers. Li et al. [81], small set of training data is available. Li et al. [57] also uti- Liu et al. [82], and Wang et al. [105] train an autoencoder, lize style transfer in their proposed transfer learning recon- VAE, and Gaussian Mixture VAE [132] respectively on im- struction framework. In another approach using style trans- ages of microstructures. They map the latent variables to fer, this time using a vanilla GAN, Yang et al. [64] expand the property space using a DNN, CNN, and Gaussian Pro- on the reconstruction task by attempting inverse design on cess Regressor, respectively. Li et al. [81] directly optimize the structure-property link. After training their GAN with the properties using the DNN, while Liu et al. [82], and style transfer loss and an additional loss to penalize mode Wang et al. [105] optimize using a Genetic Algorithm. collapse, they treat the noise vectors as input variables and Though optimizing latent variables is a common ap- optimize them with Bayesian optimization, using the gener- proach, several other DGMs have been proposed for pho- ator to translate between design vectors and microstructure tonics/phononics design that optimize performance in other images. The authors choose to optimize microstructures for ways. For instance, Malkiel et al. [56] uniquely use a para- energy absorption which can be evaluated from images us- metric representation and a bidirectional DNN to simulta- ing coupled-wave analysis [130], providing them a conve- neously learn a bidirectional mapping between nanostruc- nient structure-property link that would be more challenging tures and the property space. Ma et al. [91] propose a novel if optimizing for other objectives. VAE-based framework consisting of feature extraction, pre- More recently, Fokina et al. [104] adapt StyleGAN [7] diction, recognition, and generation networks. The proposed to the microstructure image synthesis domain. Other stud- method is capable of both forward prediction of properties ies impose the “style” of generated microstructure images based on metamaterial structure as well as the inverse (gen- through enforcement of physical properties. For example, erative) prediction task based on properties. Furthermore, Chen et al. [89] propose an approach to generate Random the framework supports self-supervised learning where the Heterogeneous Material (RHM) microstructure images using model trains on unlabeled metamaterial pattern images with- a cGAN conditioned on target images. Their cGAN uses an out corresponding property labels. The authors also demon- augmented loss based on the matching of perimeter, volume, strate transfer learning to other microstructure shape classes and Euler characteristics. DGM studies in microstructure as well as the inverse design of multiple microstructures design have also used other advanced methods introduced forming a meta-mirror with desired properties. DGMs have in computer vision such as super-resolution [58] and image also been applied to nano-scale photonic devices, such as an translation [67, 4, 67, 124]. optic broadband power splitter as in Tang et al. [88]. Although 2D images are a widely used means to rep- resent microstructures, Zhang et al. [65] expand the mi- Unit-cell-based metamaterials: Research in generative crostructure reconstruction task to the 3D domain. Their design for microstructures has also focused on the devel- ScaffoldGAN method generates scaffold materials to mimic opment of unit cell structures for metamaterials. For in- real-world examples of human bone scaffolds as well as stance, Wang et al. [32] ﬁt a VAE to metamaterial unit foam metal scaffolds in both 3D and 2D data. They employ cells and demonstrate that latent space parameters can re- the conventional GAN approach with style loss for better vi- ﬂect the physical properties of the unit cells. They then sual similarity between generated scaffolds and real ones. demonstrate several downstream tasks using their VAE, such However, the authors additionally introduce a novel “struc- as diverse subset selection and generation, targeted genera- tural loss” term to speciﬁcally address spatial coherence, a tion to match desired stiffness matrix values, and metama- limitation of GANs in emulating scaffolds. This additional terial family design. The authors also demonstrate several loss helps them achieve better results that mimic the features 2D macro-optimization runs to design arrays of unit cells to of the data more realistically. Other studies also consider match macro-level deﬂection targets in an approach similar DGMs on 3D microstructures, such as Mosser et al. [66] and to classic TO. Xue et al. [83] train a VAE to generate unit 9 Copyright © by ASME cells, then perform Bayesian Optimization in the latent space in DGMs. Dering et al. [95], for example, apply an iter- of the VAE to attain desired macroscopic elastic properties. ative retraining approach to boat sketches using an adapta- tion of the Long Short-Term Memory (LSTM)-based Sketch- RNN [135] as the generator. To evaluate candidate designs, 5.3 2D Shape Synthesis the performance is scored using a simulated environment in a game engine, within which the “behavior” (motion) of the design is learned. Fig. 4: Airfoil synthesis using DGMs usually entails employ- A recent work by Chen and Ahmed [15] addresses both ing 2D shape generation approaches. performance evaluation and design novelty through their pro- Designing features with critical geometric considera- posed Performance Augmented Diverse GAN (PaDGAN) tions is a task that appears in several engineering ﬁelds framework. Conventional GANs are trained to mimic the such as aerospace and automobile design. DGMs have been design space they are trained in and as a consequence, widely applied to these problems. Within aerospace, the de- are penalized for generating novel designs. PaDGAN ex- sign of an airfoil, which is the cross-sectional shape of a pands on the conventional GAN architecture by modeling wing, is of particular interest to many researchers. Airfoils design performance and design diversity using a Determi- have a wide variety of uses in engineering domains such as nantal Point Process (DPP) kernel [136]. By promoting di- propeller, rotor, and turbine blade design. Since airfoil per- versity, PaDGAN directly addresses mode collapse in GANs formance parameters are of key interest, most of the research too (see Sec. 2.1). The authors demonstrated that PaDGAN focuses on performance-aware conditional generation. For is capable of generating high-quality and previously unseen example, Yilmaz and German [74] apply a cGAN to the novel designs for the UIUC airfoil database, using the lift to airfoil design problem, conditioning their network on vari- drag ratio as the quality metric. [137] extends this work to ous stall parameters. While the overwhelming majority of multi-objective generation. research in this domain uses DGMs trained to learn shape parameterization based on spline interpolation of Cartesian 5.4 3D Shape Synthesis points, equation-based parameterization is also sometimes used. For example, Li et al. [94] propose a performance- aware RL framework where the RL agent learns the opti- mal equation coefﬁcients using Proximal Policy Optimiza- tion (PPO) [133]. Fig. 5: DGM models, such as Range-GAN [77], can be used Other works generalize their applications to general 2D to generate 3D models of aircrafts with constraints. shape synthesis. For example, Chen and Fuge [75] propose Bezier ´ GAN, a framework that learns shape representations 3D object generation using deep learning is an active through Bezier ´ curves, featuring InfoGAN style condition- ﬁeld in computer science, with signiﬁcant research efforts ing [17]. Chen et al. [76] expand on this work with Bayesian being dedicated to generating realistic-looking shapes and Optimization to maximize lift/drag ratio. Classic test data for objects. 3D shapes and objects are typically represented 2D shape synthesis methods are the UIUC airfoil dataset as by voxels, point clouds, or meshes. Advancements in 3D well as artiﬁcially-designed shapes like superformulas [134]. shape synthesis in the computer graphics community have While the emphasis of certain papers is on shapes or airfoils leaned heavily on GANs or Autoencoders, as well as other themselves, several papers primarily propose methodology machine learning advancements like Recurrent Neural Net- advancements that they choose to demonstrate on the 2D works (RNNs), Transformers, and Graph Neural Networks shape synthesis domain. For example, Chen and Fuge [33] (GNNs). This research is tangentially relevant to engineer- address the challenging problem of multi-component design ing design but is typically focused more on visual appear- generation using a GAN to synthesize parts using inter-part ance and aesthetics rather than design considerations like dependencies. The authors assume inter-part dependencies functional performance derived from simulations or manu- in a design are known and propose modeling them using di- facturability. For this reason, we do not discuss these meth- rected acyclic graphs. They propose a hierarchical adapta- ods in detail and opt to list a few of the many noteworthy tion of the InfoGAN using a single discriminator for the en- papers in this ﬁeld from 2016 to 2021 for the reader to fur- tire design and one generator/auxiliary network pair for each ther explore at their discretion: [138, 139, 140, 141, 142, 143]. part in the design, which they term the hierarchical GAN It is important to note that many of these proposed DGM (HGAN). The method is tested on a medley of synthetic architectures could be adapted to the design domain too. De- datasets generated using BezierGAN ´ [75] and demonstrated spite the prominence of 3D object generation papers in the to form meaningful and interpretable latent spaces. Although computer graphics community, a few have arisen within the the paper assumes that part dependency graphs for the ob- engineering design community as well. In an early work, ject class are well deﬁned, this paper takes a signiﬁcant step for example, Brock et al. [84] implement a 3D cVAE to re- towards multi-component design synthesis and interpretable construct and interpolate between voxelized models from the GANs. ModelNet-10 Dataset. Several other papers use the 2D shape synthesis do- The majority of 3D shape synthesis work in engineering main to address the challenges of performance evaluation design, however, has implemented some kind of design per- 10 Copyright © by ASME formance consideration. Zhang et al. [92] propose the use able different tasks. For example, Wang et al. [78] look at of a Genetic Algorithm (GA) to optimize latent space design the problem of generating samples that are visually pleasing embeddings of a trained VAE variant called the Variational for any given human subject. To do this, they train a modi- Shape Learner (VSL) [144], which they demonstrate on the ﬁed version of the Auxiliary Classiﬁer GAN (ACGAN) [148] optimization of 3D aircraft models. Shu et al. [69] take an which classiﬁes the type of image being generated at the iterative retraining approach by retraining a GAN on high- same time as learning to generate images. The authors, how- performance models evaluated using Computational Fluid ever, do not opt for a cGAN architecture. Instead, they condi- Dynamics (CFD) evaluation. The authors apply the pro- tion the GAN using Electroencephalography (EEG) signals posed method to point cloud aircraft models from ShapeNet from human subjects. They train an encoder that takes EEG and use minimization of aerodynamic drag as the perfor- signals and transforms them into a set of input features for mance objective of choice. The authors use a standard GAN the GAN. By exposing human subjects to the images as they loss and use a discriminator architecture from [145]. No- are being generated, the encoder introduces the human re- bari et al. [77] introduce another self-augmentation approach sponse to the design into the generation process. This ef- to train on skewed or sparse datasets. They couple it with fectively allows for a human subject to extract more visually a “range loss” to encourage designs to comply with design pleasing samples from the GAN model. Several other stud- constraints related to parameter bounds and apply their ap- ies such as [73, 85, 96] propose frameworks that incorporate proach to generate 3D aircraft models. humans into the design process. Kinematic synthesis: In machine design, generating a 5.5 Other Applications mechanism to generate speciﬁc motion curves or transmit In this section, we discuss several application domains power is a common design task. Kinematic synthesis in- for DGMs with only a few relevant papers identiﬁed in our volves selecting the size, type, and conﬁguration of mech- search. anism components to achieve such a goal. Several works have applied DGMs to various kinematic synthesis prob- Manufacturability: Synthesizing manufacturable designs lems. Deshpande and Purwar [85], for example, propose a is an important, albeit less-explored area in DGMs. VAE-based method for planar linkage synthesis. They de- Greminger [100] proposes approaching the problem of terministically parameterize both coupler paths and linkages, manufacturability using an MSG-GAN architecture [146] then generate sample coupler paths from a variety of linkages adapted to the 3D domain. The author synthetically gen- (four-bar, slider-crank, Stephenson-type six-bar). In the ﬁrst erates topologies that are manufacturable by a 3-axis mill, of their two studies, the authors train a VAE to reconstruct trains the GAN to generate similar topologies, then optimizes trajectories. They then propose a method in which humans generated topologies using TO. A key challenge in GAN- can interact with the trained VAE to customize linkage paths based synthesis of manufacturable designs is a lack of anno- to their design goals by visualizing the effect of various latent tated datasets with design and manufacturing process details. space perturbations and selecting one that matches their de- sign vision. The authors then expand on their ﬁrst study with Fluid ﬂow shapes: DGMs have also been used to gener- a cVAE architecture, feeding in the linkage as the data sam- ate ﬂow shapes. For example, Lee et al. [96] apply a Dou- ples and the coupler curve as the condition vector. In a sim- ble Deep Q-Network (DoubleDQN) [147] RL framework to ilar work, Sharma and Purwar [86] explore spatial linkage learn the geometry of a microﬂuidic channel to generate tar- synthesis of 5-SS mechanisms using VAEs. In other papers, get ﬂow shapes. The authors also propose methods in which the same authors take a look at paths themselves as guid- a human designer can participate in the design process along- ing mediums for linkage mechanism design by ﬁrst training side the RL agent to impart design knowledge or constraints. models to encode paths, then using said models to search Other approaches that integrate humans into the design gen- within a dataset of mechanisms to obtain solutions for the eration process are discussed next. speciﬁc paths [93]. Other studies have approached the kinematic synthesis Integrating humans into the generation process: One el- task through Reinforcement Learning. Vermeer et al. [107] ement of design that is often less emphasized in DGMs for propose using Deep Q-Learning to synthesize planar link- design is human interpretation and design input during the age mechanisms to obtain mechanisms that are capable of generative process. Though datasets are often created us- producing straight lines, showing that RL can be effective in ing some human input, humans are not directly involved dur- generating linkage mechanisms using machine learning. ing the training process of most DGMs. Burnap et al. [90] note that numerical performance measures often do not cor- Truss design: Raina et al. [108] propose an RL-based ap- respond to a human’s perception of design quality, and study proach to truss design. In their approach, they learn from this phenomenon on automobile images generated through a dataset of sequential human design decisions [149]. They a conditional VAE. Supporting the involvement of humans ﬁrst train an Autoencoder to map trusses to and from a design during the training process is a potential approach to ﬁnd embedding. Sequential design embeddings are then used to the middle ground between the expertise and intuition of hu- predict a heatmap of possible next design states using a su- mans and the detailed information of datasets, which can en- pervised transition network. Interestingly, the authors choose 11 Copyright © by ASME to convert to and from images from their parametric repre- able. The most common method for dataset generation is sentation when training the autoencoder and using decoded Solid Isotropic Material with Penalisation (SIMP), which has results. Finally, a rule-based agent selects from possible next several publicly available software implementations. A few design steps. Even without knowledge of design metrics and TO datasets, however, are open source. Sosnovik and Os- performance, the RL agents were found to generate compet- eledets [52] generate a dataset of 10,000 artiﬁcially gener- itive designs compared to humans. Puentes et al. [109] ex- ated topologies providing both ﬁnal and intermediate topolo- pand on this work by learning action-sequence heuristics in- gies from the optimization process . Each included topology stead of individual design actions. Raina et al. [97] further contains one 40x40 image of the topology generated at ev- expand on this work with a goal-based reinforcement learn- ery stage of 100 steps of Topology Optimization. Hence, ing agent. a total of one million images are provided. Nie et al. [71] provide the dataset used to train their TopologyGAN frame- Hierarchical product design synthesis: Some works work . Unlike the aforementioned, this dataset consists only have applied DGMs to multi-component products, typically of ﬁnal topologies but contains 49078 generated topologies with a well-known hierarchical structure. Stump et al. [34], at a resolution of 64x128, generated from 42 unique bound- for example, propose a unique approach that simultaneously ary conditions. Both datasets discussed use the ToPy [151] combines RL, Recurrent Neural Networks (RNNs), gram- implementation of SIMP. mars, and physics-based simulation. In their approach, the RNN learns to generate modular sailing crafts by sampling 6.2 Microstructure Datasets discrete selections from a predeﬁned shape grammar. The Well-established technologies such as optical mi- training of the RNN is detailed in [110]. The control policy croscopy (OM) and scanning electron microscopy (SEM) are of the craft is then optimized using RL in a physics-based often used to visualize material microstructures. As such, simulated sailing environment. Regenwetter et al. [87, 150] numerous datasets of microstructure scan images are pub- explore the full synthesis of bicycles, a diverse class of prod- licly available, such as [152], [153], and [154]. Numer- ucts typically consisting of numerous hierarchical compo- ous datasets are also available for the design of composite nents. The authors demonstrate full generative synthesis of materials of various types, such as the NanoMine nanopoly- bikes using VAEs for both image and parametric represen- mer composite database , which contains over 20,000 data- tations. The authors note that their detailed design parame- points [155]. Compiled lists of materials science datasets terization allows AI-generated designs to be physically fab- and synthetic microstructure datasets are also available. For ricated. example, Yang et al. [64] provide a trained GAN model which generates synthetic microstructure images . Procedural content generation: Lopez et al., explore pro- cedural content generation in a speciﬁc context for virtual reality. In their work, they introduce a reinforcement learn- 6.3 Design Geometry Datasets ing model that can learn to generate 3D virtual reality (VR) The UIUC airfoil database has been used as a case content which users can explore in a VR environment [98]. study in several generative design research frameworks [76, In their approach, they speciﬁcally work towards generat- 15, 75]. The database details nearly 1,600 real-world airfoil ing manufacturing environments that are physically valid and designs using coordinates of points on the surface. Since the feasible. In an extension of this work, Cunningham et al., ex- original data provides inconsistent numbers of coordinates tend the method to generate content for multiple contexts in- along the top and bottom surfaces, Chen et al. [76] propose stead of just one at a time [99], which enables the integration a method to standardize the data with B-spline interpolation of user-speciﬁc parameters. over the airfoil which is also used in other works [33, 15]. 6.4 3D Object Datasets 6 Datasets ShapeNet [156] is one of the most commonly-used 3D In this section, we present commonly-used datasets that model datasets, consisting of over 51,300 3D models of 55 have been or have the potential to be used to train DGMs for object categories. PartNet [157] expands on ShapeNet with data-driven design tasks. We provide a more detailed list on- ﬁne-grained hierarchical semantic annotations for compo- line . While the datasets listed are not comprehensive, we nent parts of ShapeNet objects. Princeton ModelNet [158] aim to note key types of datasets that have been commonly used in engineering design applications. We hope that re- searchers can create larger well-annotated datasets and make https://github.com/ISosnovik/top them public for the community to use. https://github.com/zhenguonie/2020_TopologyGAN https://github.com/tetherless-world/ nanomine-ontology 6.1 Topology Optimization Datasets https://github.com/sedaoturak/ Most papers on DGMs for Topology Optimization gen- data-resources-for-materials-science erate their own datasets, which are often not publicly avail- https://github.com/zyz293/GAN_Materials_Design https://m-selig.ae.illinois.edu/ads/coord_ database.html https://shapenet.org/ https://decode.mit.edu/datasets/ https://modelnet.cs.princeton.edu/ 12 Copyright © by ASME is another commonly-used 3D model dataset consisting of 6.7 Sketch Datasets 127,915 voxel-based 3D models of 662 object categories. QuickDraw [165] is a sketch dataset of 50 million doo- Numerous works discussed in this review use ShapeNet or dles from 345 categories . The doodles were collected by ModelNet models [69, 77, 92, 84]. Google from user-drawn sketches in an interactive sketch- ing game. QuickDraw data is used in several works dis- Sangpil et al. [159] introduce a dataset of 58,696 models cussed [166, 95]. Toh & Miller [167] introduce a dataset of of mechanical components from 68 classes called the Me- 934 innovative milk frother design sketches with associated chanical Components Benchmark (MCB) . The MCB con- text descriptions that can potentially be used to train DGMs, tains a hierarchical label tree grouping components into sub- perhaps factoring in Natural Language Processing (NLP) . classes of different levels, such as Components ! Fasteners ! Nuts ! Wingnuts, for example. Models are represented as point clouds, voxels, and 2D views. 6.8 Sequential Human Design Datasets McComb et al. [149] provide a tabular truss design dataset taken from a truss design activity executed by six- 6.5 CAD and CAD-based Datasets teen human teams . Sequential design operations such as Willis et al. [160] introduce two datasets of Autodesk joint and member placement are recorded using geometric Fusion models . One dataset, intended for reconstruction parameters such as joint coordinates, member size, etc.. Per- tasks, contains 8,625 models and the other, intended for formance metrics such as safety factors and weights are also segmentation, contains 35,680 models. The reconstruction included. While this dataset can be used as a truss design dataset is particularly interesting for generative tasks, as it dataset, it is primarily intended as a resource to study or contains information governing the sequential CAD opera- mimic the human design process. We note that the Autodesk tion steps taken to generate a part. Fusion reconstruction dataset mentioned previously [160] Regenwetter et al. [87, 150] introduce a dataset called can also be used for learning sequential design tasks. BIKED consisting of mixed data extracted from 4,512 bi- cycle CAD models. The dataset includes bicycle assembly images, segmented subcomponent images, as well as “para- 7 Discussion, Challenges and Future Work metric” data. The “parametric” data consists of 2,395 mixed- When applying DGMs to engineering design, the ‘stan- type design parameters describing both high-level and low- dard’ objective of mimicking the training data is often insuf- level characteristics. BIKED provides an advantage over ﬁcient, or even counterproductive. Instead, real designs are conventional 3D object datasets for generative tasks in that governed by speciﬁc objectives and constraints, often includ- synthesized designs contain the necessary parametric infor- ing novelty or creativity. Thus, different objectives, such as mation to physically fabricate designs. BIKED’s paramet- real-world performance metrics, novelty, and adherence to ric data is used in [87] for bicycle synthesis and its image constraints may make for better training objectives. We dis- data in [161] for generating novel designs. The FRAMED cuss the challenges with performance evaluation, constraint dataset [162] expands on BIKED with structural perfor- violation, and incorporation of novelty in the following sec- mance data, such as weight, safety factors, and deﬂections tions, as well as pathways to potential solutions. We further under various loads for all 4512 models as well as artiﬁcially discuss some general challenges for DGMs in engineering generated bicycle frames . design, such as limited availability of data, lack of bench- mark problems and metrics, and delayed adoption of cutting edge methods from different research communities. Our dis- 6.6 Metamaterials Datasets cussion is supported by observations from our literature re- Wang et al. [32] introduce a dataset of 248,396 2D unit view. cells represented by 50x50 pixelated matrices. The unit cells have associated stiffness tensor components provided and can also be used for TO research. Chan et al. [163] introduce 7.1 Design Performance Evaluation a dataset of 3,000 3D isosurface unit cells sampled from 30 Incorporating performance evaluation into DGMs is one level-set functions, along with corresponding 3D elastic ten- of the key stepping stones toward practical applications of sor components. Wang et al. [164] introduce a dataset of 795 ML in design. Three major challenges in performance eval- unit cells generated from 10 lattice models. Associated stiff- uation are ﬁdelity, cost, and differentiability. ness tensors are provided. The three datasets can be found 1. Evaluation methods lacking ﬁdelity may result in DGM- here . generated designs that do not meet speciﬁcations. 2. Computational cost precludes compatibility with meth- ods that heavily sample performance values. https://bit.ly/3ne4gwv https://github.com/AutodeskAILab/ Fusion360GalleryDataset https://github.com/googlecreativelab/ https://decode.mit.edu/projects/biked/ quickdraw-dataset https://decode.mit.edu/projects/framed/ https://sites.psu.edu/creativitymetrics/2018/ https://ideal.mech.northwestern.edu/research/ 07/18/milkfrother/ software/ https://www.sciencedirect.com/science/article/ pii/S2352340918302014?via%3Dihub 13 Copyright © by ASME 3. Lack of differentiability makes implementation into the 7.2 Feasibility, Constraints, and Manufacturability training process difﬁcult in any machine learning mod- One key component of design performance is obeying els which rely on gradient-based optimization. explicit design constraints, as well as implicit constraints such as physical feasibility and manufacturability. DGMs Physical evaluation, which means building a product and are difﬁcult to rely upon for explicit design constraints since testing its performance in the real-world, typically has the they are generally probabilistic and may generate completely highest ﬁdelity but is rarely adopted in DGMs due to its invalid designs. This issue is a concern that many researchers prohibitive cost. Qualitative human evaluation of designs have pointed to [177,33,103]. One potential solution is to de- has also been investigated in several papers [85, 73, 96]], velop inexpensive and reliable validation methods, however, although the evaluation is typically not focused on perfor- this is a challenging task that may require signiﬁcant human mance. Medium-ﬁdelity evaluations, such as numerical sim- input. ulations often deliver satisfactory ﬁdelity, but can be costly Another underlying issue lies in design representation, and are rarely differentiable. Methods that do incorporate which is discussed in detail in Sec. 3. Representations such medium-ﬁdelity performance evaluation like [69, 62] do not as images often have no clear translation to representations rely on performance score gradients, and instead alternate with practical uses. Other representations, such as 3D mod- (re)training and performance evaluation, typically doing this els of various types can only be realistically fabricated us- retraining a handful of times. ing additive manufacturing, which precludes many design Low-cost evaluation methods like surrogate models can domains. For ML-generated designs to be physically fabri- be worked into the training objective and are by far the most cated and used in practical applications, the feasibility across common performance evaluation method seen in DGMs [15, this ‘domain gap’ must be overcome [103, 87]. Though a 20, 77, 18, 70, 79, 60]. Unfortunately, surrogate models can few works have investigated manufacturability [100], we ob- be brittle and generalize poorly to designs that differ from served that modeling it is not considered in most of the pa- the data the surrogate was trained on. This is a particular pers we reviewed. Other works have attempted to apply concern when training data for surrogates is unevenly dis- DGMs to parameterizations that encode similar parametric tributed across the performance space [77] and when nov- design data to what would be used in manufacturing draw- elty and performance are incorporated into training objec- ings. However, the same works identify that parametric data tives for DGMs. Many different approaches have attempted of this sort is challenging to learn and generate [87]. to improve the performance of low-ﬁdelity surrogate models. For example, self-supervised data augmentation [77] uses the 7.3 Creativity and Novelty DGM itself to generate samples in sparse regions (through Whereas creativity and novelty are essential aspects optimization [137] or conditioning [77]) which then can be of the classic design process, DGMs rarely explicitly con- used to train the low-ﬁdelity models to perform more accu- sider either. Most DGMs learn to mimic the data cover- rately and in turn improve methods that rely on them for de- ing the existing and already explored portions of the design sign generation. Multi-ﬁdelity modeling is another promis- space. While this emulative behavior is helpful for main- ing approach, which involves generating surrogate models taining realism and ensuring sample quality, it incentivizes that augment a few costly high-ﬁdelity samples with low- DGMs against generating creative or novel designs [178]. ﬁdelity samples to attain higher ﬁdelity surrogates with min- For DGMs to progress towards more human-like design, ad- imal expense. vancements must be made in modeling creativity as well as Advancements have also been made in using machine in developing architectures that promote creativity. learning to improve and accelerate medium-ﬁdelity physics Several recent works [137, 15, 161] have proposed meth- simulations such as Finite Element Analysis [168, 169, 170, ods to encourage creativity and novelty in DGMs. In their 171, 172] and Computational Fluid Dynamics [173, 174, 175, framework, CreativeGAN, Nobari et al., focus on identify- 172]. These works, although not yet robust enough to be ing novelty and guiding DGMs towards such behavior by di- easily applied as an alternative to high-ﬁdelity simulations, rectly introducing novel features into typical designs, thereby provide a proof of concept for machine learning-based accel- expanding the design space and novelty of the DGM’s data. eration and even replacement of higher-ﬁdelity simulations. Creativity in machine learning has also been explored out- Numerous studies have also proposed crowdsourcing side of the design community. For example, Creative Ad- methods to evaluate synthesized data from generative ma- versarial Networks (CAN) [178] introduce entropy into the chine learning methods [90, 176, 166] instead of physical training to encourage the generation of surprising images. or physics-based evaluation. Though the ﬁdelity of crowd- Readers are directed to Franceschelli et al.’s survey on the sourced evaluation is highly task- and crowd-expertise- topic for more detail [179]. dependent, the high evaluation cost and lack of differentia- bility are fairly universal. In summary, current DGMs are largely constrained in 7.4 Evaluating Model Performance & Benchmark performance evaluation by an inherent ﬁdelity vs. cost trade- Problems off, however, several directions show promise in enabling Readers may have noted that many of the works dis- faster and more accurate evaluation methods that may escape cussed have nearly identical methodologies, though they are this limitation. applied to a speciﬁc dataset and only compare their work 14 Copyright © by ASME in this respect with a few other baselines. This may be at- datasets, publicly release these datasets, and establish bench- tributed to a large variety of design applications compared to mark problems for future works. We subsequently encourage other domains such as computer vision and a tendency of de- researchers to test their methods on other researchers’ data, sign researchers to ﬁnd solutions to their particular problem, publicly release their benchmarking results, and acknowl- instead of ﬁnding generalizable solutions to many. How- edge state-of-the-art methods when possible. ever, this makes any measure of ‘state-of-the-art’ algorithm dependent on speciﬁc applications. Without a good under- 7.6 Other DGMs standing of state-of-the-art methods that are broadly applica- So far our discussion has been focused on the VAE, ble across engineering domains, practitioners will struggle to GAN, and RL-based approaches, which have dominated the select models for practical deployment. While some works ﬁeld of DGMs in engineering design. Transformer-based se- in the design community have speciﬁcally introduced bench- quential models were initially developed for natural language mark problems and datasets [87, 162], there is still a need processing and generation or translation of text, however, for larger, higher-quality, and more numerous datasets and researchers have shown that the scalability of these mod- benchmarks. els allows for them to be used for the generation of com- The difﬁculty of establishing a state-of-the-art lies in the plex data in a sequential manner [180, 181, 182, 183]. Fur- lack of benchmark problems and performance metrics within thermore, these models have an unprecedented ability to be the DGM ﬁeld and within design automation as a whole. mixed with language processing. This may enable human Design datasets tend to be small, restricted in domain, and language-based conditioning and control over the generation sparse in distribution as we discuss in Section 7.5. Further- process of data [181, 183], enabling deployment for a much more, many methods are developed on proprietary datasets. wider user base and a much wider set of tasks. The applica- Finally, the hugely different design representation methods tions of transformer-based models as well as their scalability across the ﬁeld make establishing standardized model perfor- may appeal to researchers in the design community. Despite mance benchmarks difﬁcult, even if there were good datasets this, it is important to note that these models are particularly upon which to do so. These problems are noted by many au- large and therefore require very large datasets, further high- thors in the ﬁeld as well, who ﬁnd it difﬁcult to compare their lighting the need for better datasets. approach with existing ones [60, 177]. 8 Conclusion 7.5 Data Limitations and Quality In this review, we discussed the applications of Deep Data sparsity is one of the greatest challenges facing the Generative Models (DGMs) across engineering design ﬁelds. data-driven design community and is a particular concern for To give readers a sense of the tools and methods available, researchers developing data-hungry DGMs. Broadly, there we began our review with an overview of DGMs typically are three problems when it comes to the data. The ﬁrst used for engineering design problems, emphasizing Gen- of these is a general lack of data in many design domains. Although datasets continue to cover more and more design erative Adversarial Networks (GANs), Variational Autoen- coders (VAEs), and Reinforcement Learning (RL). To help ﬁelds, there still are many that lack publicly available data. weigh different design representation methods, we then dis- There are also data representations that are underrepresented cussed the strengths and weaknesses of common parameter- in the current datasets, notably graphs as discussed in Sec. 3. ization methods. To inform readers about existing work out- The second key data-related problem that designers face in side of their subdomain, we then collected and reviewed 63 applying DGMs is the insufﬁcient size of current datasets. papers from a variety of different engineering design sub- Many of the latest breakthroughs in deep learning, notably disciplines which directly propose DGMs. To make read- in computer vision and natural language processing, have ers aware of the datasets available on which to develop and owed their success to very large models which are notori- test data-driven design methods, we review commonly-used ously data-hungry, requiring millions or billions of training datasets in the ﬁeld. Finally, to provide inspiration, we dis- examples, and cannot be properly utilized with datasets of cuss key challenges and limitations currently seen across the smaller size. Focusing resources on the generation of very ﬁeld and highlight possible solution pathways. large publicly available datasets or effective data augmenta- tion methods would open doors for the design community to leverage larger deep models. The last major limitation in existing data is sparsity and bias. These problems are com- References mon in many design datasets. How the data is distributed in [1] Chakrabarti, A., Shea, K., Stone, R., Cagan, J., Camp- the performance space can cause some challenges for many bell, M., Hernandez, N. V., and Wood, K. L., 2011. DGMs, especially in inverse design problems [77]. Diver- “Computer-Based Design Synthesis Research: An sity is important to ensure the data covers the design space Overview”. Journal of Computing and Information as evenly as possible to avoid bias in the data. Keeping di- Science in Engineering, 11(2), 06. 021003. versity and bias avoidance in mind will help researchers gen- [2] Deng, L., and Yu, D., 2014. “Deep learning: Meth- erate better datasets. ods and applications”. Found. Trends Signal Process., We highly encourage researchers to develop diverse 7(3–4), jun, p. 197–387. 15 Copyright © by ASME [3] Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, V., Radford, A., Chen, X., and Chen, X., 2016. “Im- B., Warde-Farley, D., Ozair, S., Courville, A., and proved techniques for training gans”. In Advances Bengio, Y., 2014. “Generative adversarial nets”. In in Neural Information Processing Systems, D. Lee, Proceedings of the 27th International Conference on M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, Neural Information Processing Systems - Volume 2, eds., Vol. 29, Curran Associates, Inc., pp. 2234–2242. NIPS’14, MIT Press, p. 2672–2680. [15] Chen, W., and Ahmed, F., 2021. “Padgan: Learning [4] Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A., 2017. to generate high-quality novel designs”. Journal of “Unpaired image-to-image translation using cycle- Mechanical Design, 143(3), p. 031703. consistent adversarial networks”. In 2017 IEEE In- [16] Mirza, M., and Osindero, S., 2014. “Condi- ternational Conference on Computer Vision (ICCV), tional generative adversarial nets”. arXiv preprint pp. 2242–2251. arXiv:1411.1784. [5] Choi, Y., Uh, Y., Yoo, J., and Ha, J.-W., 2020. “Star- [17] Chen, X., Duan, Y., Houthooft, R., Schulman, J., gan v2: Diverse image synthesis for multiple do- Sutskever, I., and Abbeel, P., 2016. “Infogan: Inter- mains”. In 2020 IEEE/CVF Conference on Computer pretable representation learning by information max- Vision and Pattern Recognition (CVPR), pp. 8185– imizing generative adversarial nets”. In Proceedings 8194. of the 30th International Conference on Neural Infor- [6] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehti- mation Processing Systems, pp. 2180–2188. nen, J., and Aila, T., 2020. “Analyzing and improving [18] Dong, Y., Li, D., Zhang, C., Wu, C., Wang, H., Xin, the image quality of stylegan”. In 2020 IEEE/CVF M., Cheng, J., and Lin, J., 2020. “Inverse design of Conference on Computer Vision and Pattern Recogni- two-dimensional graphene/h-bn hybrids by a regres- tion (CVPR), pp. 8107–8116. sional and conditional gan”. Carbon, 169, pp. 9–16. [7] Karras, T., Laine, S., and Aila, T., 2019. “A style- [19] Ding, X., Wang, Y., Xu, Z., Welch, W. J., and Wang, based generator architecture for generative adversar- Z. J., 2021. “Ccgan: Continuous conditional gener- ial networks”. In 2019 IEEE/CVF Conference on ative adversarial networks for image generation”. In Computer Vision and Pattern Recognition (CVPR), 9th International Conference on Learning Represen- pp. 4396–4405. tations, ICLR 2021, Virtual Event, Austria, May 3-7, [8] Creswell, A., White, T., Dumoulin, V., Arulkumaran, 2021, OpenReview.net. K., Sengupta, B., and Bharath, A. A., 2018. “Genera- [20] Heyrani Nobari, A., Chen, W., and Ahmed, F., 2021. tive adversarial networks: An overview”. IEEE Signal “Pcdgan: A continuous conditional diverse generative Processing Magazine, 35(1), pp. 53–65. adversarial network for inverse design”. Proceedings [9] Salimans, T., Goodfellow, I., Zaremba, W., Cheung, of the 27th ACM SIGKDD Conference on Knowledge V., Radford, A., and Chen, X., 2016. “Improved tech- Discovery & Data Mining, Aug. niques for training gans”. In Proceedings of the 30th [21] Kingma, D. P., and Welling, M., 2013. “Auto- International Conference on Neural Information Pro- encoding variational bayes”. arXiv preprint cessing Systems, NIPS’16, Curran Associates Inc., arXiv:1312.6114. p. 2234–2242. [22] Kullback, S., and Leibler, R. A., 1951. “On infor- [10] Arjovsky, M., and Bottou, L., 2017. “Towards princi- mation and sufﬁciency”. The annals of mathematical pled methods for training generative adversarial net- statistics, 22(1), pp. 79–86. works”. arXiv preprint arXiv:1701.04862. [23] Kingma, D. P., and Welling, M., 2019. “An introduc- [11] Arjovsky, M., Chintala, S., and Bottou, L., 2017. tion to variational autoencoders”. Foundations and “Wasserstein generative adversarial networks”. In Trends in Machine Learning, 12(4), pp. 307–392. Proceedings of the 34th International Conference on [24] Sohn, K., Yan, X., and Lee, H., 2015. “Learn- Machine Learning - Volume 70, ICML’17, JMLR.org, ing structured output representation using deep con- p. 214–223. ditional generative models”. In Proceedings of the [12] Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, 28th International Conference on Neural Information V., and Courville, A., 2017. “Improved training of Processing Systems - Volume 2, NIPS’15, MIT Press, wasserstein gans”. In Proceedings of the 31st In- p. 3483–3491. ternational Conference on Neural Information Pro- [25] Kaelbling, L. P., Littman, M. L., and Moore, A. W., cessing Systems, NIPS’17, Curran Associates Inc., 1996. “Reinforcement learning: A survey”. J. Artif. p. 5769–5779. Int. Res., 4(1), May, p. 237–285. [13] Srivastava, A., Valkov, L., Russell, C., Gutmann, [26] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., M. U., and Sutton, C., 2017. “Veegan: Reducing mode Veness, J., Bellemare, M. G., Graves, A., Ried- collapse in gans using implicit variational learning”. miller, M., Fidjeland, A. K., Ostrovski, G., et al., In Advances in Neural Information Processing Sys- 2015. “Human-level control through deep reinforce- tems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wal- ment learning”. nature, 518(7540), pp. 529–533. lach, R. Fergus, S. Vishwanathan, and R. Garnett, [27] Daneshmand, M., Helmi, A., Avots, E., Noroozi, F., eds., Vol. 30, Curran Associates, Inc., pp. 3308–3318. Alisinanoglu, F., Arslan, H. S., Gorbova, J., Haamer, [14] Salimans, T., Goodfellow, I., Zaremba, W., Cheung, R. E., Ozcinar, C., and Anbarjafari, G., 2018. “3d 16 Copyright © by ASME scanning: A comprehensive survey”. arXiv preprint Engineering Conference. V01AT02A071. arXiv:1801.08863. [40] Patalano, S., Vitolo, F., and Lanzotti, A., 2013. “A [28] Remondino, F., 2003. “From point cloud to surface: graph-based software tool for the cad modeling of me- the modeling and visualization problem”. Interna- chanical assemblies”. In GRAPP/IVAPP. tional Archives of the Photogrammetry, Remote Sens- [41] Henaff, M., Bruna, J., and LeCun, Y., 2015. Deep ing and Spatial Information Sciences, 34. convolutional networks on graph-structured data. [29] Ranjan, A., Bolkart, T., Sanyal, S., and Black, M. J., [42] Yun, S., Jeong, M., Kim, R., Kang, J., and Kim, H. J., 2018. “Generating 3d faces using convolutional mesh 2019. “Graph transformer networks”. In Advances autoencoders”. In Proceedings of the European Con- in Neural Information Processing Systems, H. Wal- ference on Computer Vision (ECCV), pp. 704–720. lach, H. Larochelle, A. Beygelzimer, F. d'Alche-Buc, ´ [30] Cheng, S., Bronstein, M., Zhou, Y., Kotsia, I., Pan- E. Fox, and R. Garnett, eds., Vol. 32, Curran Asso- tic, M., and Zafeiriou, S., 2019. “Meshgan: Non- ciates, Inc. linear 3d morphable models of faces”. arXiv preprint [43] Velick ˇ ovic, ´ P., Cucurull, G., Casanova, A., Romero, arXiv:1903.10384. A., Lio, ` P., and Bengio, Y., 2018. “Graph attention [31] Zhang, Z., Wang, Y., Jimack, P. K., and Wang, H., networks”. In International Conference on Learning 2020. “Meshingnet: a new mesh generation method Representations. based on deep learning”. In International Conference [44] Vashishth, S., Sanyal, S., Nitin, V., and Talukdar, on Computational Science, Springer, pp. 186–198. P., 2019. “Composition-based multi-relational graph [32] Wang, L., Chan, Y.-C., Ahmed, F., Liu, Z., Zhu, P., convolutional networks”. In International Conference and Chen, W., 2020. “Deep generative modeling for on Learning Representations. mechanistic-based learning and design of metamate- [45] Li, Y., Tarlow, D., Brockschmidt, M., and Zemel, R., rial systems”. Computer Methods in Applied Mechan- 2017. Gated graph sequence neural networks. ics and Engineering, 372, p. 113377. [46] Liao, R., Li, Y., Song, Y., Wang, S., Hamilton, W., [33] Chen, W., and Fuge, M., 2019. “Synthesizing designs Duvenaud, D. K., Urtasun, R., and Zemel, R., 2019. with interpart dependencies using hierarchical gener- “Efﬁcient graph generation with graph recurrent at- ative adversarial networks”. Journal of Mechanical tention networks”. In Advances in Neural Informa- Design, 141(11), p. 111403. tion Processing Systems, H. Wallach, H. Larochelle, [34] Stump, G. M., Miller, S. W., Yukish, M. A., Simp- A. Beygelzimer, F. d'Alche-Buc, ´ E. Fox, and R. Gar- son, T. W., and Tucker, C., 2019. “Spatial grammar- nett, eds., Vol. 32, Curran Associates, Inc. based recurrent neural network for design form and [47] Bojchevski, A., Shchur, O., Zugner ¨ , D., and behavior optimization”. Journal of Mechanical De- Gunnemann, ¨ S., 2018. “Netgan: Generating graphs sign, 141(12). via random walks”. In International Conference on [35] Cao, W., Robinson, T., Hua, Y., Boussuge, F., Col- Machine Learning, pp. 609–618. ligan, A. R., and Pan, W., 2020. “Graph Repre- [48] You, J., Ying, R., Ren, X., Hamilton, W., and sentation of 3D CAD Models for Machining Feature Leskovec, J., 2018. “Graphrnn: Generating realis- Recognition With Deep Learning”. Vol. Volume 11A: tic graphs with deep auto-regressive models”. In In- 46th Design Automation Conference (DAC) of Inter- ternational conference on machine learning, PMLR, national Design Engineering Technical Conferences pp. 5708–5717. and Computers and Information in Engineering Con- [49] Li, Y., Vinyals, O., Dyer, C., Pascanu, R., and ference. V11AT11A003. Battaglia, P., 2018. Learning deep generative models [36] Yang, W., Ding, H., Zi, B., and Zhang, D., of graphs. 2017. “New Graph Representation for Planetary Gear [50] Cao, N. D., and Kipf, T., 2018. Molgan: An implicit Trains”. Journal of Mechanical Design, 140(1), 11. generative model for small molecular graphs. 012303. [51] You, J., Liu, B., Ying, R., Pande, V., and Leskovec, J., [37] Hsu, C.-H., and Lam, K.-T., 1992. “A New Graph 2018. “Graph convolutional policy network for goal- Representation for the Automatic Kinematic Analysis directed molecular graph generation”. In Proceedings of Planetary Spur-Gear Trains”. Journal of Mechani- of the 32nd International Conference on Neural Infor- cal Design, 114(1), 03, pp. 196–200. mation Processing Systems, pp. 6412–6422. [38] Lee, J. Y., and Kim, K., 1996. “Geometric rea- [52] Sosnovik, I., and Oseledets, I., 2019. “Neural net- soning for knowledge-based parametric design us- works for topology optimization”. Russian Journal ing graph representation”. Computer-Aided Design, of Numerical Analysis and Mathematical Modelling, 28(10), pp. 831–841. 34(4), pp. 215–223. [39] Coatanea, ´ E., Nonsiri, S., Christophe, F., and Mokam- [53] Behzadi, M. M., and Ilies ¸, H. T., 2021. “Real-time mel, F., 2014. “Graph Based Representation and topology optimization in 3d via deep transfer learn- Analyses for Conceptual Stages”. Vol. Volume 1A: ing”. Computer-Aided Design, 135, p. 103014. 34th Computers and Information in Engineering Con- [54] Keshavarzzadeh, V., Alirezaei, M., Tasdizen, T., and ference of International Design Engineering Techni- Kirby, R. M., 2021. “Image-based multiresolution cal Conferences and Computers and Information in topology optimization using deep disjunctive nor- 17 Copyright © by ASME mal shape model”. Computer-Aided Design, 130, steels using generative adversarial networks”. Engi- p. 102947. neering Reports, 3(1), p. e12274. [55] Cang, R., Yao, H., and Ren, Y., 2019. “One-shot [68] Liu, S., Zhong, Z., Takbiri-Borujeni, A., Kazemi, M., generation of near-optimal topology through theory- Fu, Q., and Yang, Y., 2019. “A case study on ho- driven machine learning”. Computer-Aided Design, mogeneous and heterogeneous reservoir porous me- 109, pp. 12–21. dia reconstruction by using generative adversarial net- [56] Malkiel, I., Mrejen, M., Nagler, A., Arieli, U., Wolf, works”. Energy Procedia, 158, pp. 6164–6169. L., and Suchowski, H., 2018. “Plasmonic nanostruc- [69] Shu, D., Cunningham, J., Stump, G., Miller, S. W., ture design and characterization via deep learning”. Yukish, M. A., Simpson, T. W., and Tucker, C. S., Light: Science & Applications, 7(1), pp. 1–8. 2020. “3d design using generative adversarial net- [57] Li, X., Zhang, Y., Zhao, H., Burkhart, C., Brin- works and physics-based validation”. Journal of Me- son, L. C., and Chen, W., 2018. “A transfer learn- chanical Design, 142(7), p. 071701. ing approach for microstructure reconstruction and [70] Sharpe, C., and Seepersad, C. C., 2019. “Topology structure-property predictions”. Scientiﬁc reports, design with conditional generative adversarial net- 8(1), pp. 1–13. works”. In International Design Engineering Tech- [58] Jung, J., Na, J., Park, H. K., Park, J. M., Kim, G., nical Conferences and Computers and Information in Lee, S., and Kim, H. S., 2021. “Super-resolving ma- Engineering Conference, Vol. 59186, American Soci- terial microstructure image via deep learning for mi- ety of Mechanical Engineers, p. V02AT03A062. crostructure characterization and mechanical behavior [71] Nie, Z., Lin, T., Jiang, H., and Kara, L. B., 2021. analysis”. npj Computational Materials, 7(1), pp. 1– “Topologygan: Topology optimization using genera- 11. tive adversarial networks based on physical ﬁelds over [59] Li, B., Huang, C., Li, X., Zheng, S., and Hong, J., the initial domain”. Journal of Mechanical Design, 2019. “Non-iterative structural topology optimization 143(3), p. 031715. using deep learning”. Computer-Aided Design, 115, [72] Yu, Y., Hur, T., Jung, J., and Jang, I. G., 2019. “Deep pp. 172–180. learning for determining a near-optimal topological [60] Rawat, S., and Shen, M. H., 2019. Application of design without any iteration”. Structural and Multi- adversarial networks for 3d structural topology opti- disciplinary Optimization, 59(3), pp. 787–799. mization. Tech. rep., SAE Technical Paper. [73] Valdez, S., Seepersad, C., and Kambampati, S., 2021. [61] Oh, S., Jung, Y., Lee, I., and Kang, N., 2018. “De- “A framework for interactive structural design explo- sign automation by integrating generative adversarial ration”. In International Design Engineering Tech- networks and topology optimization”. In International nical Conferences and Computers and Information in Design Engineering Technical Conferences and Com- Engineering Conference, IDETC-21, ASME. puters and Information in Engineering Conference, [74] Yilmaz, E., and German, B., 2020. “Conditional gen- Vol. 51753, American Society of Mechanical Engi- erative adversarial network framework for airfoil in- neers, p. V02AT03A008. verse design”. In AIAA aviation 2020 forum, p. 3185. [62] Oh, S., Jung, Y., Kim, S., Lee, I., and Kang, N., 2019. [75] Chen, W., and Fuge, M., 2018. “Bezier ´ gan: Au- “Deep generative design: Integration of topology opti- tomatic generation of smooth curves from inter- mization and generative models”. Journal of Mechan- pretable low-dimensional parameters”. arXiv preprint ical Design, 141(11). arXiv:1808.08871. [63] Tan, R. K., Zhang, N. L., and Ye, W., 2020. “A deep [76] Chen, W., Chiu, K., and Fuge, M., 2019. “Aero- learning–based method for the design of microstruc- dynamic design optimization and shape exploration tural materials”. Structural and Multidisciplinary Op- using generative adversarial networks”. In AIAA timization, 61(4), pp. 1417–1438. Scitech 2019 Forum, p. 2351. [64] Yang, Z., Li, X., Catherine Brinson, L., Choudhary, [77] Heyrani Nobari, A., Chen, W. W., and Ahmed, A. N., Chen, W., and Agrawal, A., 2018. “Mi- F., 2021. “RANGE-GAN: Design Synthesis Under crostructural materials design via deep adversarial Constraints Using Conditional Generative Adversarial learning methodology”. Journal of Mechanical De- Networks”. Journal of Mechanical Design, 09, pp. 1– sign, 140(11). 16. [65] Zhang, H., Yang, L., Li, C., Wu, B., and Wang, W., [78] Wang, P., Peng, D., Li, L., Chen, L., Wu, C., Wang, 2021. “Scaffoldgan: Synthesis of scaffold materials X., Childs, P., and Guo, Y., 2019. “Human-in-the- based on generative adversarial networks”. Computer- loop design with machine learning”. Proceedings of Aided Design, 138, p. 103041. the Design Society: International Conference on En- [66] Mosser, L., Dubrule, O., and Blunt, M. J., 2017. “Re- gineering Design, 1(1), p. 2577–2586. construction of three-dimensional porous media using [79] Guo, T., Lohan, D. J., Cang, R., Ren, M. Y., and Alli- generative adversarial neural networks”. Physical Re- son, J. T., 2018. “An indirect design representation for view E, 96(4), p. 043309. topology optimization using variational autoencoder [67] Lee, J.-W., Goo, N. H., Park, W. B., Pyo, M., and and style transfer”. In 2018 AIAA/ASCE/AHS/ASC Sohn, K.-S., 2021. “Virtual microstructure design for Structures, Structural Dynamics, and Materials Con- 18 Copyright © by ASME ference, p. 0804. ing and exploring the product form design space us- [80] Cang, R., Li, H., Yao, H., Jiao, Y., and Ren, Y., 2018. ing deep generative models”. In International De- “Improving direct physical properties prediction of sign Engineering Technical Conferences and Com- heterogeneous materials from imaging data via con- puters and Information in Engineering Conference, volutional neural network and a morphology-aware Vol. 50107, American Society of Mechanical Engi- generative model”. Computational Materials Science, neers, p. V02AT03A013. 150, pp. 212–221. [91] Ma, W., Cheng, F., Xu, Y., Wen, Q., and Liu, Y., [81] Li, X., Ning, S., Liu, Z., Yan, Z., Luo, C., and Zhuang, 2019. “Probabilistic representation and inverse design Z., 2020. “Designing phononic crystal with antici- of metamaterials based on a deep generative model pated band gap through a deep learning based data- with semi-supervised learning strategy”. Advanced driven method”. Computer Methods in Applied Me- Materials, 31(35), p. 1901111. chanics and Engineering, 361, p. 112737. [92] Zhang, W., Yang, Z., Jiang, H., Nigam, S., Yamakawa, [82] Liu, Z., Raju, L., Zhu, D., and Cai, W., 2020. “A hy- S., Furuhata, T., Shimada, K., and Kara, L. B., 2019. brid strategy for the discovery and design of photonic “3d shape synthesis for conceptual design and opti- structures”. IEEE Journal on Emerging and Selected mization using variational autoencoders”. In Inter- Topics in Circuits and Systems, 10(1), pp. 126–135. national Design Engineering Technical Conferences [83] Xue, T., Wallin, T. J., Menguc, Y., Adriaenssens, S., and Computers and Information in Engineering Con- and Chiaramonte, M., 2020. “Machine learning gen- ference, Vol. 59186, American Society of Mechanical erative models for automatic design of multi-material Engineers, p. V02AT03A017. 3d printed composite solids”. Extreme Mechanics Let- [93] Deshpande, S., and Purwar, A., 2020. “An Image- ters, 41, p. 100992. Based Approach to Variational Path Synthesis of [84] Brock, A., Lim, T., Ritchie, J. M., and We- Linkages”. Journal of Computing and Information ston, N., 2016. “Context-aware content genera- Science in Engineering, 21(2), 10. 021005. tion for virtual environments”. In International De- [94] Li, R., Zhang, Y., and Chen, H., 2021. “Learning the sign Engineering Technical Conferences and Com- aerodynamic design of supercritical airfoils through puters and Information in Engineering Conference, deep reinforcement learning”. AIAA Journal, pp. 1– Vol. 50084, American Society of Mechanical Engi- 14. neers, p. V01BT02A045. [95] Dering, M., Cunningham, J., Desai, R., Yukish, M. A., [85] Deshpande, S., and Purwar, A., 2019. “Computational Simpson, T. W., and Tucker, C. S., 2018. “A physics- creativity via assisted variational synthesis of mecha- based virtual environment for enhancing the quality nisms using deep generative models”. Journal of Me- of deep generative designs”. In International De- chanical Design, 141(12). sign Engineering Technical Conferences and Com- [86] Sharma, S., and Purwar, A., 2020. “Path synthesis puters and Information in Engineering Conference, of defect-free spatial 5-ss mechanisms using machine Vol. 51753, American Society of Mechanical Engi- learning”. In International Design Engineering Tech- neers, p. V02AT03A015. nical Conferences and Computers and Information in [96] Lee, X. Y., Balu, A., Stoecklein, D., Ganapathysub- Engineering Conference, Vol. 83990, American Soci- ramanian, B., and Sarkar, S., 2019. “A case study ety of Mechanical Engineers, p. V010T10A034. of deep reinforcement learning for engineering de- [87] Regenwetter, L., Curry, B., and Ahmed, F., 2021. sign: Application to microﬂuidic devices for ﬂow “BIKED: A dataset and machine learning benchmarks sculpting”. Journal of Mechanical Design, 141(11), for data-driven bicycle design”. In International De- p. 111401. sign Engineering Technical Conferences and Com- [97] Raina, A., Puentes, L., Cagan, J., and McComb, C., puters and Information in Engineering Conference, 2021. “Goal-directed design agents: Integrating vi- IDETC-21, ASME. sual imitation with one-step lookahead optimization [88] Tang, Y., Kojima, K., Koike-Akino, T., Wang, Y., Wu, for generative design”. Journal of Mechanical Design, P., Tahersima, M., Jha, D., Parsons, K., and Qi, M., 143(12), p. 124501. 2020. “Generative deep learning model for a multi- [98] Lopez, C. E., Ashour, O., and Tucker, C. S., 2019. level nano-optic broadband power splitter”. In 2020 “Reinforcement Learning Content Generation for Vir- Optical Fiber Communications Conference and Exhi- tual Reality Applications”. Vol. Volume 1: 39th Com- bition (OFC), IEEE, pp. 1–3. puters and Information in Engineering Conference of [89] Chen, H., and Liu, X., 2021. “Geometry enhanced International Design Engineering Technical Confer- generative adversarial networks for random heteroge- ences and Computers and Information in Engineering neous material representation”. In International De- Conference. V001T02A009. sign Engineering Technical Conferences and Com- [99] Cunningham, J., Lopez, C., Ashour, O., and Tucker, puters and Information in Engineering Conference, C. S., 2020. “Multi-Context Generation in Virtual Re- IDETC-21, ASME. ality Environments Using Deep Reinforcement Learn- [90] Burnap, A., Liu, Y., Pan, Y., Lee, H., Gonza- ing”. Vol. Volume 9: 40th Computers and Informa- lez, R., and Papalambros, P. Y., 2016. “Estimat- tion in Engineering Conference (CIE) of International 19 Copyright © by ASME Design Engineering Technical Conferences and Com- design process: Human-inspired heuristic guidance puters and Information in Engineering Conference. through learned visual design agents”. In Proceedings V009T09A072. of the Design Society: DESIGN Conference, Vol. 1, [100] Greminger, M., 2020. “Generative adversarial net- Cambridge University Press, pp. 355–364. works with synthetic training data for enforcing man- [110] Yukish, M. A., Stump, G. M., and Miller, S. W., ufacturing constraints on topology optimization”. In 2020. “Using recurrent neural networks to model spa- International Design Engineering Technical Confer- tial grammars for design creation”. Journal of Me- ences and Computers and Information in Engineer- chanical Design, 142(10), p. 104501. ing Conference, Vol. 84003, American Society of Me- [111] Zhu, J.-H., Zhang, W.-H., and Xia, L., 2016. “Topol- chanical Engineers, p. V11AT11A005. ogy optimization in aircraft and aerospace structures [101] Fujita, K., Minowa, K., Nomaguchi, Y., Yamasaki, S., design”. Archives of Computational Methods in Engi- and Yaji, K., 2021. “Design concept generation with neering, 23(4), pp. 595–622. variational deep embedding over comprehensive opti- [112] Xia, L., and Breitkopf, P., 2017. “Recent advances on mization”. In International Design Engineering Tech- topology optimization of multiscale nonlinear struc- nical Conferences and Computers and Information in tures”. Archives of Computational Methods in Engi- Engineering Conference, IDETC-21, ASME. neering, 24(2), pp. 227–249. [102] Cang, R., Vipradas, A., and Ren, Y., 2017. [113] Borrvall, T., and Petersson, J., 2003. “Topology opti- “Scalable microstructure reconstruction with multi- mization of ﬂuids in stokes ﬂow”. International jour- scale pattern preservation”. In International De- nal for numerical methods in ﬂuids, 41(1), pp. 77– sign Engineering Technical Conferences and Com- 107. puters and Information in Engineering Conference, [114] Zhou, S., and Li, Q., 2008. “A variational level Vol. 58134, American Society of Mechanical Engi- set method for the topology optimization of steady- neers, p. V02BT03A010. state navier–stokes ﬂow”. Journal of Computational [103] Cang, R., Xu, Y., Chen, S., Liu, Y., Jiao, Y., and Physics, 227(24), pp. 10178–10195. Yi Ren, M., 2017. “Microstructure representation and [115] Zegard, T., and Paulino, G. H., 2016. “Bridging reconstruction of heterogeneous materials via deep topology optimization and additive manufacturing”. belief network for computational material design”. Structural and Multidisciplinary Optimization, 53(1), Journal of Mechanical Design, 139(7), p. 071404. pp. 175–192. [104] Fokina, D., Muravleva, E., Ovchinnikov, G., and Os- [116] Langelaar, M., 2016. “Topology optimization of eledets, I., 2020. “Microstructure synthesis using 3d self-supporting structures for additive manufactur- style-based generative adversarial networks”. Phys- ing”. Additive Manufacturing, 12, pp. 60–70. ical Review E, 101(4), p. 043308. [117] Dbouk, T., 2017. “A review about the engineering de- [105] Wang, Z., Xian, W., Baccouche, M. R., Lanzerath, sign of optimal heat transfer systems using topology H., Li, Y., and Xu, H., 2021. “A gaussian mixture optimization”. Applied Thermal Engineering, 112, variational autoencoder-based approach for designing pp. 841–854. phononic bandgap metamaterials”. In International [118] Koga, A. A., Lopes, E. C. C., Nova, H. F. V., De Lima, Design Engineering Technical Conferences and Com- C. R., and Silva, E. C. N., 2013. “Development of puters and Information in Engineering Conference, heat sink device by using topology optimization”. In- IDETC-21, ASME. ternational Journal of Heat and Mass Transfer, 64, [106] Cang, R., and Ren, M. Y., 2016. “Deep Network- pp. 759–772. Based Feature Extraction and Reconstruction of Com- [119] Gatys, L., Ecker, A. S., and Bethge, M., 2015. “Tex- plex Material Microstructures”. Vol. Volume 2B: 42nd ture synthesis using convolutional neural networks”. Design Automation Conference of International De- Advances in neural information processing systems, sign Engineering Technical Conferences and Com- 28, pp. 262–270. puters and Information in Engineering Conference. [120] Gatys, L. A., Ecker, A. S., and Bethge, M., 2016. V02BT03A008. “Image style transfer using convolutional neural net- [107] Vermeer, K., Kuppens, R., and Herder, J., 2018. works”. In Proceedings of the IEEE conference on “Kinematic Synthesis Using Reinforcement Learn- computer vision and pattern recognition, pp. 2414– ing”. Vol. Volume 2A: 44th Design Automation Con- 2423. ference of International Design Engineering Techni- [121] Berthelot, D., Schumm, T., and Metz, L., 2017. “Be- cal Conferences and Computers and Information in gan: Boundary equilibrium generative adversarial net- Engineering Conference. V02AT03A009. works”. arXiv preprint arXiv:1703.10717. [108] Raina, A., McComb, C., and Cagan, J., 2019. “Learn- [122] Jiang, Z., Zheng, Y., Tan, H., Tang, B., and Zhou, H., ing to design from humans: Imitating human design- 2017. “Variational deep embedding: an unsupervised ers through deep learning”. Journal of Mechanical and generative approach to clustering”. In Proceed- Design, 141(11). ings of the 26th International Joint Conference on Ar- [109] Puentes, L., Raina, A., Cagan, J., and McComb, tiﬁcial Intelligence, pp. 1965–1972. C., 2020. “Modeling a strategic human engineering [123] Ledig, C., Theis, L., Huszar ´ , F., Caballero, J., Cun- 20 Copyright © by ASME ningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, [136] Kulesza, A., Taskar, B., et al., 2012. “Determinantal J., Wang, Z., et al., 2017. “Photo-realistic single image point processes for machine learning”. Foundations super-resolution using a generative adversarial net- and Trends® in Machine Learning, 5(2–3), pp. 123– work”. In 2017 IEEE Conference on Computer Vision 286. and Pattern Recognition (CVPR), IEEE, pp. 105–114. [137] Chen, W., and Ahmed, F., 2021. “Mo-padgan: [124] Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A., 2017. Reparameterizing engineering designs for augmented “Image-to-image translation with conditional adver- multi-objective optimization”. Applied Soft Comput- sarial networks”. In Proceedings of the IEEE con- ing, 113, p. 107909. ference on computer vision and pattern recognition, [138] Li, J., Xu, K., Chaudhuri, S., Yumer, E., Zhang, H., pp. 1125–1134. and Guibas, L., 2017. “Grass: Generative recursive [125] He, K., Zhang, X., Ren, S., and Sun, J., 2016. “Deep autoencoders for shape structures”. ACM Transac- residual learning for image recognition”. In Proceed- tions on Graphics (TOG), 36(4), pp. 1–14. ings of the IEEE conference on computer vision and [139] Zou, C., Yumer, E., Yang, J., Ceylan, D., and Hoiem, pattern recognition, pp. 770–778. D., 2017. “3d-prnn: Generating shape primitives with [126] Hu, J., Shen, L., and Sun, G., 2018. “Squeeze-and- recurrent neural networks”. In Proceedings of the excitation networks”. In Proceedings of the IEEE con- IEEE International Conference on Computer Vision, ference on computer vision and pattern recognition, pp. 900–909. pp. 7132–7141. [140] Groueix, T., Fisher, M., Kim, V. G., Russell, B. C., [127] Long, J., Shelhamer, E., and Darrell, T., 2015. “Fully and Aubry, M., 2018. “A papier-mach ˆ e ´ approach to convolutional networks for semantic segmentation”. learning 3d surface generation”. In Proceedings of In Proceedings of the IEEE conference on computer the IEEE conference on computer vision and pattern vision and pattern recognition, pp. 3431–3440. recognition, pp. 216–224. [128] Bostanabad, R., Zhang, Y., Li, X., Kearney, T., Brin- [141] Gao, L., Yang, J., Wu, T., Yuan, Y.-J., Fu, H., Lai, son, L. C., Apley, D. W., Liu, W. K., and Chen, W., Y.-K., and Zhang, H., 2019. “Sdm-net: Deep gener- 2018. “Computational microstructure characteriza- ative network for structured deformable mesh”. ACM tion and reconstruction: Review of the state-of-the-art Transactions on Graphics (TOG), 38(6), pp. 1–15. techniques”. Progress in Materials Science, 95, pp. 1– [142] Mo, K., Guerrero, P., Yi, L., Su, H., Wonka, P., Mitra, 41. N. J., and Guibas, L. J., 2019. “Structurenet: hierar- [129] Lee, H., Grosse, R., Ranganath, R., and Ng, A. Y., chical graph networks for 3d shape generation”. ACM 2009. “Convolutional deep belief networks for scal- Transactions on Graphics, 38(6), pp. 1–19. able unsupervised learning of hierarchical representa- [143] Park, J. J., Florence, P., Straub, J., Newcombe, R., and tions”. In Proceedings of the 26th annual international Lovegrove, S., 2019. “Deepsdf: Learning continuous conference on machine learning, pp. 609–616. signed distance functions for shape representation”. In [130] Yu, S., Zhang, Y., Wang, C., Lee, W.-k., Dong, Proceedings of the IEEE/CVF Conference on Com- B., Odom, T. W., Sun, C., and Chen, W., 2017. puter Vision and Pattern Recognition, pp. 165–174. “Characterization and design of functional quasi- [144] Liu, S., Giles, L., and Ororbia, A., 2018. “Learning random nanostructured materials using spectral den- a hierarchical latent-variable model of 3d shapes”. In sity function”. Journal of Mechanical Design, 139(7), 2018 International Conference on 3D Vision (3DV), p. 071401. IEEE, pp. 542–551. [131] Molesky, S., Lin, Z., Piggott, A. Y., Jin, W., Vuckovic, ´ [145] Qi, C. R., Su, H., Mo, K., and Guibas, L. J., 2017. J., and Rodriguez, A. W., 2018. “Inverse design in “Pointnet: Deep learning on point sets for 3d classiﬁ- nanophotonics”. Nature Photonics, 12(11), pp. 659– cation and segmentation”. In Proceedings of the IEEE 670. conference on computer vision and pattern recogni- [132] Dilokthanakul, N., Mediano, P. A., Garnelo, M., Lee, tion, pp. 652–660. M. C., Salimbeni, H., Arulkumaran, K., and Shana- [146] Karnewar, A., and Wang, O., 2020. “Msg-gan: Multi- han, M., 2016. “Deep unsupervised clustering with scale gradients for generative adversarial networks”. gaussian mixture variational autoencoders”. arXiv In Proceedings of the IEEE/CVF Conference on Com- preprint arXiv:1611.02648. puter Vision and Pattern Recognition, pp. 7799–7808. [133] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., [147] van Hasselt, H., Guez, A., and Silver, D., 2016. “Deep and Klimov, O., 2017. “Proximal policy optimization reinforcement learning with double q-learning”. Pro- algorithms”. arXiv preprint arXiv:1707.06347. ceedings of the AAAI Conference on Artiﬁcial Intelli- [134] Gielis, J., 2003. “A generic geometric transforma- gence, 30(1), Mar. tion that uniﬁes a wide range of natural and abstract [148] Odena, A., Olah, C., and Shlens, J., 2017. “Condi- shapes”. American journal of botany, 90(3), pp. 333– tional image synthesis with auxiliary classiﬁer gans”. 338. In International conference on machine learning, [135] Ha, D., and Eck, D., 2018. “A neural representation PMLR, pp. 2642–2651. of sketch drawings”. In International Conference on [149] McComb, C., Cagan, J., and Kotovsky, K., 2018. Learning Representations. “Data on the design of truss structures by teams of en- 21 Copyright © by ASME gineering students”. Data in brief, 18, pp. 160–163. and Information in Engineering Conference, IDETC- [150] Regenwetter, L., Curry, B., and Ahmed, F., 2022. 21, ASME. “Biked: A dataset for computational bicycle design [162] Regenwetter, L., Weaver, C., and Ahmed, F., 2022. with machine learning benchmarks”. Journal of Me- Framed: Data-driven structural performance analysis chanical Design, 144(3). of community-designed bicycle frames. [151] Hunter, W., et al., 2017. Topy-topology optimization [163] Chan, Y.-C., Ahmed, F., Wang, L., and Chen, W., with python. 2021. “Metaset: Exploring shape and property spaces [152] Iren, D., Ackermann, M., Gorfer, J., Pujar, G., for data-driven metamaterials design”. Journal of Me- Wesselmecking, S., Krupp, U., and Bromuri, S., chanical Design, 143(3), p. 031707. 2021. “Aachen-heerlen annotated steel microstructure [164] Wang, L., van Beek, A., Da, D., Chan, Y.-C., Zhu, P., dataset”. Scientiﬁc Data, 8(1), pp. 1–9. and Chen, W., 2021. “Data-driven multiscale design [153] Larmuseau, M., Sluydts, M., Theuwissen, K., Duprez, of cellular composites with multiclass microstructures L., Dhaene, T., and Cottenier, S., 2020. “Compact for natural frequency maximization”. arXiv preprint representations of microstructure images using triplet arXiv:2106.06478. networks”. npj Computational Materials, 6(1), pp. 1– [165] Jongejan, J., Rowley, H., Kawashima, T., Kim, J., and 11. Fox-Gieg, N., 2016. “The quick, draw!-ai experi- [154] DeCost, B. L., Hecht, M. D., Francis, T., Webler, ment”. Mount View, CA, accessed Feb, 17(2018), p. 4. B. A., Picard, Y. N., and Holm, E. A., 2017. “Uhcsdb: [166] Lopez, C., Miller, S. R., and Tucker, C. S., ultrahigh carbon steel micrograph database”. Integrat- 2018. “Human validation of computer vs human ing Materials and Manufacturing Innovation, 6(2), generated design sketches”. In International De- pp. 197–205. sign Engineering Technical Conferences and Com- [155] Zhao, H., Wang, Y., Lin, A., Hu, B., Yan, R., Mc- puters and Information in Engineering Conference, Cusker, J., Chen, W., McGuinness, D. L., Schadler, Vol. 51845, American Society of Mechanical Engi- L., and Brinson, L. C., 2018. “Nanomine schema: An neers, p. V007T06A015. extensible data representation for polymer nanocom- [167] Toh, C. A., and Miller, S. R., 2013. “Exploring the posites”. APL Materials, 6(11), p. 111108. utility of product dissection for early-phase idea gen- [156] Chang, A. X., Funkhouser, T., Guibas, L., Hanra- eration”. In International Design Engineering Tech- han, P., Huang, Q., Li, Z., Savarese, S., Savva, M., nical Conferences and Computers and Information in Song, S., Su, H., et al., 2015. “Shapenet: An Engineering Conference, Vol. 55928, American Soci- information-rich 3d model repository”. arXiv preprint ety of Mechanical Engineers, p. V005T06A034. arXiv:1512.03012. [168] Liang, L., Liu, M., Martin, C., and Sun, W., 2018. “A [157] Mo, K., Zhu, S., Chang, A. X., Yi, L., Tripathi, deep learning approach to estimate stress distribution: S., Guibas, L. J., and Su, H., 2019. “Partnet: A a fast and accurate surrogate of ﬁnite-element analy- large-scale benchmark for ﬁne-grained and hierarchi- sis”. Journal of The Royal Society Interface, 15(138), cal part-level 3d object understanding”. In Proceed- p. 20170844. ings of the IEEE/CVF Conference on Computer Vi- [169] Jiang, H., Nie, Z., Yeo, R., Farimani, A. B., and sion and Pattern Recognition, pp. 909–918. Kara, L. B., 2020. “StressGAN: A Generative Deep [158] Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, Learning Model for 2D Stress Distribution Predic- X., and Xiao, J., 2015. “3d shapenets: A deep rep- tion”. Vol. Volume 11B: 46th Design Automation resentation for volumetric shapes”. In Proceedings of Conference (DAC) of International Design Engineer- the IEEE conference on computer vision and pattern ing Technical Conferences and Computers and Infor- recognition, pp. 1912–1920. mation in Engineering Conference. V11BT11A023. [159] Kim, S., Chi, H.-g., Hu, X., Huang, Q., and Ra- [170] Nie, Z., Jiang, H., and Kara, L. B., 2019. “Stress Field mani, K., 2020. “A large-scale annotated mechani- Prediction in Cantilevered Structures Using Convolu- cal components benchmark for classiﬁcation and re- tional Neural Networks”. Vol. Volume 1: 39th Com- trieval tasks with deep neural networks”. In Proceed- puters and Information in Engineering Conference of ings of 16th European Conference on Computer Vi- International Design Engineering Technical Confer- sion (ECCV). ences and Computers and Information in Engineering [160] Willis, K. D., Pu, Y., Luo, J., Chu, H., Du, T., Lam- Conference. V001T02A011. bourne, J. G., Solar-Lezama, A., and Matusik, W., [171] Nie, Z., Jiang, H., and Kara, L. B., 2019. “Stress 2021. “Fusion 360 gallery: A dataset and environ- ﬁeld prediction in cantilevered structures using con- ment for programmatic cad construction from human volutional neural networks”. Journal of Computing design sequences”. ACM Transactions on Graphics and Information Science in Engineering, 20(1), Sep. (TOG), 40(4), pp. 1–24. [172] Pfaff, T., Fortunato, M., Sanchez-Gonzalez, A., and [161] Nobari, A. H., Rashad, M. F., and Ahmed, F., 2021. Battaglia, P., 2020. “Learning mesh-based simulation “Creativegan: Editing generative adversarial networks with graph networks”. In International Conference on for creative design synthesis”. In International Design Learning Representations. Engineering Technical Conferences and Computers [173] Kochkov, D., Smith, J. A., Alieva, A., Wang, Q., 22 Copyright © by ASME Brenner, M. P., and Hoyer, S., 2021. “Machine learning–accelerated computational ﬂuid dynamics”. Proceedings of the National Academy of Sciences, 118(21). [174] Duraisamy, K., Iaccarino, G., and Xiao, H., 2019. “Turbulence modeling in the age of data”. Annual Re- view of Fluid Mechanics, 51(1), pp. 357–377. [175] Kim, B., Azevedo, V. C., Thuerey, N., Kim, T., Gross, M., and Solenthaler, B., 2019. “Deep ﬂuids: A gen- erative network for parameterized ﬂuid simulations”. Computer Graphics Forum, 38(2), May, p. 59–70. [176] Dering, M. L., and Tucker, C. S., 2017. “Generative adversarial networks for increasing the veracity of big data”. In 2017 IEEE International Conference on Big Data (Big Data), IEEE, pp. 2595–2602. [177] Panchal, J. H., Fuge, M., Liu, Y., Missoum, S., and Tucker, C., 2019. “Special Issue: Machine Learning for Engineering Design”. Journal of Mechanical De- sign, 141(11), 10. 110301. [178] Elgammal, A., Liu, B., Elhoseiny, M., and Mazzone, M., 2017. “Can: Creative adversarial networks gen- erating “art” by learning about styles and deviating from style norms”. In 8th International Conference on Computational Creativity, ICCC 2017, Georgia In- stitute of Technology. [179] Franceschelli, G., and Musolesi, M., 2021. Creativity and machine learning: A survey. [180] Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I., 2020. “Generative pre- training from pixels”. In Proceedings of the 37th In- ternational Conference on Machine Learning, H. D. III and A. Singh, eds., Vol. 119 of Proceedings of Ma- chine Learning Research, PMLR, pp. 1691–1703. [181] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I., 2021. Zero- shot text-to-image generation. [182] Dhariwal, P., Jun, H., Payne, C., Kim, J. W., Radford, A., and Sutskever, I., 2020. Jukebox: A generative model for music. [183] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I., 2021. Learning transferable visual models from natural lan- guage supervision. 23 Copyright © by ASME http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Journal of Mechanical Design Unpaywall http://www.deepdyve.com/lp/unpaywall/deep-generative-models-in-engineering-design-a-review-yEegT8VtHg

Loading next page...

References (199)

Wei Ma, Feng Cheng, Yihao Xu, Qinlong Wen, Yongmin Liu (2019)
Probabilistic Representation and Inverse Design of Metamaterials Based on a Deep Generative Model with Semi‐Supervised Learning Strategy
Advanced Materials, 31
Matthew Dering, Conrad Tucker (2017)
Generative adversarial networks for increasing the veracity of big data
2017 IEEE International Conference on Big Data (Big Data)
S. Patalano, F. Vitolo, A. Lanzotti (2016)
A Graph-based Software Tool for the CAD Modeling of Mechanical Assemblies
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei Efros (2016)
Image-to-Image Translation with Conditional Adversarial Networks
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Tero Karras, S. Laine, M. Aittala, Janne Hellsten, J. Lehtinen, Timo Aila (2019)
Analyzing and Improving the Image Quality of StyleGAN
2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Lin Gao (2019)
SDM-NET: Deep Generative Network for Structured Deformable Mesh
Jung (2021)
Super-Resolving Material Microstructure Image Via Deep Learning for Microstructure Characterization and Mechanical Behavior Analysis
npj Comput. Mater., 7
L. Mosser, O. Dubrule, M. Blunt (2017)
Reconstruction of three-dimensional porous media using generative adversarial neural networks
Physical review. E, 96 4-1
Diederik Kingma, M. Welling (2013)
Auto-Encoding Variational Bayes
CoRR, abs/1312.6114
Shrinath Deshpande, A. Purwar (2020)
An Image-Based Approach to Variational Path Synthesis of Linkages
J. Comput. Inf. Sci. Eng., 21
L. Puentes, A. Raina, J. Cagan, C. McComb (2020)
MODELING A STRATEGIC HUMAN ENGINEERING DESIGN PROCESS: HUMAN-INSPIRED HEURISTIC GUIDANCE THROUGH LEARNED VISUAL DESIGN AGENTS
Proceedings of the Design Society: DESIGN Conference, 1
(2019)
One-shot 17
Yonggyun Yu, Taeil Hur, Jaeho Jung, I. Jang (2018)
Deep learning for determining a near-optimal topological design without any iteration
Structural and Multidisciplinary Optimization, 59
Amir Hooshmand, M. Campbell (2013)
Topology Optimization of Fluid Channels Using Generative Graph Grammars
Yu (2017)
Characterization and Design of Functional Quasi-Random Nanostructured Materials Using Spectral Density Function
ASME J. Mech. Des., 139
A. Nobari, Wei Chen, Faez Ahmed (2021)
PcDGAN: A Continuous Conditional Diverse Generative Adversarial Network For Inverse Design
Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining
Augustus Odena, C. Olah, Jonathon Shlens (2016)
Conditional Image Synthesis with Auxiliary Classifier GANs
M. Langelaar (2016)
Topology optimization of 3D self-supporting structures for additive manufacturing
Additive manufacturing, 12
L. Xia, P. Breitkopf (2017)
Recent Advances on Topology Optimization of Multiscale Nonlinear Structures
Archives of Computational Methods in Engineering, 24
A. Elgammal, Bingchen Liu, Mohamed Elhoseiny, Marian Mazzone (2017)
CAN: Creative Adversarial Networks, Generating "Art" by Learning About Styles and Deviating from Style Norms
ArXiv, abs/1706.07068
Cun-20 Copyright ©
L. Deng, Dong Yu (2014)
Deep Learning: Methods and Applications
Found. Trends Signal Process., 7
Wentai Zhang, Zhangsihao Yang, Haoliang Jiang, Suyash Nigam, Soji Yamakawa, T. Furuhata, K. Shimada, L. Kara (2019)
3D Shape Synthesis for Conceptual Design and Optimization Using Variational Autoencoders
ArXiv, abs/1904.07964
Tero Karras, S. Laine, Timo Aila (2018)
A Style-Based Generator Architecture for Generative Adversarial Networks
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Diederik Kingma, M. Welling (2019)
An Introduction to Variational Autoencoders
ArXiv, abs/1906.02691
Jongejan (2016)
The Quick, Draw!-ai Experiment
Mount View, CA, Accessed Feb, 17
Ruijin Cang, Hechao Li, Hope Yao, Yang Jiao, Yi Ren (2017)
Improving direct physical properties prediction of heterogeneous materials from imaging data via convolutional neural network and a morphology-aware generative model
Computational Materials Science
Animesh Karnewar, Oliver Wang (2019)
MSG-GAN: Multi-Scale Gradients for Generative Adversarial Networks
2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Amaresh Chakrabarti, K. Shea, R. Stone, J. Cagan, M. Campbell, N. Hernandez, K. Wood (2011)
Computer-Based Design Synthesis Research: An Overview
J. Comput. Inf. Sci. Eng., 11
C. Qi, Hao Su, Kaichun Mo, L. Guibas (2016)
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Weijuan Cao, T. Robinson, Yang Hua, F. Boussuge, Andrew Colligan, Wanbin Pan (2020)
Graph Representation of 3D CAD Models for Machining Feature Recognition With Deep Learning
Volume 11A: 46th Design Automation Conference (DAC)
Kaiming He, X. Zhang, Shaoqing Ren, Jian Sun (2015)
Deep Residual Learning for Image Recognition
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Sharad Rawat, M. Shen (2019)
Application of Adversarial Networks for 3D Structural Topology Optimization
SAE Technical Paper Series
I. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio (2014)
Generative adversarial networks
Communications of the ACM, 63
Kaichun Mo, Shilin Zhu, Angel Chang, L. Yi, Subarna Tripathi, L. Guibas, Hao Su (2018)
PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Evan Shelhamer, Jonathan Long, Trevor Darrell (2014)
Fully convolutional networks for semantic segmentation
2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Zhaocheng Liu, L. Raju, Dayu Zhu, W. Cai (2019)
A Hybrid Strategy for the Discovery and Design of Photonic Structures
IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 10
Zhirong Wu, Shuran Song, A. Khosla, F. Yu, Linguang Zhang, Xiaoou Tang, Jianxiong Xiao (2014)
3D ShapeNets: A deep representation for volumetric shapes
2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Chuhang Zou, Ersin Yumer, Jimei Yang, Duygu Ceylan, Derek Hoiem (2017)
3D-PRNN: Generating Shape Primitives with Recurrent Neural Networks
2017 IEEE International Conference on Computer Vision (ICCV)
Ivan Sosnovik, I. Oseledets (2017)
Neural networks for topology optimization
Russian Journal of Numerical Analysis and Mathematical Modelling, 34
L. Liang, Minliang Liu, C. Martin, Wei Sun (2018)
A deep learning approach to estimate stress distribution: a fast and accurate surrogate of finite-element analysis
Journal of The Royal Society Interface, 15
Lyle Regenwetter, Brent Curry, Faez Ahmed (2021)
BIKED: A Dataset and Machine Learning Benchmarks for Data-Driven Bicycle Design
ArXiv, abs/2103.05844
M. Daneshmand, A. Helmi, Egils Avots, F. Noroozi, Fatih Alisinanoglu, Hasan Arslan, Jelena Gorbova, R. Haamer, C. Ozcinar, G. Anbarjafari (2018)
3D Scanning: A Comprehensive Survey
ArXiv, abs/1801.08863
Ruijin Cang, Max Ren (2016)
Deep Network-Based Feature Extraction and Reconstruction of Complex Material Microstructures
Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei Efros (2017)
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks
2017 IEEE International Conference on Computer Vision (ICCV)
Kochkov (2021)
Machine Learning–Accelerated Computational Fluid Dynamics
Proc. Natl. Acad. Sci. USA, 118
Yu-Chin Chan, Faez Ahmed, Liwei Wang, Wei Chen (2020)
METASET: Exploring Shape and Property Spaces for Data-Driven Metamaterials Design
ArXiv, abs/2006.02142
Vahid Keshavarzzadeh, Mitra Alirezaei, T. Tasdizen, R. Kirby (2021)
Image-Based Multiresolution Topology Optimization Using Deep Disjunctive Normal Shape Model
Comput. Aided Des., 130
Hongrui Chen, Xingchen Liu (2021)
Geometry Enhanced Generative Adversarial Networks for Random Heterogeneous Material Representation
Volume 3A: 47th Design Automation Conference (DAC)
Nat Dilokthanakul, P. Mediano, M. Garnelo, M. Lee, Hugh Salimbeni, Kai Arulkumaran, M. Shanahan (2016)
Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders
ArXiv, abs/1611.02648
Ruijin Cang, Hope Yao, Yi Ren (2018)
One-shot generation of near-optimal topology through theory-driven machine learning
Comput. Aided Des., 109
David Berthelot, Tom Schumm, Luke Metz (2017)
BEGAN: Boundary Equilibrium Generative Adversarial Networks
ArXiv, abs/1703.10717
The Engineering Conference
Nature, 76
Hui Zhang, Lei Yang, Changjian Li, Bojian Wu, Wenping Wang (2021)
ScaffoldGAN: Synthesis of Scaffold Materials based on Generative Adversarial Networks
Comput. Aided Des., 138
Daria Fokina, E. Muravleva, G. Ovchinnikov, I. Oseledets (2019)
Microstructure synthesis using style-based generative adversarial networks.
Physical review. E, 101 4-1
Xian Lee, Aditya Balu, D. Stoecklein, B. Ganapathysubramanian, S. Sarkar (2019)
A Case Study of Deep Reinforcement Learning for Engineering Design: Application to Microfluidic Devices for Flow Sculpting
Journal of Mechanical Design
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, P. Talukdar (2019)
Composition-based Multi-Relational Graph Convolutional Networks
ArXiv, abs/1911.03082
Siyan Liu, Zhi Zhong, Ali Takbiri-Borujeni, Mohammad Kazemi, Qinwen Fu, Yuhao Yang (2019)
A case study on homogeneous and heterogeneous reservoir porous media reconstruction by using generative adversarial networks
Energy Procedia
Runze Li, Yufei Zhang, Haixin Chen (2020)
Learning the aerodynamic design of supercritical airfoils through deep reinforcement learning
ArXiv, abs/2010.03651
H. Hasselt, A. Guez, David Silver (2015)
Deep Reinforcement Learning with Double Q-Learning
Shiyang Cheng, M. Bronstein, Yuxiang Zhou, I. Kotsia, M. Pantic, S. Zafeiriou (2019)
MeshGAN: Non-linear 3D Morphable Models of Faces
ArXiv, abs/1903.10384
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Kim, Alec Radford, I. Sutskever (2020)
Jukebox: A Generative Model for Music
ArXiv, abs/2005.00341
Christopher McComb, J. Cagan, K. Kotovsky (2018)
Data on the design of truss structures by teams of engineering students
Data in Brief, 18
A. Ramesh, Mikhail Pavlov, Gabriel Goh, S. Gray, Chelsea Voss, Alec Radford, Mark Chen, I. Sutskever (2021)
Zero-Shot Text-to-Image Generation
ArXiv, abs/2102.12092
Zijiang Yang, Xiaolin Li, L. Brinson, A. Choudhary, Wei Chen, Ankit Agrawal (2018)
Microstructural Materials Design Via Deep Adversarial Learning Methodology
Journal of Mechanical Design
Shashank Sharma, A. Purwar (2020)
Path Synthesis of Defect-Free Spatial 5-SS Mechanisms Using Machine Learning
Volume 10: 44th Mechanisms and Robotics Conference (MR)
He Zhao, Yixing Wang, Anqi Lin, Bingyin Hu, Rui Yan, Jamie McCusker, Wei Chen, D. McGuinness, L. Schadler, L. Brinson (2018)
NanoMine schema: An extensible data representation for polymer nanocomposites
APL Materials
M. Yukish, G. Stump, S. Miller (2020)
Using Recurrent Neural Networks to Model Spatial Grammars for Design Creation
Journal of Mechanical Design, 142
James Cunningham, C. López, O. Ashour, Conrad Tucker (2020)
Multi-Context Generation in Virtual Reality Environments Using Deep Reinforcement Learning
Volume 9: 40th Computers and Information in Engineering Conference (CIE)
Dule Shu, James Cunningham, G. Stump, S. Miller, M. Yukish, T. Simpson, Conrad Tucker (2020)
3D Design Using Generative Adversarial Networks and Physics-Based Validation
Journal of Mechanical Design, 142
T. Dbouk (2017)
A review about the engineering design of optimal heat transfer systems using topology optimization
Applied Thermal Engineering, 112
Copyright © by ASME
Lyle Regenwetter, Brent Curry, Faez Ahmed (2021)
BIKED: A Dataset for Computational Bicycle Design with Machine Learning Benchmarks
Journal of Mechanical Design
Aleksandar Bojchevski, Oleksandr Shchur, Daniel Zügner, Stephan Günnemann (2018)
NetGAN: Generating Graphs via Random Walks
ArXiv, abs/1803.00816
John Schulman, F. Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov (2017)
Proximal Policy Optimization Algorithms
ArXiv, abs/1707.06347
Honglak Lee, R. Grosse, R. Ranganath, A. Ng (2009)
Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations
Christine Toh, Scarlett Miller (2013)
Exploring the Utility of Product Dissection for Early-Phase Idea Generation
Wenjian Yang, H. Ding, Bin Zi, Dan Zhang (2018)
New Graph Representation for Planetary Gear Trains
Journal of Mechanical Design, 140
Jae Lee, Kwangsoo Kim (1996)
Geometric reasoning for knowledge-based parametric design using graph representation
Comput. Aided Des., 28
G. Stump, S. Miller, M. Yukish, T. Simpson, Conrad Tucker (2019)
Spatial Grammar-Based Recurrent Neural Network for Design Form and Behavior Optimization
Journal of Mechanical Design
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, P. Lio’, Yoshua Bengio (2017)
Graph Attention Networks
ArXiv, abs/1710.10903
Renjie Liao, Yujia Li, Yang Song, Shenlong Wang, C. Nash, William Hamilton, D. Duvenaud, R. Urtasun, R. Zemel (2019)
Efficient Graph Generation with Graph Recurrent Attention Networks
Design Engineering Technical Conferences and Computers and Information
S. Molesky, Zin Lin, Alexander Piggott, Weiliang Jin, J. Vučković, Alejandro Rodriguez (2018)
Inverse design in nanophotonics
Nature Photonics, 12
Jie Hu, Li Shen, Samuel Albanie, Gang Sun, E. Wu (2017)
Squeeze-and-Excitation Networks
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
W. Chen, Faez Ahmed (2021)
PaDGAN: Learning to Generate High-Quality Novel Designs
Journal of Mechanical Design, 143
Simon Hessner, Saquib Sarfraz (2018)
Image Style Transfer using Convolutional Neural Networks
Xiang Li, Shaowu Ning, Zhanli Liu, Ziming Yan, C. Luo, Zhuang Zhuo (2020)
Designing phononic crystal with anticipated band gap through a deep learning based data-driven method
Computer Methods in Applied Mechanics and Engineering, 361
L. Kaelbling, M. Littman, A. Moore (1996)
Reinforcement Learning: A Survey
J. Artif. Intell. Res., 4
Andrew Brock, Theodore Lim, J. Ritchie, Nick Weston (2016)
Context-Aware Content Generation for Virtual Environments
Wei Chen, Faez Ahmed (2020)
MO-PaDGAN: Reparameterizing Engineering Designs for Augmented Multi-objective Optimization
ArXiv, abs/2009.07110
Liwei Wang, Yu-Chin Chan, Faez Ahmed, Zhao Liu, Ping Zhu, Wei Chen (2020)
Deep Generative Modeling for Mechanistic-based Learning and Design of Metamaterial Systems
ArXiv, abs/2006.15274
Sangeun Oh, Yongsu Jung, Seongsin Kim, Ikjin Lee, Namwoo Kang (2019)
Deep Generative Design: Integration of Topology Optimization and Generative Models
Journal of Mechanical Design
Angel Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Qi-Xing Huang, Zimo Li, S. Savarese, M. Savva, Shuran Song, Hao Su, Jianxiong Xiao, L. Yi, F. Yu (2015)
ShapeNet: An Information-Rich 3D Model Repository
ArXiv, abs/1512.03012
Huaiyu Zhu (1997)
On Information and Sufficiency
Renkai Tan, N. Zhang, W. Ye (2020)
A deep learning–based method for the design of microstructural materials
Structural and Multidisciplinary Optimization, 61
C. López, O. Ashour, Conrad Tucker (2019)
Reinforcement Learning Content Generation for Virtual Reality Applications
Volume 1: 39th Computers and Information in Engineering Conference
Giorgio Franceschelli, Mirco Musolesi (2021)
Creativity and Machine Learning: A Survey
ArXiv, abs/2104.02726
T. Borrvall, J. Petersson (2003)
Topology optimization of fluids in Stokes flow
International Journal for Numerical Methods in Fluids, 41
Prafulla Dhariwal, Alex Nichol (2021)
Diffusion Models Beat GANs on Image Synthesis
ArXiv, abs/2105.05233
Baotong Li, Congjia Huang, Xin Li, Shuai Zheng, Jun Hong (2019)
Non-iterative structural topology optimization using deep learning
Comput. Aided Des., 115
Tomás Zegard, G. Paulino (2016)
Bridging topology optimization and additive manufacturing
Structural and Multidisciplinary Optimization, 53
Volodymyr Mnih, K. Kavukcuoglu, David Silver, Andrei Rusu, J. Veness, Marc Bellemare, Alex Graves, Martin Riedmiller, A. Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, D. Kumaran, Daan Wierstra, S. Legg, D. Hassabis (2015)
Human-level control through deep reinforcement learning
Nature, 518
Jihong Zhu, Weihong Zhang, L. Xia (2016)
Topology Optimization in Aircraft and Aerospace Structures Design
Archives of Computational Methods in Engineering, 23
Leon Gatys, Alexander Ecker, M. Bethge (2016)
Image Style Transfer Using Convolutional Neural Networks
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
E. Coatanéa, Sarayut Nonsiri, François Christophe, Faisal Mokammel (2014)
Graph based representation and analyses for conceptual stages
Karl Willis, Yewen Pu, Jieliang Luo, Hang Chu, Tao Du, J. Lambourne, Armando Solar-Lezama, W. Matusik (2020)
Fusion 360 Gallery: A Dataset and Environment for Programmatic CAD Construction from Human Design Sequences
Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, Hyunwoo Kim (2019)
Graph Transformer Networks
Yujia Li, Daniel Tarlow, Marc Brockschmidt, R. Zemel (2015)
Gated Graph Sequence Neural Networks
arXiv: Learning
Shiwei Zhou, Qing Li (2008)
A variational level set method for the topology optimization of steady-state Navier-Stokes flow
J. Comput. Phys., 227
Yingheng Tang, K. Kojima, T. Koike-Akino, Ye Wang, Pengxiang Wu, Mohammad Tahersima, Devesh Jha, K. Parsons, M. Qi (2020)
Generative Deep Learning Model for a Multi-Level Nano-Optic Broadband Power Splitter
2020 Optical Fiber Communications Conference and Exhibition (OFC)
Jitesh Panchal, M. Fuge, Y. Liu, S. Missoum, Conrad Tucker (2019)
Special Issue: Machine Learning for Engineering Design
Journal of Mechanical Design
A. Raina, L. Puentes, J. Cagan, Christopher McComb (2021)
Goal-Directed Design Agents: Integrating Visual Imitation with One-Step Lookahead Optimization for Generative Design
ArXiv, abs/2110.03223
Kaz Vermeer, Reinier Kuppens, J. Herder (2018)
Kinematic Synthesis Using Reinforcement Learning
Volume 2A: 44th Design Automation Conference
Wei Chen, Kevin Chiu, M. Fuge (2019)
Aerodynamic Design Optimization and Shape Exploration using Generative Adversarial Networks
AIAA Scitech 2019 Forum
Sofia Valdez, C. Seepersad, Sandilya Kambampati (2021)
A Framework for Interactive Structural Design Exploration
Volume 3B: 47th Design Automation Conference (DAC)
Wei Chen, M. Fuge (2018)
BézierGAN: Automatic Generation of Smooth Curves from Interpretable Low-Dimensional Parameters
ArXiv, abs/1808.08871
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, Hanning Zhou (2016)
Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering
Alex Burnap, Ye Liu, Yanxin Pan, Honglak Lee, Rich Gonzalez, P. Papalambros (2016)
Estimating and Exploring the Product Form Design Space Using Deep Generative Models
Tinghao Guo, Danny Lohan, James Allison, Ruijin Cang, Yi Ren (2018)
An indirect design representation for topology optimization using variational autoencoder and style transfer
Mark Chen, Alec Radford, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, D. Luan, I. Sutskever (2020)
Generative Pretraining From Pixels
David Ha, D. Eck (2017)
A Neural Representation of Sketch Drawings
ArXiv, abs/1704.03477
Zheyan Zhang, Yongxing Wang, P. Jimack, He Wang (2020)
MeshingNet: A New Mesh Generation Method Based on Deep Learning
Computational Science – ICCS 2020, 12139
Jiaxuan You, Bowen Liu, Rex Ying, V. Pande, J. Leskovec (2018)
Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation
Tianju Xue, Thomas Wallin, Yigit Menguc, S. Adriaenssens, Maurizio Chiaramonte (2020)
Machine learning generative models for automatic design of multi-material 3D printed composite solids
Extreme Mechanics Letters, 41
Jeong Park, Peter Florence, Julian Straub, Richard Newcombe, S. Lovegrove (2019)
DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Matthew Dering, James Cunningham, Raj Desai, M. Yukish, T. Simpson, Conrad Tucker (2018)
A Physics-Based Virtual Environment for Enhancing the Quality of Deep Generative Designs
Volume 2A: 44th Design Automation Conference
Jonathan Ho, Ajay Jain, P. Abbeel (2020)
Denoising Diffusion Probabilistic Models
ArXiv, abs/2006.11239
M. Greminger (2020)
Generative Adversarial Networks With Synthetic Training Data for Enforcing Manufacturing Constraints on Topology Optimization
Volume 11A: 46th Design Automation Conference (DAC)
M. Behzadi, H. Ilies (2021)
Real-Time Topology Optimization in 3D via Deep Transfer Learning
ArXiv, abs/2102.07657
Alec Radford, Jong Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, I. Sutskever (2021)
Learning Transferable Visual Models From Natural Language Supervision
Jin-Woong Lee, Nam Goo, W. Park, Myungho Pyo, K. Sohn (2020)
Virtual microstructure design for steels using generative adversarial networks
Engineering Reports, 3
Byungsoo Kim, V. Azevedo, N. Thürey, Theodore Kim, M. Gross, B. Solenthaler (2018)
Deep Fluids: A Generative Network for Parameterized Fluid Simulations
Computer Graphics Forum, 38
Mikael Henaff, Joan Bruna, Yann LeCun (2015)
Deep Convolutional Networks on Graph-Structured Data
ArXiv, abs/1506.05163
A. Raina, Christopher McComb, J. Cagan (2019)
Learning to design from humans: Imitating human designers through deep learning
ArXiv, abs/1907.11813
Cheng-Ho Hsu, Kin-Tak Lam (1992)
A New Graph Representation for the Automatic Kinematic Analysis of Planetary Spur-Gear Trains
Journal of Mechanical Design, 114
Goodfellow (2014)
Generative Adversarial Nets
Xiaolin Li, Yichi Zhang, He Zhao, Craig Burkhart, L. Brinson, Wei Chen (2018)
A Transfer Learning Approach for Microstructure Reconstruction and Structure-property Predictions
Scientific Reports, 8
Remondino (2003)
From Point Cloud to Surface: The Modeling and Visualization Problem
Lyle Regenwetter, Colin Weaver, Faez Ahmed (2022)
FRAMED: Data-Driven Structural Performance Analysis of Community-Designed Bicycle Frames
ArXiv, abs/2201.10459
Thibault Groueix, Matthew Fisher, Vladimir Kim, Bryan Russell, Mathieu Aubry (2018)
A Papier-Mache Approach to Learning 3D Surface Generation
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Zhenguo Nie, Haoliang Jiang, L. Kara (2018)
Deep Learning for Stress Field Prediction Using Convolutional Neural Networks
J. Comput. Inf. Sci. Eng., 20
Zhenguo Nie, Tong Lin, Haoliang Jiang, L. Kara (2020)
TopologyGAN: Topology Optimization Using Generative Adversarial Networks Based on Physical Fields Over the Initial Domain
ArXiv, abs/2003.04685
Leon Gatys, Alexander Ecker, M. Bethge (2015)
Texture Synthesis Using Convolutional Neural Networks
Kaichun Mo, Paul Guerrero, L. Yi, Hao Su, Peter Wonka, N. Mitra, L. Guibas (2019)
StructureNet
ACM Transactions on Graphics (TOG), 38
Li (2017)
Grass: Generative Recursive Autoencoders for Shape Structures
ACM Trans. Graph., 36
Ruijin Cang, Aditya Vipradas, Yi Ren (2017)
Scalable Microstructure Reconstruction With Multi-Scale Pattern Preservation
C. López, Scarlett Miller, Conrad Tucker (2018)
Human Validation of Computer vs Human Generated Design Sketches
Volume 7: 30th International Conference on Design Theory and Methodology
Haoliang Jiang, Zhenguo Nie, Roselyn Yeo, A. Farimani, L. Kara (2020)
StressGAN: A Generative Deep Learning Model for 2D Stress Distribution Prediction
ArXiv, abs/2006.11376
Dmitrii Kochkov, Jamie Smith, M. Brenner (2021)
Supplementary Information for Machine learning accelerated computational ﬂuid dynamics
Emre Yilmaz, Brian German (2020)
Conditional Generative Adversarial Network Framework for Airfoil Inverse Design
AIAA AVIATION 2020 FORUM
Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, B. Sengupta, A. Bharath (2017)
Generative Adversarial Networks: An Overview
IEEE Signal Processing Magazine, 35
Wei Chen, M. Fuge (2019)
Synthesizing Designs With Interpart Dependencies Using Hierarchical Generative Adversarial Networks
Journal of Mechanical Design
Jun Li, Kai Xu, S. Chaudhuri, Ersin Yumer, Hao Zhang, L. Guibas (2017)
Deformation-driven shape correspondence via shape recognition
ACM Transactions on Graphics (TOG), 36
Shuangcheng Yu, Yichi Zhang, Chen Wang, Won-Kyu Lee, B. Dong, T. Odom, Cheng Sun, Wei Chen (2016)
Characterization and Design of Functional Quasi-Random Nanostructured Materials Using Spectral Density Function
Kihyuk Sohn, Honglak Lee, Xinchen Yan (2015)
Learning Structured Output Representation using Deep Conditional Generative Models
Martín Arjovsky, Soumith Chintala, L. Bottou (2017)
Wasserstein Generative Adversarial Networks
Liwei Wang, A. Beek, D. Da, Yu-Chin Chan, Ping Zhu, Wei Chen (2021)
Data-Driven Multiscale Design of Cellular Composites with Multiclass Microstructures for Natural Frequency Maximization
ArXiv, abs/2106.06478
Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, Aaron Courville (2017)
Improved Training of Wasserstein GANs
Michiel Larmuseau, Michael Sluydts, K. Theuwissen, L. Duprez, T. Dhaene, S. Cottenier (2020)
Compact representations of microstructure images using triplet networks
npj Computational Materials, 6
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, I. Sutskever, P. Abbeel (2016)
InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
Lee (2021)
Virtual Microstructure Design for Steels Using Generative Adversarial Networks
Eng. Rep., 3
Shikun Liu, C. Giles, Alexander Ororbia (2017)
Learning a Hierarchical Latent-Variable Model of 3D Shapes
2018 International Conference on 3D Vision (3DV)
Jaimyun Jung, Juwon Na, Hyung Park, Jeong-Min Park, Gyuwon Kim, Seungchul Lee, Hyoung-Seop Kim (2021)
Super-resolving material microstructure image via deep learning for microstructure characterization and mechanical behavior analysis
npj Computational Materials, 7
J. Gielis (2003)
A generic geometric transformation that unifies a wide range of natural and abstract shapes.
American journal of botany, 90 3
C. Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Aitken, A. Tejani, J. Totz, Zehan Wang, Wenzhe Shi (2016)
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
A. Nobari, Wei Chen, Faez Ahmed (2021)
RANGE-GAN: Design Synthesis Under Constraints Using Conditional Generative Adversarial Networks
Journal of Mechanical Design
Yujia Li, O. Vinyals, Chris Dyer, Razvan Pascanu, P. Battaglia (2018)
Learning Deep Generative Models of Graphs
ArXiv, abs/1803.03324
Nicola Cao, Thomas Kipf (2018)
MolGAN: An implicit generative model for small molecular graphs
ArXiv, abs/1805.11973
Mehdi Mirza, Simon Osindero (2014)
Conditional Generative Adversarial Nets
ArXiv, abs/1411.1784
Itzik Malkiel, M. Mrejen, Achiya Nagler, U. Arieli, Lior Wolf, H. Suchowski (2018)
Plasmonic nanostructure design and characterization via Deep Learning
Light, Science & Applications, 7
Li (2021)
Learning the Aerodynamic Design of Supercritical Airfoils Through Deep Reinforcement Learning
AIAA J., 59
Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, Michael Black (2018)
Generating 3D faces using Convolutional Mesh Autoencoders
ArXiv, abs/1807.10267
Conner Sharpe, C. Seepersad (2019)
Topology Design With Conditional Generative Adversarial Networks
Volume 2A: 45th Design Automation Conference
Martín Arjovsky, L. Bottou (2017)
Towards Principled Methods for Training Generative Adversarial Networks
ArXiv, abs/1701.04862
Yunjey Choi, Youngjung Uh, Jaejun Yoo, Jung-Woo Ha (2019)
StarGAN v2: Diverse Image Synthesis for Multiple Domains
2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Tim Salimans, I. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen (2016)
Improved Techniques for Training GANs
ArXiv, abs/1606.03498
Sangpil Kim, Hyung-gun Chi, Xiao Hu, Qi-Xing Huang, Karthik, Ramani (2020)
A Large-Scale Annotated Mechanical Components Benchmark for Classification and Retrieval Tasks with Deep Neural Networks
Ruijin Cang, Yaopengxiao Xu, Shaohua Chen, Yongming Liu, Yang Jiao, Max Ren (2016)
Microstructure Representation and Reconstruction of Heterogeneous Materials via Deep Belief Network for Computational Material Design
ArXiv, abs/1612.07401
Deniz Iren, M. Ackermann, Julian Gorfer, Gaurav Pujar, S. Wesselmecking, U. Krupp, S. Bromuri (2021)
Aachen-Heerlen annotated steel microstructure dataset
Scientific Data, 8
R. Bostanabad, Yichi Zhang, Xiaolin Li, T. Kearney, L. Brinson, D. Apley, Wing Liu, Wei Chen (2018)
Computational microstructure characterization and reconstruction: Review of the state-of-the-art techniques
Progress in Materials Science
Shrinath Deshpande, A. Purwar (2019)
Computational Creativity Via Assisted Variational Synthesis of Mechanisms Using Deep Generative Models
Journal of Mechanical Design
Alex Kulesza, B. Taskar (2012)
Determinantal Point Processes for Machine Learning
Found. Trends Mach. Learn., 5
Sangeun Oh, Yongsu Jung, Ikjin Lee, Namwoo Kang (2018)
Design Automation by Integrating Generative Adversarial Networks and Topology Optimization
Volume 2A: 44th Design Automation Conference
International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. XXXIV-5/W10 FROM POINT CLOUD TO SURFACE: THE MODELING AND VISUALIZATION PROBLEM
Brian DeCost, M. Hecht, T. Francis, B. Webler, Y. Picard, E. Holm (2017)
UHCSDB: UltraHigh Carbon Steel Micrograph DataBase
Integrating Materials and Manufacturing Innovation, 6
Akash Srivastava, Lazar Valkov, Chris Russell, Michael Gutmann, Charles Sutton (2017)
VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning
Yuan Dong, Dawei Li, Chi Zhang, Chuhan Wu, Hong Wang, Ming Xin, Jianlin Cheng, Jian Lin (2020)
Inverse design of two-dimensional graphene/h-BN hybrids by a regressional and conditional GAN
Carbon, 169
Pan Wang, Danlin Peng, Ling Li, Liuqing Chen, Chao Wu, Xiaoyi Wang, P. Childs, Yike Guo (2019)
Human-in-the-Loop Design with Machine Learning
Proceedings of the Design Society: International Conference on Engineering Design
A. Koga, Edson Lopes, Helcio Nova, C. Lima, E. Silva (2013)
Development of heat sink device by using topology optimization
International Journal of Heat and Mass Transfer, 64
Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, J. Leskovec (2018)
GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models
A. Nobari, Muhammad Rashad, Faez Ahmed (2021)
CreativeGAN: Editing Generative Adversarial Networks for Creative Design Synthesis
ArXiv, abs/2103.06242
K. Duraisamy, G. Iaccarino, Heng Xiao (2018)
Turbulence Modeling in the Age of Data
Annual Review of Fluid Mechanics
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, Illia Polosukhin (2017)
Attention is All you Need
Xin Ding, Yongwei Wang, Zuheng Xu, W. Welch, Z. Wang (2020)
CcGAN: Continuous Conditional Generative Adversarial Networks for Image Generation
ArXiv, abs/2011.07466
T. Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, P. Battaglia (2020)
Learning Mesh-Based Simulation with Graph Networks
ArXiv, abs/2010.03409
Zihan Wang, Weikang Xian, M. Baccouche, H. Lanzerath, Ying Li, Hongyi Xu (2021)
A Gaussian Mixture Variational Autoencoder-Based Approach for Designing Phononic Bandgap Metamaterials
Volume 3B: 47th Design Automation Conference (DAC)
Nie (2021)
Topologygan: Topology Optimization Using Generative Adversarial Networks Based on Physical Fields Over the Initial Domain
ASME J. Mech. Des., 143
K. Fujita, K. Minowa, Yutaka Nomaguchi, S. Yamasaki, K. Yaji (2021)
Design Concept Generation With Variational Deep Embedding Over Comprehensive Optimization
Volume 3B: 47th Design Automation Conference (DAC)

Publisher: Unpaywall
ISSN: 1050-0472
DOI: 10.1115/1.4053859
Publisher site: See Article on Publisher Site

Abstract

Deep Generative Models in Engineering Design: A Review Lyle Regenwetter Amin Heyrani Nobari Dept. of Mechanical Engineering Dept. of Mechanical Engineering Massachusetts Institute of Technology Massachusetts Institute of Technology Cambridge, MA 02139 Cambridge, MA 02139 Email: [email protected] Email: [email protected] Faez Ahmed Dept. of Mechanical Engineering Massachusetts Institute of Technology Cambridge, MA 02139 Email: [email protected] Automated design synthesis has the potential to revolu- 1 Introduction tionize the modern engineering design process and improve The human design process is a ubiquitous element of access to highly optimized and customized products across modern society, playing a critical role in the technologies countless industries. Successfully adapting generative Ma- producing the food we eat, the products we use, and the chine Learning to design engineering may enable such au- spaces in which we live. Accelerating the design process tomated design synthesis and is a research subject of great through automation can reduce cost and increase industrial importance. We present a review and analysis of Deep productivity, which would be immensely desirable for global Generative Machine Learning models in engineering de- productivity and prosperity. Integrating AI into the design sign. Deep Generative Models (DGMs) typically leverage process can alleviate dependence on human experts and rev- deep networks to learn from an input dataset and synthesize olutionize user customizability, providing specialized prod- new designs. Recently, DGMs such as feedforward Neural ucts for individual users without the prohibitive cost of man- Networks (NNs), Generative Adversarial Networks (GANs), ual design. Driven by the widespread potential to advance Variational Autoencoders (VAEs), and certain Deep Rein- global equity and prosperity through design automation, forcement Learning (DRL) frameworks have shown promis- methods such as “generative design” have recently emerged ing results in design applications like structural optimiza- alongside advanced computing and automation technologies. tion, materials design, and shape synthesis. The prevalence “Generative design” is the process in which algorithms of DGMs in engineering design has skyrocketed since 2016. directly synthesize designs either via explicit programming Anticipating continued growth, we conduct a review of re- or implicit learning. Early generative design methods leaned cent advances to beneﬁt researchers interested in DGMs for heavily on explicit programming of human design exper- design. We structure our review as an exposition of the tise through manually-deﬁned design representation methods algorithms, datasets, representation methods, and applica- like grammars [1]. While practical for explicitly encoding tions commonly used in the current literature. In particu- design constraints and objectives, these rule-based frame- lar, we discuss key works that have introduced new tech- works ignored opportunities for implicit leaning on infor- niques and methods in DGMs, successfully applied DGMs mation and knowledge encoded in the vast expanse of exist- to a design-related domain, or directly supported the devel- ing designs. As the availability of computational resources opment of DGMs through datasets or auxiliary methods. We increased over the past decade, data-intensive methods like further identify key challenges and limitations currently seen deep learning opened doors to successfully automate com- in DGMs across design ﬁelds, such as design creativity, han- plex human tasks such as image processing and natural lan- dling constraints and objectives, and modeling both form guage processing. and functional performance simultaneously. In our discus- In deep learning, data is propagated through sequential sion, we identify possible solution pathways as key areas on layers to learn progressively higher-level meaning, an ar- which to target future work. chitecture generally known as an Artiﬁcial Neural Network (ANN) or just Neural Network (NN) [2]. Most of the deep learning-based approaches pioneered during the 2010s lever- 1 Copyright © by ASME arXiv:2110.10863v4 [cs.LG] 16 Mar 2022 aged extensive quantities of data to avoid explicit feature and creativity into DGMs. Despite these advancements, engineering. This trend is mirrored in engineering design DGMs for engineering design are still in their infancy and with algorithms learning data distributions instead of requir- will require further efforts to effectively overcome these fun- ing them to be predeﬁned. Among these algorithms are Deep damental challenges. Generative Models (DGMs) — deep learning models that Our primary goal in this work is to help build cohe- can approximate complicated, high-dimensional probability sion between the countless active researchers in the design distributions using a large dataset. In this review paper, we ﬁeld working with DGMs and furthermore provide a start- speciﬁcally deﬁne “Deep Generative Models” as algorithms ing point for researchers entering the ﬁeld. In particular, we that are capable of generating new samples using deep learn- seek to provide researchers with a reference guide in plan- ing. Generative Adversarial Networks (GANs) and Varia- ning projects in the data-driven generative design space. To tional Autoencoders (VAEs) are two classes of DGMs that this end, we provide an overview of common methods and have demonstrated compelling synthesis of images, text, and tools (Sec. 2), a discussion of different data parameterization tabular data in numerous domains. Considering that images, methods (Sec. 3), a review of potentially relevant research text, and tabular data are all common representation methods across various design domains (Sec. 5), an overview of rele- for design, one might assume that DGMs should be capable vant datasets (Sec. 6), and an analysis of common challenges of synthesizing full designs as well with relative ease. How- in the ﬁeld (Sec. 7). Figure 1 provides an overview of the ever, several unique properties of the generative design task standard process to apply DGMs in engineering design. pose particular challenges for DGMs. Many of these chal- lenges are so fundamental that the future success of DGMs in engineering design is largely contingent on the ability to 2 Overview of Deep Generative Models overcome them. We list four of these challenges below: Deep Generative Machine Learning approaches share the goal of high-quality synthesis but signiﬁcantly vary in 1. Modeling design performance: Real-world functional methodology. In practice, we identify four common ap- performance is critical in many engineering design proaches to generate designs: direct generation using deep tasks. Developing performance-aware DGMs capable neural networks (DNN), adversarial generation with Gener- of synthesizing designs for a given set of target require- ative Adversarial Networks (GAN), generation from embed- ments (a process termed as inverse design) is a challeng- ding vectors using Variational Autoencoders (VAE), and se- ing task that is exacerbated by the computational cost of quential generation using Reinforcement learning (RL). In numerical simulation and the even greater difﬁculty of the design community, we observe that GANs, VAEs, and real-world evaluation. RL are most commonly used for design synthesis. While 2. Data sparsity: Compared to other research ﬁelds like DNNs as well as extensions like recurrent neural networks Computer Vision, which have massive publicly avail- (RNNs) are occasionally used for direct design synthesis, able datasets, the availability of large, well-annotated, they are more frequently used for non-generative tasks. In public datasets in engineering is severely lacking. Fur- this section, we brieﬂy discuss the background and method- thermore, even when data is available, the distribution ology of GANs, VAEs, and RL. of the data often does not cover the design space evenly, with much sparsity often observed in the data. 3. The creativity gap: In conventional DGM applications, 2.1 Generative Adversarial Networks the overarching goal is to mimic the training data and Originally introduced in 2014, Generative Adversarial emulate existing designs. In engineering design, emula- Networks (GANs) [3] found initial success with convincing tion of existing products is often undesirable. Designers image synthesis performance [4, 5, 6, 7]. We provide an in- typically aim to introduce products with novel features troduction of GANs but refer the reader to [8] for a detailed to target new market segments. overview. A generative adversarial network [3] consists of 4. Usability and feasibility: For synthesized designs to be two models — a generator and a discriminator. The genera- physically fabricated, they must be physically feasible. tor G maps an arbitrary noise distribution to the data distribu- Furthermore, designs must be encoded in a data repre- tion, in our case the distribution of designs, and can thus gen- sentation that contains enough parametric detail to be erate new data; simultaneously, the discriminator D learns to converted into a representation usable for fabrication. distinguish between real and generated data. Both models are usually built with deep neural networks. As D improves, Over the past few years, the design community has made G also improves as it learns to generate data that fools D. substantial progress in using generative machine learning (ML) models to create new designs. DGMs have been ap- plied to a broad range of design tasks such as structural opti- Common challenges in training GAN models: GANs mization, materials design, and shape synthesis. Over time, are often considered difﬁcult to train, suffering from train- researchers have introduced increasingly advanced methods, ing instability stemming from several sources [9, 10]. One which have begun to address some of the above challenges. common issue in GAN training occurs when the discrimi- For example, many works have proposed approaches to in- nator overpowers the generator, easily distinguishing gener- corporate design performance and optimization into DGM ated samples, and causing the gradient to the generator to training. Other works have explored incorporating novelty vanish, effectively halting the generator’s training. This is- 2 Copyright © by ASME Fig. 1: This ﬁgure outlines the typical components of design synthesis problems using Deep Generative Models. Some design problems are more suitable for speciﬁc design representation methods, which also inﬂuences the type of deep generative model architectures required. sue has been addressed by many researchers. For example, GAN conditioning: In the design domain, we often have the WGAN replaces the discriminator with a critic, modiﬁes design constraints, requirements, and objectives that any the GAN’s loss function to estimate the Wasserstein (Earth generated design should satisfy. To use DGMs for such prob- Mover’s) distance between the original data and generated lems, these requirements should be imposed on them. For data distributions, and modiﬁes the training process [11, 12]. example, we may seek to train a DGM to generate bikes, but depending on our user, we may want to constrain it to gener- Another problem that GANs face is the issue of “mode ate only roadbikes or mountain bikes without retraining for collapse,” where the generator fails to encompass all modes each generation task. Model conditioning is one method to in the data distribution or even generates only a handful of do this. Several proposed approaches add conditioning to unique samples that are capable of fooling the discrimina- the GAN using a condition vector which is intended to be tor. To overcome these issues, researchers have developed interpretable. Typically GANs are discretely conditioned by novel algorithmic techniques [13, 14, 15] to reward diversity feeding the condition vector into both the generator and dis- in samples. 3 Copyright © by ASME criminator, in a conﬁguration known as a Conditional GAN actor after taking actions, based on the effects of said actions (cGAN) [16]. Instead of feeding the condition vector into on the environment. In this scenario, the actor’s goal is to the discriminator, an auxiliary network and cross entropy maximize the rewards it receives by making decisions (i.e., loss can instead be used to reconstruct the condition vec- taking actions) such that the total reward is maximized. From tor from the generated samples in a conﬁguration known as this point of view, reinforcement learning can be thought of an Information Maximizing GAN (InfoGAN) [17]. Con- as an approach similar to optimization, where an objective ditioning is also essential in design applications where in- (maximizing the reward) is being optimized. verse design is being done on performance metrics, which One of the ﬁrst attempts at introducing deep learning to often exist in continuous spaces (e.g., stiffness, lift coef- the reinforcement learning approach was done in 2013 by ﬁcient, drag coefﬁcient, density, etc.). Researchers have Mnih et al., when they introduced deep learning to a rein- come up with continuous conditioning solutions for GANs forcement learning process known as Q-Learning [26]. Q- such as the Regressional GAN [18], continuous conditional Learning refers to learning the state-action value function GAN (CcGAN) [19] and performance conditioned diverse or Q-function, which is a progressively updated estimate of GAN (PcDGAN) [20]. the expected reward to be received from taking a particular action in a particular state. Mnih et al., attempted to learn the Q-function using convolutional neural networks (CNN). 2.2 Variational Autoencoders Many Deep RL techniques have been introduced since this Introduced in 2013, Variational Autoencoders found sig- ﬁrst work by Mnih et al.. Further exploration of the details niﬁcant success in many machine learning applications. Au- of these approaches is left to the reader. toencoders are unsupervised embedding algorithms consist- In practice, when applying RL to design applications, ing of an encoder that maps an input design into a (typi- the design process is usually broken down into a sequential cally) lower-dimensional latent space and a decoder that re- process of building a design or altering existing designs in constructs the design as accurately as possible from the la- steps (i.e., actions taken to alter or expand the current state tent space. The encoder and decoder are conventionally im- of a design being generated) and the reward is measured by plemented using deep neural networks. To generate new the quality or performance of the resulting design (i.e., the samples, latent vectors are sampled from the latent space environment). While RL requires no dataset, this advantage and fed through the decoder. Typically, the distribution of is balanced by dependence on meaningful and reliable re- the real data mapped to the latent space of an autoencoder ward signals, which may often require a high-ﬁdelity simu- is sparse, meaning that sampling a realistic latent vector lation environment. One major beneﬁt of RL over GANs and is difﬁcult. This limitation is addressed with the introduc- VAEs is the fact that the reward function can be set based on tion of the Variational Autoencoder (VAE), ﬁrst proposed by any objective which does not need to be differentiable. In Kingma et al. [21]. The Variational Autoencoder adds in a contrast, any objective added to the loss function of a GAN probabilistic sampling in the latent space that regularizes the or VAE must be differentiable since GANs and VAEs are latent distribution. In practice, the VAEs’s encoder outputs n trained using the gradient-based optimization [15]. means and n variances, from which n-dimensional latent vec- tors are sampled before decoding. To maintain a predictable latent space distribution, The VAE adds a Kullback-Liebler 3 Overview of Design Representation Methods (KL) divergence [22] loss between the distribution of the la- In this section, we discuss common design representa- tent space and a standard Gaussian. Interested readers are tion methods seen in DGMs for engineering design which encouraged to refer to the literature [23] for a more detailed are visualized in Section 2 of Figure 1. We include a deﬁni- overview. tion and discuss the pros and cons of each method. Conditional VAEs: Just as we do for GANs, we may also seek to condition VAE training on design constraints or user 3.1 Images preferences. The VAE has a natural advantage over the GAN Design data often comes in the form of images (e.g. in that its latent space is typically already structured. Since microstructure scans) or can be represented in image form this structure may be fairly weak and difﬁcult to interpret, ex- (e.g. Topology Optimization). An image consists of a rect- plicitly conditioning VAEs may still be desirable. The Con- angular grid of pixels, each of which contains a color pa- ditional VAE (cVAE) [24] extends on the conventional VAE rameter. They are commonly represented by third-order ten- by adding a conditioning vector as an input to both the en- sors (height widt h channels). Common color schemes coder and decoder and helps achieve this goal. are black-and-white (boolean color channel), grayscale (in- teger color channel), and color (3-4 integer color chan- 2.3 Reinforcement Learning nels). Pros: The image is an information-rich represen- Reinforcement Learning fundamentally differs from the tation and can capture many details of a design. The use other DGMs discussed in that it learns without a dataset in an of convolution/convolution-transpose ﬁlters in deep learn- unsupervised fashion through a large set of trial and error in- ing provides a convenient tool for learning/generation of teractions between an actor and an environment [25]. This is both high-level and low-level features as well as upsam- typically done through some reward signal being sent to the pling/downsampling. Many cutting-edge ML techniques are 4 Copyright © by ASME pioneered in the computer vision domain and are often di- 3.4 Meshes rectly applicable to images. Cons: Representing designs us- Meshes are a common method to represent objects in 3D ing pixels means that the generated design images can be space. Triangular meshes are by far the most commonly used infeasible for downstream tasks. Accurately fabricating de- form. Triangular meshes are the native representation used in signs based on images can be difﬁcult or impossible. Even many computer graphics algorithms and software, as well as performance evaluation using conventional simulation tools many Finite Element tools. Pros: Meshes can be directly vi- like FEA or CFD can require an intermediate conversion sualized and simulated in many FEA or CFD tools, enabling from an image to a 3D model. The poor usability of im- easy pipelines for performance evaluation using numerical ages is exacerbated by the prevalence of artifacts in many methods. A mesh can be considered a specialized type of applications (hanging pixels, disconnected geometry, etc.). graph and can leverage graph operators like graph convolu- Artifacts are especially common when training on (typically) tional operators. Cons: In contrast to other representations small datasets in the design domain since training is often ter- like voxelizations and point clouds, meshes are more chal- minated early due to over-ﬁtting concerns. All in all, images lenging to directly generate using Machine Learning meth- can be considered surrogate representations of engineering ods, despite recent advances in algorithms that directly gen- designs and may lack domain knowledge and information on erate meshes [29, 30, 31]. the physical realization of the design. Therefore, DGMs us- ing images as representations often have a gap between the 3.5 Signed Distance Functions generated images and the actual design they are representing. The Signed Distance Function/Field (SDF) is a repre- sentation method that consists of a (typically 3D) functional map from a coordinate point to an SDF value. The magni- 3.2 Voxelizations tude of this value indicates the distance to the nearest point Voxels are 3D grid points that are effectively the 3D on the surface of the object and the sign indicates whether the equivalent of pixels. As such, voxelizations share many char- point is inside or outside the object. SDFs themselves can acteristics with images. In practice, voxels aren’t conducive be represented in many ways, for example, as a rasterized to ‘color’ parameterization and are typically represented as grid in which each ‘voxel’ contains a continuous numerical booleans (space vs. object). This effectively makes them value denoting the SDF value at that point. Pros: SDFs can third-order tensors with dimension (height widt h de pt h). serve as a convenient intermediate parameterization for many Pros: Voxelizations support 3D convolution which can learn learning tasks. Cons: Like point clouds or voxels, SDFs are high-level and low-level features in 3D. Cons: Compared to difﬁcult to use in downstream tasks without ﬁrst converting images and other representations, the curse of dimensional- to BRep or polygonal representations. ity is especially pronounced with voxels, with the number of parameters scaling with the cube of spatial resolution. 3.6 Parameterizations Voxelizations share the same issues as images. Their us- We use the term “parametric” data to encompass any de- ability is limited in downstream tasks and artifacts are very sign representation consisting of a collection of design pa- prevalent. Like images, voxelizations serve as surrogate rep- rameters where any spatial or temporal signiﬁcance of pa- resentations (often representing CAD models which origi- rameters is unknown. Most parametric data can be orga- nate from parametric representations or 3D shapes which nized in tabular form with each row being a collection of originate from meshes). Like many other representations parameters representing a single design and each column de- such as point clouds and Signed Distance Fields, voxeliza- scribing a design parameter. Tabular design data often con- tions often require conversion before they can be used in sists of a collection of mixed-datatype parameters where re- downstream tasks. For example, they are often converted lations between these parameters may be unclear or nonex- to Boundary Representation (BRep) or polygonal represen- istent. Since parametric data may come in many varieties, tations, which are often the native parameterizations of ren- the pros and cons discussed may not apply to every case. dering and graphics software, Finite Element Analysis, and Pros: Quality parametric data is typically very information- Computational Fluid Dynamics simulation. dense (i.e. requiring fewer parameters to encode the same level of geometric detail). Whereas spatially-organized rep- resentations such as pixels or voxels encode designs with 3.3 Point Clouds uniform information density, parametric data can contain Point Clouds are simple collections of points, often in more detail in design-critical areas without the need for up- 3D space, which are deﬁned to be within some object. Pros: sampling the entire representation. This information density Point Clouds can represent arbitrarily complex geometry often comes with a lower dimensionality which can make with a ﬁnite number of points, though ﬁdelity may vary. optimization of parametrically represented designs signiﬁ- Point Clouds are often the native output of 3D scanning soft- cantly easier. Parametric data may also be more support- ware, making them relatively easy to create [27]. Cons: Like ive of downstream tasks, especially if design parameters are Voxelizations and Signed Distance Functions, Point Clouds human-interpretable. For example, a detailed enough design often require conversion to BRep or polygonal representa- parameterization may allow generated designs to be directly tions such as meshes [28] for downstream tasks. fabricated using conventional (non-additive) manufacturing 5 Copyright © by ASME techniques. Human-interpretable parameterizations can also and their success in molecular graph generation [50, 51], give human designers a tractable method to interact with gen- there is less usage of graph-based DGMs in the design erative methods to allow for human-in-loop design. Finally, community, possibly due to the lack of graph-based design design parameters can sometimes be directly linked to the datasets. latent space of a generative method, as demonstrated in nu- merous works [32, 33], creating a pipeline to directly condi- tion design generation on high-level design goals. Linking 4 Literature Review Methodology design parameters with latent variables has several potential Sec. 5 discusses speciﬁc works that apply Deep Genera- advantages, such as enabling more effective optimization or tive Models to engineering design or make advancements to inverse design using generative methods. Cons: Learning existing generative ML methods in the context of engineer- parametric data can be particularly challenging. Parametric ing design. We consider works based on a predeﬁned scope, data commonly uses mixed datatypes and inherits the train- with each work we discuss meeting the following selection ing challenges of the constituent components. Multimodal criteria. Note that these criteria only apply to the engineer- distributions, skewed categories, non-Gaussian distributions, ing design papers presented and that the works we cite to add data sparsity, and poor data scaling additionally make the ap- context to or substantiate the discussion of fundamentals, ap- plication of DGMs and training very difﬁcult. Since methods plications, and datasets need not adhere to these rules. that are robust to all of the mentioned challenges are difﬁcult 1. We limit our consideration speciﬁcally to papers involv- to come by, successfully applying existing methods to the ing Deep Generative Models, focusing on Variational parametric data domain can be hard. Finally, since paramet- Autoencoders, Generative Adversarial Networks, and ric data may be nontrivial to convert to 3D models, generated Reinforcement Learning in particular. Works must uti- parametric designs may be challenging to evaluate using nu- lize deep learning. Works only considering design opti- merical simulations or through qualitative visualization. mization are excluded. 2. We only consider work speciﬁcally relevant to engineer- 3.7 Grammars ing design and not other domains (such as computer sci- Grammars are representation methods consisting of ence). variables, terminal symbols, nonterminal symbols, and a set 3. We consider only work published between Jan. 2014 and of rules. Rules describe how non-terminal symbols can ex- Sep. 2021, when we conclude our review, as many piv- pand into other terminal and nonterminal symbols. The most otal works in deep learning (CNNs, VAEs, GANs) were prevalent grammars in engineering design are graph and spa- introduced in this period. tial grammars [1], and they have been applied in a wide To identify works, we speciﬁcally searched for “Generative variety of applications, such as the design of satellites and Adversarial Network,” “Variational Autoencoder,” and “Re- electro-mechanical systems. Grammars can be especially inforcement Learning” in Google Scholar. We initially con- useful to dictate feasible assembly hierarchies of design com- ﬁned our search to a set of known design venues, speciﬁ- ponents as in [34]. Pros: Grammars can explicitly con- cally the Journal of Mechanical Design, the Proceedings of strain design spaces to feasible or desirable regions by nature the International Design Engineering Technical Conferences, of their construction, thereby encoding domain knowledge. Computer-Aided Design Journal, International Conference Cons: Grammars are challenging to implicitly learn and of- on Engineering Design, and Artiﬁcial Intelligence for Engi- ten must be manually deﬁned. Grammars can also restrict the neering Design, Analysis, and Manufacturing Journal. This exploration of the design space. A survey on grammar-based helped us identify an initial set of seed papers. From all design synthesis approaches is provided in [1]. search results identiﬁed through these methods, 41 papers were deemed to be relevant and included. The relevance was 3.8 Graphs decided by independent assessment by two raters, who are Graphs are a highly ﬂexible representation method con- also authors of this paper. For papers where there was a sisting of nodes and edges, which can be directed or undi- disagreement, all authors discussed them and mutually de- rected. Graph-based representations have proven success- cided on their classiﬁcation. Next, papers cited in these seed ful for many different aspects of design generation and opti- papers were considered and added to this paper. 22 papers mization and provided avenues for describing complex sys- from other venues were deemed to be relevant and included tems efﬁciently. Pros: Graphs are highly adaptable and are as well. Of the 63 papers, 48 were published between Jan. capable of representing many different kinds of complex sys- 2019 and Aug. 2021, while a mere 15 were published be- tems and designs [35, 36, 37, 38]. Graphs also provide repre- tween Jan. 2016 and Dec. 2018, indicating strong growth in sentations for design processes and modeling complex inter- the ﬁeld in recent years. actions in systems [39,40] which may enable methods for au- tomating the design of systems or modeling inter-part depen- dencies. Graph neural networks (GNNs) [41, 42, 43, 44, 45] 5 Application Domains in Engineering Design provide an excellent tool for machine learning on graphs. Engineering design encompasses a wide variety of ap- Cons: Despite the developments of graph neural networks plications, ranging from designing aircraft models to small- (GNNs) in the computer science community [46, 47, 48, 49] scale metamaterials. To structure different types of applica- 6 Copyright © by ASME Table 1: Characterization of application studies by domain and architecture. Some works use multiple architectures or ﬁt into multiple application domains and are listed more than once. We additionally classify works by datatype used: Image , c v m s g h p Point Cloud , Voxelelization , Mesh , Signed Distance Field , Grammar , Graph , and Parametric (other parameterization) Topology Optimization Materials 2D Shape Synthesis 3D shape synthesis Other Domains i vi p i p i i Plain NN [52] [53] [54] [55] [56] [57] [58] i i i i i vi v i vi c GAN [59] [60] [61] [62] [63] [64] [65] [66] [67] [68] [69] i i i i p p p ph GAN+Conditioning [70] [71] [72] [73] [74] [75] [76] [33] i p v i New GAN-based [72] [15] [77] [78] i i i i i i v p p p AE/VAE [79] [80] [81] [82] [32] [83] [84] [85] , [86] , [87] i † i p i cVAE [88] [89] [85] [90] i s i p New VAE-based [91] [92] [93] p p i p g i p p p RL-based [94] [95] [96] [34] [97] [98] [99] v i i i i i i i p v p g i p i p g Other Method [100] [101] [102] [103] [67] [104] [105] [106] [95] [100] [107] [34] [108] [109] [110] i i vi i i i +Style Transfer [79] [64] [65] [80] [57] [104] i i s +Genetic Alg. [82] [105] [92] i i p +Bayesian Opt. [64] [83] [76] *Model-based reconstruction using parameters in a lower-dimensional space † Technically not images, but use an image-like parameterization structure tions using DGMs, we grouped them into a few categories of TO algorithms to rapidly converge, saving computational application areas, which are discussed in this section and are cost. We consider a baseline for DGMs in TO, in which reported in Table 1. an existing generative architecture is applied to a dataset of TO-generated designs and the network is trained to gener- ate samples mimicking the training data. A few papers fall 5.1 Deep Generative Models in Topology Optimization within this category of methods: Rawat and Chen [60] train a WGAN on a dataset generated through TO and train an auxiliary network to additionally predict performance met- rics. Sharpe and Seepersad [70] expand on this baseline with a cGAN conditioned on volume fraction and load location. Guo et al. [79] train a VAE with an additional style trans- Fig. 2: Sample topologies generated by 3D Topology Opti- fer [119,120] loss on a TO-generated dataset for heat transfer mization and propose iterative strategies for targeted design optimiza- Topology Optimization (TO) is a research ﬁeld with a tion using the VAE’s latent space. Style transfer is discussed long history of research and methods. TO searches a de- further in Sec. 5.2. sign space to ﬁnd an ideal spatial distribution of material to optimize some predeﬁned objective. Common areas of ap- Iterative DGM training, ﬁltering, and human guidance plication include solid mechanics [111, 112], ﬂuid dynam- in TO: Oh et al. [62, 61] propose a method that expands ics [113, 114], additive manufacturing [115, 116], and heat on the baseline generative-network-ﬁtting by iteratively syn- transfer [117, 118]. While Topology Optimization is con- thesizing new designs, optimizing them in TO, then drop- sidered a method for generative design, standard TO does ping designs that are too similar from the full collection of not leverage deep learning and is not considered a DGM. In designs. The authors use a modiﬁed Boundary Equilibrium recent years, however, several papers have proposed meth- GAN (BEGAN) [121], which extends on the WGAN and be- ods to use TO and DGMs together, in many cases to address gin training on a collection of existing TO-generated topolo- the computational cost of TO on large amounts of data. De- gies. However, unlike the previously discussed generative- spite differences between architectures, we found that many network-ﬁtting approaches used in literature [60, 70, 79], the DGMs for TO share certain characteristics. In particular, retraining and re-optimization allows the framework to gen- due to the predominant use of voxelized or pixelized rep- erate, optimize, and explore new areas of the design space. resentations, many hybrid TO-Generative ML models use The proposed method is applied to the problem of wheel de- methods originally developed for computer vision appli- sign, with the key motivation being to ﬁnd a trade-off be- cations, such as convolutional neural networks and super- tween the aesthetics of real designs and the structural per- resolution [59, 53]. formance of designs found through TO. The authors present strong empirical results on this wheel design problem. Supervised generation of optimized topologies: To avoid the computational cost of Topology Optimization, DGMs The issue of gaps in the design space of topolo- have been used to predict ﬁnal optimized topologies di- gies is also addressed by Fujita et al. [101], who pro- rectly. This approach can also be used as an initialization pose an approach leveraging a Variational Deep Embedding technique for conventional TO, which allows conventional (VaDE) [122]. An initial dataset is generated using TO. The 7 Copyright © by ASME proposed method sequentially identiﬁes voids in the design culating optimal solutions (using TO) for proposed solutions space using the VaDE, decodes designs from the void, opti- that are in greatest violation of these conditions. mizes them using TO, then adds them to the training dataset. Although the above works have proposed methods with Other DGM approaches in topology generation: Sos- automated retraining steps, human input can also be injected novik and Oseledets [52] propose to use gradient informa- into the training process. For example, Valdez et al. [73] tion from a topology distribution to inform estimates of a propose a cGAN-based human-in-loop topology design gen- ﬁnal topology through a DNN. Their DNN takes both the eration framework in which the designer iteratively selects density distribution and gradient of this density distribution design clusters to gradually hone in on preferable designs. from some intermediate step of a TO process to generate an estimate of the ﬁnal topology without waiting for TO con- DGMs for TO that utilize super-resolution: Since one vergence. The authors demonstrate binary accuracy of 98% of the key limitations of Topology Optimization is computa- when predicting the ﬁnal topology after only ﬁve iterations tional cost, which scales with the resolution, a major focus of TO, when the full TO process would take 100 iterations to in DGMs for TO has focused on attaining high-resolution complete. TO-like results. A common approach uses super-resolution, In a different kind of approach, Keshavarz- which is a technique used in computer vision to convert low- zadeh et al. [54], address the generalization problem resolution images to high-resolution ones. Several works of training for different scales and different domains. expand on the baseline DGM-in-TO framework by learn- They parametrically represent shapes in both 2D and 3D ing from a low-resolution TO-generated dataset, then per- using their proposed “Disjunctive Normal Shape Model” forming super-resolution on synthesized topologies, such (DNSM) [54]. Using this DNSM, they create a platform as Yu et al. [72], who add a cGAN for upscaling. A for resolution-independent shape reconstruction. They then similar approach proposed by Li et al. [59] uses a Super- train an NN model to generate optimal topologies given Resolution GAN (SRGAN) [123] to generate high-resolution boundary conditions and other problem-speciﬁc information optimal topologies for heat transfer problems after generat- in the DNSM space, which then can be reconstructed at any ing low-dimensional topologies on a different GAN. Other resolution using the DNSM. They demonstrate that their researchers have proposed transfer learning for the super- DNSM method overcomes the super-resolution problem resolution task instead of using GANs or VAEs for super- across a variety of problems and domains. resolution. For example, Behzadi et al. [53], train a feedfor- ward NN-based model to predict optimized topologies di- 5.2 Microstructure, Nanostructure, and Metamaterials rectly without any discriminator and using the MSE loss. Once they have trained their model on the low-resolution data, they lock the model weights and add a few layers to the model to increase the output resolution and only train said layers to obtain high-resolution samples. This avoids the need for a large quantity of high-resolution data or the extra Fig. 3: Deep Generative Models are often employed to gen- time required to train a high-resolution model from scratch. erate 3D Metamaterial Unit cells Many papers that have applied super-resolution techniques demonstrate that after initial training, their framework con- Many design applications not only require the need to sistently generates near-optimal topologies many times faster design the topology or shape of the artifact, but also the ma- than classic TO [72, 59]. terial properties within it. Inverse design of materials is one of the key elements of Computational Materials Science. The Improving DGM-based topology generation using physi- most common approach to developing inverse materials de- cal properties: Several papers have succeeded in improv- sign frameworks is the development of Process-Structure- ing the baseline performance of DGMs for topology gen- Property (PSP) links, i.e. understanding how a particular eration through the use of physical properties of the de- material processing approach impacts its microstructure and sign domain. For example, Nie et al. [71] propose to adapt how the corresponding microstructure impacts its physical the cGAN architecture Pix2Pix [124] to generate synthetic properties [128]. Designing material microstructures for di- topologies based on spatial ﬁelds of various physical param- rect use is difﬁcult, as there must then be some fabrication eters (displacement, strain energy density, and Von Mises process to generate a material with the target microstructure. stress) as input to the generator. Topologies generated by A common goal in computational materials science is mi- TO are taken as ground truth for training. The authors crostructure image “reconstruction,” in other words, generat- also propose a new generator architecture combining the ing microstructure images that exhibit certain characteristics. Squeeze and Excitation ResNet [125, 126] with U-Net [127]. Bostanabad et al. [128] give an overview of the Microstruc- Cang et al. [55] use a neural network to generate optimized ture Characterization and Reconstruction (MCR) ﬁeld. Re- topologies from loading conditions. They propose an ap- construction can accelerate downstream tasks like augment- proach to rapidly evaluate the deviation of proposed solu- ing the training data of networks that attempt to model PSP tions from the problem’s optimality conditions. They then links. Better modeling PSP links can in turn create more ac- progressively augment their dataset during training by recal- curate generative material design pipelines. Many classes of 8 Copyright © by ASME DGMs have been applied to this reconstruction task, includ- Liu et al. [68]. ing GANs [64], VAEs [80] and Convolutional Deep Belief Networks (CDBNs) [129] [103, 102, 106]. Other studies at- Photonics and phononics: Within microstructure design, tempt to bridge the gap between microstructure and proper- the design of materials with photonic or phononic proper- ties in generative tasks using trained black-box surrogates, ties is an area of interest in numerous industries including such as Tan et al.’s work [63]. Most research focuses on 2D sensing, communications, and display technology. The work microstructure images, though several also consider 3D vox- by Yang et al. [64] is an example of a subﬁeld of genera- elizations [66, 68]. tive microstructure design targeting the development of mi- Since reconstruction often involves mimicking exist- crostructures with particular photonic or phononic proper- ing microstructures, several papers have applied style trans- ties. Molesky et al. [131] also provide a review of inverse fer [119, 120] to the problem as part of a DGM architecture. design in nanophotonics. Style transfer is in essence a loss between two images that In the photonics and phononics subﬁeld, performance attempts to capture the difference in “style” between the im- evaluations such as the one used in [64] are frequently ages. It is typically calculated by comparing the interme- used to incorporate performance into DGMs. For exam- diate layers of an auxiliary style convolutional neural net- ple, the technique of optimization using learned mappings work. Cang et al. [80], for example, propose a VAE with from the latent space of a trained autoencoder to the property style transfer which is targeted at applications where only a space has been employed in several papers. Li et al. [81], small set of training data is available. Li et al. [57] also uti- Liu et al. [82], and Wang et al. [105] train an autoencoder, lize style transfer in their proposed transfer learning recon- VAE, and Gaussian Mixture VAE [132] respectively on im- struction framework. In another approach using style trans- ages of microstructures. They map the latent variables to fer, this time using a vanilla GAN, Yang et al. [64] expand the property space using a DNN, CNN, and Gaussian Pro- on the reconstruction task by attempting inverse design on cess Regressor, respectively. Li et al. [81] directly optimize the structure-property link. After training their GAN with the properties using the DNN, while Liu et al. [82], and style transfer loss and an additional loss to penalize mode Wang et al. [105] optimize using a Genetic Algorithm. collapse, they treat the noise vectors as input variables and Though optimizing latent variables is a common ap- optimize them with Bayesian optimization, using the gener- proach, several other DGMs have been proposed for pho- ator to translate between design vectors and microstructure tonics/phononics design that optimize performance in other images. The authors choose to optimize microstructures for ways. For instance, Malkiel et al. [56] uniquely use a para- energy absorption which can be evaluated from images us- metric representation and a bidirectional DNN to simulta- ing coupled-wave analysis [130], providing them a conve- neously learn a bidirectional mapping between nanostruc- nient structure-property link that would be more challenging tures and the property space. Ma et al. [91] propose a novel if optimizing for other objectives. VAE-based framework consisting of feature extraction, pre- More recently, Fokina et al. [104] adapt StyleGAN [7] diction, recognition, and generation networks. The proposed to the microstructure image synthesis domain. Other stud- method is capable of both forward prediction of properties ies impose the “style” of generated microstructure images based on metamaterial structure as well as the inverse (gen- through enforcement of physical properties. For example, erative) prediction task based on properties. Furthermore, Chen et al. [89] propose an approach to generate Random the framework supports self-supervised learning where the Heterogeneous Material (RHM) microstructure images using model trains on unlabeled metamaterial pattern images with- a cGAN conditioned on target images. Their cGAN uses an out corresponding property labels. The authors also demon- augmented loss based on the matching of perimeter, volume, strate transfer learning to other microstructure shape classes and Euler characteristics. DGM studies in microstructure as well as the inverse design of multiple microstructures design have also used other advanced methods introduced forming a meta-mirror with desired properties. DGMs have in computer vision such as super-resolution [58] and image also been applied to nano-scale photonic devices, such as an translation [67, 4, 67, 124]. optic broadband power splitter as in Tang et al. [88]. Although 2D images are a widely used means to rep- resent microstructures, Zhang et al. [65] expand the mi- Unit-cell-based metamaterials: Research in generative crostructure reconstruction task to the 3D domain. Their design for microstructures has also focused on the devel- ScaffoldGAN method generates scaffold materials to mimic opment of unit cell structures for metamaterials. For in- real-world examples of human bone scaffolds as well as stance, Wang et al. [32] ﬁt a VAE to metamaterial unit foam metal scaffolds in both 3D and 2D data. They employ cells and demonstrate that latent space parameters can re- the conventional GAN approach with style loss for better vi- ﬂect the physical properties of the unit cells. They then sual similarity between generated scaffolds and real ones. demonstrate several downstream tasks using their VAE, such However, the authors additionally introduce a novel “struc- as diverse subset selection and generation, targeted genera- tural loss” term to speciﬁcally address spatial coherence, a tion to match desired stiffness matrix values, and metama- limitation of GANs in emulating scaffolds. This additional terial family design. The authors also demonstrate several loss helps them achieve better results that mimic the features 2D macro-optimization runs to design arrays of unit cells to of the data more realistically. Other studies also consider match macro-level deﬂection targets in an approach similar DGMs on 3D microstructures, such as Mosser et al. [66] and to classic TO. Xue et al. [83] train a VAE to generate unit 9 Copyright © by ASME cells, then perform Bayesian Optimization in the latent space in DGMs. Dering et al. [95], for example, apply an iter- of the VAE to attain desired macroscopic elastic properties. ative retraining approach to boat sketches using an adapta- tion of the Long Short-Term Memory (LSTM)-based Sketch- RNN [135] as the generator. To evaluate candidate designs, 5.3 2D Shape Synthesis the performance is scored using a simulated environment in a game engine, within which the “behavior” (motion) of the design is learned. Fig. 4: Airfoil synthesis using DGMs usually entails employ- A recent work by Chen and Ahmed [15] addresses both ing 2D shape generation approaches. performance evaluation and design novelty through their pro- Designing features with critical geometric considera- posed Performance Augmented Diverse GAN (PaDGAN) tions is a task that appears in several engineering ﬁelds framework. Conventional GANs are trained to mimic the such as aerospace and automobile design. DGMs have been design space they are trained in and as a consequence, widely applied to these problems. Within aerospace, the de- are penalized for generating novel designs. PaDGAN ex- sign of an airfoil, which is the cross-sectional shape of a pands on the conventional GAN architecture by modeling wing, is of particular interest to many researchers. Airfoils design performance and design diversity using a Determi- have a wide variety of uses in engineering domains such as nantal Point Process (DPP) kernel [136]. By promoting di- propeller, rotor, and turbine blade design. Since airfoil per- versity, PaDGAN directly addresses mode collapse in GANs formance parameters are of key interest, most of the research too (see Sec. 2.1). The authors demonstrated that PaDGAN focuses on performance-aware conditional generation. For is capable of generating high-quality and previously unseen example, Yilmaz and German [74] apply a cGAN to the novel designs for the UIUC airfoil database, using the lift to airfoil design problem, conditioning their network on vari- drag ratio as the quality metric. [137] extends this work to ous stall parameters. While the overwhelming majority of multi-objective generation. research in this domain uses DGMs trained to learn shape parameterization based on spline interpolation of Cartesian 5.4 3D Shape Synthesis points, equation-based parameterization is also sometimes used. For example, Li et al. [94] propose a performance- aware RL framework where the RL agent learns the opti- mal equation coefﬁcients using Proximal Policy Optimiza- tion (PPO) [133]. Fig. 5: DGM models, such as Range-GAN [77], can be used Other works generalize their applications to general 2D to generate 3D models of aircrafts with constraints. shape synthesis. For example, Chen and Fuge [75] propose Bezier ´ GAN, a framework that learns shape representations 3D object generation using deep learning is an active through Bezier ´ curves, featuring InfoGAN style condition- ﬁeld in computer science, with signiﬁcant research efforts ing [17]. Chen et al. [76] expand on this work with Bayesian being dedicated to generating realistic-looking shapes and Optimization to maximize lift/drag ratio. Classic test data for objects. 3D shapes and objects are typically represented 2D shape synthesis methods are the UIUC airfoil dataset as by voxels, point clouds, or meshes. Advancements in 3D well as artiﬁcially-designed shapes like superformulas [134]. shape synthesis in the computer graphics community have While the emphasis of certain papers is on shapes or airfoils leaned heavily on GANs or Autoencoders, as well as other themselves, several papers primarily propose methodology machine learning advancements like Recurrent Neural Net- advancements that they choose to demonstrate on the 2D works (RNNs), Transformers, and Graph Neural Networks shape synthesis domain. For example, Chen and Fuge [33] (GNNs). This research is tangentially relevant to engineer- address the challenging problem of multi-component design ing design but is typically focused more on visual appear- generation using a GAN to synthesize parts using inter-part ance and aesthetics rather than design considerations like dependencies. The authors assume inter-part dependencies functional performance derived from simulations or manu- in a design are known and propose modeling them using di- facturability. For this reason, we do not discuss these meth- rected acyclic graphs. They propose a hierarchical adapta- ods in detail and opt to list a few of the many noteworthy tion of the InfoGAN using a single discriminator for the en- papers in this ﬁeld from 2016 to 2021 for the reader to fur- tire design and one generator/auxiliary network pair for each ther explore at their discretion: [138, 139, 140, 141, 142, 143]. part in the design, which they term the hierarchical GAN It is important to note that many of these proposed DGM (HGAN). The method is tested on a medley of synthetic architectures could be adapted to the design domain too. De- datasets generated using BezierGAN ´ [75] and demonstrated spite the prominence of 3D object generation papers in the to form meaningful and interpretable latent spaces. Although computer graphics community, a few have arisen within the the paper assumes that part dependency graphs for the ob- engineering design community as well. In an early work, ject class are well deﬁned, this paper takes a signiﬁcant step for example, Brock et al. [84] implement a 3D cVAE to re- towards multi-component design synthesis and interpretable construct and interpolate between voxelized models from the GANs. ModelNet-10 Dataset. Several other papers use the 2D shape synthesis do- The majority of 3D shape synthesis work in engineering main to address the challenges of performance evaluation design, however, has implemented some kind of design per- 10 Copyright © by ASME formance consideration. Zhang et al. [92] propose the use able different tasks. For example, Wang et al. [78] look at of a Genetic Algorithm (GA) to optimize latent space design the problem of generating samples that are visually pleasing embeddings of a trained VAE variant called the Variational for any given human subject. To do this, they train a modi- Shape Learner (VSL) [144], which they demonstrate on the ﬁed version of the Auxiliary Classiﬁer GAN (ACGAN) [148] optimization of 3D aircraft models. Shu et al. [69] take an which classiﬁes the type of image being generated at the iterative retraining approach by retraining a GAN on high- same time as learning to generate images. The authors, how- performance models evaluated using Computational Fluid ever, do not opt for a cGAN architecture. Instead, they condi- Dynamics (CFD) evaluation. The authors apply the pro- tion the GAN using Electroencephalography (EEG) signals posed method to point cloud aircraft models from ShapeNet from human subjects. They train an encoder that takes EEG and use minimization of aerodynamic drag as the perfor- signals and transforms them into a set of input features for mance objective of choice. The authors use a standard GAN the GAN. By exposing human subjects to the images as they loss and use a discriminator architecture from [145]. No- are being generated, the encoder introduces the human re- bari et al. [77] introduce another self-augmentation approach sponse to the design into the generation process. This ef- to train on skewed or sparse datasets. They couple it with fectively allows for a human subject to extract more visually a “range loss” to encourage designs to comply with design pleasing samples from the GAN model. Several other stud- constraints related to parameter bounds and apply their ap- ies such as [73, 85, 96] propose frameworks that incorporate proach to generate 3D aircraft models. humans into the design process. Kinematic synthesis: In machine design, generating a 5.5 Other Applications mechanism to generate speciﬁc motion curves or transmit In this section, we discuss several application domains power is a common design task. Kinematic synthesis in- for DGMs with only a few relevant papers identiﬁed in our volves selecting the size, type, and conﬁguration of mech- search. anism components to achieve such a goal. Several works have applied DGMs to various kinematic synthesis prob- Manufacturability: Synthesizing manufacturable designs lems. Deshpande and Purwar [85], for example, propose a is an important, albeit less-explored area in DGMs. VAE-based method for planar linkage synthesis. They de- Greminger [100] proposes approaching the problem of terministically parameterize both coupler paths and linkages, manufacturability using an MSG-GAN architecture [146] then generate sample coupler paths from a variety of linkages adapted to the 3D domain. The author synthetically gen- (four-bar, slider-crank, Stephenson-type six-bar). In the ﬁrst erates topologies that are manufacturable by a 3-axis mill, of their two studies, the authors train a VAE to reconstruct trains the GAN to generate similar topologies, then optimizes trajectories. They then propose a method in which humans generated topologies using TO. A key challenge in GAN- can interact with the trained VAE to customize linkage paths based synthesis of manufacturable designs is a lack of anno- to their design goals by visualizing the effect of various latent tated datasets with design and manufacturing process details. space perturbations and selecting one that matches their de- sign vision. The authors then expand on their ﬁrst study with Fluid ﬂow shapes: DGMs have also been used to gener- a cVAE architecture, feeding in the linkage as the data sam- ate ﬂow shapes. For example, Lee et al. [96] apply a Dou- ples and the coupler curve as the condition vector. In a sim- ble Deep Q-Network (DoubleDQN) [147] RL framework to ilar work, Sharma and Purwar [86] explore spatial linkage learn the geometry of a microﬂuidic channel to generate tar- synthesis of 5-SS mechanisms using VAEs. In other papers, get ﬂow shapes. The authors also propose methods in which the same authors take a look at paths themselves as guid- a human designer can participate in the design process along- ing mediums for linkage mechanism design by ﬁrst training side the RL agent to impart design knowledge or constraints. models to encode paths, then using said models to search Other approaches that integrate humans into the design gen- within a dataset of mechanisms to obtain solutions for the eration process are discussed next. speciﬁc paths [93]. Other studies have approached the kinematic synthesis Integrating humans into the generation process: One el- task through Reinforcement Learning. Vermeer et al. [107] ement of design that is often less emphasized in DGMs for propose using Deep Q-Learning to synthesize planar link- design is human interpretation and design input during the age mechanisms to obtain mechanisms that are capable of generative process. Though datasets are often created us- producing straight lines, showing that RL can be effective in ing some human input, humans are not directly involved dur- generating linkage mechanisms using machine learning. ing the training process of most DGMs. Burnap et al. [90] note that numerical performance measures often do not cor- Truss design: Raina et al. [108] propose an RL-based ap- respond to a human’s perception of design quality, and study proach to truss design. In their approach, they learn from this phenomenon on automobile images generated through a dataset of sequential human design decisions [149]. They a conditional VAE. Supporting the involvement of humans ﬁrst train an Autoencoder to map trusses to and from a design during the training process is a potential approach to ﬁnd embedding. Sequential design embeddings are then used to the middle ground between the expertise and intuition of hu- predict a heatmap of possible next design states using a su- mans and the detailed information of datasets, which can en- pervised transition network. Interestingly, the authors choose 11 Copyright © by ASME to convert to and from images from their parametric repre- able. The most common method for dataset generation is sentation when training the autoencoder and using decoded Solid Isotropic Material with Penalisation (SIMP), which has results. Finally, a rule-based agent selects from possible next several publicly available software implementations. A few design steps. Even without knowledge of design metrics and TO datasets, however, are open source. Sosnovik and Os- performance, the RL agents were found to generate compet- eledets [52] generate a dataset of 10,000 artiﬁcially gener- itive designs compared to humans. Puentes et al. [109] ex- ated topologies providing both ﬁnal and intermediate topolo- pand on this work by learning action-sequence heuristics in- gies from the optimization process . Each included topology stead of individual design actions. Raina et al. [97] further contains one 40x40 image of the topology generated at ev- expand on this work with a goal-based reinforcement learn- ery stage of 100 steps of Topology Optimization. Hence, ing agent. a total of one million images are provided. Nie et al. [71] provide the dataset used to train their TopologyGAN frame- Hierarchical product design synthesis: Some works work . Unlike the aforementioned, this dataset consists only have applied DGMs to multi-component products, typically of ﬁnal topologies but contains 49078 generated topologies with a well-known hierarchical structure. Stump et al. [34], at a resolution of 64x128, generated from 42 unique bound- for example, propose a unique approach that simultaneously ary conditions. Both datasets discussed use the ToPy [151] combines RL, Recurrent Neural Networks (RNNs), gram- implementation of SIMP. mars, and physics-based simulation. In their approach, the RNN learns to generate modular sailing crafts by sampling 6.2 Microstructure Datasets discrete selections from a predeﬁned shape grammar. The Well-established technologies such as optical mi- training of the RNN is detailed in [110]. The control policy croscopy (OM) and scanning electron microscopy (SEM) are of the craft is then optimized using RL in a physics-based often used to visualize material microstructures. As such, simulated sailing environment. Regenwetter et al. [87, 150] numerous datasets of microstructure scan images are pub- explore the full synthesis of bicycles, a diverse class of prod- licly available, such as [152], [153], and [154]. Numer- ucts typically consisting of numerous hierarchical compo- ous datasets are also available for the design of composite nents. The authors demonstrate full generative synthesis of materials of various types, such as the NanoMine nanopoly- bikes using VAEs for both image and parametric represen- mer composite database , which contains over 20,000 data- tations. The authors note that their detailed design parame- points [155]. Compiled lists of materials science datasets terization allows AI-generated designs to be physically fab- and synthetic microstructure datasets are also available. For ricated. example, Yang et al. [64] provide a trained GAN model which generates synthetic microstructure images . Procedural content generation: Lopez et al., explore pro- cedural content generation in a speciﬁc context for virtual reality. In their work, they introduce a reinforcement learn- 6.3 Design Geometry Datasets ing model that can learn to generate 3D virtual reality (VR) The UIUC airfoil database has been used as a case content which users can explore in a VR environment [98]. study in several generative design research frameworks [76, In their approach, they speciﬁcally work towards generat- 15, 75]. The database details nearly 1,600 real-world airfoil ing manufacturing environments that are physically valid and designs using coordinates of points on the surface. Since the feasible. In an extension of this work, Cunningham et al., ex- original data provides inconsistent numbers of coordinates tend the method to generate content for multiple contexts in- along the top and bottom surfaces, Chen et al. [76] propose stead of just one at a time [99], which enables the integration a method to standardize the data with B-spline interpolation of user-speciﬁc parameters. over the airfoil which is also used in other works [33, 15]. 6.4 3D Object Datasets 6 Datasets ShapeNet [156] is one of the most commonly-used 3D In this section, we present commonly-used datasets that model datasets, consisting of over 51,300 3D models of 55 have been or have the potential to be used to train DGMs for object categories. PartNet [157] expands on ShapeNet with data-driven design tasks. We provide a more detailed list on- ﬁne-grained hierarchical semantic annotations for compo- line . While the datasets listed are not comprehensive, we nent parts of ShapeNet objects. Princeton ModelNet [158] aim to note key types of datasets that have been commonly used in engineering design applications. We hope that re- searchers can create larger well-annotated datasets and make https://github.com/ISosnovik/top them public for the community to use. https://github.com/zhenguonie/2020_TopologyGAN https://github.com/tetherless-world/ nanomine-ontology 6.1 Topology Optimization Datasets https://github.com/sedaoturak/ Most papers on DGMs for Topology Optimization gen- data-resources-for-materials-science erate their own datasets, which are often not publicly avail- https://github.com/zyz293/GAN_Materials_Design https://m-selig.ae.illinois.edu/ads/coord_ database.html https://shapenet.org/ https://decode.mit.edu/datasets/ https://modelnet.cs.princeton.edu/ 12 Copyright © by ASME is another commonly-used 3D model dataset consisting of 6.7 Sketch Datasets 127,915 voxel-based 3D models of 662 object categories. QuickDraw [165] is a sketch dataset of 50 million doo- Numerous works discussed in this review use ShapeNet or dles from 345 categories . The doodles were collected by ModelNet models [69, 77, 92, 84]. Google from user-drawn sketches in an interactive sketch- ing game. QuickDraw data is used in several works dis- Sangpil et al. [159] introduce a dataset of 58,696 models cussed [166, 95]. Toh & Miller [167] introduce a dataset of of mechanical components from 68 classes called the Me- 934 innovative milk frother design sketches with associated chanical Components Benchmark (MCB) . The MCB con- text descriptions that can potentially be used to train DGMs, tains a hierarchical label tree grouping components into sub- perhaps factoring in Natural Language Processing (NLP) . classes of different levels, such as Components ! Fasteners ! Nuts ! Wingnuts, for example. Models are represented as point clouds, voxels, and 2D views. 6.8 Sequential Human Design Datasets McComb et al. [149] provide a tabular truss design dataset taken from a truss design activity executed by six- 6.5 CAD and CAD-based Datasets teen human teams . Sequential design operations such as Willis et al. [160] introduce two datasets of Autodesk joint and member placement are recorded using geometric Fusion models . One dataset, intended for reconstruction parameters such as joint coordinates, member size, etc.. Per- tasks, contains 8,625 models and the other, intended for formance metrics such as safety factors and weights are also segmentation, contains 35,680 models. The reconstruction included. While this dataset can be used as a truss design dataset is particularly interesting for generative tasks, as it dataset, it is primarily intended as a resource to study or contains information governing the sequential CAD opera- mimic the human design process. We note that the Autodesk tion steps taken to generate a part. Fusion reconstruction dataset mentioned previously [160] Regenwetter et al. [87, 150] introduce a dataset called can also be used for learning sequential design tasks. BIKED consisting of mixed data extracted from 4,512 bi- cycle CAD models. The dataset includes bicycle assembly images, segmented subcomponent images, as well as “para- 7 Discussion, Challenges and Future Work metric” data. The “parametric” data consists of 2,395 mixed- When applying DGMs to engineering design, the ‘stan- type design parameters describing both high-level and low- dard’ objective of mimicking the training data is often insuf- level characteristics. BIKED provides an advantage over ﬁcient, or even counterproductive. Instead, real designs are conventional 3D object datasets for generative tasks in that governed by speciﬁc objectives and constraints, often includ- synthesized designs contain the necessary parametric infor- ing novelty or creativity. Thus, different objectives, such as mation to physically fabricate designs. BIKED’s paramet- real-world performance metrics, novelty, and adherence to ric data is used in [87] for bicycle synthesis and its image constraints may make for better training objectives. We dis- data in [161] for generating novel designs. The FRAMED cuss the challenges with performance evaluation, constraint dataset [162] expands on BIKED with structural perfor- violation, and incorporation of novelty in the following sec- mance data, such as weight, safety factors, and deﬂections tions, as well as pathways to potential solutions. We further under various loads for all 4512 models as well as artiﬁcially discuss some general challenges for DGMs in engineering generated bicycle frames . design, such as limited availability of data, lack of bench- mark problems and metrics, and delayed adoption of cutting edge methods from different research communities. Our dis- 6.6 Metamaterials Datasets cussion is supported by observations from our literature re- Wang et al. [32] introduce a dataset of 248,396 2D unit view. cells represented by 50x50 pixelated matrices. The unit cells have associated stiffness tensor components provided and can also be used for TO research. Chan et al. [163] introduce 7.1 Design Performance Evaluation a dataset of 3,000 3D isosurface unit cells sampled from 30 Incorporating performance evaluation into DGMs is one level-set functions, along with corresponding 3D elastic ten- of the key stepping stones toward practical applications of sor components. Wang et al. [164] introduce a dataset of 795 ML in design. Three major challenges in performance eval- unit cells generated from 10 lattice models. Associated stiff- uation are ﬁdelity, cost, and differentiability. ness tensors are provided. The three datasets can be found 1. Evaluation methods lacking ﬁdelity may result in DGM- here . generated designs that do not meet speciﬁcations. 2. Computational cost precludes compatibility with meth- ods that heavily sample performance values. https://bit.ly/3ne4gwv https://github.com/AutodeskAILab/ Fusion360GalleryDataset https://github.com/googlecreativelab/ https://decode.mit.edu/projects/biked/ quickdraw-dataset https://decode.mit.edu/projects/framed/ https://sites.psu.edu/creativitymetrics/2018/ https://ideal.mech.northwestern.edu/research/ 07/18/milkfrother/ software/ https://www.sciencedirect.com/science/article/ pii/S2352340918302014?via%3Dihub 13 Copyright © by ASME 3. Lack of differentiability makes implementation into the 7.2 Feasibility, Constraints, and Manufacturability training process difﬁcult in any machine learning mod- One key component of design performance is obeying els which rely on gradient-based optimization. explicit design constraints, as well as implicit constraints such as physical feasibility and manufacturability. DGMs Physical evaluation, which means building a product and are difﬁcult to rely upon for explicit design constraints since testing its performance in the real-world, typically has the they are generally probabilistic and may generate completely highest ﬁdelity but is rarely adopted in DGMs due to its invalid designs. This issue is a concern that many researchers prohibitive cost. Qualitative human evaluation of designs have pointed to [177,33,103]. One potential solution is to de- has also been investigated in several papers [85, 73, 96]], velop inexpensive and reliable validation methods, however, although the evaluation is typically not focused on perfor- this is a challenging task that may require signiﬁcant human mance. Medium-ﬁdelity evaluations, such as numerical sim- input. ulations often deliver satisfactory ﬁdelity, but can be costly Another underlying issue lies in design representation, and are rarely differentiable. Methods that do incorporate which is discussed in detail in Sec. 3. Representations such medium-ﬁdelity performance evaluation like [69, 62] do not as images often have no clear translation to representations rely on performance score gradients, and instead alternate with practical uses. Other representations, such as 3D mod- (re)training and performance evaluation, typically doing this els of various types can only be realistically fabricated us- retraining a handful of times. ing additive manufacturing, which precludes many design Low-cost evaluation methods like surrogate models can domains. For ML-generated designs to be physically fabri- be worked into the training objective and are by far the most cated and used in practical applications, the feasibility across common performance evaluation method seen in DGMs [15, this ‘domain gap’ must be overcome [103, 87]. Though a 20, 77, 18, 70, 79, 60]. Unfortunately, surrogate models can few works have investigated manufacturability [100], we ob- be brittle and generalize poorly to designs that differ from served that modeling it is not considered in most of the pa- the data the surrogate was trained on. This is a particular pers we reviewed. Other works have attempted to apply concern when training data for surrogates is unevenly dis- DGMs to parameterizations that encode similar parametric tributed across the performance space [77] and when nov- design data to what would be used in manufacturing draw- elty and performance are incorporated into training objec- ings. However, the same works identify that parametric data tives for DGMs. Many different approaches have attempted of this sort is challenging to learn and generate [87]. to improve the performance of low-ﬁdelity surrogate models. For example, self-supervised data augmentation [77] uses the 7.3 Creativity and Novelty DGM itself to generate samples in sparse regions (through Whereas creativity and novelty are essential aspects optimization [137] or conditioning [77]) which then can be of the classic design process, DGMs rarely explicitly con- used to train the low-ﬁdelity models to perform more accu- sider either. Most DGMs learn to mimic the data cover- rately and in turn improve methods that rely on them for de- ing the existing and already explored portions of the design sign generation. Multi-ﬁdelity modeling is another promis- space. While this emulative behavior is helpful for main- ing approach, which involves generating surrogate models taining realism and ensuring sample quality, it incentivizes that augment a few costly high-ﬁdelity samples with low- DGMs against generating creative or novel designs [178]. ﬁdelity samples to attain higher ﬁdelity surrogates with min- For DGMs to progress towards more human-like design, ad- imal expense. vancements must be made in modeling creativity as well as Advancements have also been made in using machine in developing architectures that promote creativity. learning to improve and accelerate medium-ﬁdelity physics Several recent works [137, 15, 161] have proposed meth- simulations such as Finite Element Analysis [168, 169, 170, ods to encourage creativity and novelty in DGMs. In their 171, 172] and Computational Fluid Dynamics [173, 174, 175, framework, CreativeGAN, Nobari et al., focus on identify- 172]. These works, although not yet robust enough to be ing novelty and guiding DGMs towards such behavior by di- easily applied as an alternative to high-ﬁdelity simulations, rectly introducing novel features into typical designs, thereby provide a proof of concept for machine learning-based accel- expanding the design space and novelty of the DGM’s data. eration and even replacement of higher-ﬁdelity simulations. Creativity in machine learning has also been explored out- Numerous studies have also proposed crowdsourcing side of the design community. For example, Creative Ad- methods to evaluate synthesized data from generative ma- versarial Networks (CAN) [178] introduce entropy into the chine learning methods [90, 176, 166] instead of physical training to encourage the generation of surprising images. or physics-based evaluation. Though the ﬁdelity of crowd- Readers are directed to Franceschelli et al.’s survey on the sourced evaluation is highly task- and crowd-expertise- topic for more detail [179]. dependent, the high evaluation cost and lack of differentia- bility are fairly universal. In summary, current DGMs are largely constrained in 7.4 Evaluating Model Performance & Benchmark performance evaluation by an inherent ﬁdelity vs. cost trade- Problems off, however, several directions show promise in enabling Readers may have noted that many of the works dis- faster and more accurate evaluation methods that may escape cussed have nearly identical methodologies, though they are this limitation. applied to a speciﬁc dataset and only compare their work 14 Copyright © by ASME in this respect with a few other baselines. This may be at- datasets, publicly release these datasets, and establish bench- tributed to a large variety of design applications compared to mark problems for future works. We subsequently encourage other domains such as computer vision and a tendency of de- researchers to test their methods on other researchers’ data, sign researchers to ﬁnd solutions to their particular problem, publicly release their benchmarking results, and acknowl- instead of ﬁnding generalizable solutions to many. How- edge state-of-the-art methods when possible. ever, this makes any measure of ‘state-of-the-art’ algorithm dependent on speciﬁc applications. Without a good under- 7.6 Other DGMs standing of state-of-the-art methods that are broadly applica- So far our discussion has been focused on the VAE, ble across engineering domains, practitioners will struggle to GAN, and RL-based approaches, which have dominated the select models for practical deployment. While some works ﬁeld of DGMs in engineering design. Transformer-based se- in the design community have speciﬁcally introduced bench- quential models were initially developed for natural language mark problems and datasets [87, 162], there is still a need processing and generation or translation of text, however, for larger, higher-quality, and more numerous datasets and researchers have shown that the scalability of these mod- benchmarks. els allows for them to be used for the generation of com- The difﬁculty of establishing a state-of-the-art lies in the plex data in a sequential manner [180, 181, 182, 183]. Fur- lack of benchmark problems and performance metrics within thermore, these models have an unprecedented ability to be the DGM ﬁeld and within design automation as a whole. mixed with language processing. This may enable human Design datasets tend to be small, restricted in domain, and language-based conditioning and control over the generation sparse in distribution as we discuss in Section 7.5. Further- process of data [181, 183], enabling deployment for a much more, many methods are developed on proprietary datasets. wider user base and a much wider set of tasks. The applica- Finally, the hugely different design representation methods tions of transformer-based models as well as their scalability across the ﬁeld make establishing standardized model perfor- may appeal to researchers in the design community. Despite mance benchmarks difﬁcult, even if there were good datasets this, it is important to note that these models are particularly upon which to do so. These problems are noted by many au- large and therefore require very large datasets, further high- thors in the ﬁeld as well, who ﬁnd it difﬁcult to compare their lighting the need for better datasets. approach with existing ones [60, 177]. 8 Conclusion 7.5 Data Limitations and Quality In this review, we discussed the applications of Deep Data sparsity is one of the greatest challenges facing the Generative Models (DGMs) across engineering design ﬁelds. data-driven design community and is a particular concern for To give readers a sense of the tools and methods available, researchers developing data-hungry DGMs. Broadly, there we began our review with an overview of DGMs typically are three problems when it comes to the data. The ﬁrst used for engineering design problems, emphasizing Gen- of these is a general lack of data in many design domains. Although datasets continue to cover more and more design erative Adversarial Networks (GANs), Variational Autoen- coders (VAEs), and Reinforcement Learning (RL). To help ﬁelds, there still are many that lack publicly available data. weigh different design representation methods, we then dis- There are also data representations that are underrepresented cussed the strengths and weaknesses of common parameter- in the current datasets, notably graphs as discussed in Sec. 3. ization methods. To inform readers about existing work out- The second key data-related problem that designers face in side of their subdomain, we then collected and reviewed 63 applying DGMs is the insufﬁcient size of current datasets. papers from a variety of different engineering design sub- Many of the latest breakthroughs in deep learning, notably disciplines which directly propose DGMs. To make read- in computer vision and natural language processing, have ers aware of the datasets available on which to develop and owed their success to very large models which are notori- test data-driven design methods, we review commonly-used ously data-hungry, requiring millions or billions of training datasets in the ﬁeld. Finally, to provide inspiration, we dis- examples, and cannot be properly utilized with datasets of cuss key challenges and limitations currently seen across the smaller size. Focusing resources on the generation of very ﬁeld and highlight possible solution pathways. large publicly available datasets or effective data augmenta- tion methods would open doors for the design community to leverage larger deep models. The last major limitation in existing data is sparsity and bias. These problems are com- References mon in many design datasets. How the data is distributed in [1] Chakrabarti, A., Shea, K., Stone, R., Cagan, J., Camp- the performance space can cause some challenges for many bell, M., Hernandez, N. V., and Wood, K. L., 2011. DGMs, especially in inverse design problems [77]. Diver- “Computer-Based Design Synthesis Research: An sity is important to ensure the data covers the design space Overview”. Journal of Computing and Information as evenly as possible to avoid bias in the data. Keeping di- Science in Engineering, 11(2), 06. 021003. versity and bias avoidance in mind will help researchers gen- [2] Deng, L., and Yu, D., 2014. “Deep learning: Meth- erate better datasets. ods and applications”. Found. Trends Signal Process., We highly encourage researchers to develop diverse 7(3–4), jun, p. 197–387. 15 Copyright © by ASME [3] Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, V., Radford, A., Chen, X., and Chen, X., 2016. “Im- B., Warde-Farley, D., Ozair, S., Courville, A., and proved techniques for training gans”. In Advances Bengio, Y., 2014. “Generative adversarial nets”. In in Neural Information Processing Systems, D. Lee, Proceedings of the 27th International Conference on M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, Neural Information Processing Systems - Volume 2, eds., Vol. 29, Curran Associates, Inc., pp. 2234–2242. NIPS’14, MIT Press, p. 2672–2680. [15] Chen, W., and Ahmed, F., 2021. “Padgan: Learning [4] Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A., 2017. to generate high-quality novel designs”. Journal of “Unpaired image-to-image translation using cycle- Mechanical Design, 143(3), p. 031703. consistent adversarial networks”. In 2017 IEEE In- [16] Mirza, M., and Osindero, S., 2014. “Condi- ternational Conference on Computer Vision (ICCV), tional generative adversarial nets”. arXiv preprint pp. 2242–2251. arXiv:1411.1784. [5] Choi, Y., Uh, Y., Yoo, J., and Ha, J.-W., 2020. “Star- [17] Chen, X., Duan, Y., Houthooft, R., Schulman, J., gan v2: Diverse image synthesis for multiple do- Sutskever, I., and Abbeel, P., 2016. “Infogan: Inter- mains”. In 2020 IEEE/CVF Conference on Computer pretable representation learning by information max- Vision and Pattern Recognition (CVPR), pp. 8185– imizing generative adversarial nets”. In Proceedings 8194. of the 30th International Conference on Neural Infor- [6] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehti- mation Processing Systems, pp. 2180–2188. nen, J., and Aila, T., 2020. “Analyzing and improving [18] Dong, Y., Li, D., Zhang, C., Wu, C., Wang, H., Xin, the image quality of stylegan”. In 2020 IEEE/CVF M., Cheng, J., and Lin, J., 2020. “Inverse design of Conference on Computer Vision and Pattern Recogni- two-dimensional graphene/h-bn hybrids by a regres- tion (CVPR), pp. 8107–8116. sional and conditional gan”. Carbon, 169, pp. 9–16. [7] Karras, T., Laine, S., and Aila, T., 2019. “A style- [19] Ding, X., Wang, Y., Xu, Z., Welch, W. J., and Wang, based generator architecture for generative adversar- Z. J., 2021. “Ccgan: Continuous conditional gener- ial networks”. In 2019 IEEE/CVF Conference on ative adversarial networks for image generation”. In Computer Vision and Pattern Recognition (CVPR), 9th International Conference on Learning Represen- pp. 4396–4405. tations, ICLR 2021, Virtual Event, Austria, May 3-7, [8] Creswell, A., White, T., Dumoulin, V., Arulkumaran, 2021, OpenReview.net. K., Sengupta, B., and Bharath, A. A., 2018. “Genera- [20] Heyrani Nobari, A., Chen, W., and Ahmed, F., 2021. tive adversarial networks: An overview”. IEEE Signal “Pcdgan: A continuous conditional diverse generative Processing Magazine, 35(1), pp. 53–65. adversarial network for inverse design”. Proceedings [9] Salimans, T., Goodfellow, I., Zaremba, W., Cheung, of the 27th ACM SIGKDD Conference on Knowledge V., Radford, A., and Chen, X., 2016. “Improved tech- Discovery & Data Mining, Aug. niques for training gans”. In Proceedings of the 30th [21] Kingma, D. P., and Welling, M., 2013. “Auto- International Conference on Neural Information Pro- encoding variational bayes”. arXiv preprint cessing Systems, NIPS’16, Curran Associates Inc., arXiv:1312.6114. p. 2234–2242. [22] Kullback, S., and Leibler, R. A., 1951. “On infor- [10] Arjovsky, M., and Bottou, L., 2017. “Towards princi- mation and sufﬁciency”. The annals of mathematical pled methods for training generative adversarial net- statistics, 22(1), pp. 79–86. works”. arXiv preprint arXiv:1701.04862. [23] Kingma, D. P., and Welling, M., 2019. “An introduc- [11] Arjovsky, M., Chintala, S., and Bottou, L., 2017. tion to variational autoencoders”. Foundations and “Wasserstein generative adversarial networks”. In Trends in Machine Learning, 12(4), pp. 307–392. Proceedings of the 34th International Conference on [24] Sohn, K., Yan, X., and Lee, H., 2015. “Learn- Machine Learning - Volume 70, ICML’17, JMLR.org, ing structured output representation using deep con- p. 214–223. ditional generative models”. In Proceedings of the [12] Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, 28th International Conference on Neural Information V., and Courville, A., 2017. “Improved training of Processing Systems - Volume 2, NIPS’15, MIT Press, wasserstein gans”. In Proceedings of the 31st In- p. 3483–3491. ternational Conference on Neural Information Pro- [25] Kaelbling, L. P., Littman, M. L., and Moore, A. W., cessing Systems, NIPS’17, Curran Associates Inc., 1996. “Reinforcement learning: A survey”. J. Artif. p. 5769–5779. Int. Res., 4(1), May, p. 237–285. [13] Srivastava, A., Valkov, L., Russell, C., Gutmann, [26] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., M. U., and Sutton, C., 2017. “Veegan: Reducing mode Veness, J., Bellemare, M. G., Graves, A., Ried- collapse in gans using implicit variational learning”. miller, M., Fidjeland, A. K., Ostrovski, G., et al., In Advances in Neural Information Processing Sys- 2015. “Human-level control through deep reinforce- tems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wal- ment learning”. nature, 518(7540), pp. 529–533. lach, R. Fergus, S. Vishwanathan, and R. Garnett, [27] Daneshmand, M., Helmi, A., Avots, E., Noroozi, F., eds., Vol. 30, Curran Associates, Inc., pp. 3308–3318. Alisinanoglu, F., Arslan, H. S., Gorbova, J., Haamer, [14] Salimans, T., Goodfellow, I., Zaremba, W., Cheung, R. E., Ozcinar, C., and Anbarjafari, G., 2018. “3d 16 Copyright © by ASME scanning: A comprehensive survey”. arXiv preprint Engineering Conference. V01AT02A071. arXiv:1801.08863. [40] Patalano, S., Vitolo, F., and Lanzotti, A., 2013. “A [28] Remondino, F., 2003. “From point cloud to surface: graph-based software tool for the cad modeling of me- the modeling and visualization problem”. Interna- chanical assemblies”. In GRAPP/IVAPP. tional Archives of the Photogrammetry, Remote Sens- [41] Henaff, M., Bruna, J., and LeCun, Y., 2015. Deep ing and Spatial Information Sciences, 34. convolutional networks on graph-structured data. [29] Ranjan, A., Bolkart, T., Sanyal, S., and Black, M. J., [42] Yun, S., Jeong, M., Kim, R., Kang, J., and Kim, H. J., 2018. “Generating 3d faces using convolutional mesh 2019. “Graph transformer networks”. In Advances autoencoders”. In Proceedings of the European Con- in Neural Information Processing Systems, H. Wal- ference on Computer Vision (ECCV), pp. 704–720. lach, H. Larochelle, A. Beygelzimer, F. d'Alche-Buc, ´ [30] Cheng, S., Bronstein, M., Zhou, Y., Kotsia, I., Pan- E. Fox, and R. Garnett, eds., Vol. 32, Curran Asso- tic, M., and Zafeiriou, S., 2019. “Meshgan: Non- ciates, Inc. linear 3d morphable models of faces”. arXiv preprint [43] Velick ˇ ovic, ´ P., Cucurull, G., Casanova, A., Romero, arXiv:1903.10384. A., Lio, ` P., and Bengio, Y., 2018. “Graph attention [31] Zhang, Z., Wang, Y., Jimack, P. K., and Wang, H., networks”. In International Conference on Learning 2020. “Meshingnet: a new mesh generation method Representations. based on deep learning”. In International Conference [44] Vashishth, S., Sanyal, S., Nitin, V., and Talukdar, on Computational Science, Springer, pp. 186–198. P., 2019. “Composition-based multi-relational graph [32] Wang, L., Chan, Y.-C., Ahmed, F., Liu, Z., Zhu, P., convolutional networks”. In International Conference and Chen, W., 2020. “Deep generative modeling for on Learning Representations. mechanistic-based learning and design of metamate- [45] Li, Y., Tarlow, D., Brockschmidt, M., and Zemel, R., rial systems”. Computer Methods in Applied Mechan- 2017. Gated graph sequence neural networks. ics and Engineering, 372, p. 113377. [46] Liao, R., Li, Y., Song, Y., Wang, S., Hamilton, W., [33] Chen, W., and Fuge, M., 2019. “Synthesizing designs Duvenaud, D. K., Urtasun, R., and Zemel, R., 2019. with interpart dependencies using hierarchical gener- “Efﬁcient graph generation with graph recurrent at- ative adversarial networks”. Journal of Mechanical tention networks”. In Advances in Neural Informa- Design, 141(11), p. 111403. tion Processing Systems, H. Wallach, H. Larochelle, [34] Stump, G. M., Miller, S. W., Yukish, M. A., Simp- A. Beygelzimer, F. d'Alche-Buc, ´ E. Fox, and R. Gar- son, T. W., and Tucker, C., 2019. “Spatial grammar- nett, eds., Vol. 32, Curran Associates, Inc. based recurrent neural network for design form and [47] Bojchevski, A., Shchur, O., Zugner ¨ , D., and behavior optimization”. Journal of Mechanical De- Gunnemann, ¨ S., 2018. “Netgan: Generating graphs sign, 141(12). via random walks”. In International Conference on [35] Cao, W., Robinson, T., Hua, Y., Boussuge, F., Col- Machine Learning, pp. 609–618. ligan, A. R., and Pan, W., 2020. “Graph Repre- [48] You, J., Ying, R., Ren, X., Hamilton, W., and sentation of 3D CAD Models for Machining Feature Leskovec, J., 2018. “Graphrnn: Generating realis- Recognition With Deep Learning”. Vol. Volume 11A: tic graphs with deep auto-regressive models”. In In- 46th Design Automation Conference (DAC) of Inter- ternational conference on machine learning, PMLR, national Design Engineering Technical Conferences pp. 5708–5717. and Computers and Information in Engineering Con- [49] Li, Y., Vinyals, O., Dyer, C., Pascanu, R., and ference. V11AT11A003. Battaglia, P., 2018. Learning deep generative models [36] Yang, W., Ding, H., Zi, B., and Zhang, D., of graphs. 2017. “New Graph Representation for Planetary Gear [50] Cao, N. D., and Kipf, T., 2018. Molgan: An implicit Trains”. Journal of Mechanical Design, 140(1), 11. generative model for small molecular graphs. 012303. [51] You, J., Liu, B., Ying, R., Pande, V., and Leskovec, J., [37] Hsu, C.-H., and Lam, K.-T., 1992. “A New Graph 2018. “Graph convolutional policy network for goal- Representation for the Automatic Kinematic Analysis directed molecular graph generation”. In Proceedings of Planetary Spur-Gear Trains”. Journal of Mechani- of the 32nd International Conference on Neural Infor- cal Design, 114(1), 03, pp. 196–200. mation Processing Systems, pp. 6412–6422. [38] Lee, J. Y., and Kim, K., 1996. “Geometric rea- [52] Sosnovik, I., and Oseledets, I., 2019. “Neural net- soning for knowledge-based parametric design us- works for topology optimization”. Russian Journal ing graph representation”. Computer-Aided Design, of Numerical Analysis and Mathematical Modelling, 28(10), pp. 831–841. 34(4), pp. 215–223. [39] Coatanea, ´ E., Nonsiri, S., Christophe, F., and Mokam- [53] Behzadi, M. M., and Ilies ¸, H. T., 2021. “Real-time mel, F., 2014. “Graph Based Representation and topology optimization in 3d via deep transfer learn- Analyses for Conceptual Stages”. Vol. Volume 1A: ing”. Computer-Aided Design, 135, p. 103014. 34th Computers and Information in Engineering Con- [54] Keshavarzzadeh, V., Alirezaei, M., Tasdizen, T., and ference of International Design Engineering Techni- Kirby, R. M., 2021. “Image-based multiresolution cal Conferences and Computers and Information in topology optimization using deep disjunctive nor- 17 Copyright © by ASME mal shape model”. Computer-Aided Design, 130, steels using generative adversarial networks”. Engi- p. 102947. neering Reports, 3(1), p. e12274. [55] Cang, R., Yao, H., and Ren, Y., 2019. “One-shot [68] Liu, S., Zhong, Z., Takbiri-Borujeni, A., Kazemi, M., generation of near-optimal topology through theory- Fu, Q., and Yang, Y., 2019. “A case study on ho- driven machine learning”. Computer-Aided Design, mogeneous and heterogeneous reservoir porous me- 109, pp. 12–21. dia reconstruction by using generative adversarial net- [56] Malkiel, I., Mrejen, M., Nagler, A., Arieli, U., Wolf, works”. Energy Procedia, 158, pp. 6164–6169. L., and Suchowski, H., 2018. “Plasmonic nanostruc- [69] Shu, D., Cunningham, J., Stump, G., Miller, S. W., ture design and characterization via deep learning”. Yukish, M. A., Simpson, T. W., and Tucker, C. S., Light: Science & Applications, 7(1), pp. 1–8. 2020. “3d design using generative adversarial net- [57] Li, X., Zhang, Y., Zhao, H., Burkhart, C., Brin- works and physics-based validation”. Journal of Me- son, L. C., and Chen, W., 2018. “A transfer learn- chanical Design, 142(7), p. 071701. ing approach for microstructure reconstruction and [70] Sharpe, C., and Seepersad, C. C., 2019. “Topology structure-property predictions”. Scientiﬁc reports, design with conditional generative adversarial net- 8(1), pp. 1–13. works”. In International Design Engineering Tech- [58] Jung, J., Na, J., Park, H. K., Park, J. M., Kim, G., nical Conferences and Computers and Information in Lee, S., and Kim, H. S., 2021. “Super-resolving ma- Engineering Conference, Vol. 59186, American Soci- terial microstructure image via deep learning for mi- ety of Mechanical Engineers, p. V02AT03A062. crostructure characterization and mechanical behavior [71] Nie, Z., Lin, T., Jiang, H., and Kara, L. B., 2021. analysis”. npj Computational Materials, 7(1), pp. 1– “Topologygan: Topology optimization using genera- 11. tive adversarial networks based on physical ﬁelds over [59] Li, B., Huang, C., Li, X., Zheng, S., and Hong, J., the initial domain”. Journal of Mechanical Design, 2019. “Non-iterative structural topology optimization 143(3), p. 031715. using deep learning”. Computer-Aided Design, 115, [72] Yu, Y., Hur, T., Jung, J., and Jang, I. G., 2019. “Deep pp. 172–180. learning for determining a near-optimal topological [60] Rawat, S., and Shen, M. H., 2019. Application of design without any iteration”. Structural and Multi- adversarial networks for 3d structural topology opti- disciplinary Optimization, 59(3), pp. 787–799. mization. Tech. rep., SAE Technical Paper. [73] Valdez, S., Seepersad, C., and Kambampati, S., 2021. [61] Oh, S., Jung, Y., Lee, I., and Kang, N., 2018. “De- “A framework for interactive structural design explo- sign automation by integrating generative adversarial ration”. In International Design Engineering Tech- networks and topology optimization”. In International nical Conferences and Computers and Information in Design Engineering Technical Conferences and Com- Engineering Conference, IDETC-21, ASME. puters and Information in Engineering Conference, [74] Yilmaz, E., and German, B., 2020. “Conditional gen- Vol. 51753, American Society of Mechanical Engi- erative adversarial network framework for airfoil in- neers, p. V02AT03A008. verse design”. In AIAA aviation 2020 forum, p. 3185. [62] Oh, S., Jung, Y., Kim, S., Lee, I., and Kang, N., 2019. [75] Chen, W., and Fuge, M., 2018. “Bezier ´ gan: Au- “Deep generative design: Integration of topology opti- tomatic generation of smooth curves from inter- mization and generative models”. Journal of Mechan- pretable low-dimensional parameters”. arXiv preprint ical Design, 141(11). arXiv:1808.08871. [63] Tan, R. K., Zhang, N. L., and Ye, W., 2020. “A deep [76] Chen, W., Chiu, K., and Fuge, M., 2019. “Aero- learning–based method for the design of microstruc- dynamic design optimization and shape exploration tural materials”. Structural and Multidisciplinary Op- using generative adversarial networks”. In AIAA timization, 61(4), pp. 1417–1438. Scitech 2019 Forum, p. 2351. [64] Yang, Z., Li, X., Catherine Brinson, L., Choudhary, [77] Heyrani Nobari, A., Chen, W. W., and Ahmed, A. N., Chen, W., and Agrawal, A., 2018. “Mi- F., 2021. “RANGE-GAN: Design Synthesis Under crostructural materials design via deep adversarial Constraints Using Conditional Generative Adversarial learning methodology”. Journal of Mechanical De- Networks”. Journal of Mechanical Design, 09, pp. 1– sign, 140(11). 16. [65] Zhang, H., Yang, L., Li, C., Wu, B., and Wang, W., [78] Wang, P., Peng, D., Li, L., Chen, L., Wu, C., Wang, 2021. “Scaffoldgan: Synthesis of scaffold materials X., Childs, P., and Guo, Y., 2019. “Human-in-the- based on generative adversarial networks”. Computer- loop design with machine learning”. Proceedings of Aided Design, 138, p. 103041. the Design Society: International Conference on En- [66] Mosser, L., Dubrule, O., and Blunt, M. J., 2017. “Re- gineering Design, 1(1), p. 2577–2586. construction of three-dimensional porous media using [79] Guo, T., Lohan, D. J., Cang, R., Ren, M. Y., and Alli- generative adversarial neural networks”. Physical Re- son, J. T., 2018. “An indirect design representation for view E, 96(4), p. 043309. topology optimization using variational autoencoder [67] Lee, J.-W., Goo, N. H., Park, W. B., Pyo, M., and and style transfer”. In 2018 AIAA/ASCE/AHS/ASC Sohn, K.-S., 2021. “Virtual microstructure design for Structures, Structural Dynamics, and Materials Con- 18 Copyright © by ASME ference, p. 0804. ing and exploring the product form design space us- [80] Cang, R., Li, H., Yao, H., Jiao, Y., and Ren, Y., 2018. ing deep generative models”. In International De- “Improving direct physical properties prediction of sign Engineering Technical Conferences and Com- heterogeneous materials from imaging data via con- puters and Information in Engineering Conference, volutional neural network and a morphology-aware Vol. 50107, American Society of Mechanical Engi- generative model”. Computational Materials Science, neers, p. V02AT03A013. 150, pp. 212–221. [91] Ma, W., Cheng, F., Xu, Y., Wen, Q., and Liu, Y., [81] Li, X., Ning, S., Liu, Z., Yan, Z., Luo, C., and Zhuang, 2019. “Probabilistic representation and inverse design Z., 2020. “Designing phononic crystal with antici- of metamaterials based on a deep generative model pated band gap through a deep learning based data- with semi-supervised learning strategy”. Advanced driven method”. Computer Methods in Applied Me- Materials, 31(35), p. 1901111. chanics and Engineering, 361, p. 112737. [92] Zhang, W., Yang, Z., Jiang, H., Nigam, S., Yamakawa, [82] Liu, Z., Raju, L., Zhu, D., and Cai, W., 2020. “A hy- S., Furuhata, T., Shimada, K., and Kara, L. B., 2019. brid strategy for the discovery and design of photonic “3d shape synthesis for conceptual design and opti- structures”. IEEE Journal on Emerging and Selected mization using variational autoencoders”. In Inter- Topics in Circuits and Systems, 10(1), pp. 126–135. national Design Engineering Technical Conferences [83] Xue, T., Wallin, T. J., Menguc, Y., Adriaenssens, S., and Computers and Information in Engineering Con- and Chiaramonte, M., 2020. “Machine learning gen- ference, Vol. 59186, American Society of Mechanical erative models for automatic design of multi-material Engineers, p. V02AT03A017. 3d printed composite solids”. Extreme Mechanics Let- [93] Deshpande, S., and Purwar, A., 2020. “An Image- ters, 41, p. 100992. Based Approach to Variational Path Synthesis of [84] Brock, A., Lim, T., Ritchie, J. M., and We- Linkages”. Journal of Computing and Information ston, N., 2016. “Context-aware content genera- Science in Engineering, 21(2), 10. 021005. tion for virtual environments”. In International De- [94] Li, R., Zhang, Y., and Chen, H., 2021. “Learning the sign Engineering Technical Conferences and Com- aerodynamic design of supercritical airfoils through puters and Information in Engineering Conference, deep reinforcement learning”. AIAA Journal, pp. 1– Vol. 50084, American Society of Mechanical Engi- 14. neers, p. V01BT02A045. [95] Dering, M., Cunningham, J., Desai, R., Yukish, M. A., [85] Deshpande, S., and Purwar, A., 2019. “Computational Simpson, T. W., and Tucker, C. S., 2018. “A physics- creativity via assisted variational synthesis of mecha- based virtual environment for enhancing the quality nisms using deep generative models”. Journal of Me- of deep generative designs”. In International De- chanical Design, 141(12). sign Engineering Technical Conferences and Com- [86] Sharma, S., and Purwar, A., 2020. “Path synthesis puters and Information in Engineering Conference, of defect-free spatial 5-ss mechanisms using machine Vol. 51753, American Society of Mechanical Engi- learning”. In International Design Engineering Tech- neers, p. V02AT03A015. nical Conferences and Computers and Information in [96] Lee, X. Y., Balu, A., Stoecklein, D., Ganapathysub- Engineering Conference, Vol. 83990, American Soci- ramanian, B., and Sarkar, S., 2019. “A case study ety of Mechanical Engineers, p. V010T10A034. of deep reinforcement learning for engineering de- [87] Regenwetter, L., Curry, B., and Ahmed, F., 2021. sign: Application to microﬂuidic devices for ﬂow “BIKED: A dataset and machine learning benchmarks sculpting”. Journal of Mechanical Design, 141(11), for data-driven bicycle design”. In International De- p. 111401. sign Engineering Technical Conferences and Com- [97] Raina, A., Puentes, L., Cagan, J., and McComb, C., puters and Information in Engineering Conference, 2021. “Goal-directed design agents: Integrating vi- IDETC-21, ASME. sual imitation with one-step lookahead optimization [88] Tang, Y., Kojima, K., Koike-Akino, T., Wang, Y., Wu, for generative design”. Journal of Mechanical Design, P., Tahersima, M., Jha, D., Parsons, K., and Qi, M., 143(12), p. 124501. 2020. “Generative deep learning model for a multi- [98] Lopez, C. E., Ashour, O., and Tucker, C. S., 2019. level nano-optic broadband power splitter”. In 2020 “Reinforcement Learning Content Generation for Vir- Optical Fiber Communications Conference and Exhi- tual Reality Applications”. Vol. Volume 1: 39th Com- bition (OFC), IEEE, pp. 1–3. puters and Information in Engineering Conference of [89] Chen, H., and Liu, X., 2021. “Geometry enhanced International Design Engineering Technical Confer- generative adversarial networks for random heteroge- ences and Computers and Information in Engineering neous material representation”. In International De- Conference. V001T02A009. sign Engineering Technical Conferences and Com- [99] Cunningham, J., Lopez, C., Ashour, O., and Tucker, puters and Information in Engineering Conference, C. S., 2020. “Multi-Context Generation in Virtual Re- IDETC-21, ASME. ality Environments Using Deep Reinforcement Learn- [90] Burnap, A., Liu, Y., Pan, Y., Lee, H., Gonza- ing”. Vol. Volume 9: 40th Computers and Informa- lez, R., and Papalambros, P. Y., 2016. “Estimat- tion in Engineering Conference (CIE) of International 19 Copyright © by ASME Design Engineering Technical Conferences and Com- design process: Human-inspired heuristic guidance puters and Information in Engineering Conference. through learned visual design agents”. In Proceedings V009T09A072. of the Design Society: DESIGN Conference, Vol. 1, [100] Greminger, M., 2020. “Generative adversarial net- Cambridge University Press, pp. 355–364. works with synthetic training data for enforcing man- [110] Yukish, M. A., Stump, G. M., and Miller, S. W., ufacturing constraints on topology optimization”. In 2020. “Using recurrent neural networks to model spa- International Design Engineering Technical Confer- tial grammars for design creation”. Journal of Me- ences and Computers and Information in Engineer- chanical Design, 142(10), p. 104501. ing Conference, Vol. 84003, American Society of Me- [111] Zhu, J.-H., Zhang, W.-H., and Xia, L., 2016. “Topol- chanical Engineers, p. V11AT11A005. ogy optimization in aircraft and aerospace structures [101] Fujita, K., Minowa, K., Nomaguchi, Y., Yamasaki, S., design”. Archives of Computational Methods in Engi- and Yaji, K., 2021. “Design concept generation with neering, 23(4), pp. 595–622. variational deep embedding over comprehensive opti- [112] Xia, L., and Breitkopf, P., 2017. “Recent advances on mization”. In International Design Engineering Tech- topology optimization of multiscale nonlinear struc- nical Conferences and Computers and Information in tures”. Archives of Computational Methods in Engi- Engineering Conference, IDETC-21, ASME. neering, 24(2), pp. 227–249. [102] Cang, R., Vipradas, A., and Ren, Y., 2017. [113] Borrvall, T., and Petersson, J., 2003. “Topology opti- “Scalable microstructure reconstruction with multi- mization of ﬂuids in stokes ﬂow”. International jour- scale pattern preservation”. In International De- nal for numerical methods in ﬂuids, 41(1), pp. 77– sign Engineering Technical Conferences and Com- 107. puters and Information in Engineering Conference, [114] Zhou, S., and Li, Q., 2008. “A variational level Vol. 58134, American Society of Mechanical Engi- set method for the topology optimization of steady- neers, p. V02BT03A010. state navier–stokes ﬂow”. Journal of Computational [103] Cang, R., Xu, Y., Chen, S., Liu, Y., Jiao, Y., and Physics, 227(24), pp. 10178–10195. Yi Ren, M., 2017. “Microstructure representation and [115] Zegard, T., and Paulino, G. H., 2016. “Bridging reconstruction of heterogeneous materials via deep topology optimization and additive manufacturing”. belief network for computational material design”. Structural and Multidisciplinary Optimization, 53(1), Journal of Mechanical Design, 139(7), p. 071404. pp. 175–192. [104] Fokina, D., Muravleva, E., Ovchinnikov, G., and Os- [116] Langelaar, M., 2016. “Topology optimization of eledets, I., 2020. “Microstructure synthesis using 3d self-supporting structures for additive manufactur- style-based generative adversarial networks”. Phys- ing”. Additive Manufacturing, 12, pp. 60–70. ical Review E, 101(4), p. 043308. [117] Dbouk, T., 2017. “A review about the engineering de- [105] Wang, Z., Xian, W., Baccouche, M. R., Lanzerath, sign of optimal heat transfer systems using topology H., Li, Y., and Xu, H., 2021. “A gaussian mixture optimization”. Applied Thermal Engineering, 112, variational autoencoder-based approach for designing pp. 841–854. phononic bandgap metamaterials”. In International [118] Koga, A. A., Lopes, E. C. C., Nova, H. F. V., De Lima, Design Engineering Technical Conferences and Com- C. R., and Silva, E. C. N., 2013. “Development of puters and Information in Engineering Conference, heat sink device by using topology optimization”. In- IDETC-21, ASME. ternational Journal of Heat and Mass Transfer, 64, [106] Cang, R., and Ren, M. Y., 2016. “Deep Network- pp. 759–772. Based Feature Extraction and Reconstruction of Com- [119] Gatys, L., Ecker, A. S., and Bethge, M., 2015. “Tex- plex Material Microstructures”. Vol. Volume 2B: 42nd ture synthesis using convolutional neural networks”. Design Automation Conference of International De- Advances in neural information processing systems, sign Engineering Technical Conferences and Com- 28, pp. 262–270. puters and Information in Engineering Conference. [120] Gatys, L. A., Ecker, A. S., and Bethge, M., 2016. V02BT03A008. “Image style transfer using convolutional neural net- [107] Vermeer, K., Kuppens, R., and Herder, J., 2018. works”. In Proceedings of the IEEE conference on “Kinematic Synthesis Using Reinforcement Learn- computer vision and pattern recognition, pp. 2414– ing”. Vol. Volume 2A: 44th Design Automation Con- 2423. ference of International Design Engineering Techni- [121] Berthelot, D., Schumm, T., and Metz, L., 2017. “Be- cal Conferences and Computers and Information in gan: Boundary equilibrium generative adversarial net- Engineering Conference. V02AT03A009. works”. arXiv preprint arXiv:1703.10717. [108] Raina, A., McComb, C., and Cagan, J., 2019. “Learn- [122] Jiang, Z., Zheng, Y., Tan, H., Tang, B., and Zhou, H., ing to design from humans: Imitating human design- 2017. “Variational deep embedding: an unsupervised ers through deep learning”. Journal of Mechanical and generative approach to clustering”. In Proceed- Design, 141(11). ings of the 26th International Joint Conference on Ar- [109] Puentes, L., Raina, A., Cagan, J., and McComb, tiﬁcial Intelligence, pp. 1965–1972. C., 2020. “Modeling a strategic human engineering [123] Ledig, C., Theis, L., Huszar ´ , F., Caballero, J., Cun- 20 Copyright © by ASME ningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, [136] Kulesza, A., Taskar, B., et al., 2012. “Determinantal J., Wang, Z., et al., 2017. “Photo-realistic single image point processes for machine learning”. Foundations super-resolution using a generative adversarial net- and Trends® in Machine Learning, 5(2–3), pp. 123– work”. In 2017 IEEE Conference on Computer Vision 286. and Pattern Recognition (CVPR), IEEE, pp. 105–114. [137] Chen, W., and Ahmed, F., 2021. “Mo-padgan: [124] Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A., 2017. Reparameterizing engineering designs for augmented “Image-to-image translation with conditional adver- multi-objective optimization”. Applied Soft Comput- sarial networks”. In Proceedings of the IEEE con- ing, 113, p. 107909. ference on computer vision and pattern recognition, [138] Li, J., Xu, K., Chaudhuri, S., Yumer, E., Zhang, H., pp. 1125–1134. and Guibas, L., 2017. “Grass: Generative recursive [125] He, K., Zhang, X., Ren, S., and Sun, J., 2016. “Deep autoencoders for shape structures”. ACM Transac- residual learning for image recognition”. In Proceed- tions on Graphics (TOG), 36(4), pp. 1–14. ings of the IEEE conference on computer vision and [139] Zou, C., Yumer, E., Yang, J., Ceylan, D., and Hoiem, pattern recognition, pp. 770–778. D., 2017. “3d-prnn: Generating shape primitives with [126] Hu, J., Shen, L., and Sun, G., 2018. “Squeeze-and- recurrent neural networks”. In Proceedings of the excitation networks”. In Proceedings of the IEEE con- IEEE International Conference on Computer Vision, ference on computer vision and pattern recognition, pp. 900–909. pp. 7132–7141. [140] Groueix, T., Fisher, M., Kim, V. G., Russell, B. C., [127] Long, J., Shelhamer, E., and Darrell, T., 2015. “Fully and Aubry, M., 2018. “A papier-mach ˆ e ´ approach to convolutional networks for semantic segmentation”. learning 3d surface generation”. In Proceedings of In Proceedings of the IEEE conference on computer the IEEE conference on computer vision and pattern vision and pattern recognition, pp. 3431–3440. recognition, pp. 216–224. [128] Bostanabad, R., Zhang, Y., Li, X., Kearney, T., Brin- [141] Gao, L., Yang, J., Wu, T., Yuan, Y.-J., Fu, H., Lai, son, L. C., Apley, D. W., Liu, W. K., and Chen, W., Y.-K., and Zhang, H., 2019. “Sdm-net: Deep gener- 2018. “Computational microstructure characteriza- ative network for structured deformable mesh”. ACM tion and reconstruction: Review of the state-of-the-art Transactions on Graphics (TOG), 38(6), pp. 1–15. techniques”. Progress in Materials Science, 95, pp. 1– [142] Mo, K., Guerrero, P., Yi, L., Su, H., Wonka, P., Mitra, 41. N. J., and Guibas, L. J., 2019. “Structurenet: hierar- [129] Lee, H., Grosse, R., Ranganath, R., and Ng, A. Y., chical graph networks for 3d shape generation”. ACM 2009. “Convolutional deep belief networks for scal- Transactions on Graphics, 38(6), pp. 1–19. able unsupervised learning of hierarchical representa- [143] Park, J. J., Florence, P., Straub, J., Newcombe, R., and tions”. In Proceedings of the 26th annual international Lovegrove, S., 2019. “Deepsdf: Learning continuous conference on machine learning, pp. 609–616. signed distance functions for shape representation”. In [130] Yu, S., Zhang, Y., Wang, C., Lee, W.-k., Dong, Proceedings of the IEEE/CVF Conference on Com- B., Odom, T. W., Sun, C., and Chen, W., 2017. puter Vision and Pattern Recognition, pp. 165–174. “Characterization and design of functional quasi- [144] Liu, S., Giles, L., and Ororbia, A., 2018. “Learning random nanostructured materials using spectral den- a hierarchical latent-variable model of 3d shapes”. In sity function”. Journal of Mechanical Design, 139(7), 2018 International Conference on 3D Vision (3DV), p. 071401. IEEE, pp. 542–551. [131] Molesky, S., Lin, Z., Piggott, A. Y., Jin, W., Vuckovic, ´ [145] Qi, C. R., Su, H., Mo, K., and Guibas, L. J., 2017. J., and Rodriguez, A. W., 2018. “Inverse design in “Pointnet: Deep learning on point sets for 3d classiﬁ- nanophotonics”. Nature Photonics, 12(11), pp. 659– cation and segmentation”. In Proceedings of the IEEE 670. conference on computer vision and pattern recogni- [132] Dilokthanakul, N., Mediano, P. A., Garnelo, M., Lee, tion, pp. 652–660. M. C., Salimbeni, H., Arulkumaran, K., and Shana- [146] Karnewar, A., and Wang, O., 2020. “Msg-gan: Multi- han, M., 2016. “Deep unsupervised clustering with scale gradients for generative adversarial networks”. gaussian mixture variational autoencoders”. arXiv In Proceedings of the IEEE/CVF Conference on Com- preprint arXiv:1611.02648. puter Vision and Pattern Recognition, pp. 7799–7808. [133] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., [147] van Hasselt, H., Guez, A., and Silver, D., 2016. “Deep and Klimov, O., 2017. “Proximal policy optimization reinforcement learning with double q-learning”. Pro- algorithms”. arXiv preprint arXiv:1707.06347. ceedings of the AAAI Conference on Artiﬁcial Intelli- [134] Gielis, J., 2003. “A generic geometric transforma- gence, 30(1), Mar. tion that uniﬁes a wide range of natural and abstract [148] Odena, A., Olah, C., and Shlens, J., 2017. “Condi- shapes”. American journal of botany, 90(3), pp. 333– tional image synthesis with auxiliary classiﬁer gans”. 338. In International conference on machine learning, [135] Ha, D., and Eck, D., 2018. “A neural representation PMLR, pp. 2642–2651. of sketch drawings”. In International Conference on [149] McComb, C., Cagan, J., and Kotovsky, K., 2018. Learning Representations. “Data on the design of truss structures by teams of en- 21 Copyright © by ASME gineering students”. Data in brief, 18, pp. 160–163. and Information in Engineering Conference, IDETC- [150] Regenwetter, L., Curry, B., and Ahmed, F., 2022. 21, ASME. “Biked: A dataset for computational bicycle design [162] Regenwetter, L., Weaver, C., and Ahmed, F., 2022. with machine learning benchmarks”. Journal of Me- Framed: Data-driven structural performance analysis chanical Design, 144(3). of community-designed bicycle frames. [151] Hunter, W., et al., 2017. Topy-topology optimization [163] Chan, Y.-C., Ahmed, F., Wang, L., and Chen, W., with python. 2021. “Metaset: Exploring shape and property spaces [152] Iren, D., Ackermann, M., Gorfer, J., Pujar, G., for data-driven metamaterials design”. Journal of Me- Wesselmecking, S., Krupp, U., and Bromuri, S., chanical Design, 143(3), p. 031707. 2021. “Aachen-heerlen annotated steel microstructure [164] Wang, L., van Beek, A., Da, D., Chan, Y.-C., Zhu, P., dataset”. Scientiﬁc Data, 8(1), pp. 1–9. and Chen, W., 2021. “Data-driven multiscale design [153] Larmuseau, M., Sluydts, M., Theuwissen, K., Duprez, of cellular composites with multiclass microstructures L., Dhaene, T., and Cottenier, S., 2020. “Compact for natural frequency maximization”. arXiv preprint representations of microstructure images using triplet arXiv:2106.06478. networks”. npj Computational Materials, 6(1), pp. 1– [165] Jongejan, J., Rowley, H., Kawashima, T., Kim, J., and 11. Fox-Gieg, N., 2016. “The quick, draw!-ai experi- [154] DeCost, B. L., Hecht, M. D., Francis, T., Webler, ment”. Mount View, CA, accessed Feb, 17(2018), p. 4. B. A., Picard, Y. N., and Holm, E. A., 2017. “Uhcsdb: [166] Lopez, C., Miller, S. R., and Tucker, C. S., ultrahigh carbon steel micrograph database”. Integrat- 2018. “Human validation of computer vs human ing Materials and Manufacturing Innovation, 6(2), generated design sketches”. In International De- pp. 197–205. sign Engineering Technical Conferences and Com- [155] Zhao, H., Wang, Y., Lin, A., Hu, B., Yan, R., Mc- puters and Information in Engineering Conference, Cusker, J., Chen, W., McGuinness, D. L., Schadler, Vol. 51845, American Society of Mechanical Engi- L., and Brinson, L. C., 2018. “Nanomine schema: An neers, p. V007T06A015. extensible data representation for polymer nanocom- [167] Toh, C. A., and Miller, S. R., 2013. “Exploring the posites”. APL Materials, 6(11), p. 111108. utility of product dissection for early-phase idea gen- [156] Chang, A. X., Funkhouser, T., Guibas, L., Hanra- eration”. In International Design Engineering Tech- han, P., Huang, Q., Li, Z., Savarese, S., Savva, M., nical Conferences and Computers and Information in Song, S., Su, H., et al., 2015. “Shapenet: An Engineering Conference, Vol. 55928, American Soci- information-rich 3d model repository”. arXiv preprint ety of Mechanical Engineers, p. V005T06A034. arXiv:1512.03012. [168] Liang, L., Liu, M., Martin, C., and Sun, W., 2018. “A [157] Mo, K., Zhu, S., Chang, A. X., Yi, L., Tripathi, deep learning approach to estimate stress distribution: S., Guibas, L. J., and Su, H., 2019. “Partnet: A a fast and accurate surrogate of ﬁnite-element analy- large-scale benchmark for ﬁne-grained and hierarchi- sis”. Journal of The Royal Society Interface, 15(138), cal part-level 3d object understanding”. In Proceed- p. 20170844. ings of the IEEE/CVF Conference on Computer Vi- [169] Jiang, H., Nie, Z., Yeo, R., Farimani, A. B., and sion and Pattern Recognition, pp. 909–918. Kara, L. B., 2020. “StressGAN: A Generative Deep [158] Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, Learning Model for 2D Stress Distribution Predic- X., and Xiao, J., 2015. “3d shapenets: A deep rep- tion”. Vol. Volume 11B: 46th Design Automation resentation for volumetric shapes”. In Proceedings of Conference (DAC) of International Design Engineer- the IEEE conference on computer vision and pattern ing Technical Conferences and Computers and Infor- recognition, pp. 1912–1920. mation in Engineering Conference. V11BT11A023. [159] Kim, S., Chi, H.-g., Hu, X., Huang, Q., and Ra- [170] Nie, Z., Jiang, H., and Kara, L. B., 2019. “Stress Field mani, K., 2020. “A large-scale annotated mechani- Prediction in Cantilevered Structures Using Convolu- cal components benchmark for classiﬁcation and re- tional Neural Networks”. Vol. Volume 1: 39th Com- trieval tasks with deep neural networks”. In Proceed- puters and Information in Engineering Conference of ings of 16th European Conference on Computer Vi- International Design Engineering Technical Confer- sion (ECCV). ences and Computers and Information in Engineering [160] Willis, K. D., Pu, Y., Luo, J., Chu, H., Du, T., Lam- Conference. V001T02A011. bourne, J. G., Solar-Lezama, A., and Matusik, W., [171] Nie, Z., Jiang, H., and Kara, L. B., 2019. “Stress 2021. “Fusion 360 gallery: A dataset and environ- ﬁeld prediction in cantilevered structures using con- ment for programmatic cad construction from human volutional neural networks”. Journal of Computing design sequences”. ACM Transactions on Graphics and Information Science in Engineering, 20(1), Sep. (TOG), 40(4), pp. 1–24. [172] Pfaff, T., Fortunato, M., Sanchez-Gonzalez, A., and [161] Nobari, A. H., Rashad, M. F., and Ahmed, F., 2021. Battaglia, P., 2020. “Learning mesh-based simulation “Creativegan: Editing generative adversarial networks with graph networks”. In International Conference on for creative design synthesis”. In International Design Learning Representations. Engineering Technical Conferences and Computers [173] Kochkov, D., Smith, J. A., Alieva, A., Wang, Q., 22 Copyright © by ASME Brenner, M. P., and Hoyer, S., 2021. “Machine learning–accelerated computational ﬂuid dynamics”. Proceedings of the National Academy of Sciences, 118(21). [174] Duraisamy, K., Iaccarino, G., and Xiao, H., 2019. “Turbulence modeling in the age of data”. Annual Re- view of Fluid Mechanics, 51(1), pp. 357–377. [175] Kim, B., Azevedo, V. C., Thuerey, N., Kim, T., Gross, M., and Solenthaler, B., 2019. “Deep ﬂuids: A gen- erative network for parameterized ﬂuid simulations”. Computer Graphics Forum, 38(2), May, p. 59–70. [176] Dering, M. L., and Tucker, C. S., 2017. “Generative adversarial networks for increasing the veracity of big data”. In 2017 IEEE International Conference on Big Data (Big Data), IEEE, pp. 2595–2602. [177] Panchal, J. H., Fuge, M., Liu, Y., Missoum, S., and Tucker, C., 2019. “Special Issue: Machine Learning for Engineering Design”. Journal of Mechanical De- sign, 141(11), 10. 110301. [178] Elgammal, A., Liu, B., Elhoseiny, M., and Mazzone, M., 2017. “Can: Creative adversarial networks gen- erating “art” by learning about styles and deviating from style norms”. In 8th International Conference on Computational Creativity, ICCC 2017, Georgia In- stitute of Technology. [179] Franceschelli, G., and Musolesi, M., 2021. Creativity and machine learning: A survey. [180] Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I., 2020. “Generative pre- training from pixels”. In Proceedings of the 37th In- ternational Conference on Machine Learning, H. D. III and A. Singh, eds., Vol. 119 of Proceedings of Ma- chine Learning Research, PMLR, pp. 1691–1703. [181] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I., 2021. Zero- shot text-to-image generation. [182] Dhariwal, P., Jun, H., Payne, C., Kim, J. W., Radford, A., and Sutskever, I., 2020. Jukebox: A generative model for music. [183] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I., 2021. Learning transferable visual models from natural lan- guage supervision. 23 Copyright © by ASME

Journal

Journal of Mechanical Design – Unpaywall

Published: Feb 16, 2022

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 7-Day Trial for You or Your Team.

Learn More →

Deep Generative Models in Engineering Design: A Review

Deep Generative Models in Engineering Design: A Review

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 7-Day Trial for You or Your Team.

Learn More →

Deep Generative Models in Engineering Design: A Review

Deep Generative Models in Engineering Design: A Review

References (199)

Abstract

Journal

Recommended Articles

There are no references for this article.

Our policy towards the use of cookies