TY - JOUR
AU - 
AB - PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation Bin Bi, Chenliang Li, Chen Wu, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, Luo Si Alibaba Group fb.bi, lcl193798, wuchen.wc, ym119608g@alibaba-inc.com fhebian.ww, songfang.hsf, f.huang, luo.sig@alibaba-inc.com Abstract generating natural language sentences, including tasks like neural machine translation (Bahdanau Self-supervised pre-training, such as et al., 2015; Vaswani et al., 2017), abstractive sum- BERT (Devlin et al., 2018), MASS (Song marization (Rush et al., 2015; See et al., 2017a; et al., 2019) and BART (Lewis et al., 2019), Gehrmann et al., 2018), generative question an- has emerged as a powerful technique for nat- ural language understanding and generation. swering (QA) (Tan et al., 2017; Bi et al., 2019), Existing pre-training techniques employ au- question generation (Zhao et al., 2018) and con- toencoding and/or autoregressive objectives to versational response generation (Vinyals and Le, train Transformer-based models by recovering 2015). Many of the language generation tasks re- original word tokens from corrupted text with quire the models to read and to comprehend a given some masked tokens. The training goals of document, based on which output text is generated. existing techniques are often inconsistent In this paper, we present PALM, a novel approach with 
TI - PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation
JO - Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
DO - 10.18653/v1/2020.emnlp-main.700
DA - 2020-01-01
UR - https://www.deepdyve.com/lp/unpaywall/palm-pre-training-an-autoencoding-autoregressive-language-model-for-TyJsFrPYAI
DP - DeepDyve
ER -