Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

A unified architecture for natural language processing: deep neural networks with multitask learning

A unified architecture for natural language processing: deep neural networks with multitask learning A Uni ed Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning Ronan Collobert Jason Weston NEC Labs America, 4 Independence Way, Princeton, NJ 08540 USA collober@nec-labs.com jasonw@nec-labs.com Abstract We describe a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense (grammatically and semantically) using a language model. The entire network is trained jointly on all these tasks using weight-sharing, an instance of multitask learning. All the tasks use labeled data except the language model which is learnt from unlabeled text and represents a novel form of semi-supervised learning for the shared tasks. We show how both multitask learning and semi-supervised learning improve the generalization of the shared tasks, resulting in stateof-the-art performance. Currently, most research analyzes those tasks separately. Many systems possess few characteristics that would help develop a uni ed architecture which would presumably be necessary for deeper semantic tasks. In particular, many systems possess three failings in this regard: (i) they are shallow in the sense that the classi er is often linear, (ii) for good performance http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png

A unified architecture for natural language processing: deep neural networks with multitask learning

Association for Computing Machinery — Jul 5, 2008

Loading next page...
/lp/association-for-computing-machinery/a-unified-architecture-for-natural-language-processing-deep-neural-SqaQ5B5R1S

References

References for this paper are not available at this time. We will be adding them shortly, thank you for your patience.

Datasource
Association for Computing Machinery
Copyright
Copyright © 2008 by ACM Inc.
ISBN
978-1-60558-205-4
doi
10.1145/1390156.1390177
Publisher site
See Article on Publisher Site

Abstract

A Uni ed Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning Ronan Collobert Jason Weston NEC Labs America, 4 Independence Way, Princeton, NJ 08540 USA collober@nec-labs.com jasonw@nec-labs.com Abstract We describe a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense (grammatically and semantically) using a language model. The entire network is trained jointly on all these tasks using weight-sharing, an instance of multitask learning. All the tasks use labeled data except the language model which is learnt from unlabeled text and represents a novel form of semi-supervised learning for the shared tasks. We show how both multitask learning and semi-supervised learning improve the generalization of the shared tasks, resulting in stateof-the-art performance. Currently, most research analyzes those tasks separately. Many systems possess few characteristics that would help develop a uni ed architecture which would presumably be necessary for deeper semantic tasks. In particular, many systems possess three failings in this regard: (i) they are shallow in the sense that the classi er is often linear, (ii) for good performance

There are no references for this article.