LD-RNN - Label-Dependency Aware Recurrent Neural Networks
Index des sujets :
Description
LD-RNN tool is a raw software package allowing to train and test several types of RNNs, including the new variant of RNN, named LD-RNN by the way, described in Improving Recurrent Neural Networks for Sequence Labelling, Label-Dependencies Aware Recurrent Neural Networks (coming soon) and Étude de réseaux de neurones récurrents pour l’étiquetage de séquences (French publication).
The LD-RNN variant is an evolution of the Jordan RNN where labels are given back as input to the network, and thus converted into embeddings the same way as words, as shown in the following image.
While very simple, this modification provides a big advantage over other more traditional RNNs, including LSTM and GRU, as I show in the CICling 2017 paper Label-Dependencies Aware Recurrent Neural Networks (pdf file coming soon).
Note that results reported in Improving Recurrent Neural Networks for Sequence Labelling and Étude de réseaux de neurones récurrents pour l’étiquetage de séquences are obtained using a sigmoid and L2 regularization. In the CICling 2017 paper a ReLU hidden layer and dropout regularization have been added, giving better results. Adding also a character-level convolution layer further improves results.
Saying raw software I mean that LD-RNN is made of a bunch of scripts. The core part is coded in Octave. Some Perl scripts are used to convert input data and output data format. Some Bash scripts are used to make RNNs training and testing easier.
Note that using advanced BLAS library support, in particular OpenBLAS or nvBLAS, Octave can run very very fast without having to change any single line of code. Read for example the following post to see how Octave can speed-up using these advanced BLAS libraries: Drop-in Acceleration of GNU Octave.
The LD-RNN variant is an evolution of the Jordan RNN where labels are given back as input to the network, and thus converted into embeddings the same way as words, as shown in the following image.
While very simple, this modification provides a big advantage over other more traditional RNNs, including LSTM and GRU, as I show in the CICling 2017 paper Label-Dependencies Aware Recurrent Neural Networks (pdf file coming soon).
Note that results reported in Improving Recurrent Neural Networks for Sequence Labelling and Étude de réseaux de neurones récurrents pour l’étiquetage de séquences are obtained using a sigmoid and L2 regularization. In the CICling 2017 paper a ReLU hidden layer and dropout regularization have been added, giving better results. Adding also a character-level convolution layer further improves results.
Saying raw software I mean that LD-RNN is made of a bunch of scripts. The core part is coded in Octave. Some Perl scripts are used to convert input data and output data format. Some Bash scripts are used to make RNNs training and testing easier.
Note that using advanced BLAS library support, in particular OpenBLAS or nvBLAS, Octave can run very very fast without having to change any single line of code. Read for example the following post to see how Octave can speed-up using these advanced BLAS libraries: Drop-in Acceleration of GNU Octave.
Features
- You can train LD-RNN variant and Jordan RNN (I will dismiss Jordan RNN soon however, and I'm going to integrate GRU hidden layer)
- L2 and dropout regularization
- Character-level convolution layer
- Deep networks
- Forward, backward and bidirectional models
- Early stopping based on the prediction accuracy on the development data
Download
Please send me an email @ens.
After first few downloads and once I will be sure everybody can install and use the package easily and without problems, I will put a direct link right here.
Your comments/remarks are welcome. Thank you
After first few downloads and once I will be sure everybody can install and use the package easily and without problems, I will put a direct link right here.
Your comments/remarks are welcome. Thank you
Licence
LD-RNN is provided under the 3-clause BSD licence (this may change once I will have more details from the CNRS copyright bureau):
Copyright (c) 2016 - 2017, CNRS
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
- Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
- Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
Installation and usage
See the README file in the package.