Marco Dinarelli with his first journal publication in a IEEE review    Marco Dinarelli
Web site of Marco Dinarelli in English  Site web de Marco Dinarelli en français  Sito web di Marco Dinarelli in italiano 


LaTTiCe-CNRS
UMR 8094
Office 9
1 rue Maurice Arnoux
92120 Montrouge, France

Email:
marco [dot] dinarelli [at] ens [dot] fr
marco [dot] dinarelli [at] gmail [dot] com

                        Curriculum Vitae           Profile of Marco Dinarelli in LinkedIn


Latest news

2017 / 06 / 28:
Chair of the session Vector representations at the French conference TALN 2017

2017 / 05 / 22:
Paper accepted at the Interspeech 2017 conference

LD-RNN - Label-Dependency Aware Recurrent Neural Networks

Content index:

Description

LD-RNN tool is a raw software package allowing to train and test several types of RNNs, including the new variant of RNN, named LD-RNN by the way, described in Improving Recurrent Neural Networks for Sequence Labelling, Label-Dependencies Aware Recurrent Neural Networks (coming soon) and Étude de réseaux de neurones récurrents pour l’étiquetage de séquences (French publication).
The LD-RNN variant is an evolution of the Jordan RNN where labels are given back as input to the network, and thus converted into embeddings the same way as words, as shown in the following image.
LD-RNN detailed architecture


While very simple, this modification provides a big advantage over other more traditional RNNs, including LSTM and GRU, as I show in the CICling 2017 paper Label-Dependencies Aware Recurrent Neural Networks (pdf file coming soon).
Note that results reported in Improving Recurrent Neural Networks for Sequence Labelling and Étude de réseaux de neurones récurrents pour l’étiquetage de séquences are obtained using a sigmoid and L2 regularization. In the CICling 2017 paper a ReLU hidden layer and dropout regularization have been added, giving better results. Adding also a character-level convolution layer further improves results.

Saying raw software I mean that LD-RNN is made of a bunch of scripts. The core part is coded in Octave. Some Perl scripts are used to convert input data and output data format. Some Bash scripts are used to make RNNs training and testing easier.
Note that using advanced BLAS library support, in particular OpenBLAS or nvBLAS, Octave can run very very fast without having to change any single line of code. Read for example the following post to see how Octave can speed-up using these advanced BLAS libraries: Drop-in Acceleration of GNU Octave.

Features

  • You can train LD-RNN variant and Jordan RNN (I will dismiss Jordan RNN soon however, and I'm going to integrate GRU hidden layer)
  • L2 and dropout regularization
  • Character-level convolution layer
  • Deep networks
  • Forward, backward and bidirectional models
  • Early stopping based on the prediction accuracy on the development data

Download

Please send me an email @ens.

After first few downloads and once I will be sure everybody can install and use the package easily and without problems, I will put a direct link right here.

Your comments/remarks are welcome. Thank you

Licence

LD-RNN is provided under the 3-clause BSD licence (this may change once I will have more details from the CNRS copyright bureau):

Copyright (c) 2016 - 2017, CNRS
All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
  2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
  3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Installation and usage

See the README file in the package.