OpenGrm Advanced Usage

Below are a variety of topics covered in greater depth or of more specialized interest than found in the Quick Tour. Reading the Quick Tour first is recommended.

Distributed Computation Work in progress, under construction

The C++ operations in OpenGrm offer extensive distributed computation support. N-gram counting can readily be parallelized by sharding the data and producing a count FST Md for each data shard d. These can then be count-merged to produce a single count FST. Alternatively and with more parallelism, each Md can be further split by context c, which restricts each sub-model Md,c to a specific range of n-grams. The Md,c in the same context c can then be count-merged to produce one model Mc for each context. The Mc are constructed to be in proper n-gram model format and so can be processed in parallel by the estimation and pruning operations and then the pruned model components can be model-merged into a single model at the end of this pipeline.

We have implemented a complete distributed version of OpenGrm NGram in C++ Flume, however this system is currently not released. Instead, we provide here some added functionality to our convenience script, While this does not perform parallel computation, it can construct a pruned model as described above using data and context sharding. This allows processing corpora that would otherwise exceed available memory provided adequate disk space (under $TMPDIR) and computation time are provided. The script also could serve as a starting point for a fully distributed implementation by parallelizing the calls internal to the script, which should linearly speed up the pipeline with the degree of parallelism. An implementation that didn't use the file system for all data sharing/transfer like would also help greatly.

Multiple data shards are supported by specifying multiple input files to with --ifile="in.txt[0-9]". Multiple contexts are supported by specifying a context file with --contexts=context.txt. The best way to create a context file with, say, ten contexts balanced in size is with:

ngramcontext --context=10 lm.fst >context.txt
where lm.fst is a n-gram LM that was built on a sample of the corpus (e..g, small enough to build unshared). Note you must provide a context file (even if it only has one context) if you want to use data sharding. If you wish the shared context models to be merged when the pipeline finishes, you should provide the --merge_contexts flag, otherwise the component models will be returned.

-- MichaelRiley - 07 Aug 2013

Topic revision: r2 - 2013-08-09 - MichaelRiley
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback