OpenGrm Advanced Usage
Below are a variety of topics covered in greater depth or of more specialized interest than found in the Quick Tour. Reading the
first is recommended.
The C++ operations in OpenGrm offer extensive distributed computation support. N-gram counting can readily be parallelized by sharding
the data and producing a count FST Md
for each data shard d
. These can then be count-merged
to produce a single count FST. Alternatively and with more parallelism,
can be further split by context c
, which restricts each sub-model Md,c
to a specific range of n-grams. The Md,c
in the same context c
can then be count-merged to produce
one model Mc
for each context. The Mc
are constructed to be in proper n-gram model format
and so can be processed in parallel by the estimation and pruning operations
and then the pruned model components can be model-merged
into a single model at the end of this pipeline.
We have implemented a complete distributed version of OpenGrm NGram in C++ Flume
, however this system is currently not released. Instead, we provide here some added
functionality to our convenience script, ngram.sh
. While this does not perform parallel computation, it can construct a pruned model as described above using data and context sharding. This allows processing corpora that would otherwise exceed available memory provided adequate disk space (under
) and computation time are provided. The script also could serve as a starting point for a fully distributed implementation by parallelizing the calls internal to the script, which should linearly speed up the pipeline with the degree of parallelism. An implementation that didn't use the file system for all data sharing/transfer like
would also help greatly.
Multiple data shards are supported by specifying multiple input files to
. Multiple contexts are supported by specifying a context file with
. The best way to create a context file with, say, ten contexts balanced in size is with:
ngramcontext --context=10 lm.fst >context.txt
is a n-gram LM that was built on a sample of the corpus (e..g, small enough to build unshared). Note you must provide a context file (even if it only has one context) if you want to use data sharding.
If you wish the shared context models to be merged when the pipeline finishes, you should provide the
flag, otherwise the component models will be returned.
- 07 Aug 2013