This operation merges
two n-gram language models or two n-gram count
FSTs. The operation provides options for weighting the two input
FSTs, and for using the smoothing while merging.
ngrammerge [--options] in1.fst in2.fst [out.fst]
--alpha: type = double, default = 1.0, weight for in1.fst in real semiring
--beta: type = double, default = 1.0, weight for in2.fst in real semiring
--normalize: type = bool, default = false, whether to normalize the resulting model
--use_smoothing: type = bool, default = false, whether to use model smoothing when merging
--fixedorder: type = bool, default false, whether to merge in the given argument order
class NGramMerge(StdMutableFst *infst1, StdMutableFst *infst2, double alpha, double beta);
In addition to the simple C++ usage above, optional arguments permit the passing of non-default values for various parameters similar to the command-line version.
Suppose we split our corpus up into two parts, earnest.aa and earnest.ab, e.g., by using split:
$ split -844 earnest.txt earnest.
If we count each half independently, we can then merge the counts to get the same counts as derived above from the full corpus (earnest.cnts):
$ farcompilestrings -symbols=earnest.syms -keep_symbols=1 earnest.aa >earnest.aa.far
$ ngramcount -order=5 earnest.aa.far >earnest.aa.cnts
$ farcompilestrings -symbols=earnest.syms -keep_symbols=1 earnest.ab >earnest.ab.far
$ ngramcount -order=5 earnest.ab.far >earnest.ab.cnts
$ ngrammerge earnest.aa.cnts earnest.ab.cnts >earnest.merged.cnts
$ fstequal earnest.cnts earnest.merged.cnts
Note that, unlike our example merging unnormalized counts above, merging two smoothed models that have been built from half a corpus each will result in a different model than one built from the corpus as a whole, due to the smoothing and mixing.
Each of the two model or count FSTs can be weighted, using the --alpha
switch for the first input FST, and the --beta
switch for the second input FST. These weights are interpreted in the real semiring and both default to one, meaning that by default the original counts or probabilities are not scaled. For an n-gram w1 ... wk
, the default count merging approach will yield
C(w1 ... wk) = <alpha> C1(w1 ... wk) + <beta> C2(w1 ... wk)
To merge two smoothed models, the --use_smoothing=true
option provides non-zero probability from each input language model to any in-vocabulary n-gram; and the --normalize=true
option ensures that the resulting model is fully normalized. For example, to produce a merged model that weights the contribution of the first model by a factor of 3 and the contribution of the second model by a factor of 2:
$ ngrammerge --use_smoothing --normalize --alpha=3 --beta=2 earnest.aa.mod earnest.ab.mod >earnest.merged.mod