Paul R. Dixon

OpenFst 1.3.3 mapped files

One new addition in OpenFst 1.3.3 is memory mapping of const and compact Fsts. Part of the reading code has changed and there is now a new enumeration that switches the reading mode between MAP and READ modes.

enum FileReadMode { READ, MAP };

By default the value is set to READ and this will read the Fsts into RAM as normal. The shell tools introduce a new command line option --fst_read_mode which can be set to map to attempt memory map the file. The process of enabling memory mapping is slightly more complicated as the code requires the Fst files are 16 byte aligned. The files can be aligned by compiling or converting the Fst with the --fst_align=true flag. Here are the commands to convert to a const type that is aligned and then memory map the Fst into a command line tool. Adding the --v=1 will display logging information indicating if the Fst was mapped or code fell back to reading mode.

fstconvert --fst_type=const --fst_align=true in.ofst out.ofst
fstinfo --fst_read_mode=map --v=1 out.ofst

In addition to displaying the Fst information there should be some additional logging messages like the following:

INFO: mmap'ed region of 7477860 at offset 80 from test.ofst to addr 0000003C
INFO: mmap'ed region of 9084064 at offset 7477952 from test.ofst to addr 00000044

Page 1 / 1

social