A LetterAnalyzer creates a TokenStream that splits the input up into maximal strings of characters as recognized by the current locale. If implemented in Ruby it would look like;
class LetterAnalyzer def initialize(lower = true) @lower = lower end def token_stream(field, str) return LetterTokenizer.new(str, @lower) end end
As you can see it makes use of the LetterTokenizer.
Create a new LetterAnalyzer which downcases tokens by default but can optionally leave case as is. Lowercasing will be done based on the current locale.
lower | set to false if you don’t want the field’s tokens to be downcased |
static VALUE frb_letter_analyzer_init(int argc, VALUE *argv, VALUE self) { Analyzer *a; GET_LOWER(true); #ifndef POSH_OS_WIN32 if (!frb_locale) frb_locale = setlocale(LC_CTYPE, ""); #endif a = mb_letter_analyzer_new(lower); Frt_Wrap_Struct(self, NULL, &frb_analyzer_free, a); object_add(a, self); return self; }
Disabled; run with --debug to generate this.
Generated with the Darkfish Rdoc Generator 1.1.6.