Handling Natural Languages

Algolia supports all languages

The Algolia engine supports all languages out of the box. It’s language agnostic and matches the text in the search bar with the text in the index—this is called textual matching.

For example, if you have an index with only English text, and a user searches with Japanese text, the engine won’t return any results because the Latin alphabet doesn’t match Japanese characters. If your users are searching in Japanese, your index should likely contain Japanese text. If you want to support multiple languages, the most common solution is to create one index per language.

To support all languages, Algolia uses a wide array of natural language techniques. These techniques range from general, such as finding words or using Unicode, to specific, including distinguishing letters from logograms, breaking down compound words, and using single-language dictionaries for vocabulary.

This page is organized around five natural language understanding strategies:

• Engine-level processing (normalization)
• Dictionaries
• Query Processing
• Natural Language Processing (NLP) with Rules
• Configuring Typo Tolerance

Some language-based techniques (such as normalization) play an integral role in the engine, and are performed with every indexing and search operation. These are generally not configurable. Other techniques rely on specialized dictionaries, which facilitate word and word-root detection. These come with several configurable options. Finally, Algolia offers many other techniques (like typo tolerance, Rules) that can be turned on, turned off, or fine-tuned according to the use case. These are also configurable using Algolia’s API settings and suggested best practices.

How the engine normalizes data

The engine performs normalization both at indexing and query time, ensuring consistency in how your data is represented as well as matched.

You can’t globally turn off normalization, but you can do it for certain special characters using the keepDiacriticsOnCharacters setting. Additionally, the normalization process is language-agnostic and equally applies to all languages.

What does normalization mean?

• Turn all characters to lower case
• Remove all diacritics, for example, accents, umlauts, Arabic short vowels-however, you can define diacritics to keep using the keepDiacriticsOnCharacters index setting
• Remove punctuation within words, for example, apostrophes
• Manage punctuation between words
• Use word separators, such as spaces or other characters
• Include or exclude non-alphanumeric characters (separatorsToIndex)
• Transform traditional Chinese to modern

Some of these actions, such as removing punctuation within words, managing punctuation between words, and handling non-alphanumeric characters in general are part of the tokenization process. Understanding how Algolia handles this process is key to understanding how Algolia concatenates and split words at indexing and query time.

Adding language specificity using dictionaries

As already suggested, some of these automated techniques don’t work in all languages. This section describes language-specific functionalities of the engine.

No automatic language detection

Algolia doesn’t attempt to detect the language of your records nor the language of your end users as they type in queries.

Therefore, to benefit from language-specific algorithms, you need to tell the engine in what language you want your records to be interpreted.

• If you don’t pick a language, the engine assumes that you want to support all languages. The drawback here is that you create ambiguities by mixing every language’s peculiarities. For example, plurals in Italian are applied to plurals in English, causing problems such as the following: “paste”, the plural of “pasta” in Italian, will also be considered the plural of “pasta” in English, which isn’t the case, as “paste” in English is a word in its own right (to spread).
• It’s okay mix two or three languages in a single index, and specify them in your settings. However, you should prepare your indices and records appropriately. For more on this, please refer to the multiple languages tutorial.

Even though the engine can do most tasks without knowing the language of an index, there are some tasks that require knowledge of the language. For example, the engine can only compare plural to singular forms by knowing the language. The same applies to removing small words like “to” and “the” (stop words).

Because the default language of an index is all languages, enabling removeStopWords or ignorePlurals without setting an index’s language will ignore the wrong plurals and remove the wrong stop words. It’s therefore very important to set the query languages of all your indices.

Using dictionaries

Several of the language-related methods require the use of dictionaries. With dictionaries, the engine can apply language-specific, word-based logic to your data and your end user’s queries. Algolia maintains separate, language-specific dictionaries for:

• removing stop words
• detecting pluralized and other declined forms (alternative forms of words due to number, case, or gender)
• splitting compound words (also known as decompounding)
• handling Asian logograms (CJK)

Algolia provides default dictionaries for all supported query languages. While Algolia regularly updates these dictionaries, you can also customize the stop words dictionaries, declensions dictionaries, and decompounding dictionaries for your use case.

Disabling typo tolerance and prefix search on specific words

The advancedSyntax parameter lets you turn off typo tolerance on specific words in a query by using double quotes. For example, the query “foot problems” is typo tolerant on both query words, while ““foot” problems” is only typo tolerant on “problems”.

This parameter also disables prefix searching on words inside the double quotes.

Typo tolerance and languages

What’s a typo?

• A missing letter in a word, “hllo” → “hello”
• An extraneous letter, “heello” → “hello”
• Inverted letters: “hlelo” → “hello”
• Substituted letter: “heilo” → “hello”

Typo tolerance allows users to make mistakes while typing and still find the words they’re looking for. This is done by matching words that are close in spelling.

Other spelling errors

Extra or missing spaces and punctuation doesn’t count as typos. Algolia only handles them if typoTolerance is enabled (set to true, min or strict). For example:

• A missing space between two words that is handled with splitting: “helloworld” → “hello world”
• An extraneous space or punctuation that is handled with concatenation: “hel lo” → “hello”

Typos as language-dependent

To illustrate the principle, English is a suitable language, because it’s phonemic: it uses single characters to represent sounds to form a word. It makes spelling errors possible.

Algolia doesn’t support typo-tolerance for logogram-based languages (such as Chinese hanzi and Japanese kanji), as these languages use pictorial characters to represent partial or full words instead of single letters to represent sounds.

Natural language processing with rules

You can set up Rules that tell the engine to look for specific words or phrases in a query, and take a specific action or change its default behavior when it finds them.

For example, the engine can convert some query terms into filters. If a user types in a filter value—say, “red”—you can use this term as a filter instead of a search term. With the query “red dress”, then the engine could therefore only look at the “red” records (based on a filter attribute) for the word “dress”. Removing filter values from the query string and using them directly as filters is called dynamic filtering.

Dynamic filtering is only one way that Rules can understand and detect the intent of the user.