Indexing
Indexing
To provide fast and relevant search, Algolia’s engine restructures your data in a special way through a process called indexing. Algolia’s Magento extension runs through all your data - products, categories, and pages - and creates indexable objects out of it. These objects are then uploaded to Algolia’s servers, either automatically via the extension’s queue, or manually via the Magento console, or the command-line. Once pushed to Algolia’s servers, the objects go through an indexing process that transforms them into searchable data.
To learn about the indexing process, have a look at the documentation.
If you are having any issues with your data, indexes, or queue, please check the troubleshooting guide.
To provide the best search experiences for your users, the extension automatically keeps all your data up to date, offering two indexing mechanisms in Magento:
- Section reindex - An entire section of the catalog (products, categories, pages) is pushed to Algolia’s servers and reindexed.
- Single item reindex - A single resource (product, category, page) is pushed to the servers and reindexed. This happens when a resource is updated.
By default, the indexing operations run synchronously. This means the Magento administrator has to wait until the indexing process is finished before continuing. Since this is inconvenient, and can cause unexpected issues, Algolia created the indexing queue. This will process all index operations in the background, and has some fail-safes built-in.
Section reindex with an indexing queue
With the indexing queue enabled, products are reindexed using temporary indices. Instead of sending all data to the production index, a copy will be created and swapped with the production index only when ready. This approach has several advantages:
- High reindexing speed
- Avoids potential inaccuracies with deleted products
- Lower number of indexing operations needed
Any changes to the index will be visible when the swap of the temporary index has been completed.
Section reindex without an indexing queue
Without the indexing queue enabled, the reindex process has to handle the complete catalog synchronously. This means everything has to wait until the process is completed. Any update has to be pushed to Algolia’s servers to have up-to-date data.
Processing large indices synchronously may trigger PHP timeouts.
Without a queue, reindexing takes more time and resources. It’s also less reliable as some product updates may not be handled if they were applied during the process.
Enabling the indexing queue is highly recommended for doing any full reindexing, especially on large catalogs.
Automatic indexing
The Algolia Magento extension sends every update and deletion on products or categories to Algolia’s servers to keep all data up to date. You can change the indexers’ behavior to prevent these update calls, and only update the data through manual reindexing. For this to work, the indexers’ mode should be set to ‘Manual Update’.
Manual indexing
With Manual indexing enabled, the command-line has to be used to send updates of the data to Algolia’s server. For example, this is the command to completely reindex all products:
$
php path/to/magento/bin/magento indexer:reindex algolia_products
You can use the same command for all other indices created by the extension:
algolia_products- Reindexes all productsalgolia_categories- Reindexes all categoriesalgolia_pages- Reindexes all CMS pagesalgolia_suggestions- Reindexes all search query suggestionsalgolia_additional_sections- Reindexes all additional sectionsalgolia_queue_runner- Process jobs in the indexing queuealgolia_delete_products- Removes inactive products from Algolia indices
Indexing products
It’s essential for ecommerce businesses to have exact and up-to-date product data in the search.
Full reindex command
$
php path/to/magento/bin/magento indexer:reindex algolia_products
Indexable products
To prevent too many indexed products and save indexing operations on your indices, Algolia only indexes products that will actually show up in the online store. This results in a set of requirements a product has to meet before indexing the product.
Algolia only indexes products that are:
- Visible - either in the catalog, the search, or both
- Enabled
- Not deleted
- In stock - unless Magento is configured to show out-of-stock products, too.
If there’s ever a missing product in your index, make sure the product meets all these requirements.
See the troubleshooting guide for missing data for more information.
Searchable attributes
It’s possible to configure which attributes should be searched when users type their query. To configure the list of searchable attributes, navigate to the products tab through Stores > Configuration > Algolia Search.
In the products tab, it’s possible to configure per attribute if it’s searchable, retrievable or ordered. By default, all attributes are set to be searched as unordered. In general, this is better for the relevance of the search and you shouldn’t change it without a specific reason.
Read the dedicated documentation to learn more about the difference between ordered and unordered search.
Default searchable attributes
Some attributes are indexed, regardless of what’s specified in the configuration. These attributes aren’t all searchable, but can be used for filtering, sorting, customizing the ranking and building the results page.
These attributes are always indexed:
| name | The product’s name |
| url | The product’s URL |
| visibility_search | The product’s visibility in the search |
| visibility_catalog | The product’s visibility in the catalog |
| categories | The product’s categories, formatted as a tree path |
| categories_without_path | The product’s categories, without the tree path |
| thumbnail_url | The product’s thumbnail image |
| image_url | The product’s main image |
| in_stock | The product’s stock availability |
| price | The product’s price |
| type_id | The product’s type (simple, configurable, bundled, etc.) |
Facets
Facets are the attributes that will be used as filters on the results page. Common facets include price, color, categories, and brand. However, this doesn’t work for every store. The facets have to be tuned to the products being sold and the way the end user searches for these products.
There are a couple of things that need to be specified for each facet:
- The attribute
- The label - this will be displayed above the filter
- The type of facet
Facets can be set to be searchable. This will allow a user to search for a facet value by providing a search box in the filter. This is useful when a facet has a lot of different values, like with brands, for example.
Facets can also be attached to Rules. With this setting set to ‘Yes’, Algolia will dynamically filter on a facet when a user searches a certain value in the query. For example, assume you attached a Rule to the color attribute. Anytime the users’ query contains a color, like in “red shorts”, Algolia will internally filter all results with the color ‘red’ as one of their attributes. This leads to more relevant results.
When a Rule is attached to an attribute, it’s applied in both the autocomplete search and instant search results.
Rules have a limited quota. Make sure your plan supports the amount of rules desired.
By default, the extension provides facets on the price, categories and color attributes.
Any numeric attribute (like price) will be shown as a slider.
Attributes that are specified as facets are automatically indexed as retrievable but not searchable. There’s no need to specify them in the Searchable Attributes configuration.
Sorting strategies
Sorting is only available on the InstantSearch results page.
When searching for products, users may expect multiple ways to sort the result set. For example, they want to sort by relevance, popularity, price, or date.
The default sorting strategy when searching is sorting by relevance. Any other sorting strategy needs to be defined in the Sort Settings. For each strategy, an attribute, sort order (ascending or descending) and label should be defined.
By default, there are three sorting strategies:
- From lowest price to highest price
- From highest price to lowest price
- From newest to oldest
Each sorting strategy will create a new index, which will increase the amount of records. You can find more information in the FAQ.
Attributes that are configured in a sorting strategy are automatically indexed as retrievable, but not searchable. There’s no need to specify them in the Searchable Attributes configuration.
Removing a sorting strategy doesn’t automatically remove the index replica in Algolia. This has to be done manually through the dashboard.
Index settings
You can configure these settings in the Magento dashboard:
searchableAttributescustomRankingunretrievableAttributesattributesForFacetingmaxValuesPerFacetremoveWordsIfNoResults
You can manage additional index settings in the Algolia dashboard.
It’s also possible to change the settings programmatically, by hooking into the algolia_products_index_before_set_settings event provided by the extension.
See Custom Back End Events for a list of events provided by the extension.
Any changes done in the Algolia dashboard overrides these settings until a full reindex is performed from the Magento dashboard.
Indexing categories
To keep the amount of records and indexing operations as low as possible, Algolia only indexes categories that are actually active. You can change this behavior through the settings.
If set to ‘Yes’, all categories are shown in the autocomplete search and InstantSearch results page.
Full reindex command
1
php path/to/magento/bin/magento indexer:reindex algolia_categories
Searchable attributes
It’s possible to configure which attributes should be searched when users type their query. To configure the list of searchable attributes, navigate to the Category configuration through Stores > Configuration > Algolia Search > Categories.
It’s possible to configure per attribute if it’s searchable, retrievable or ordered. By default, all attributes are set to be searched as unordered. In general, this is better for the relevance of the search and we don’t recommend changing it without a specific reason.
Read the dedicated documentation to learn more about the difference between ordered and unordered search.
Default searchable category attributes
Some attributes are indexed, regardless of what’s specified in the configuration. These attributes aren’t all searchable, but can be used for filtering, sorting, customizing the ranking and building the results page.
These attributes are always indexed:
| name | The category’s name |
| url | The category’s URL |
| path | The category’s path (parent categories) |
| level | The category’s level in the category tree |
| include_in_menu | The category’s visibility in the menu |
| _tags | Filled automatically by the extension |
| popularity | The category’s popularity |
| product_count | The category’s amount of products |
Index settings
Through the Magento dashboard, the following setting for an index can be configured:
You can manage additional index settings in the Algolia dashboard.
It’s also possible to change the settings programmatically, by hooking into the algolia_categories_index_before_set_settings event provided by the extension.
See Custom Back End Events for a list of events provided by the extension.
Any changes done in the Algolia dashboard override these settings until a full reindex is performed from the Magento Dashboard.
Indexing pages
CMS pages will be automatically indexed by the extension, allowing the users to search for pages in the autocomplete menu. By default, all active pages are indexed.
The settings provide an option to exclude certain pages, like error pages, so they don’t show up in the search results.
You can turn off the indexing of pages altogether by navigating to the Additional Sections configuration, as shown below.
Full reindex command
1
php path/to/magento/bin/magento indexer:reindex algolia_pages
Searchable attributes
You can’t configure the searchable attributes for pages in the admin interface.
However, you can change them programmatically, by hooking into the algolia_after_create_page_object event provided by the extension.
See Custom Back End Events for a list of events provided by the extension.
Default searchable page attributes
These attributes are indexed by default, and aren’t all searchable (some are). You can use them for filtering, sorting, custom ranking and building the search results page.
These attributes are always indexed:
| name | The page’s name |
| url | The page’s URL |
| slug | The page’s slug |
| content | The page’s content |
Since records for the Algolia search engine have to be smaller than 10 kilobytes, any page with more than 10,000 characters won’t be indexed. In this case, only the page’s name would be searchable.
For more information about the engine’s record limit, see Index and Record Size and Usage Limitations
Index settings
The following settings are always set and can’t be changed through the admin interface:
searchableAttributes:unordered(slug),unordered(name)andunordered(content)attributesToSnippet:content:7
You can manage additional index settings in the Algolia dashboard.
It’s also possible to change the settings programmatically, by hooking into the algolia_pages_index_before_set_settings event provided by the extension.
See Custom Back End Events for a list of events provided by the extension.
Any changes done in the Algolia dashboard override these settings until a full reindex is performed from the Magento Dashboard.
Indexing suggestions
Every query that is being executed on the Magento installation is stored by Magento in the database.
Magento automatically stores the query, the number of results and the number of searches in the catalogsearch_query table, without any involvement from Algolia’s extension.
Only back-end searches are stored by Magento. Search-as-you-type searches and instant search queries aren’t stored.
Algolia’s Magento extension offers the possibility to index queries that are performed regularly on the Magento installation. In the settings, it’s possible to filter relevant queries (for example by minimum number of results, minimum popularity, etc.). The resulting queries can be pushed into the suggestions index, providing an autocomplete on the most relevant queries for the Magento installation.
To ensure the data in the suggestion index is good and relevant, the data in the catalogsearch_query table must be relevant as well.
This can be achieved by enabling back-end search with the extension, by turning on the Enable Search and Make SEO Request settings in the Magento Administration.
With these options enabled, the extension processes back-end searches.
Since the data in catalogsearch_query is updated as well, the queries in this table will become more relevant over time.
By default, suggestions aren’t indexed. When enabling the indexing of suggestions, you need to trigger a manual reindex. Another way to start the indexing of suggestions is by adding a recurring job to the cron table:
1
1 * * * * php path/to/magento/bin/magento indexer:reindex algolia_suggestions
You can also implement Algolia’s Query Suggestions instead of using Magento Suggestions. Read the dedicated documentation to learn more about it.
Full reindex command
1
php path/to/magento/bin/magento indexer:reindex algolia_suggestions
Searchable attributes
For suggestions, it’s impossible to configure the searchable attributes through the admin interface.
However, it’s possible to change them programmatically, by hooking into the algolia_after_create_suggestion_object event provided by the extension.
See Custom Back End Events for a list of events provided by the extension.
Default searchable query attributes
These attributes are indexed by default, and aren’t all searchable (some are). You can use these attributes for filtering, sorting, custom ranking, and building the search results page.
These attributes are always indexed:
| query | The query’s value |
| number_of_results | The query’s number of results |
| popularity | The query’s number of searches |
| updated_at | The query’s last update timestamp |
Index settings
The following settings are always set to configure the index, and can’t be changed through the admin interface:
searchableAttributes:unordered(query)customRanking:desc(popularity),desc(number_of_results)typoTolerance:falseattributesToRetrieve:queryremoveWordsIfNoResults:lastWords
You can manage additional settings in the Algolia dashboard.
It’s also possible to change the settings programmatically, by hooking into the algolia_suggestions_index_before_set_settings event provided by the extension.
See Custom Back End Events for a list of events provided by the extension.
Any changes done in the Algolia dashboard override these settings until a full reindex is performed from the Magento Dashboard.
Indexing additional sections
The autocomplete menu offers the possibility to display other sections from attributes, like colors and brands for example.
For this feature to work, you must enable the instant search page.
The attributes used for the additional sections have to be set as attributes for faceting.
1
php path/to/magento/bin/magento indexer:reindex algolia_additional_sections
Searchable attributes
You can manage additional index settings in the Algolia dashboard.
It’s also possible to change the settings programmatically, by hooking into the algolia_additional_sections_index_before_set_settings event provided by the extension.
See Custom Back End Events for a list of events provided by the extension.
Default searchable attributes
These attributes are indexed by default, and aren’t all searchable (some are). They can be used for filtering, sorting, customizing the ranking and building the results page.
The attributes that are always indexed:
| value | The attributes’s value, for example, Red, XL, Nike, etc. |
Index settings
The following settings are always set and can’t be changed through the admin interface:
searchableAttributes:unordered(value)
You can manage additional index settings in the Algolia dashboard.
It’s also possible to change the settings programmatically, by hooking into the algolia_additional_sections_index_before_set_settings event provided by the extension.
See Custom Back End Events for a list of events provided by the extension.
Any changes done in the Algolia dashboard overrides these settings until a full reindex is performed from the Magento Dashboard.
Removing inactive products
If you want to remove products from your Algolia index, you can use the algolia_delete_products indexer:
1
php path/to/magento/bin/magento indexer:reindex algolia_delete_products
This indexer removes all products from Algolia indices, that you don’t want to include in your search. This can be useful when you delete or deactivate products directly in Magento’s database and the extension could reindex and remove them regularly.
The reindexer doesn’t delete anything from Magento. Products are removed only from Algolia indices.
