How to Build a Search Engine for Your Website

When we speak about search, we automatically think about Google. Google has become a synonym for searching to the extent that we use the verb to google in daily communication. It helps us find the necessary information in the endless tangle of pages found on the web. Indexes them to be able to determine the results’ relevance when users enter their queries in the search.

However, we are not only searching on the web but also on individual sites, and Google is indexing these pages primarily based on full-text compliance. Such search is, however, not sufficient for performing a comfortable and helpful search on websites or e-shops in our case. As a result, the entire SaaS segment of the companies developing e-shop search arose. These companies can help customers find what they are looking for in a variety of complex ways. Artificial intelligence can apply information about customers’ interaction with web content to rank the search results correctly.

Note: The data used in this article are from the online publication Search benchmarks & insights we created in Luigi’s box based on our clients’ e-shops data. The current PDF version of the publication is available in Slovak or English at ebook.luigisbox.com.

Why is search so important

Search accounts for an average of 20% of the e-shop traffic. The average per segment ranges from 8% to 43% (Fig. 1). It is the least in the fashion segment (7.8%). Because when purchasing clothes, the customer is willing to view a large variety of products. They want to give serious consideration to what could suit them the best. Product names do not represent products sufficiently, hence it is challenging to find clothes using text search. If the customer looks for a blue T-shirt, they will also need a visual display to decide on the final product. There can easily be hundreds of blue t-shirts in the e-shop. Customers want to see as broad range as possible and gradually narrow it down using filters. Therefore, they are more likely to search through navigation.

On the contrary, most customers use search in the book and games segment (43.3%). It is because the products have distinct names, namely in a unique combination of the author and the title. The image is not so important. It only serves as a visual confirmation that the customer has found the right product. Therefore, it is natural for them to only search for the product title. It is the fastest way to get the right product here because, when purchasing in the book and game segment, the customer very often knows exactly which product they want, unlike when shopping in the fashion segment.

The importance of search is also increased by the fact that the conversion rate for customers who use search is higher than the conversion rate for those who use navigation (Figure 2). The customer, who uses the search, follows the purchasing process. They know what they want and are determined to buy it. A graph comparing the two segments illustrates this.

In every segment, the likelihood of conversion grows when using search. In the consumer electronics, fashion and sports segments, there are two reasons for this. As mentioned above, customers want to see a broad range of products, especially in segments where the share of visits with the use of search is lower.

The second reason may be that customers use the search to compare prices, especially in the consumer electronics segment, so they visit more e-shops and copy the product code or the full name into the search to compare where they can buy the desired product for the cheapest.

Search may form a significant part of the e-shop’s total turnover. It differs across segments, but our data shows that these may indeed be considerable amounts, depending on the size of the e-shop (Figure 3). In the case of the book and game segment, it can be up to 45 %. It is, therefore, essential for the search to function properly. Search is not only a means of improving the UX but also a significant trading channel whose sales power rapidly increases in e-shops which dedicate time and energy to its optimization. In the next chapter, we will cover what functions should a good search have.

What should a good search do for you?

Visibility

The ‘Google generation’ is growing up fast, subconsciously looking for a search immediately upon visiting any website to save time when finding the necessary information. E-shops, however, tend to hide the search box.

To interact with the search bar, customers often need to first search for the magnifier icon in the right corner of the page, and they can start typing their query only after clicking on it. It causes errors, particularly on mobile devices where users often need to use the navigation display button.

We often encounter the view that customers only use mobiles to explore products, but they complete purchases on desktops. Our data confirm it to a large extent, but the pandemic brought significant changes in customer behavior. One of them is that the number of purchases made by phone is getting closer to the number of purchases made through a desktop (fig. 4). Customers are increasingly less and less afraid of mobile payments. We can see that also on this chart comparing conversion rates from a mobile search in 2019 and 2020. We can see growth in almost all segments.

Autocomplete

Autocomplete function (also known as predictive text or look-as-you-type) is, in practice, already a common thing in e-shops. This tool offers search results when the customer enters the first letters of the phrase, even before they press enter. It can dramatically reduce the time needed to find desired products. Autocomplete performs several primary functions:

1. Confirms that the query is correct, and e-shop offers the products, which the customer is looking for.
2. Relieves the user from the need to write the entire phrase, which is even more useful on mobile devices.
3. Guides the user directly into the category he is looking for, even in the third level of navigation. Therefore, the user does not need to search categories in the navigation.
4. Completes the context of user’s query. Users often know only part of the product name. Autocomplete can show them the categories and brands associated with the phrase and thus effectively communicate them with the broader offer related to their query.
5. Shows the user the required product directly, if the query is specific enough (product code, exact name).

However, not all autocompletes work the same way. The basis of a quality one lies in its heart. Autocomplete cannot preprocess an expression (e.g., proper analysis of individual words in the query, extraction of product names, categories, brands, etc.) in the same way as the search itself. We often encounter autocomplete functions that respond slowly and not from the very first letter, that the user has typed in. That way its importance is being lost. Properly functioning autocomplete should respond within a maximum of 0.3 seconds, from the very first character and after each key press.

Autocomplete works in a visually restricted area where it must display several “top” products, product categories, brands, articles, frequently searched words, etc. The content of the autocomplete must be adapted to the individual e-commerce segments. When we search for books, we prefer to look for authors or titles. In the fashion segment, we opt for product categories, and for traveling, it could be destinations, etc.

You can see the importance of the autocomplete feature on a graph (fig. 5). It compares the conversion rate of visits when customers used autocomplete and those when the autocomplete suggested results, but the customer didn’t click on any of them. In all segments, the conversion rate is more than double for visits during which customers used autocomplete.

Typos

Visit your e-shop, try to use your site search, and intentionally make a typo. For instance, they look for ‘Ipohne’ or ‘refrigderator’. If you have not received any results, it means that you regularly lose customers due to search engine flaws. 30% of visitors leave the e-shop after searching with no result. What do you think: where do they go after leaving your e-shop? Most probably, directly to your competition, where they can find and buy the product.

A good search can cope with typos and provide relevant results even if the customer does not use a perfect query (fig. 6). However, there are cases when typo tolerance can do more harm than good. For instance, if the customer looks for iPhone 13, they should not see results for iPhone 11. They are interested in a particular newer phone type, and older models are irrelevant for them.

It is, therefore, crucial to know how the search works with individual queries.

If the input is not accurate enough, it is better to correct the typo before finishing the search and to get relevant results than to display more less relevant option for the query.

Slang and synonyms

Problems often arise when the customer doesn’t know the exact name of the product or when they know a different name than the one listed in your catalog. That includes various regional versions of the word, slang terms or a phonetic transcription from foreign languages Sweater/jumper, bike/bicycle, etc. – these are just a few examples where a poorly optimized customer search will not show any results if the query differs from the name in your product catalog.

An effective search is able to work with synonyms. If you are aware of such words, you can assign synonyms to them and enable the search to display desired products for queries different from the names in your product catalog. A good analyst will tell you which phrases are problematic in your e-shop – customers use them but do not receive any results or do not click on them since they appear irrelevant. This can give you an idea about which products need synonyms. Moreover, the system should be able to produce new tailor-made recommendations for any e-shop based on customer behavior.

Context recognition

A good search can also work with longer phrases containing several pieces of information, saving time for the customer. An example is a phrase containing the product color (e.g., yellow t-shirt, green bag). Instead of displaying a category and letting the user click through filters, the search engine will apply it automatically (fig. 7).

There are two approaches to the technical solution of this function. One is to extract the product category from the query and apply it as a filter. However, for this approach, the data in the product catalog needs to be of excellent quality. E-shops often offer thousands of products and lack the capacity to deal with a proper data structure. Therefore, it is more appropriate for this function if the system can identify a color from a description, name or other product attributes.

Pagination

Pagination, the distribution of results on several pages, is another significant feature of search. Data shows that up to 88.7% of customers only look at the first page of the results (fig. 8). Not even 10% of users look at the second page. Therefore, it is recommended not to use the classic pagination (distribute search results on numbered pages). It is more effective to place the “load” button at the end of the displayed results. User experience is at the core of effective pagination strategies, which aim to make the search process more intuitive and satisfying. Website pagination techniques aid in improving user navigation and effectively managing large sets of data.

After clicking on the button, new products load. If this topic caught your interest, read our blog about pagination.

Product Ranking

An effective search also takes into account other variables than the full-text score when ranking search results. Artificial intelligence or e-shop administrator can change it. When using artificial intelligence, the search may rank the results according to seasonality, margin, seller, or other metrics considering the business benefits for the e-shop, thereby creating an optimal product ranking that will promote the likelihood of conversion.

When ranking products manually, a good search should allow for so-called product prioritization. This means that the administrator can manually set which products/brands/categories should appear on the first positions of search results. The key can also be the saleability/clickability of products or an effort to highlight new products. Or on the contrary, to highlight products, that need to boost sales. Products performance needs a constant follow-up so that if highlighting a product didn’t help, you can stop it anytime. It would thus unnecessarily occupy the place of products with better sales potential in the search results.

Filters

Filters also help customers choose products. They can be used to narrow down the selection when there are too many search results. The more different kinds of products there are in an e-shop, the more filters are needed. However, when there are too many filters, they become intricate and worsen the e-shop UX. Therefore, it is best to use dynamic and, ideally, also personalized filters. Filtering by size will not be relevant for fruits, as well as the consumption date will not make sense for books.

No-result search

Unoptimized search achieves an average no-results search rate of approximately 13% across various segments. While with optimized searches it is 1–5%.

Except for the typo tolerance, various NLP (natural language processing) modules prevent searches from showing no results. Thanks to these tools, when a user enters a too-specific query, the search can distinguish which part of the phrase is more important and which can be omitted, thus providing the customer with similar products instead of showing no results at all.

For example, if the customer looks for a pink jumper, but the e-shop only has jumpers in other colors, the search can distinguish which part of the phrase is less important (in this case, it is pink) and shows jumpers in different colors. Of course, it is possible to optimize the result up to zero. However, it does not really make sense. It would not bring the desired effect if the e-shop displayed books when the customer looked for pots. Our data shows that providing irrelevant results has the same effect as providing no results. It is crucial to view the relevant results.

We can see that from the fact that the search exit rate correlates with the no-results exit rate. The problem only “transforms” into the increase in searches without clicking on any of the results.

Why it makes no sense to develop a site search internally

1. Cost

Developing a search engine is very costly and complex. It involves not just creating a search engine but also determining how to set it up to find exactly what the company needs. This complexity goes beyond technical development. It encompasses setting parameters, understanding important data, and more.

2. Expertise

E-commerce businesses should rely on the expertise of partners who specialize in search technology. This allows them to focus on their core business while a partner with expertise handles the specialized component of search. The philosophy is not to “reinvent the wheel” but rather to seek advice from those who are knowledgeable in the field. By outsourcing, they can contribute their ideas and integrate these with the partner’s expertise, creating a win-win situation where the e-shop gets the features they want, and the vendor gets product insights that help them advance the product.

3. Maintenance

The effort required for the maintenance of an in-house search solution is significant. You’d have to dedicate 30-40% of the time to maintaining an in-house solution, and it does not make sense. Maintenance and continuous updates are crucial, and specialized providers like Luigi’s Box efficiently handle this. The cost and effort of maintaining and updating the search solution internally would not be justifiable, especially when considering the company operates in multiple languages.

4. Innovation

Relying on external experts allows companies to benefit from innovations in search technology, such as the integration of generative AI and large language models. In-house development might not keep pace with these rapid technological advancements.

Summary

Search is the fastest way to bring the customer the desired products. It is not only a pleasant UX addition to the e-shop, but it is also a significant sales channel. Customers who use a search usually know what they want, and they are determined to buy. Thanks to artificial intelligence, modern searches can also use various techniques to help customers with purchase decisions. They complete the query, optimize ranking of results, correct typos, understand multiple product names, personalize search results, etc.

Search makes shopping easier for customers and therefore increases the chance of purchase. It can thus significantly contribute to the e-shop’s turnover. Non-text forms of search are also becoming increasingly popular. Such as voice search or image search. In addition to them, artificial intelligence and machine learning will continue to push the boundaries of what search can bring to an e-shop.

Frequently asked questions