Everything You Need to Know about Elasticsearch Pagination

Imagine browsing an online library full of articles, books, and resources without any guidance. The result? You’d quickly get frustrated.

Luckily, pagination is here to help. It organizes large datasets and makes navigation smoother, especially in complex search systems like Elasticsearch.

In this article, we’ll explore how Elasticsearch pagination works, the different methods available, and its key limitations.

When sending queries to Elasticsearch, default values return the ten most important documents, improving performance and user experience.

Types of pagination

There are multiple pagination approaches, each suitable for different needs. Let’s break them down.

Offset pagination

This is the simplest pagination form, where results are divided into numbered pages. Each page has a fixed number of results, and users can navigate using the “Next” and “Previous” buttons or a numbered page selection.

Imagine an online bookstore displaying 10 books per page. If a user is on page 3, the system skips (offsets) the first 20 books and fetches books 21–30.

Cursor-based pagination

Instead of relying on numbered pages, this pagination approach uses a reference point (cursor) to fetch the next result set. Cursor-based pagination is common in social media feeds and chat apps, where new data continuously enters the system.

Keyset pagination

The keyset approach is optimized for situations where results must be consistently ordered. Rather than fetching a batch of results based on an arbitrary position, it retrieves results using specific keys, such as timestamps or IDs.

For example, consider a stock market dashboard displaying trades in real time. The system retrieves trades using the last recorded timestamp as a reference, instead of paginating by arbitrary numbers.

And now, let’s discuss the pagination approaches Elasticsearch uses the most:

Scroll pagination

It is most suitable for large datasets. Once initiated, a scroll context is maintained to fetch the next batch of results without skipping. This pagination approach is often used for exporting data or processing extensive search results.

For example, imagine you need to export all customer records from a database. The system loads 1,000 customers at a time, preserving the scroll context to retrieve the next batch until all records are collected.

Scroll pagination and scroll API

The Scroll API allows you to retrieve large volumes of documents that match a query. Despite its name, it should not be used to implement infinite scrolling or to handle frequent end-user search requests.

This search operation can handle a scroll request, but Scroll API is completely stateful, meaning all index updates are ignored during the scroll request. To maintain this snapshot, Elasticsearch keeps the search context open for the duration of the scroll operation, which can be resource-intensive, especially for actively updated indexes.

Using the Scroll API requires a scroll_id, which is obtained by including a scroll parameter in the initial search request. This ID is then used in subsequent requests to retrieve the remaining results.

Why use Elasticsearch pagination

Elasticsearch is a distributed analytics search engine. It’s currently one of the most popular search engines for operational intelligence, business analytics, log analytics, security intelligence, and more. It indexes data as JSON documents and allows users to retrieve information using queries.

Elasticsearch stores original documents automatically and adds searchable references for them within the cluster’s index. Users can also employ the Elasticsearch API to retrieve documents efficiently using query strings or predefined filters.

If your webpage needs to display a large amount of data stored in Elasticsearch, efficiency is crucial. If the index contains too much information, the API Gateway may struggle to handle it. The best solution is to paginate search results, so you can control the volume of data retrieved in each request, ensuring consistent response sizes. This way, the client receives predictable data every time.

Before implementing pagination on the client side, you must first paginate the backend storage data. Luckily, most data storage platforms, including Elasticsearch, offer built-in functions for pagination, filtering, and sorting data.

Besides delivering results in smaller chunks, pagination also helps with performance bottlenecks and optimizes system efficiency. By setting appropriate limits on data retrieval and leveraging techniques like result skipping, developers can improve response times for end users. Well-optimized search bar design can further enhance user experience during search queries.

Your data structure and requirements are vital to determining what paginating methods to use.

Elasticsearch typical pagination process

When sending search queries to Elasticsearch, it uses the default values to send the first or most relevant documents with a maximum limit of 10 results.

This pagination method sets the upper default limit at 10,000. Pagination doesn’t allow returning over 10,000 documents. However, you can change this configuration by using the index.max_result_window command line, but be aware of potential issues down the line if you’re modifying indexes, such as high memory usage.

The search requests consist of two phases:

The query phase

• Data nodes execute the search query across the cluster.
• Each node calculates the relevance score for the matching documents.
• The matching document IDs and their scores are compiled into a list.
• This list is sent back to the coordinating node, which merges, sorts, and holds the results in memory.

A challenge you can encounter during this stage can occur when dealing with large offsets (e.g., deep pagination). The score-ID list grows significantly, consuming a lot of memory and slowing down the search process.

The fetch phase

• The coordinating node asks the data nodes to retrieve the actual documents.
• Each data node locates the documents based on their IDs and fetches the JSON source of each document.
• The nodes bundle these documents into a single “Multi Get-request”.
• The documents are sent back to the coordinating node, which assembles the final result and returns it to the client.

Although this phase is highly efficient, the query context and scoring information must be held in memory throughout the entire process until the response is delivered.

Large page sizes, or deep offsets can strain memory resources, leading to performance degradation.

Live index updates and pagination

Elasticsearch is good for supporting live index updates without causing performance issues. It’s easy to add documents, delete, or update while performing the same-index queries.

Even though this can be useful, it could lead to inconsistencies with search result pages when pagination is involved.

For example, if you insert a document relevant to the query and the user clicks on the second page, they will likely get the last document viewed displayed on the top. Search_after and pagination are stateless, so there’s no guarantee the order of the search hits will be the same when users change pages.

You need to use stateful pagination to ensure that the search experience is consistent after a specific number of repetitions. That brings us to the next search method called Point in Time API.

Using the Point in Time API with pagination or search_after

Using the Point in Time (PIT) API with search_after or traditional pagination helps maintain consistency. PIT creates a temporary, fixed view of the index, so users receive results from the same index version, even if data is updated during pagination. Any new documents or changes are ignored while the PIT is active, ensuring a stable and predictable search experience.

What is deep pagination and why avoid it

Deep pagination in Elasticsearch refers to requesting search results from extensive offsets, such as page 500 or beyond.

This practice leads to extensive memory usage, causes cluster latency, and disrupts the performance of your cluster overall. With this approach, you’re allowing access to all pages. That might sound good in theory, but even Google has a limited number of pages displayed.

Elasticsearch always recalculates hits while sorting and storing the whole Score-ID list within the memory. Instead, the focus should be on providing relevant scoring, filters, and UI to optimize search results on the first page.

On the other side of the spectrum, there is shallow pagination. It allows easy navigation through pages but becomes inefficient for deep pagination, as Elasticsearch must scan and skip records before returning the requested data.

Elasticsearch pagination over 10,000

As you already know, in Elasticsearch, the default pagination limit is 10,000 results. This is controlled by the setting index.max_result_window. If you try to request results beyond this limit using the standard from + size pagination, Elasticsearch will return an error.

So why does this limit exist?

Fetching results deep into large datasets is complicated because Elasticsearch needs to collect and sort all results up to the requested offset in memory. For example, asking for results from 50,000 to 50,010 requires sorting the first 50,010 results, degrading performance.

You should avoid increasing the limit unless necessary. Use search_after for deep pagination or scroll for bulk data extraction.

How to work around the 10,000 limit

If you want to increase this limit, you have multiple options:

• Increase max_result_window: You can increase this limit, but it can lead to high memory usage and performance issues.
• Use scroll API: It is designed for retrieving large datasets.
• Use search_after: Instead of offsetting, it returns results after a specific document, based on a unique, sortable field.
• Point-in-Time (PIT): Can be combined with search_after for consistent views of the data while paginating.

What are the limitations of Elasticsearch

While Elasticsearch is generally a reliable solution, there are some limitations you should keep in mind.

Scalability challenges

Elasticsearch is designed to scale horizontally, so managing large clusters with high data volumes and frequent updates can become complex. Potential issues include shard management (a shard is a fundamental data unit for storage and distribution), balancing, and data consistency.

Additionally, as the size of your indexes grows, Elasticsearch can experience slower search and indexing speeds.

Memory and resource requirements

When dealing with complex queries, Elasticsearch relies heavily on memory. This can lead to high resource consumption and require extensive and powerful hardware setups.

Consistency limitations

This system utilizes a distributed architecture, which leads to eventual consistency issues. In some situations, stale data might be returned because of the distributed nature of the system and replication delays.

Query complexity

When multistep queries are involved, these complexities can lead to performance bottlenecks, resulting in slower response times. While Elasticsearch effectively manages full-test search, it may struggle with complex aggregations. It’s crucial to simplify aggregation search and optimize query structures to enhance performance while still delivering valuable insights.

Operational maintenance

As the cluster grows, Elasticsearch becomes harder to maintain. Tasks like upgrading versions, handling node failures, and monitoring health can become unmanageable.

Not ideal for complex transactions

Elasticsearch does not support full ACID (Atomicity, Consistency, Isolation, Durability) transactions. If an application requires complex transactional operations (like in a traditional relational database), Elasticsearch might struggle.

Limited reporting

To get more advanced analytics and reporting, Elasticsearch needs to be integrated with external tools such as Kibana for visualization or utilize third-party reporting tools to address more complex requirements.

Conclusion

Effective pagination helps you deliver seamless search experiences, especially when working with large datasets in Elasticsearch. Depending on your app’s specific needs, picking the right pagination method can significantly improve performance and usability.

For instances where users need to navigate freely between pages, using traditional pagination is the best course of action.

However, if users require “Next” and “Previous” buttons and need easy access to multiple pages, the search_after method is your best option.

For scenarios where consistent search result pages are required, the Point in Time API is ideal. On the other hand, if you’re aiming to list all query hits or maintain consistent ordering across multiple result pages, the Scroll method is your go-to solution.

Remember, although Elasticsearch is a powerful solution, it comes with its own set of challenges, such as issues with scalability, operational inefficiencies, and limited analytics.

Understanding these pagination methods and picking the one that aligns best with your specific requirements helps you create a more efficient, scalable, and consistent search experience for your users.

Frequently asked questions