Idiomatic pagination using latest features

With this new release, what is the idiomatic way to do sorted query with pagination?

It seems like index-pull is the answer but I wonder if there’s a way to do pagination with qseq?

That would be more flexible (in terms of where conditions) than using an index.

3 Likes

It depends on your specific needs, e.g.

  • are you paginating for a user browsing results, or to chunk up some batch job, or other?
  • do you need sorting before pagination?
  • can you split work on known-in-advance split values, or do you want the “next n”?
  • do you need arbitrary query power, or just to walk an attribute?

Then there are number of capabilities you might use:

index-pull is fully incremental and sorts by an index.
q/qseq does not (yet) have sorting, but has arbitrary query power.
q/qseq with range predicates uses indexes to avoid considering values outside the requested range.

My use case is for a UI sorted table. It’s sent via graphql but the transport doesn’t really matter here. It need to be sorted, then paged, then hydrated (pull). As with most UI tables, it will need runtime provided where conditions. I believe this is a pretty common UI requirement. It would be the same if the UI was infinite scroll vs classic paging.

Today I do this with:

  1. a sorted datalog query (sort field and id only)
  2. drop n, take n
  3. run N pulls for the page of records (could alternatively be a second query)

The core constraint is the sorted first step. I’m hoping there’s now a better way to do this.

My current solution could become a problem if the dataset becomes large because it will effectively be a full scan to find/sort the results.

Thanks for the feedback

1 Like

If the first query is sort field and id only, is index-pull a suitable replacement?

that was my thought but I don’t think this supports the runtime filtering i.e. add extra where clauses at runtime based on user input.

my current use-case is a table that allows filtering based on first (few) letters of a column. in future I’m sure I will need different filters, hence the more general question.

Here’s what it looks like in the UI…

the letter links at the top allow the user to filter the last name field. the whole table is paginated with next/prev links.

Hi Steve,

index-pull is a great fit if the limiting clause is a prefix of a single attribute – just start pulling at that prefix.

Thanks Stu. I’ll give it a try for that requirement.

Is it correct to say that there is no solution for sorted pagination for queries with arbitrary where conditions? I also need more complex filters for my lists e.g. entities matching N different clauses, all configured by the user at runtime, not using string prefixes.

For this I believe we still need to use the old (non-lazy) design but I’ll be delighted to be corrected.

And is this likely to be possible in future?