I have a hypothetical situation that I’d like to solve, but I can’t find the ideal answer. Suppose you have a huge data set that could be returned from a query, how do you paginate it, so that the impact on memory is minimal? The datoms API, iterating over the datoms and filtering one by one? The index-range API, but I would have to do the same thing as in the datoms API, iterate over the items and filter one by one? Perform an initial query that would return only ids, and the paginate those ids so that they could be used in another query to retrieve the entire data set?
In SQL you usually can define a pagination in the query itself:
SELECT col1, col2, ... FROM ... WHERE ... ORDER BY -- this is a MUST there must be ORDER BY statement -- the paging comes here OFFSET 10 ROWS -- skip 10 rows FETCH NEXT 10 ROWS ONLY; -- take 10 rows