Slow pagination over tons of records in mongodb

Question

I have over 300k records in one collection in Mongo.

When I run this very simple query:

db.myCollection.find().limit(5);

It takes only few miliseconds.

But when I use skip in the query:

db.myCollection.find().skip(200000).limit(5)

It won't return anything... it runs for minutes and returns nothing.

How to make it better?

Answer 1

One approach to this problem, if you have large quantities of documents and you are displaying them in sorted order (I'm not sure how useful skip is if you're not) would be to use the key you're sorting on to select the next page of results.

So if you start with

db.myCollection.find().limit(100).sort({created_date:true});

and then extract the created date of the last document returned by the cursor into a variable max_created_date_from_last_result, you can get the next page with the far more efficient (presuming you have an index on created_date) query

db.myCollection.find({created_date : { $gt : max_created_date_from_last_result } }).limit(100).sort({created_date:true});