node.js - TransientTransactionError/NoSuchTransaction issue on long/large write operations using Mongoose transactions

Question

I have been running a NodeJS codebase for several months, where once a day, it grabs a relatively large amount of data from external APIs, then writes the data to a hosted MongoDB Atlas instance. This write operation has worked fine for months without issue, and in the last couple of days, I have tried to migrate the code to do the write operations using sessions/transactions within Mongoose. However, I cannot get the operation to reliably commit, as it often fails due to a TransientTransactionError, with reason NoSuchTransaction.

It fails consistently on one specific part, where I attempt to write roughly 80mb/36,000 documents to the DB. The operation takes 3-5 minutes normally to complete fully, and I've previously never had any issues with the write when not using transactions. I've also tried splitting the array up into a For Loop, where it does many small write operations of its constituent parts, but this also failed for the same reasons. Is there a timeout parameter, or maximum time the connection can be open that I'm not seeing? I feel I am following the code example in the docs pretty closely, but cannot figure out why its having trouble with this. I've also tried the session.withTransaction() pattern outlined in the mongo docs, and that also failed for the same reason.

The write operation works fine without transactions, and there are also other write operations that complete successfully using insertMany() within transactions, but i've left those out of the code snippets below because they don't cause any failures, and just isolated the piece that is throwing the exceptions.

var mongoose = require('mongoose');
var dailyDataCollection = mongoose.model('DailyDataCollection')

( async () => {
    var dbUrl = config.MONGODB_PREFIX+'://'+config.MONGODB_USERNAME + ':' + config.MONGODB_PASSWORD + '@' + config.MONGODB_HOSTNAME+'/'+config.MONGODB_DATABASE + '?retryWrites=true&w=majority';
    var database = await mongoose.connect(dbUrl, { useNewUrlParser: true, user: config.MONGODB_USER, pass: config.MONGODB_PASSWORD, useCreateIndex: true, useUnifiedTopology: true})
    var session = await database.startSession();
    try{
        await performDailyDataPull(session)
    }
    finally{
        await session.endSession();
        await database.disconnect()
    }
} 
)();

const performDailyDataPull = async(session) => {
    var veryLargeArray = await externalAPI.getTodaysData()

    await session.startTransaction()
    
    await writeVeryLargeArrayToDB(veryLargeArray, session)
    
    await session.commitTransaction()
}

const writeVeryLargeArrayToDB = async(veryLargeArray, session) => {
    try
    {
        await dailyDataCollection.insertMany(veryLargeArray, { session: session })
    }
    catch(e)
    {
        console.log(e)
    }   
}

Trace snippet

[Symbol(errorLabels)]: Set { 'TransientTransactionError' } }
codeName: NoSuchTransaction
message:'Transaction 2 has been aborted.'

question from:https://stackoverflow.com/questions/66051131/transienttransactionerror-nosuchtransaction-issue-on-long-large-write-operations

Answer 1

You may need to extend your transaction life time:

 db.adminCommand( { setParameter: 1, transactionLifetimeLimitSeconds: 600 } )