Question: In my Cell Ranger log, I keep seeing messages like this:
[jobmngr] Need 6144 MB of memory to start the next job (2048 available). Waiting for jobs to complete.
At the time this message was printed, I had > 100 GB of RAM available. Why is Martian waiting for jobs to complete when I have plenty of memory available?
Answer: This message does not reflect memory available relative to how much memory is installed on the system, but rather memory available relative to how much memory you've indicated that Cell Ranger will have available to use. This amount of memory is determined by the --localmem option, which defaults to 90% of the memory installed on the system.
You may see messages like this when the individual job chunks spawned by Martian (the pipeline framework used by Cell Ranger and other 10X pipelines) have a memory footprint that is not an integer multiple of the available memory. For example, if --localmem is set to 100, and Martian wants to spawn a large number of jobs that each require 6 GB, once the 16th job is spawned, you will theoretically only have 4 GB available (100 - (16*6) = 4). When Martian wants to spawn the 17th job, it will hold off on doing so and emit a message indicating that the available memory is less than what it needs. This will be true no matter how much physical memory is installed on the system, as again it is relative to the limit set by --localmem.
With this in mind, you should treat these messages as being for diagnostic purposes only.