Question: Cell Ranger was crashing on my Linux server mid-way through the pipeline, and after emailing email@example.com, you advised me to increase my ulimits and restrict core and memory usage with the
--localmem options. How does this keep my pipeline from crashing?
Answer: By default, Cell Ranger runs in local mode, and uses all of the cores and 90% of the memory in the system. Each core can spawn dozens of processes, and the more memory/cores Cell Ranger thinks it has, the more files/processes will be open/running at the same time. Meanwhile, your server has user limits on the max number of open processes/files, which when exceeded, can lead to a crash which has multiple phenotypes depending on what stage the pipeline is at when it happens. For example, if you were using 24 cores, each of which could spawn up to 64 processes, you would end up with 1536 processes, exceeding your default ulimit of 1024. Therefore, reducing the cores and memory being used using
--localcores options, increasing the user limits, or both, commonly resolves these crashes.
A reasonable starting point for restricting cores and memory is 8 cores and 64 GB of RAM (our minimum requirement), and scale up from there once you have achieved success.
Here are our recommendations for user limits:
|user open files (ulimit -n)||16,000|
|system max files (/proc/sys/fs/file-max)||10,000 per GB RAM available to Cell Ranger|
|user processes (ulimit -u)||64 per core available to Cell Ranger|
For more information on our system requirements for Cell Ranger, please see our support site.