Dear users,

this is to inform you that we are performing a rolling reboot of all compute nodes. Over the past days, a bug in Slurm’s job profiling plugin caused some finished jobs to leave processes running on the nodes. These processes kept using GPU memory, which could make other jobs fail with out-of-memory errors.

No running job will be killed. Each node will stop accepting new jobs, wait for its current jobs to finish, reboot and come back online automatically. Until then, pending jobs may wait longer than usual. If one of your jobs recently failed with an out-of-memory error, please resubmit it.

We apologize for the inconvenience.

Best regards,
Lorenzo.