Submitting Slurm jobs

Using a computer cluster is optional but highly recommended. Below is an example using Northwestern Quest (other institutions may have similar computational resources).

It's assumed that you: 1. Have a Quest allocation; 2. Have installed the neuraspeech environment on your allocation; 3. Have some basic working knowledge about using Quest.

Check the configuration file

The configuration file is located in conf/conf.yaml inside the trf folder. To use the same settings as in the Running TRF models section, specify the corresponding fields in the configuration file as shown below:

EEG:
  metadata:
    path: misc/example_metadata.csv
    filename_col: 'filename'

TRF_estimation:
  eeg_path: eeg/prepared
  predictor_set_path: predictors/acou_pred_set
  results_path: results
  use_parallel_processing: True
  n_workers: 2 # NULL for using all available workers
  include:
    sub: ['3000_04', '3042_04'] 
  exclude:
  models:
    full: [gammatone-8, gammatone-on-8]
    onsets_only: [gammatone-on-8]
    envelope_only: [gammatone-8]
  skip_estimated_models: True 
  boosting_cfgs:
    tstart: -0.1
    tstop: 0.5
    error: l1 # i.e., Lasso penalty
    delta: 0.005
    basis: 0.05
    partitions: 5
    test: 1
    selective_stopping: 1

random_seed: 2026

Note that you don't need change any fields not listed above, as they won't be used by the ns.trf.run_boosting() function. You can leave these other fields unspecified or at their default values.

Also, note that you may want to set n_workers to NULL and remove the whole sub: ['3000_04', '3042_04'] line for your own data and analyses. So please check if every field is specified correctly.

Job script

See misc/example_job_script.sh in the trf folder for an example Slurm job script. The comments in the script indicate which settings need to be changed for your own project.

In most cases, using either the 'short' partition, which has a maximum runtime of 4 hours, or the 'normal' partition, which has a maximum runtime of 48 hours, should be sufficient. Setting ntasks-per-node to somewhere between 20 and 40 is usually a reasonable starting point; this controls how many workers are used for parallel processing.

You may need to adjust these settings depending on the size of your dataset, the number of predictors, etc.

Once the script looks OK, log in to Quest and submit it in the terminal:

sbatch example_job_script.sh

You can check the status of the submitted jobs by running:

sacct -X

The job status may initially remain PENDING for some time, which means the job is waiting in the queue. Once resources become available, the status should change to RUNNING. If the code finishes successfully, the final status will be COMPLETED.

However, a job may fail because of an error in the code or because it exceeds the allotted runtime. In these cases, check the output log to identify the issue. It may take some trial and error to adjust the script, requested resources, or runtime settings before the job runs successfully.