Predictors from TextGrids¶
In addition to acoustic envelope and onset predictors, you can include other event-based predictors in your TRF model, as long as you have annotations for those events. Below we show how to create event predictors from TextGrid files of your stimulus audio tracks.
Making predictors using TextGrids¶
The make_predictors_from_textgrids() function in the trf module generates unit impulse predictors for selected events in TextGrid files. For example, if you include 10 event types for a stimulus track with T time points, the resulting predictor for that track will be a sparse 2D matrix with shape 10 × T. Values are set to 1 at time points where an event occurs, and all other values are set to 0.
Below is an example using phoneme predictors:
import neuraspeech as ns
# Include these phonemes/events
selected_phonemes = [
'AA', 'AO', 'AE', 'AW', 'AY', 'OW', 'EY', 'EH', 'ER', 'IY', 'IH', 'UW',
'W', 'L', 'R', 'M', 'N', 'NG',
'S', 'SH', 'F', 'HH', 'Z', 'V', 'DH',
'P', 'T', 'K', 'B', 'D', 'G'
]
ns.trf.make_predictors_from_textgrids(
eeg_sr = 128,
textgrid_path = 'stim/alice_textgrids',
stim_path = 'stim/alice',
predictor_set = 'acou_pred_set',
predictor_name = 'phoneme',
target_tier = 'phones',
select_events = selected_phonemes,
remove_regex = '[012]$',
apply_regex_before_select = True,
alignment = 'start')
Generating predictors using TextGrids in: C:\Users\xyc6648\OneDrive - Northwestern University\Desktop\scripts\neuraspeech\neuraspeech\trf\stim\alice_textgrids Output path: C:\Users\xyc6648\OneDrive - Northwestern University\Desktop\scripts\neuraspeech\neuraspeech\trf\predictors\acou_pred_set\phoneme Number of events in the predictors: 31 Processing track1.wav (target tier: phones; tier type: Interval)... Processing track10.wav (target tier: phones; tier type: Interval)... Processing track11.wav (target tier: phones; tier type: Interval)... Processing track12.wav (target tier: phones; tier type: Interval)... Processing track13.wav (target tier: phones; tier type: Interval)... Processing track14.wav (target tier: phones; tier type: Interval)... Processing track15.wav (target tier: phones; tier type: Interval)... Processing track16.wav (target tier: phones; tier type: Interval)... Processing track17.wav (target tier: phones; tier type: Interval)... Processing track18.wav (target tier: phones; tier type: Interval)... Processing track19.wav (target tier: phones; tier type: Interval)... Processing track2.wav (target tier: phones; tier type: Interval)... Processing track20.wav (target tier: phones; tier type: Interval)... Processing track21.wav (target tier: phones; tier type: Interval)... Processing track22.wav (target tier: phones; tier type: Interval)... Processing track23.wav (target tier: phones; tier type: Interval)... Processing track24.wav (target tier: phones; tier type: Interval)... Processing track25.wav (target tier: phones; tier type: Interval)... Processing track26.wav (target tier: phones; tier type: Interval)... Processing track27.wav (target tier: phones; tier type: Interval)... Processing track28.wav (target tier: phones; tier type: Interval)... Processing track29.wav (target tier: phones; tier type: Interval)... Processing track3.wav (target tier: phones; tier type: Interval)... Processing track30.wav (target tier: phones; tier type: Interval)... Processing track31.wav (target tier: phones; tier type: Interval)... Processing track32.wav (target tier: phones; tier type: Interval)... Processing track33.wav (target tier: phones; tier type: Interval)... Processing track34.wav (target tier: phones; tier type: Interval)... Processing track35.wav (target tier: phones; tier type: Interval)... Processing track36.wav (target tier: phones; tier type: Interval)... Processing track37.wav (target tier: phones; tier type: Interval)... Processing track38.wav (target tier: phones; tier type: Interval)... Processing track39.wav (target tier: phones; tier type: Interval)... Processing track4.wav (target tier: phones; tier type: Interval)... Processing track40.wav (target tier: phones; tier type: Interval)... Processing track41.wav (target tier: phones; tier type: Interval)... Processing track42.wav (target tier: phones; tier type: Interval)... Processing track43.wav (target tier: phones; tier type: Interval)... Processing track44.wav (target tier: phones; tier type: Interval)... Processing track45.wav (target tier: phones; tier type: Interval)... Processing track46.wav (target tier: phones; tier type: Interval)... Processing track47.wav (target tier: phones; tier type: Interval)... Processing track48.wav (target tier: phones; tier type: Interval)... Processing track49.wav (target tier: phones; tier type: Interval)... Processing track5.wav (target tier: phones; tier type: Interval)... Processing track50.wav (target tier: phones; tier type: Interval)... Processing track51.wav (target tier: phones; tier type: Interval)... Processing track52.wav (target tier: phones; tier type: Interval)... Processing track53.wav (target tier: phones; tier type: Interval)... Processing track54.wav (target tier: phones; tier type: Interval)... Processing track55.wav (target tier: phones; tier type: Interval)... Processing track56.wav (target tier: phones; tier type: Interval)... Processing track57.wav (target tier: phones; tier type: Interval)... Processing track58.wav (target tier: phones; tier type: Interval)... Processing track59.wav (target tier: phones; tier type: Interval)... Processing track6.wav (target tier: phones; tier type: Interval)... Processing track60.wav (target tier: phones; tier type: Interval)... Processing track7.wav (target tier: phones; tier type: Interval)... Processing track8.wav (target tier: phones; tier type: Interval)... Processing track9.wav (target tier: phones; tier type: Interval)... All done.
Key arguments you may want to change or check:
textgrid_path: Path to the folder containing the TextGrid files for the stimuli. The TextGrid files should have the same filenames as their corresponding audio files except for the file extension. For example,track1.TextGridshould correspond totrack1.wav.predictor_name: Name of the predictor to generate. In this example, we usephoneme.target_tier: Name of the tier in the TextGrid files that contains the events you want to use.remove_regex: Optional argument for cleaning TextGrid annotations before generating the predictor. This allows you to remove specific strings from the annotations using a regular expression. In this example, the phoneme annotations include stress-level indicators, such as0,1, or2, at the end of vowels (e.g.,AH0orEH1). Because we want to ignore lexical stress here, we specify a regular expression pattern to remove these suffixes.apply_regex_before_select: Whether to applyremove_regexbefore selecting events. When this is set toTrue, the regular expression cleaning is applied first, so the selected event labels do not need to include the stress suffixes. For example, you can includeAHinselected_phonemesinstead of separately listingAH0,AH1, andAH2.alignment: Specifies how the impulse predictor should be aligned to each event. Using 'start' places the impulse at the beginning of the event or phoneme. Other options include 'mid' (midpoint), 'end', and 'fill'. The 'fill' option fills the entire event interval with 1s. This argument only applies to interval tiers. For point tiers, impulses are placed at the time points where the events occur.
The phoneme predictors will be named like track1~phoneme-31.pickle. The -31 suffix indicates the total number of phonemes included.
You can do a plot to visualize the phoneme features (see Plotting EEG and stimuli):
eeg_path = 'examples/sub-3000_04_ses-1_task-alice.pickle'
stim_path = '../../neuraspeech/trf/stim/alice'
predictor_set_path = '../../neuraspeech/trf/predictors/acou_pred_set'
stim_to_plot = 'track1.wav'
ns.trf.plot_schematic(
to_plot = ['eeg', 'waveform', 'gammatone-8', 'phoneme-31'],
colors = ['Spectral', None, 'inferno', 'Grays'],
height_ratios = [0.15, 0.15, 0.3, 1],
figsize = (10, 8),
eeg_path = eeg_path,
stim_to_plot = stim_to_plot,
time_window = (0, 3),
stim_path = stim_path,
predictor_set_path = predictor_set_path
)
Then in your TRF analysis, you can specficy a model like {'acou_phoneme': [gammatone-8, gammatone-on-8, phoneme-31]} to include phoneme events as predictors. Note that if you set duration_mode = 'concat' during EEG data preparation, you'll need to run the EEG data preparation function again to concatenate the newly genearated predictors.
Pro tips¶
1. Recoding event labels¶
The regex processing is intended for cleaning unwanted characters from event labels. If you want to recode or transform event labels, you can instead pass an optional recode dictionary. In this dictionary, each key is a new label, and the corresponding values are the old labels that should be mapped to that new label. Note that recoding is always applied at the end, after event selection and regular-expression processing. Therefore, the old labels listed in the recode dictionary should match the labels as they appear after those earlier processing steps.
The example below illustrates how to transform phoneme annotations into manner-of-articulation annotations.
selected_phonemes = [
'AA', 'AO', 'AE', 'AW', 'AY', 'OW', 'EY', 'EH', 'ER', 'IY', 'IH', 'UW',
'W', 'L', 'R', 'M', 'N', 'NG',
'S', 'SH', 'F', 'HH', 'Z', 'V', 'DH',
'P', 'T', 'K', 'B', 'D', 'G'
]
# Specify the recode dict as: {'new_label_1': ['old_label_1, old_label_2'], ...}
recode = {
'Vowel': ['AA', 'AO', 'AE', 'AW', 'AY', 'OW', 'EY', 'EH', 'ER', 'IY', 'IH', 'UW'],
'Nasal-approximant': ['W', 'L', 'R', 'M', 'N', 'NG'],
'Fricative': ['S', 'SH', 'F', 'HH', 'Z', 'V', 'DH'],
'Stop': ['P', 'T', 'K', 'B', 'D', 'G']
}
ns.trf.make_predictors_from_textgrids(
eeg_sr = 128,
textgrid_path = 'stim/alice_textgrids',
stim_path = 'stim/alice',
predictor_set = 'acou_pred_set',
predictor_name = 'manner', # New manner predictors
target_tier = 'phones',
select_events = selected_phonemes,
remove_regex = '[012]$',
apply_regex_before_select = True,
recode = recode, # Recode dictionary here
alignment = 'start')
Generating predictors using TextGrids in: C:\Users\xyc6648\OneDrive - Northwestern University\Desktop\scripts\neuraspeech\neuraspeech\trf\stim\alice_textgrids Output path: C:\Users\xyc6648\OneDrive - Northwestern University\Desktop\scripts\neuraspeech\neuraspeech\trf\predictors\acou_pred_set\manner Number of events in the predictors: 4 Processing track1.wav (target tier: phones; tier type: Interval)... Processing track10.wav (target tier: phones; tier type: Interval)... Processing track11.wav (target tier: phones; tier type: Interval)... Processing track12.wav (target tier: phones; tier type: Interval)... Processing track13.wav (target tier: phones; tier type: Interval)... Processing track14.wav (target tier: phones; tier type: Interval)... Processing track15.wav (target tier: phones; tier type: Interval)... Processing track16.wav (target tier: phones; tier type: Interval)... Processing track17.wav (target tier: phones; tier type: Interval)... Processing track18.wav (target tier: phones; tier type: Interval)... Processing track19.wav (target tier: phones; tier type: Interval)... Processing track2.wav (target tier: phones; tier type: Interval)... Processing track20.wav (target tier: phones; tier type: Interval)... Processing track21.wav (target tier: phones; tier type: Interval)... Processing track22.wav (target tier: phones; tier type: Interval)... Processing track23.wav (target tier: phones; tier type: Interval)... Processing track24.wav (target tier: phones; tier type: Interval)... Processing track25.wav (target tier: phones; tier type: Interval)... Processing track26.wav (target tier: phones; tier type: Interval)... Processing track27.wav (target tier: phones; tier type: Interval)... Processing track28.wav (target tier: phones; tier type: Interval)... Processing track29.wav (target tier: phones; tier type: Interval)... Processing track3.wav (target tier: phones; tier type: Interval)... Processing track30.wav (target tier: phones; tier type: Interval)... Processing track31.wav (target tier: phones; tier type: Interval)... Processing track32.wav (target tier: phones; tier type: Interval)... Processing track33.wav (target tier: phones; tier type: Interval)... Processing track34.wav (target tier: phones; tier type: Interval)... Processing track35.wav (target tier: phones; tier type: Interval)... Processing track36.wav (target tier: phones; tier type: Interval)... Processing track37.wav (target tier: phones; tier type: Interval)... Processing track38.wav (target tier: phones; tier type: Interval)... Processing track39.wav (target tier: phones; tier type: Interval)... Processing track4.wav (target tier: phones; tier type: Interval)... Processing track40.wav (target tier: phones; tier type: Interval)... Processing track41.wav (target tier: phones; tier type: Interval)... Processing track42.wav (target tier: phones; tier type: Interval)... Processing track43.wav (target tier: phones; tier type: Interval)... Processing track44.wav (target tier: phones; tier type: Interval)... Processing track45.wav (target tier: phones; tier type: Interval)... Processing track46.wav (target tier: phones; tier type: Interval)... Processing track47.wav (target tier: phones; tier type: Interval)... Processing track48.wav (target tier: phones; tier type: Interval)... Processing track49.wav (target tier: phones; tier type: Interval)... Processing track5.wav (target tier: phones; tier type: Interval)... Processing track50.wav (target tier: phones; tier type: Interval)... Processing track51.wav (target tier: phones; tier type: Interval)... Processing track52.wav (target tier: phones; tier type: Interval)... Processing track53.wav (target tier: phones; tier type: Interval)... Processing track54.wav (target tier: phones; tier type: Interval)... Processing track55.wav (target tier: phones; tier type: Interval)... Processing track56.wav (target tier: phones; tier type: Interval)... Processing track57.wav (target tier: phones; tier type: Interval)... Processing track58.wav (target tier: phones; tier type: Interval)... Processing track59.wav (target tier: phones; tier type: Interval)... Processing track6.wav (target tier: phones; tier type: Interval)... Processing track60.wav (target tier: phones; tier type: Interval)... Processing track7.wav (target tier: phones; tier type: Interval)... Processing track8.wav (target tier: phones; tier type: Interval)... Processing track9.wav (target tier: phones; tier type: Interval)... All done.
The generated predictions are called 'manner-4' as there are 4 manenrs of articulation. Again, you can do a plot to visualize them:
eeg_path = 'examples/sub-3000_04_ses-1_task-alice.pickle'
stim_path = '../../neuraspeech/trf/stim/alice'
predictor_set_path = '../../neuraspeech/trf/predictors/acou_pred_set'
stim_to_plot = 'track1.wav'
ns.trf.plot_schematic(
to_plot = ['eeg', 'waveform', 'gammatone-8', 'manner-4'],
colors = ['Spectral', None, 'inferno', 'Grays'],
height_ratios = [0.5, 0.5, 0.7, 1],
figsize = (10, 6),
eeg_path = eeg_path,
stim_to_plot = stim_to_plot,
time_window = (0, 3),
stim_path = stim_path,
predictor_set_path = predictor_set_path
)
