Unfortunately, labeling layers using AE's label dropdown is non-visual and a bit time-consuming. When executing Relabel Layer/s, however, a panel comes up with all label colors that you can see visually. Click on the desired color to apply it. When a group layer, or the group header, is selected and a label color is applied, the whole group gets relabeled. When the setting Label Layers Outside Groups to 'None' is deactivated, you can relabel non-group layers too.