I don't want to spoil anything but it is easier to get yourself in a pickle by losing advisors than you realize. Once you get to Chapter 5 or 6 you will see just how easy it is to find yourself with no advisor roles filled and no way to fix it.
That is true even if you NEVER take them out to adventure.
That is true even if you NEVER take them out to adventure.