May I ask, why not just use a small rotary switch? If size is an issue, you can use a binary-coded rotary switch - it is small, uses only 3 pins for all positions, and you won't need to use more pins for LED indication.
Right now, an encoder can't be properly used for just any custom mode selector. It may be used if all modes are switched in the right order (0-1-2-3...), but an encoder won't do it if the mode order is different.
Also, you can use one single button to switch the mode (a button will switch them in the required order).