you might want to consider using a portal with several QPs instead. i.e. if you have overlapping groupmembership.
otherwise why not just use several QPs for the different groups?
i think there is no way to filter the users shown in the TO: selection by their room access.