When should I use the Testvisible Annotation?

I understand that @TestVisible is used to allow Unit Tests access to private and protected methods.

What I was wondering is, when is it appropriate to use this annotation?

For example, I have a list generated in a wrapper class that allows selection and processing of Users as below:

public class uUser {
    public User user {
      get;
      set;
    }
    public Boolean selected {
      get;
      set;
    }
    public uUser(User u) {
      user = u;
      selected = false;
    }
  }
I then used the following Apex to select all or deselect all these Users:
private Boolean isAllUsersSelected {
  get {
    if (isAllUsersSelected == null) {
      isAllUsersSelected = false;
    }
    return isAllUsersSelected;
  }
  set;
}
public PageReference selectAllUsers() {
  if (isAllUsersSelected == false) {
    isAllUsersSelected = true;
    for (uUser user : userList) {
      user.selected = true;
    }
  }
  else {
    isAllUsersSelected = false;
    for (uUser user : userList) {
      user.selected = false;
    }
  }
  return null;
}

Because this depends a lot on whether the isAllUsersSelected variable is true or false, I wondered whether or not it is a good idea and appropriate to use a @TestVisible method, given its access modifier is private?