Tell your best friend your dating his sister
Maybe flirt a bit and pick up on her signals and even have a chat with her about your feelings.If she doesn't feel the same then leave it there and continue to be friends, nothing more.Tell your friend that you don't want your friendship to change, or for there to be any awkwardness between the three of you.
Tell your friend before anything establish a line of communication before he/she feels betraded. He wholeheartedly takes on the role of 'protective brother' and I respect that, but I also know, as a protective brother myself, that we don't always have the most rational thought processes.Make sure the sister actually returns your affection.If there is no chemistry between the two of you, and you have never even talked to her, perhaps you should get to know her first before you ask her out on a date, so as to keep yourself from an awkward situation if she were to turn you down. Reassure her that you have talked to her sibling first, and that you just want to get to know her better on your own.In that case, I can see problems arising indeed, but maybe the friendship isn't all that important.I've never really been in your shoes, but I can give you something like your brother's perspective: Are you an asshole? It takes some reflection to realize you are one, and sometimes your friends won't tell you straight up about it.