How windy is too windy?

How windy does it need to be to make you decide not to go kayaking?

Edit: I'm just asking for a paddle down my local river. What wind speed is just too annoying? Not so much a safety issue.