Thanks! Well it really does depend on skill and experience in the kayak - as well as how flat, wide and stable the hull is. I would go out and try it without all your tackle first and see how your balance is as the chop height increases. You may also want to consider a simple sea anchor from the bow to keep the kayak pointed into the waves and wind. It will be much more stable bow to the wind and waves. (as opposed to beam on to the waves) And, Beaufort 4 might be ok if the fetch is short, and the wind hasn't been blowing for long (1 to 2 foot chop)- but if fetch is long - the wind has been blowing for quite a while - waves could build to 3 to 5 feet in Beaufort 4! That would be a challenge in any small boat!
There are a few variables for sure -- the kayak hull design and stability being one. How wide and stable is the hull shape? I would definitely go out without your fishing tackle in a rising wind to see how it feels as the chop builds. You may wish to rig up a "sea anchor" on the the bow to hold the nose of the kayak into the wind and waves. Even a plastic pail on ten to fifteen feet of line could work for that. Nose into the waves would be a lot more stable than beam on, especially given your hands will be busy fishing. It also depends on the fetch and how long the wind has been blowing. Beaufort 4 might be manageable if the fetch is short and the wind has not been blowing too long, say in wind developed chop of one to two feet in height. But if the fetch is long, and the wind has been blowing at Beaufort 4 for a while -- the chop can build to three to five feet. (this is the theoretical maximum in the "unconstrained" case.) This would be a real challenge for most small kayaks and canoes. And then the subjective factors, like experience, skill and balance certainly come into play -- being able to anticipate the motion and counter balance accordingly. Practice, practice practice! The white water kayakers and canoeists have tremendous balance skills. Go and an try it on a windy day without your kit, and see how you fare! Experiment with the sea anchor (you will likely need a retrieval line on it as well or you may not be able to reach it from the seat in the kayak. If you are on an ocean, don't forget that the swell height will be a factor as well: Wind waves add to swell heights for a combined wind wave and swell height. Good luck!
I would say that it depends on your swimming ability. Beginners under supervision should stay in shallow water in in Beaufort 0 or 1 so the superviser can still have good visibility of the swimmer. Strong and expert swimmers can handle more, but it totally depends on the situation and other factors. Are there also currents and tide rips? What is the fetch, how long has the wind been blowing? Is there also swell? What is the developed wave height from wind and swell? What temperature is the water? Even expert swimmers can get into trouble in light wind if some of these other factors are present and turn extreme. I suggest using extra caution, and get local knowledge from people that swim at the location to ask about conditions and hazards. Be honest with yourself about your swimming ability and capacity to remain calm and rescue yourself and your friends if things turn for the worse. Is it a popular spot? Is it isolated? Is help available nearbye? If you have any doubt, don't go out....