I reckon this is is such a misunderstood debate, mostly driven by something akin to snobbery by the rudderless cadres (sorry to all my snooty mates...!) and a lack of understanding of what rudderless paddling is about, from the rudderphiles. Phils initial post reveals two common threads. The first is that there are some - read most - kayak designs that won't handle adverse conditions without a rudder. The second is that his mate couldnt steer with a rudder, which is the way things go in a sport where people invariably start paddling, before they seek instruction.
On the first thred, the only thing that can really affect a kayak's directional stability, regardless of your skill level, is the wind. A quartering sea can too, but as your skills rise you should begin to deal with them better. Almost all kayaks weathercock, that is, turn towards a beam wind. Rudders were developed to counteract this tendency, & interestingly weren't initially intended as a steering device, but rather, like a modern retractable skeg, as a device purely for tracking. If a boat is designed not to have a rudder, it will have some other feature - a harder chine or skeg - that will help it to offset the effects of wind on directional stability. If it's designed to have a rudder, it doesn't really matter how good you are, you will have a bastard of a time in a beam wind if you think you can pull the rudder up & still comfortably say in a straight line. There are only a handful of designs around; among them Mick's Maelstrom, it's close cousin the Nadgee & Nigel Dennis' Explorer, that really & truly can be paddled without skeg or rudder.
On the second topic, 90% of paddling in both my rudderless sea kayak, & rudderred racing kayak, is done using my under-deck muscles. I use my hips, knees, feet to alter the shape of my hull using edges, in order to control the directional stability of my kayak. It's a choice I make, because with a bit of dedication, it gives me a huge amount of control over my craft in all conditions. I also have a sea kayak which is designed to be paddled in a more - sorry for this awful word - holistic way. When I'm instructing boat control, I always stress this body control for directional stability, discouraging Phil's mate's double dipping in favour of better use of edges to effect turns. Rudder dependency leads paddlers to become reliant on their top half for power, & their toes for steering, leaving out all the big muscle groups in between that have such a pronounced effect on your kayaking if they can be engaged properly. You can see the difference on the water, there is so much more fun to be had when your whole body gets involved, increasing your skills definitely increases the enjoyment, in my humble opinion. It's something like the difference between driving a manual & an automoatic car. If your choice is to keep it simple, bushwalk on the sea, use your boat as a device to get from A to B, without being too bothered about this high falutin' skills stuff, than that's great, it's just not the way I choose to paddle. My advice to rudder boat paddlers if they do have ambitions to increase their skills, is to swap over the rudder cables. That way you're using a much more correct technique for directional control, push on your left foot to turn right & vice versa. Edging skills flow quite naturally from that base.
In my 20ft long, 42cm wide racing kayak, I set the rudder pedals back so I have to reach to get them, using the rudder as a tracking device so I can concentrate more on my forward stroke. Without the rudder I would expend more energy on corrections, and in that particular boat all I want to do is go fast. Having a rudder on that boat is great, as it is designed to have one. I still use my edges (and hence my below deck muscles) to take off on a wave & steer on steeper waves when the rudder is poking 2 feet into thin air, but the rudder is there to dull the unforseen direction changes.
Pulling the rudder up, or off your boat won't generally do you any favours in the fun stakes, becuase in all likelihood if it has a rudder, it's meant to be used. What it will teach you though, is how to control the boat without the rudder when it fails, & even forgetting about the hydraulics of a sideways surf, they do occassionally fail. That said, the Mirage rudder is one seriously clever, innovative & above all proven design; I've only ever seen them come unstuck when the owner has been tardy in the minimal maintenance required.