I have several inflatables and folding boats, and have been a floater for about 30 years, both in guiding and in my own pursuits. Like everything, they all have their advantages and disadvantages.
I have two Ally Pac Canoes and one Russian army surplus folding Kayak, a 21 footer with three cockpits. They all work well in smooth water.
In rough water, an inflatable is preferable. I personally wouldn't even think about one of the inexpensive inflatables for reasons already mentioned (except to try the kayak experience without soaking too much money into it at first). The inexpensive ones are susceptible to popping, and who wants to mess with that? A good quality inflatable is almost unsinkable in regular fishing situations. They bounce off of rocks and pointy stobs with no problems. Hooks won't penetrate unless you do something really silly. They generally have at least three compartments and you'd have to pop them all to sink the dang thing. We carry knives with rounded tips (but with sharp edges) when using inflatables. Inflatables go together faster and more easily than the folding boats. If you should damage an inflatable, the repairs are not difficult, just a little trickier than patching a hip boot. If you damage a folding boat, (like break a rib) repairs are more of a problem.
One draw back to kayaks is that you are seated very low in them and some people find it difficult to sit with legs straight out in front of them for long periods of time. Fly casting can be a little more difficult in a kayak than in other boats. On the other hand, you don't have to bend over very far to net or release a fish.