Like all organizational models, waterfall is mainly a theory of collaboration. By putting the most serious planning at the beginning, with subsequent work derived from the plan, the waterfall method amounts to a pledge by all parties not to learn anything while doing the actual work. Instead, waterfall insists that the participants will understand best how things should work before accumulating any real-world experience, and that planners will always know more than workers.Clay Shirky – Healthcare.gov and the Gulf Between Planning and Reality
This quote, especially the part I have highlighted in bold, is the best description I’ve come across of why waterfall methods fail when developing complex systems. Waterfall works great when your measure of success is “Did we execute this plan well”, not so great if your measure of success is “Does this product do what we need it to do”.