Since optimizing for ambitious objectives does not work, we must embrace the process of discovery when attempting to achieve ambitious goals.
This means we must be willing to follow our instinct for novelty and embracing the inherent uncertainty of the journey. For it is in venturing into the unmapped territories of the interesting that we stumble upon the verdant oases of transformative ideas.
Objective-driven optimization aims to incrementally improve performance on a defined metric or goal. However, for ambitious challenges, rigidly pursuing an objective can lead to deceptive local optima that prevent us from finding groundbreaking solutions.
Novelty search, on the other hand, rewards exploring the new and the unusual without regard for any specific objective. By incentivizing constant divergence from the familiar, it propels the search process outward into the uncharted, forcing it to accumulate discoveries and continually expand its repertoire of possibilities. This captures the essence of how human creativity often works - through unexpected connections and abandoning constraining assumptions.
This approach produces very different orderings of discoveries compared to objective-based search. Instead of greedily climbing toward a peak of “better” solutions, novelty search meanders through a more varied terrain, vaulting from one island of complexity and interestingness to the next.
Moreover, the novelties uncovered through this wandering inevitably shed new light on the problem domain itself, catalyzing insights and expanding our understanding.
Just as biological evolution incrementally encoded information about the universe into organisms’ complex features, so too can novelty-driven searches acquire profound new knowledge.
By letting go of the siren song of narrow ambition, we can better navigate the “wild borderland between the known and unknown” - that twilight realm where the most remarkable discoveries await.
Ironically, it is the willingness to embrace uncertainty that unlocks access to the richest veins of innovation. Surrendering to novelty may be our surest path to revolutionary ideas that can reshape technology, science, and society itself.
Source: Why Greatness Cannot Be Planned