Why Greatness Cannot Be Planned By Kenneth O. Stanley and Joel Lehman
The document “Why Greatness Cannot Be Planned” delves into the concept that innovation and discovery are often hindered by rigid objectives, emphasizing the importance of novelty and exploration without predefined goals. It discusses how the pursuit of novelty and interesting paths can lead to unexpected breakthroughs, challenging the common belief that progress requires clear objectives. By highlighting the limitations of both objective-driven and non-objective approaches, the document suggests that a balance between the two may be necessary for tackling ambitious challenges and fostering creativity. Ultimately, it advocates for embracing uncertainty and open-minded exploration to unlock new possibilities and avoid stagnation in the quest for innovation.
Highlights
There’s an assumption behind these pursuits that isn’t often stated but that few would think to question: We assume that any worthy social accomplishment is best achieved by first setting it as an objective and then pursuing it together with conviction.
That momentous first day that we enter kindergarten is the gateway to an endless cycle of assessment that will track us deep into adulthood.
maybe we’ve become so used to objectives defining everything we do that we’ve forgotten that their value can even be questioned.
The idea that all our pursuits can be distilled into neatly-defined objectives and then almost mechanically pursued offers a kind of comfort against the harsh unpredictability of life.
Sometimes, the best way to change the world is to stop trying to change it—perhaps you’ve noticed that your best ideas are often those you were not seeking.
“Staring at the wall” will not meet much respect on a college application, even if it’s when you have your best ideas.
It’s useful to think of achievement as a process of discovery.
In a sense, the places we’ve visited, whether in our lives or just in our minds, are stepping stones to new ideas.
It’s no surprise then that computer scientists even have a term, search space, that refers to this very concept—it’s the general idea that creation and discovery happen within a space of possibilities that contains stepping stones leading from one discovery to another.
Objectives are well and good when they are sufficiently modest, but things get a lot more complicated when they’re more ambitious. In fact, objectives actually become obstacles towards more exciting achievements, like those involving discovery, creativity, invention, or innovation—or even achieving true happiness. thesis (View Highlight)
if only you could see the connection
They dangle a false promise of achievement if we pursue them purposefully. But strangely in the end we often must give them up ever to have the chance of reaching them.
we need to let go
The real innovators did their innovating by just being themselves.
In other words, there is a third way—just because you don’t have an objective doesn’t mean you have to be wandering.
In other words, greatness is possible if you are willing to stop demanding what that greatness should be.
In short, objectives are a pillar of our culture, but they’re also a prison around our potential. It’s time to break out and discover what’s outside.
In an interesting experiment by Richard Wiseman [12], subjects were asked to count the number of photographs in a newspaper. It turns out that those who focused on the goal of counting the photographs took significantly longer to complete the task than those who were less focused on the objective. Why? The more open-minded participants noticed that on the inside of page two Wiseman had written, “Stop counting: There are 43 photographs in this newspaper.” While some might say that noticing the answer on page two is only luck, the deeper lesson is that focusing too much on your goal can actually prevent you from making useful unexpected discoveries.
Indeed, psychologists have noted that children need time to explore without specific tasks or objectives set for them by adults [29, 30].
The system works as a whole because it has no unified objective—everyone is following their own instincts. And the most successful users of all are those with open minds, who avoid looking for only one thing in particular.
no matter how tempting it is to believe in it, the distant objective cannot guide you to itself—it is the ultimate false compass.
In a sense, the human condition is that we’re marooned on a stepping stone with only the insight of the human mind to guide us. Though the mind is a powerful force in search, it’s still difficult to see farther than one stepping stone away, no matter our intelligence.
To make this idea more concrete, there’s a great example of this kind of deception called the Chinese finger trap. The trap is a harmless-looking puzzle shaped as an open tube. To begin, you stick your index fingers into each end. But once inserted, your fingers become trapped—and the harder you try to pull them out, the tighter the trap ensnares them. The deception of the Chinese finger trap is that the path to freedom is to push inward, away from freedom. In other words, the stepping stone to freedom is to become less free. This situation illustrates deception well because it shows how wrong it can be to measure progress towards your goal: If your objective is freedom from the Chinese finger trap, then measuring progress by how close you are to freedom is exactly the wrong approach.
The moral is that we can’t expect to achieve anything great without overcoming some level of deception.
To arrive somewhere remarkable we must be willing to hold many paths open without knowing where they might lead.
It created them because nature is a stepping-stone collector, accumulating steps towards ever-more complicated novelties, marching onward onto the mist-cloaked lake of possible life-forms, heading eternally both everywhere and nowhere in particular.
The future that the past created was not the vision of the past, but instead what the past unexpectedly enabled.
Great invention is defined by the realization that the prerequisites are in place, laid before us by predecessors with entirely unrelated ambitions, just waiting to be combined and enhanced. The flash of insight is seeing the bridge to the next stepping stone by building from the old ones.
this is the thesis of Objectives
The prolific creativity of these kinds of processes is difficult to overstate. After all, these are the same processes that created us, and through which we conquered the skies and networked the world.
we are the product of searching through this search space and Q* allows AI to accelerate through idea space
Non-objective search is the true source of much that gives value to our lives. When we unleash search from the trap of the objective, liberating it from the requirement to move only towards where we hope to arrive, it becomes a kind of treasure-hunter that finds needles in the haystack of what’s possible. So then why are so many of our efforts still dominated by the mythical objective? Whatever your goals are, from finding the perfect partner to creating the next great invention, when objectives are ambitious, the only reward you’re likely to receive is deception. That’s what you get for traveling by the false compass.
We should be concerned by the disconnect between how the world is supposed to work and the way it really does work.
There’s much we cannot achieve by trying to achieve it.
But it’s only when the objective is ignored, when the reigns of exploration swing free, that the farthest frontiers are conquered.
Instead of judging our progress towards a goal, the past allows us to judge our liberation from the outdated.
And while it might sound wishy-washy to go looking for “interesting” things, interestingness is a surprisingly deep and important concept. In the words of the famous philosopher Alfred Whitehead [42]: “It is more important that a proposition be interesting than it be true.” W.T. Stace [43], another philosopher, adds that “the criticism that interestingness is a trivial end proceeds from a scale of values thus perverted and turned upside down.”
Rather than thinking of the future as a destination, it becomes a road, a path of undefined potential. This non-objective perspective captures better the spirit of processes like Picbreeder, evolution in nature, and human innovation—ratcheting processes that build stepping stone upon stepping stone, branching and diverging ever outward to everywhere and nowhere in particular.
It was worth exploring the path opened by the Alien Face because it held potential—not the kind of objective potential measured by an intelligence test or performance assessment—but the potential that hangs in the sunset viewed over an endless sea, the transcendent possibility of an open future.
Rather than relying on a false compass, novelty only asks us to compare where we are with where we’ve been. In short, objectives mean sailing to a distant destination with an unknown path while novelty requires only steering away from where we’ve been already
The reality is that we humans have a nose for the interesting. We understand that if we take the interesting path, it may yet lead somewhere important, even though we might not know where. The history of serendipitous discovery supports this idea. If serendipitous discovery was simply accidental, then it wouldn’t take any particular special education or intellect to make such discoveries. For all we know, being a little disorganized or crazy might even be the best way to start.
As you might expect, novelty search leads to one of these different orderings. It doesn’t search from bad to good because without an objective it doesn’t even know what “good” is. Although it might appear that being more novel is “better” than being less novel, it all depends on what you’ve seen so far—someone else with different experiences might come up with exactly the opposite judgment. When rewarding novelty, “better” doesn’t stay “better” over time. The reason is that as soon as a novel behavior is discovered, it quickly becomes less novel as similar behaviors are discovered.
While most people are familiar with thinking of progress as a journey from worse to better, a journey from simplicity to complexity without a clear destination is more exotic. But in a way it’s also more sensible because it’s not subject to deception (because you’re not trying to get anywhere in particular).
That new knowledge is the magic step when a novelty search climbs out of ignorance into meaning. Eventually doing something genuinely novel always requires learning something about the world.
Information accumulation and increasing complexity are the tell-tale signs of any kind of search without an explicit objective.
As Stephen Jay Gould has pointed out in evolution, once all the simple ways to live are exhausted, the only way to create a new species or niche is to become more complex [48]. In other words, there are only so many ways of being a bacteria. That’s why increasing complexity is almost inevitable if evolution is to continue. But these increases in complexity are not arbitrary. Rather, they reflect the properties of the world in which evolution takes place: Eyes represent the presence of light in the universe. Ears signify mechanical vibration. Legs are reflections of gravity, and lungs of oxygen.
In a sense, over eons our bodies have become a kind of encyclopedia of facts about the universe in which they exist.
The environment we evolved in is reflected in our DNA. We are living computers that have been collecting information about our world for eons now, storing it all in our DNA, within ourselves.
The stepping stones to walking aren’t necessarily good walking, or even balancing. Falling down and kicking your legs may be a better stepping stone than trying to take a step (because kicking your legs is the foundation of oscillation, which is how walking works), but if walking is the objective, falling down is considered one of the worst things you can do.
So it’s often possible to achieve more by not trying to achieve it, and now we have experimental evidence to back it up.
Countless treasures are buried along the path to nowhere in particular. We can still dig them out and enjoy them even if we can’t control what they are or when we find them. That’s the real lesson of the interesting and the novel. But to see it most clearly, we need to appreciate the futility that lurks behind all methods of discovery, so that we can finally liberate ourselves from the fantasy of the elusive magic bullet and embrace the reality of the much more powerful treasure hunter.
Impossibility isn’t a popular topic in our culture, where we say that you can be anything that you want to be, do anything if you put your mind to it. But in this chapter we’ll stare impossibility straight in the eye—though we won’t sacrifice our optimism in exchange. Instead, we’ll discover a source of optimism that embraces the uncertainty of the far future rather than fearing or denying it.
This strange paradox, where trying is a curse and not trying is a blessing, sets the stage for a more realistic understanding of what is achievable and how.
The treasure hunter is an opportunistic explorer—searching for anything and everything of value, without a care for what might be found. To be a treasure hunter, you have to collect as many stepping stones as you can, because you never know which one might lead somewhere valuable.
Novelty search shows that it’s possible to capture the process of open-ended innovation and divergent thinking even within a computer. So it can’t be a mystical form of voodoo but rather a principled and logical process that we can understand and even capture.
Instead of something to fear it can be something to embrace.
No, the way to unleash the treasure hunter is actually through separating people from each other
But any ambitious societal effort will end up confronting the same frustrating paradox. When the quest for progress is packaged into a measure, the result is an objective-driven approach. If the objective is ambitious, then a drive to increase objective performance is likely to produce deception, preventing the best possible result from being discovered.
For example, when India was under British rule, the British government tried to exterminate poisonous snakes by paying citizens for every dead snake they handed over. But it didn’t work out the way it was intended: Instead it led to citizens literally breeding cobras just to kill them for the bounty. Ultimately, the number of venomous snakes in India actually increased [63]. So the incentive system produced the opposite of the intended effect.
The greatest danger is when the logic is wrapped within a lofty objective, lending it instant credibility and seemingly putting it beyond question.
A lot of common assumptions, like the supposed danger of aimlessness, start looking more uncertain if you accept that the objective compass is a myth.
Contrary to intuitions, we can conclude that assessment accuracy is unimportant in these kinds of problems. While it sounds strange, it starts to make sense if you realize that objectivity itself is sometimes counterproductive. In such cases, of course accuracy no longer serves a useful master.
Instead, the lesson is that silencing diversity is a sure-fire way to slow down progress. So ultimately we’re left with the conclusion that uniformity may be as meaningless an ideal as accuracy, especially when you have a specific ambitious problem in mind like improving education.
The problem is that once we see the fundamental flaw in objectives, we’re ejected into a wildly uncertain world and lose our usual bearings. So at least striving for higher objective performance, accuracy, and uniformity provide some bearings for improving education, even if they’re faulty. But once again there’s a simple truth that can push us past the fear: We don’t need objectives to find great things. We don’t need to seek top performance or perfect accuracy to discover something amazing. It’s like when we traded objectives for novelty in Chap. 5—we weren’t left without principles, but with different principles that better reflect how discovery really works. And the key principle to keep in mind is that the alternative to objective deception is the treasure hunter. And the treasure hunter is about collecting stepping stones. So when we’re dealing with societal efforts like education, we might be able to make good progress if we as a society help expose each other to potential stepping stones to new ideas.
But beyond geography, there remain important unknowns left to challenge those still willing to explore: those unknowns within the space of ideas. And the benefits from human innovation can overshadow wealth or glory. In fact, new ideas and technologies have the power to reshape our world and society entirely.
It’s in this wild borderland between the known and the unknown that we should want our greatest minds probing, rather than within the comfortable vacation-spot of maximal consensus. Just think, which project is likely more revolutionary, one that receives excellent, excellent, poor, poor, or one that receives excellent, excellent, excellent, excellent? Splitting experts may be more of an achievement than unifying them.
Science is among the greatest explorations of humankind—that we would reward primarily consensus in deciding where to step next is as stifling to discovery here as in any creative endeavor.
Rather, the point is that disunity among research groups and within science as a whole can actually drive progress. In this way, disunity’s power can help us better structure scientific and other creative endeavors.
Because while there are many elegant and important discoveries left to uncover, unearthing them requires constant ingenuity and openmindedness rather than simple brute force.
Though it likely requires passing through many unrelated steps first, following interestingness instead of narrow ambition may better reveal the stepping stones to transformative science and great economic growth.
Not knowing where we’re going is the way of the information accumulator, it’s the treasure hunter, it’s the stepping-stone collector, it’s the path to everywhere and nowhere, it’s the tunnel to the future. We don’t know where we’re going and that’s why we produce great things.
It’s fun to play the rebel and attack what is always taken for granted, but truly saying goodbye to a familiar habit is no easy task.
When people do actually achieve amazing feats, they almost always build upon eons of accumulated innovations during which the final outcome was not the objective. If you think about that fact, then conceding control of your destination is like throwing away an emotional crutch.
When you’re standing on the edge of possibility looking out over the unknown, objectives become false beacons, but interestingness is different. Interestingness forms a network of roads leading from one treasure to another.
The successful inventor asks where we can get from here rather than how we can get there.
All of us can transform the present into the future. None can transform the future into the present.
Letting go of objectives is also difficult because it means letting go of the idea that there’s a right path.
Instead of judging every activity for its potential to succeed, we should judge our projects for their potential to spawn more projects. If we really behave as treasure hunters and stepping stone collectors, then the only important thing about a stepping stone is that it leads to more stepping stones, period.
That’s why we need to beware of the seduction of consensus.
Disagreement and divergence are virtues that deserve to be protected.
So if you’re wondering how to escape the myth of the objective, just do things because they’re interesting. Not everything needs to be guided by rigid objectives. If you have a strong feeling, go with it. If you don’t have a clear objective, then you can’t be wrong, because wherever you end up is okay.
If you set out to program computers but you’re now making movies, you’re probably doing something right.
It’s in your interest that some do not follow the path you think is right, because one day they will build the stepping stones that lead to your greatest discovery.
To achieve our highest goals, we must be willing to abandon them.
The problem with competition as a general explanation for creativity is that it usually drives towards everything converging to the best. And the best is only one thing, not an endless series of stepping stones.
The formula for the universal law of gravity is interesting, but the explanation behind it is more profound: Matter attracts matter, a simple and elegant truth
Relate gravity to law of attraction
The point is that interpretation isn’t just an academic exercise. If the interpretation is clear enough then it can lead to insights so concrete that they can be formalized and applied to many other contexts.
Put another way, evolution is creative despite the competition it tolerates—not because of competition.
Evolution is the ultimate treasure hunter, searching for nothing and finding everything as it spills through the space of all possible organisms. It’s the world’s most prolific inventor.
New highlights added April 6, 2024 at 9:56 AM
So the twist is that the field of AI is itself a search for algorithms, and AI researchers are themselves experts on thinking about searching. In other words, AI researchers are searching for algorithms that search.
We can think of the gradients followed by AI researchers as the information they use to decide which algorithms are the best. A technical term for a rule of thumb that guides search is a heuristic.
While no one can be sure which heuristics will work the best for getting to high-level AI, the AI community has settled upon two in particular. The first, which we’ll call the experimentalist heuristic, follows the rule of thumb that an algorithm’s promise is given by how well it performs.
Besides the experimentalist heuristic, the second main gradient in AI research is the theoretical heuristic. This heuristic suggests that algorithms are better if they can be proven to have desirable properties.
Importantly, these two heuristics hold a profound sway over the field. Even if you don’t like them or have your own philosophy, you won’t publish many of your ideas without respecting them.
It would be one thing if criticisms about performance were just criticisms—and not a standard reason for rejection. But because performance is so widely accepted as a filter for new ideas, it becomes a classic deceptive objective function.
So the experimentalist heuristic is short-sighted: It judges Weird for its present value rather than the value of the future it opens up for AI research.
It may be inconvenient, but what makes for a good stepping stone is more slippery than we’d like.
While a theorist might respond that the benefit of the theorem is that any future algorithm that honors the same assumptions will inherit the same guarantees, this fact is not necessarily good news for encouraging the exploration of new ideas. It means that the community becomes restricted only to those algorithms that honor the same growing set of assumptions, blinding the meta-search to every path forward that breaks the assumptions. In the end, the effect is less exploration—the objective paradox takes it hold.
The algorithms that are remembered, just as with all human inventions, are those that lay the foundation for future trailblazers. They lead to creating new algorithms and perhaps even entirely new fields.
If we can only think objectively, then we aren’t thinking much at all. There’s no substitute for using our minds because science can’t provide a fixed method to discover the next great idea. The greatest ideas are unlike those that came before them. Every stepping stone has a unique story of discovery. And every human being who found one has a story too.
In case it’s hard to think of alternatives to arguing about performance or guarantees, there are in fact many other important clues we can consider: inspiration, elegance, potential to provoke further creativity, thought-provoking construction, challenge to the status quo, novelty, analogy to nature, beauty, simplicity, and imagination.
Anyone can say that performance should improve, but who has the courage to see the beauty of an idea? We could use a few more brave experts like that.