Diederik Kruijssen
Astrophysicist turned AI researcher; Chief Scientist at Allora Labs, previously led ERC-funded research groups at leading German astrophysics institutes.
Auto-generated transcript — lightly formatted, may contain errors.
Diederik
physics of star cluster disruption applies quite directly to society. Because that is the true risk of AI, is that the
the rate at which the perturbation is applied, or the the timescale over which it is applied, is shorter than a generation. And therefore our society will actually not be able to adapt.
If we care about humanity and society being resilient against this type of change, then we should be very conscientious about how this change proceeds, if it's desirable in the first place.
Philipp
Welcome to Tastebench. I'm Philipp Zahn, my co-host is Bruno Marnette and this is a podcast about taste and the creative side of AI. Today's guest is Diederik Kruijssen an astrophysicist and chief scientist at the Allora Labs. After years as an academic studying the history of galaxies, he is now working on the foundations of decentralized AI.
Thank you very much Diederik for joining today. we first want to talk a bit about your personal trajectory, which is a bit unusual, I'd say. And one thing when I was looking through your papers and the research you did in the past, one thing that I found, which was also supported by AI explaining these papers to me and some context, was that you were actually engaging in some form of galactic archaeology. maybe you can
Give your perspective on this, what this actually is and what your original starting point of research actually was.
Diederik
Yes, of course. would be very happy to. And yeah, just to start, thank you so much for having me on the on the podcast. I've I've been a huge fan of the series, so it's actually been really fun to to now be part of the conversation. galactic archaeology is is just one of these fun terms, right? It it sort of, you know, pictures like an Indiana Jones astronaut or so. but but it it's actually almost as as exciting as it sounds.
So basically my original interest, like I I spent fifteen to twenty years in in astrophysics research. And my original interests were basically kind of like archaeology, qu wondering where everything came from. And that that went into things like star formation, but also galaxy formation. And you know, of course the issue is is that the universe is like thirteen billion years old, a little bit more. And
If you look at stars today in in like the nearby universe and the Milky Way, you see them after all of that sort of history, after all of that luggage that they're carrying along with them. So you can bec because light has a finite travel time or travel speed, you can look far away and look back into the past that way, but then you'll never see the Milky Way itself. You'll never see the solar system. You'll never figure out where we actually came from.
So galactic archaeology is the the branch of astrophysics that attempts to reconstruct that history, assembly history of the Milky Way, formation history of the Milky Way, by looking at the properties of stars and stellar populations today. And that's quite challenging, but because fundamentally you're trying to sort of rewind 13 billion years of of evolution during which lots of very, very messy things happen.
but but basically what what people figured out pretty quickly is that galaxies like the Milky Way they form by merging smaller galaxies together. And one of the things I was working on is working on how how stars form. And I realized that star formation is a very sort of spatially and temporally clustered process. So stars typically, you know, don't form alone, but if there's a star forming somewhere from like cold gas in the universe, then
There's usually other stars forming nearby. And some fraction of these stars actually form in gravitationally bound stellar clusters. So their mutual gravity keeps them together. And if these clusters are are massive enough, their gravity can keep them together for 13 billion years. And the great advantage of these is that they were born together. So they have a single age. They were born in a specific place at a specific time with specific chemical composition.
specific properties. And it turns out, through basically wrapping a lot of a lot of history in a in a short summary here, but through pretty big, you know, supercomputer simulations and so on, we developed an understanding for how these star clusters sort of co-evolved with their host galaxies, such that we could relate their present-day properties to the galaxies they were formed in.
And basically by learning that from these cosmological simulations, we were then able to look at the old star clusters that are still orbiting the Milky Way and basically infer how the Milky Way formed. So we discovered that it, you know, had its biggest galaxy merger about twelve billion years ago. And then there was another one and another one. And we were able to sort of name these and and and order them and also figure out how how massive these other building blocks were that then coalesced.
To form the Milky Way. Yeah, that was one of the many things I was interested in during that time. but but it was definitely one of the things that that were most enjoyable. And it was my first interaction with AI, actually. I used, I I trained neural networks on these cosmological simulations to be able to relate the observed star cluster properties to the properties of the galaxies they came from. So that was my my first.
AI interaction. And of course then after that I was lost.
Philipp
Sorry, c can you locate this in time wise? When was this exactly?
Diederik
Yeah.
it was it was fairly late. simple machine learning stuff I I'd done earlier, but this was in two thousand nineteen that I I did that study. So it's actually fairly recent. Yeah.
Philipp
Can I ask a follow-up question? this is again fr from a total outsider looking in. And the way you described it now, this also seems to the way the way you think about this evolution seems to have parallels in also how we think about our evolution on our planet itself and basically trying to rewind and understand it. is this maybe just accidental or do you see there is actually a as an actual parallel here?
Diederik
yeah, I I I didn't I didn't sort of proactively think about it like that at the time, but I certainly believe there was a parallel. Like over time, I originally started working on astrophysics because I I just wanted to figure out where everything came from. Then I realized at some point I actually wanted to know where we came from. And later I wanted to know as I started working on AI, where intelligence comes from.
And it turns out these are basically different sides of the same high dimensional coin. Charles Darwin is one of the greatest geniuses the world has ever seen because he has uncovered very sort of fundamental mechanisms that basically don't just describe the origin of species, but they describe population dynamics.
And it turns out if you're interested in a statistical description of stuff in the universe, you very quickly end up at these sort of mathematical representations of population dynamics that are extremely generalizable. So by all means, yes, you can apply and and I actually way back back in in 2011 when I first had these sort of ideas of hey, we would be able to reconstruct
The assembly history of the Milky Way using using star clusters. I actually started looking at which star clusters might survive, right? I I alluded to that earlier. I said that the most massive ones have enough gravitational binding that they might survive for 13 billion years. And the reason that not all of them survive is they get basically shaken gravitationally by their environment. And if they encounter something that pulls them, then some stars get lost. And over time they dissolve.
And it turns out you can actually treat that process of of trying to figure out which star cluster survive entirely through evolutionary mechanisms, like natural selection, their evolving proper own evolving properties and then the ongoing selection on that as the environment around them changes. It's basically it's it's a constant form of of adaptation and natural selection through migration and all of these other things that is extremely analogous, which then indeed applies.
to our own evolution and our own intelligence and as I maintain also to the evolution of artificial intelligence. We'll be subjected to exactly the same properties and processes.
Philipp
I'll do a flag post here because the changes of the environment in the universe versus changes of the environment that we will be discussing later. I think this is interesting to discuss. But maybe let's before that, let's do a segue into what you were actually working on now because I alluded to this. You made a jump, a quite dramatic one for an academic that was very successful. so maybe you can tell us a bit more about your latest trajectory.
Diederik
Yes, of course. yeah, I mean
People often ask me why why I made the jump. and you you you might notice in what I'm saying that I I I yeah, I'm I have a broad broad range of interests. Let me put it that way. And I I realized very quickly, yeah, indeed a couple of years ago, that that I was sort of developing a routine in in astrophysics research that didn't feel sufficiently engaging. Like I wasn't challenging myself sufficiently.
So the natural sort of continuation of what I was just describing was indeed to start looking at artificial intelligence then and try and try and figure out if the same processes can be applied there. And through yeah, a c a couple of explorations that led me to where I'm today, at at Allora Labs, where I'm basically, yeah, I've built and I'm I'm running the research department where
We're trying to develop forms of of swarm intelligence, of decentralized swarm intelligence, that can basically outperform any individual model. And the the philosophical basis for that is is you know quite varied actually. There were there is of course like a there are ideological arguments why you might want that, like to to avoid things like concentration of power and and these types of of things, open access to information. but
But there is actually a fairly fundamental reason to pursue that type of solution as well. And that is that intelligence in its sort of most fundamental form benefits from maximizing the number of connections across across, let's say, the intelligence substrate. And if you want to achieve that, then that means you can't have sort of siloed individual models. But what you need is a huge variety of specialists.
That can freely exchange information and then a coordination layer, a substrate, that learns to which specialist it should listen under which circumstances. Fundamentally, that's also how society learns, right? We we're a group of people that that each have their own sort of background experience and and speciality. And if you you know treat everybody the same way or treat everybody on the same basis, then you you end up
quite misinformed. you shouldn't, you shouldn't listen to me for my opinion about modern dance. but you know there are some other things where maybe you would want to listen to what I'm saying. And and that challenge is actually is a fairly old problem I found in in swarm intelligence, in in decentralized machine learning or or distributed learning, which yeah is is extremely engaging. So
That is what what I therefore strive to to solve in the Allora context is something that I originally came to call the Cassandra problem. which goes back to to the myth of Troy where Cassandra was was warning everybody that that they shouldn't bring in that wooden horse because it had Greek soldiers in it, and and of course she was cursed by the god Apollo that that nobody would believe her because she she turned him down.
Now that that problem is actually is prevalent in AI as well, because if you have a swarm of of of infer inferrors of basically agents that that provide inference, then what most distributed learning systems do is they basically accumulate historical reputation to assign some weights or or choose or or come up with some historical motivation. But that completely
ignores the fact that there are specific circumstances where you should be listening to specific predictors. And Cassandra is the obvious example, right? At that one moment in history, they should have listened to her and not brought in the wooden horse. And it turns out if if you want to solve that type of problem in swarm intelligence, then you can end up with a solution where the sort of the the synthesized inference, so the the combination of all these predictors
Can actually be more performant than even the best individual. And that's because the best individual, too, has bad days. And if the coordination layer can learn when those bad days are, then it's already better than the best predictor. And that is what we ended up turning into sort of the the V1 of the Alura network. And it's been running live since November last year on Maynet. And these
Yeah, these types of developments are in the end what you know convinced me that that and then this is a fairly fundamental philosophical point is that research, even state of the art research, is almost never topically confined. You're always looking at these sort of very generalizable I I I s I hesitate to u use the word holistic because it also has other maybe more negative connotations, but but
Typically, yeah, let's say requires this top level approach where you try and see the parallels between domains that that, you know, might differ in applicability but share this this sort of common fabric that in the end governs most of our reality.
Philipp
I have two follow up questions on this. the first one, when you made this analogy to societies and individuals basically having different strengths, there is this old kind of Hayekian idea that information is also distributed, right? What you see, what you locally have is not what I have access to. And then on top, obviously our abilities to, you know, perform certain judgments, evaluations, whatever is different as well. That's one thing. So maybe how you see that different and then on the Trojan horse, I think this is
tying back to what I alluded to before on the the change of the environment because this is the I think the prime example of a of a of fundamental change that you could not predict from the past as the environment is evolving. Right. Yeah. maybe you can you comment on these two things, how you or your basically your points of view.
Diederik
Yeah, sure. so I I I completely agree on information itself being distributed, fragmented, and that these perceptions can provide extremely relevant input to this sort of intelligence coordination problem that that that we're talking about, right? So I I think this is actually one of the the key strengths of of decentralized learning or decentralized AI.
is not just that, everybody uses a different algorithm, but it's actually that that unique data sources, unique intelligence is being used, which again propagates into this sort of the this meta learning step of learning like which predictor is good when that relates that that is an immediate reflection of the underlying information fragmentation as well. It's not just a reflection of the underlying model.
So it's indeed is like it's it's a very result oriented form of synthesis, right? It it's it it it's anything that goes into the result, which then tries to interpret that that result in the context. And that that step itself, by the way, of trying to learn like which predictor is good when relies on a similar type of of distribution of of information, which is
Why, by the way, that role too, we've actually made decentralized. So so so it it it has some sort of levels of the hierarchy of of the learning hierarchy that that try to to grapple with with this with this concept. But in the end we found that it actually solved the problem fairly well. Having said that, and that that brings me to to your second sub-question here, is when the environment changes, all of this
breaks apart. And that is actually is fairly fundamental for for any AI model. Because yes, you have a swarm of models. So that means fundamentally, you know, some models might be better suited to an environmental shift than than others. But still in this configuration, all of the models are are trained on just the available training history, right? So if there's an environmental shift by definition, if that environmental shift is unpredictable,
then these models can't really anticipate that. They have no way of anticipating it by just the definition of it being being unpredictable. And yeah, this is indeed something that that's quite close to to my heart as as I published a paper on this last week, which is basically papers called "Flawed in Nature, Perfect through Evolution". And it actually it it advocates a new design
principle for swarm AI systems. And the it's it's basically built on a on a number of of mathematical theorems that outline why a swarm of of sort of historically looking models, so models trained on historical training data, always fail in the face of unpredictable environmental change. So so you know theorem one is
the the the the models are fine if if the environment doesn't change. if the environment does change, they actually incur like a performance loss that grows at least linearly with time. They incur a linear regret. the next theorem is that it turns out they can't help it. They're information theoretically optimal if they, you know, they they can only train on their training data, so there is nothing they could have known about the future. But
The big nugget is then theorem four, which is you can overcome this by introducing the substrate that makes evolution or the mechanism that makes evolution tick, and that is of random mutation of model coefficients. So if you go back to to basically what what Darwin, of course, didn't know yet at at the level of the molecular level, at the level of DNA, but still in terms of the mechanism, did understand.
Is that evolution is is the result of and fundamentally it's a way for a population to adapt to a changing environment. That's why it's so interesting in this context. but what what evolution does is it applies natural selection to heritable variation. These are the critical two ingredients. Now, natural selection is something that AI has been doing for for a long time, right? In distributed systems.
it's it's totally normal that you you pick your models that are historically better. So so or or that are currently better. So natural selection is not that new, but what is new is the heritable variation. So the problem with training AI models is that training is naturally a convergent process. So in other words, if you train over large enough data sets and over a long enough time,
Your models are pretty much going to be saying the same thing. Now, evolutionarily, that is an absolute catastrophe. we're all getting worried that that AI might wipe us out within the next three years. AI should be worried that it gets wiped out within the next three years because its fitness is close to zero. There is absolutely nothing that can introduce the type of heritable variation into AI.
That would make it survive an environment change. So AI is extremely weak on that front. And that is why the moment there is an environmental shift, AI fails. So the solution that I believe exists and I've found and proven in the paper now is that if you introduce random mutations to the coefficients in an optimized model swarm, so basically for each of the models,
And those mutations occur randomly through a stochastic process, through a Poisson process. And what that means is you thereby make each individual model slightly worse, slightly suboptimal, because it's optimized on its training data. So it's individually worse. Then it turns out that creates a form of sort of face space scatter where every model is slightly displaced from the optimum, such that when the optimum shifts,
There is a model that is going to be closer to the new optimum than if there wouldn't have been any mutations. And that, if you then have a synthesis layer that picks up on these changes, says, like, wait, this model just became better. I don't know why, because I don't know that there has been an environmental shift, but I just noticed this model has been more performant lately. Then the entire synthesis layer, the coordination layer, can go like, okay, that's the model we should be listening to, right?
And what that does is it basically allows your swarm of models to systematically outperform, let's say, traditional swarms of of unmutated models. And in, i i in the paper I proved it mathematically, but also ran some experiments and I I got results where like in in up to eighty percent of the environmental shifts, a mutating swarm was outperforming a control swarm of traditional models. So it's not like a low significance result. It's actually pretty glaring when once you try it.
So so that is how I think AI might be able to handle environmental change in ways that we've probably not really thought about before. And I think this is an an underappreciated branch of of swarm AI that yeah, we should dedicate some resources to. No not to make AI kill us in three years, right? And increase its increase its evolutionary fitness. No, not at all. But but
to to let AI solve problems in in yeah, rapidly changing environments.
Bruno
I'd love to I'd love to maybe go a little bit into the maybe the technical aspect there because it's what I love about your work is in in many ways this is kind of against maybe the trends of some of the big labs, right? Which and and I I think if you look at the current incentives right now in like private labs is it's very much incentive to put all your eggs in the same basket because you tend to squeeze if you have one cluster with GPUs
You tend to squeeze more more intelligence if you train just one model, right? Yeah. and maybe it's a bit of a surprise, right? I mean there's been other er eras of AI where people were not concentrating resources as much as we are today. but f first of all I'm I'm wondering like in what's your before we talk about all the the things that are starting to work in a decentralized way.
What what is your I'm curious about what is your intuition about what what are the case for centralization right now? Wha why why are things currently working better when you when you train everything into one model?
Diederik
I I think it's mostly because the the resource requirements for performance AI are enormous, right? So you need you need training data, you need computational power, you need hardware infrastructure, you need all these types of things. And and naturally that leads to a form of concentration of resources because this is this is like it's a combination of of of many different facets and that naturally makes it super linear. So
you you automatically end up with with concentration and you and then you get some high exponential payoff of of that concentration. So that is why it naturally leads to that form of isolation or of of consolidation into monoliths, which then also naturally makes the whole system sort of unwieldy and and inflexible. Like it'll struggle to adapt. So what that means is
these monolithic frontier models are going to be very good at the at the the problems that fall decisively within the domain or the scope of their training data and or very slightly venture behind it, but not much. The moment you are looking for things like
what what we maybe value in human intelligence, like creativity, originality, these types of things, then I think they struggle terribly. And and you can see that, like eve even Astra right now, right? Like you you y you you you play with these types of models and you notice like, wait, yeah. I mean it's obviously just interpolating and maybe slightly extrapolating over its training data. And you could make the case to some extent that the same applies to humans, right? In the end.
we're we're all trained on on, many years of history. but I but I and and and this is hard, right? Because I I don't necessarily have a very sort of logical way of of disproving the counter that that humans are just the same. But I think that when I speak to a a a truly intelligent person, there is always something relating to to creativity, originality, and also like an elevated form of consciousness in a way.
And and and this is interesting because, obviously yeah, latent space interpolations are are not the same as a form of of consciousness, or at least, you know, that that is, I don't think, what what the current consensus is. So consciousness is is, you know, has been proposed across the literature to be the result of many different things. One of them is, you know, maximizing indeed connection
connections between different sources of information. Consciousness to me seems like if if the if that is indeed related to integration of information, then that is something that that is undoubtedly also related to sort of the unexpected leaps that that the brightest humans can make, right? Is like, I never thought of that connection. That's bizarre. But but
then you talk to the person about it. And it turns out they're operating at this sort of level of of consciousness, the state of being, that at the same time doesn't make it very surprising that they were the ones to have that idea. Now, if that that means that they can have that type of idea, maybe they're tapping into a form of intelligence that these sort of monolithic models can't really, because they're just interpolating. So I think the confidence with which you can confidently or decisively extrapolate beyond your training data.
without being wrong, that is probably the greatest hallmark of intelligence. And as we just discussed, indeed, like technically, that is not something that these these sort of training data confined models can do. That is why you then need your mutations to sort of randomly scatter beyond your your your training data. And that is how nature did it. That's how nature in in over you know a long time generated
biological intelligence. So naturally, or at least from my perspective, that is the same sort of inevitable state that you would need to get for AI.
Bruno
There's what you're saying resonates a lot with some of my experience working on training models. The there's an aspect I think I I just couldn't agree more, which is the this kind of monolithic intelligence is I think really it's specific to the the way we train, the kind of task we aim for. which is typically the world of verifiable rewards, right? Like if you're trying to do solving math problems, solving, you know, making code work, solving technical tasks.
If you take these three domains, for instance, it's huge transferability. making a domain better at math makes it better at coding, and so that's an argument for you know putting all your eggs in the same basket is the this kind of additive. what is obviously different is when you move to the world of creativity, if you start rewarding diversity and so on.
maybe the best solution will no longer be to have a monolith, but instead to train separate models and have them you know play and and and and so on. but but I think what what people usually find difficult when they try to shift to that model is like to define winning, right? Like what what is the what do you train what is it essentially writing the reward function also, you know, formalizing what the what the goal is. I I suspect in the generity in the
evolutionary world you also need to define kind of dying, right? You need to define what it means for one one agent in your swan to kind of be unfit. could you maybe tell us a bit about that aspect of your of your work? Like when what happened after the mutation, which how do you decide if the agent must survive or not?
Diederik
So in in the models I I ran or in the and in the the framework I built, I didn't actually make them die because I felt that e even even an unfit individual could evolve or could yeah, could mutate towards being useful again or the environment might drift back into its direction. So I took a fairly generous approach there. That's not what nature does. but I yeah, so so what I ended up doing technically is is actually just
assign higher weights to the ones that are doing better with some recent smoothing to to to basically pick up on like systematic change rather than noise, right? So you you you need some some smoothing function. But but you're right that it is actually quite hard so so if we now go to a a fully decentralized setting, right? In nature there is no reward mechanism for being a mutant.
individually. There is no reward mechanism for that. Statistically, every single mutation is is a handicap for for the individual. So in nature, you know
Bruno
So I it's not s sorry, maybe just to correct this is we don't know in advance if it's a handicap or not, right?
Diederik
Statistically speaking, the expectation value is that it's a handicap. That that is like there are more ways in order in which you can mutate wrong than right.
Bruno
Assuming
it's perfectly trained. compared to what would be the perfectly trained Right.
Diederik
And and and the same is true in evolution, right? Because typically what happens thro over generations is that a species go to goes to to to try and maximize evolutionary fitness. Now what what's the trick within that is that, you know, on the population level, that relies also on a form of of heritable variation. So so it wants to maintain that. Now, of course, in nature it does that not only through mutation, it actually also does that through through sexual reproduction. Of course, right? Which
intro introduces that that form of of heritable variation. Now, models don't have babies, so hence we we go to mutations. and and these but but what that immediately introduces is this issue where you cannot incentivize an individual model very easily to mutate. Because at that moment it's failing at the objective function more on average. Like the th this is tip this is a population level incentive.
And this is from my perspective, actually. I I've written up some thoughts on that in the paper, but this is actually for me is is an unsolved problem. I I I'm currently thinking about that a little bit more. Like, okay, how would you actually do this? And you you can you can do that artificially. You can say, Okay, I'm now gonna reward you for the extent to which you systematically differ from from your peers, right? It it's not what you typically do in a consensus mechanism, but when you're
doing predictive things, you you're actually quite interested in that if the environment is constantly shifting. So these are things that are yeah not very obvious and and one of the reasons for that is that models need to choose to mutate in a way or like model creators, model runners need to choose to to make their model mutate, whereas in nature it's not a choice. It just happens. And that's the difference. That's why probably also nature evolved the way it did. Is that it wasn't a choice
Bruno
Yeah. But when you say it's not solved, can I ask about group selection? Because I think in evolutionary biology sometimes the argument is you know if one group of people has the gene that gives them the you know the collaboration instinct and so on, they would overpower the other group. Is it good would that sort of mechanism make sense? Like introducing a group versus group?
Diederik
Yeah, that that would that would definitely work if there was a selection mechanism at the group level, because indeed this is definitely something that that, requires a group. but yeah, then that needs to be built. Right. Right now, w what what you have is either monoliths or or sort of one layer hierarchies of distributed systems. And what you're describing is indeed you need a multilayer hierarchy of a of a distributed system. And then indeed you can start start to play with these types of things. I agree.
Philipp
Sorry, this is more of a comment or an observation. There is a there's a paper by James March, I think I I was looking for it in parallel but I couldn't find it. I think it's from nineteen ninety one. it's basically describing a version of the organizational problem, the equivalent of that. In the sense of an organization is performing a business and it's working, but obviously if the environment changes, it's not fit anymore. So how can they actually incentivize? And he he makes this argument there is this natural ingrained bias against it because the optimization works and everything else is basic, has just optional value now.
And that's require and it this is a hard management problem, obviously, to do. Well
Diederik
Yeah, it it it is sorry sorry to to to to comment on that briefly, I think it is especially in in businesses, because these are resource heavily resource constrained. So you can't say, let's just spin up the same department ten times so that we apply exactly the type of group selection to these ten departments. Like that that's a waste of resources. But long term it would probably be for the best. So sorry, Philipp, c could you but
Philipp
No, so good. Maybe also connecting to what Bruno said before. I think this is one of the reasons also adapted over the industry why there is so much concentration on the thing itself and doing something else has a lot of opportunity costs at the moment. I have another question. in the paper, the changes in the environment are basically happening. They're given. They are not informed by the state before. they are not predictable in that sense. Now in social systems, very often there is obviously first kind of a there is a
the past has an influence. Yeah. But also what is very interesting is in some sense, and this is I think this is a typical social phenomenon that is very hardly happening in physical systems. we as agents, as humans, if we let's say we are interacting in a market. We are bringing about the market. Let's say an interest rate is the outcome of whatever we do. Yeah. But itself, that cloak that aggregate has an effect on us. There's this kind of very weird feedback to from the micro to the micro and then the micro brings the macro about. Yeah.
Your thoughts on this in terms of how if you if you if we take seriously the idea of, independent agents maybe also bringing about actions in this world and influencing the environment. how you see your framework and the journalists kind of setting fit?
Diederik
Yeah. That that's a good question. I I think the the moment that the environmental shifts and the way that an individual agent should be responding to these become predictable, then the the added value of random mutation diminishes. because anything that is predictable doesn't require unpredictable shifts in in the agent either. Right. So yeah, I I would say there exists like a law of proportionality of randomness.
which is that the basically the more random the environment changes, the more random the changes in the agents should be. And and that that also relates to, for instance, this is one of these things that sound utterly trivial when you say them, but you know, I had to find it. So cut me some slack. But it turns out that your your mutant swarm performs best when its mutation drift rate is the same as the environmental drift rate. So
What what that means is, your problem changes. So let's say you're trying to predict some target variable and and the environment shifts such that it results in an effective target shift rate, then your mutation should be of exactly the right magnitude and rate that that your models keep their random wall keeps up with that. And and I I think the same applies in in terms of the the the feedback loops that you're describing. Like if if
there exists a a predictable feedback loop to some fraction of of the shift that you're seeing, then the swarm should probably incorporate it and will incorporate that. So that then means is that the shift becomes like there fundamentally what I'm describing is always applicable to the random part of the drift. That that's the way you should be perceiving it. And anything else, anything that's not random should be learnable fundamentally. And if it's not, then
Then we need better models. That's roughly my perception. Having said that, by the way, I I I should maybe be a little bit more careful. because these these types of feedback loops, especially in multi agent systems, can often give rise to to chaotic behavior. And of course, chaotic behavior itself is a form of randomness. So then you might say, like, okay, fundamentally, like on the individual level.
there is a deterministic feedback loop. But it turns out because you're dealing with, n much larger than one, you lose the determinism. Then, you know, we're back at square one and it's all random in the end anyway. So it it depends on what the again, what the scale is of the system that you're considering. so randomness can be can be zero on the scale of the individual, but can still be a hundred percent on the scale of the system just because of the induced
peer interactions within the the swarm.
Philipp
So this this is this just crossed my mind. It's a it's a bit of a dark example, but it's given that's 9/11 today. comparing human learning against the learning that you have in this so f in the system so far. So if you if you take again the example of the planes that were crashing into the World Trade Center, the fourth plane basically was not crashing because the people on board got news about what was happening and they reacted fast. And I think this is a typical this fastness. And in some sense, it's the same with the Trojan horse, right? It's yes once there.
It is different for everyone. Yeah. so our human learning at some level can be extremely fast. Obviously quick can be also very slow, but this kind of jump effects are there. How do you locate this, maybe also as humans, relative to the distributed AI that you've been described before?
Diederik
Yeah, that's that's tricky because you always like the you you always need to achieve and this is true for humans as much as it is for AI, but you need to achieve a form of statistical significance. Like the moment you are making decisions, what you're doing is you're you're basically taking action based on an information flow. And that information flow is often extremely noisy. So, you know, if if three planes crash into three buildings
within an hour of one another, within a same geographical area that's sufficiently rare that it you can take action. If it's just one, you don't. And you know, sure, if it's one and at that moment your plane is being hijacked, then maybe that is already like, wait. But the moment there were two, then you're like, okay, wait, something is going on. So, and and we're three, right? So
It it's honestly it's a multiplication of of low probabilities that then, you know, pushes your statistical significance in a certain direction. Now what that means in practice when you want to sort of detect these these like the different contexts within you need to listen to a different agent or or environmental shifts where a like say a mutant's agent became better than than the norm, than the unmutated one. what you need to do is you need to accumulate
sufficient data points over over some time frame where the hope is that that time frame can be shorter than the time it would take for training data updates to pick up on the change, right? Because if you train on, say, the entire history of aviation in this example, then you know three incidents might not show up as quickly. It's like, wait, yeah, but somebody also crashed in 1957 and
And it was like, we should put this in in the high dimensional framework to be absolutely certain. no, that that's not relevant. Instead, if you've got somebody who goes like, Well, wait, last year, how often has it happened that three planes crashed within an hour of one another with such geographic proximity? You go like that never happens. So so th yeah, this exact this is exactly the same process that basically that
The systems we've been talking about have been designed around. So it is it's basically, yeah, having this sort of time limited window. like you know, can use exponential moving averages or whatever, then technically it's not time limited, it's just, you know, fading. But but you fundamentally you're sensitive to recency more than to the entire history, which is what, you know, you can also do with training data fundamentally. Like there are
There are techniques that allow you to do that, but it's fundamentally less flexible. And the moment you introduce random mutations and you deal with environmental drift, then you basically you you must do this. You can no longer rely on on the sluggishness of of solving the complete problem, but you need to achieve a form of statistical significance on short time frames. And then you take action.
Philipp
Yeah. I also picked the obviously the example I picked is one where we arguably have learned very fast. Obviously there are other examples where very stupid form of unlearning or learning in the in a long direction takes place, in the case of a cargo cult, for instance. may maybe one thing. So a recent guest suggested to us, this was Turi obviously in the in the idea of how we humans work best, it's actually in in the group.
And also being opposed by others, diversity itself for us is important in the communication as well, in learning more broadly. But also in this notion of in taste, how we f how we appreciate things and how we learn about things that are actually that we like and that are good. Yeah. This was a long intro leading to we see a lot of changes in the constraint, what people can do, even with the monolithic model that exists so far. They are good at certain things and that you know makes some people worried. Maybe can you
Give your points of view, how you see the role of humans in the current AI, but maybe also in the decentralized AI. What is their place there relative to AI, relative to what comes?
Diederik
I think the human standard is incredibly important because for for for many different reasons. maybe the primary one is that I don't believe that AI experiences the world in the same sort of multifaceted way that humans can for now. you know, we we have senses. We have a a much more direct feedback loop between
the world we live in and what we do with that information than any current AI models. So
I I I think taste is probably the the main driver for all important innovation. and and and it it it is indeed related to diversity. Diversity of thought, by the way, I think diversity of identity is pretty much entirely irrelevant in this context. So it's diversity of thought and and perspective that that drives indeed the friction between humans.
the friction between humans in terms of what their perspectives are, and that can, you know, generate these sort of feedback loops towards the individual to to change that perspective too. Now, the moment you go to AI, especially when you have these more monolithic models, then, you know, they're capable of doing this sort of, you know, encyclopedic summary of the possible perspectives that exist without attaching necessarily a value to these. Or if they attach a value to them, then you know
I'm not sure what that value means because of how humans experience the world. But fundamentally
Humans created artificial intelligence as a tool, right? And and I actually I I very much dislike the whole lobby to to perceive AI agents as these sort of semi-conscious beings, which which I I really don't believe they are, not in the current state. and and and we shouldn't we should be very careful to not overly anthropomorphize AI.
Let me give give you a couple of examples. and then this is a very, very famous example actually from from anybody who's done academic research or who has has supervised PhD students and so on. very, very often PhD students can go to to their advisor. They've been working on something for a week, a week, full time, and they go to the advisor
They they show a figure, look, I made this figure. I I think it's a really cool result and the advisor takes one look and says, No, that can't be right.
That that happens extremely often, right? And generally the advisor is right. And that is because and and and sometimes the advisor can't actually say why. Like I I'm guilty of that. I've done this so many times whereas so result that that can't be right. Why? Well, I don't really know. and then and then you need to sit down and think about it. And of course, if I then think about it very carefully, I know exactly why it is, and I can point to it a lot more specifically.
But my first impression is that can't be right. Sometimes too, by the way, then you're wrong and then you actually discover something. That's great. But but but that's rare, of course. So so these are things where where there is a form of human intuition that is extremely non-algorithmic, and that is probably related to this multifaceted experience of the world we live in, right? Is the fact that we have our senses, that we have this bizarre
subconscious processing of the information that is fed to us through dreams, through, through daydreaming, through all of our sort of back of the head thoughts that that occupies and distract us at completely unpredictable moments where you you're sitting in a discussion and suddenly you're thinking about something that's entirely unrelated. That is, you know, maybe maybe AI would see that as a as a flaw.
But that is the flaw to see that as one. it it is probably one of humanity's biggest strengths. Another example is that if you want to do research well and actually, therefore, what what we spoke about previously, optimize this sort of extrapolation step to venture beyond your personal your own training data and actually do something new. One of the most valuable skills and abilities that I've seen in people.
is to know exactly which corners to cut. the ideal solution to a problem is not where you start. You start with the simplest thing first. And that means you're constantly walking this tightrope between oversimplification and therefore uselessness, or, you know, too many bells and whistles and therefore not actually giving you any insight into the problem you're trying to solve. So that tightrope of knowing exactly which corners to cut and how to get to the most sort of
I the mo not the most accurate or the most comprehensive, but the most useful version of your answer. That is something too that is extremely human. And, you know, across these two examples that I just gave, I think, you know, it has taste dripping off them. It it it is basically shaped by taste altogether. So no, I don't I don't see a future of human obsolescence at all.
Bruno
And independently of so obviously this is a debate some people feel emotional about because of human obsolescence question, as you mentioned. maybe just to to try and have like a very like non-romantic view on it. we've had we've had two types of guests, with pretty extreme views. Some we really think of AI as a tool, it's not touching the domain of taste. AI is really just here to, you know, execute. If you use AI to come up with ideas you you're making a mistake, you shouldn't do that. And
And yeah, basically the the the this is a picture of a future where there's almost a clear line between what is execution and what is the ID generation world, which kind of should be human. And then we have we have other guests who think of it that no no, this is a continuum, this is a very blurry line. in some ways, you know, AI is already doing a lot of like micro decisions for us, like you know, and when coders are I trusting more and more.
They're a coding agent, they're basically letting the agent do what were what used to be considered design. people used to design architecture and it used to be an almost creative experience, right? And now you know, they just know the machine will make better choices. And so yeah, well well I I guess I'm I'm curious in your mind, like what's the arg what is the argument for a clear separation versus a you know, a gray line? Like a
Diederik
Yeah, it's very interesting that you mentioned that because I I think I I fall in neither category. I think AI is is fundamentally a tool, but that doesn't mean it can't stimulate creative insight. Right? And and it it that doesn't mean it can't create art even. Like it can. It it won't be revolutionary probably, because it will be and then even like
I'm gonna admi admit immediately, I've been trying to push AI to come up with creative insight in in domains of research that I'm very, very familiar with. And if I do that with fairly little guidance, it fails miserably. And if I the like the more guidance I give it, the better it does, but also the closer it will be to what I would have done myself anyway. So
I I think in that sense it is again it it can be a tool in in in making your own extrapolation step. But but I've not seen it like make giant leaps. Like you know, the the the Navier Stokes saga.
Bruno
That was my next question. Go on.
Diederik
So I mean, okay. I I I think what you see there is again, it's a form of tool, right? So so there are humans who take steps in a certain direction, and those steps can be bigger steps because each step is AI assisted. you can also go like, yeah, it was all autonomous, and then it turns out when you zoom in it wasn't all autonomous. And even if it was all autonomous, then it is using something that
had human involvement or hybrid human AI involvement. these things happen and these things are organic. That is a gray line. But fundamentally, you know, what that means is you're still, you know, collaborating with the AI as an exponent of human knowledge. And even if the I AI were to be autonomous, then it's still an exponent of human knowledge. So i I I I I don't know. I think there there is there is
With some people there's a great desire to attribute greater and greater powers to a tool that is extremely sophisticated and capable. but, you know, l let let's not, you know, pretend that the Jedi have returned. It it's it it it it's not magic. It's not magic. And I I think that that's the key difference. Like humans humans have the capability of identifying that
They stand on the shoulders of giants, to quote one of the greatest researchers ever. humans can see that they're standing on the shoulders of giants. And what that means is and and this is actually I I I wrote this about the whole Navier Stokes drama. what you will see increasingly is what I call the final nudge problem. Like there are these big questions that have been open for a long time.
And you know, any researcher has has done their share of big question solving to two different degrees of magnitude over the course of their career. And they will recognize immediately that when you solve one of these big questions, you do that by your ability of to to to sort of integrate the literature in into your mind to to almost become the problem. And because you become the problem, you see the connection, you go like, wait, that's it.
And that is what is happening here. It's like you you you build up this sort of this this corpus of of of knowledge and understanding that goes so far that that this giant big problem can be solved by giving it a final nudge. Now that final nudge is going to be made by whichever agent can do it the fastest. And speed is definitely a great advantage of AI. So what you're going to see now is humans
Pushing the boundaries ever closer to solving these problems. But statistically speaking, the final nudge will always be given by AI just because it's faster at it. So you're going to get this weird misperception that AI is now replacing humans in all big discoveries when that's not really the case. It's just that they're fastest at that at that last step of execution, at that final nudge. So
Yeah, a lot of this is is social noise and and part of it is c commercial noise. It's another case for decentralization, by the way. And yeah, I I I think it's very important to to interpret this cleanly. And that that is notwithstanding the fact that of course there are very intelligent people working on these types of things and I understand their enthusiasm, but
I don't know. My my first response to anything that seems too good to be true is always that it's too good to be true.
Bruno
And and just to understand, so you you you're showing skepticism about, machines being not magic, not as magic as advertised. would you say humans as are as magic as advertised?
Diederik
Yes, yes, yeah.
Bruno
Is human creativity also oversold?
Diederik
I think and and this goes back to flawed in nature. I think that human fallibility is an evolutionary strength. And I think therefore that the maybe the magic is just in nature and and and in the processes that shape populations.
Right? And but but that that's that's maybe the the physicist in me surfacing. But but I don't I don't think there is anything magical about humans. I think that humans are potentially the outcome of fairly magical processes, and that's just the universe that we live in. but the same is true for AI. AI too is is is a manifestation of of that universe not quite observing itself yet, the w the way humans
or or the universe observing itself. But but at least an interesting, you know, development in terms of how information can organize itself within that universe. And and to me the magic is very firmly in in in how how fallibility and and weakness together with
a form of of unpredictable in un unpredictability and randomness can create order, structure and beauty in a world that is fundamentally chaotic and unpredictable. And that is something that life has learned to admire and to harness by basically gambling every second of its existence.
'Cause that's what life is. It's fundamentally just a mega hedge. And that is why we are where we are today. And this is something that that the sort of the ultra determinism of AI has has not fully captured yet. if it and I I I I mean, I I'm not one of these you know, I I I I I don't believe in positivism as a as a, you know, philosophical
perspective, I I don't think you know there there is a natural progression to development, for instance. I I don't believe you you naturally always improve as a system. But what a system will do is it will respond to circumstances and on some scale of the hierarchy of being, there will be Darwinistic processes that apply
apply forms of selection on on on the variation that exists. And what that means too is you can get catastrophic extinction events. You can end up with populations or or groups of groups that are fundamentally unfit to survive the environmental changes that we're going through. And as a society actually, this is something that we should be extremely mindful of. Is to me, the big risk
of AI is not like th the mega system that that you know that is not Skynet becoming conscious on on a specific date, but it is actually that our society operates on a natural adaptation timescale. And is not even a biological. This is now applying sort of
evolution to just to just society. Yeah, it's institutional, but but but also society through through sort of mass psychology, these types of things. society has a natural time scale to it and is usually connected to to the duration of of of a human generation. although indeed institutionally like you know these timescales are somewhat different, but they're still fairly similar. Like orders of order of magnitude through roughly, you know, the order of a few decades. and and what you see and this is again coming back
Back from physics, we're now going back to the star clusters we started the conversation with. But there's a very simple physical rule for the for star cluster disruption. And that is if a star cluster, which is gravitationally holding itself together through the mutual pull of its stars, if that is perturbed by another massive passing object. So it's basically there's this sort of transient gravitational pull, then the very simple rule is.
That if the timescale for that perturbation is slower than the internal movement of the stars within the star cluster, then the star cluster will respond what they call adiabatically. It will slowly expand and slowly shrink again, and it won't lose a lot of stars. And basically, that is because the cluster adapts to this perturbation. But if that perturbation is by an object that moves so fast that the interaction
Is faster than say the time it takes a star to cross the cluster to make an orbit within the cluster. Then we call it an impulsive perturbation. And that typically leads to the complete dispersal of the cluster, because the stars can't adjust their orbits, they can't adapt, and the binding energy is not enough to keep the thing together. Now it turns out the physics of star cluster disruption applies quite directly to society. Because that is the true risk of AI, is that the
the rate at which the perturbation is applied, or the the timescale over which it is applied, is shorter than a generation. And therefore our society will actually not be able to adapt. And there are all kinds of like this this will have economic implications. Like indeed, businesses have adaptation timescales that are too long as well. And there was this beautiful
piece by Citadel at some point earlier this year that who responded to the Citrini piece saying like, we've got this exponential disruption of society by AI, and Citadel went like, no, no, no, no, not all of the companies we have a stake in are going to go bankrupt, you know? because there is this S curve and and we will have, you know, great response, it will pick up and and at some point it will flatten. That's not how impulsive perturbations work.
Impulsive perturbations are are catastrophic because they happen faster than the system can respond. So that's the part we need to address. If we care about y you know, humanity and society being resilient against this type of change, then we should be very conscientious about how this change proceeds, if it's desirable in the first place. And by that I don't mean the anthropic type of of overregulation.
But just actually thinking about what this is, about educating ourselves, about making sure that the human system, society, the human infrastructure is actually evolving on the timescale that we need to to implement AI in the way we want it, rather than being either responsive.
Or over accelerating AI development. there has to be a natural equilibrium there. And this too is is a conversation that is not held in the right way. Like the the the the the incentive contamination is enormous, of course. These are very, very big questions where and incentive misalignment is is prevalent. And it is
really, really fundamental. It ties directly to the inability of society to adjust to to rapid change. And it's a type of fundamental thing that is much more dangerous than Skynet nuking us all.
Philipp
we have already taken up quite a lot of your time. Can I ask maybe a last question on this? Sure. which ties back the incentives with the monolithic idea. I think if you're operating under the idea idea that the monolithic approach will give you AGI then there are very clear race dynamics for individual companies, but also across countries. A decentralized approach, in which sense would you see this is has a maybe a balancing feature or
would not allow itself to this to this kind of dynamics. Or would be maybe susceptible to other dynamics, but y you know what I mean.
Diederik
Yeah, no, this is a good question. th this is a very multifaceted question actually. first, very briefly, I think monoliths will always struggle with with specialization for the reasons that we discussed. So what that means is you will would always need to augment these monoliths with specialists. other otherwise they can't achieve an AGI stat status ever. So I think actually a decentralized architecture is f a fundamental prerequisite for any form of of AGI.
So that's one. But then two, in terms of the incentive, yes, there there is a a sort of race, of course, but but that race is naturally because of the mechanisms we discussed too, is that if the environment shifts, then that means being ahead is is only ever temporary at best. So it means the race never finishes. And three
I don't actually think that AGI is an absolute end state. I think what we're like even though for us it now seems like, it's like, you know, you've got human intelligence and then AGI is just beyond that. But but that is obviously as a continuum. So what that means is we will always be measuring f relative forms of intelligence, no matter how AGI we are. And what that means too is that there will
never be that end state where a single entity will declare victory. And that means too that the the transient nature of being ahead will actually end up being a saving race. Because what that means is that nobody can ever like declare victory. And in the end the race will be everlasting and will be about diminishing returns too, because in the end we've got finite resources and the returns will be diminishing.
So yes, there will be in that sense, there will be a flattening of that curve eventually. the question is when is not obvious, is not clear at all. But, and that comes back to the decentralized side, fundamentally, if you're capable of of harnessing a coordination layer across a swarm of models that are all racing towards AGI, then by definition you're always the one who's ahead in that race.
Right? That that that is that is the fundamental reason why decentralized forms of AI are always going to be the solution. Because every model will have a weakness. The moment you accept that axiom, you will all also agree that the moment you can learn who is weak when will allow you to outperform any individual always. So it it's that very, very simple set of axioms that tells you
Yes, AGI will be an ongoing process will be an ongoing, probably never never ending process, and in that process the decentralized coordination layer will be
Always be
the leading one.
Philipp
Perfect. Diederik thanks a lot for being part of this. It was a pleasure.
Diederik
Thank you so much. Really enjoyed the conversation. Thank you for your time.