Austin Kilroy

You know, so your your experience obviously is based on

things happening, facts, knowledge and so on. your expertise is your ability to deploy that previous experience.

Your sense of judgment might, you know, that there might be a a a metric of success in a certain situation. Did you make a good judgment call? So those those kind of things are objective. Taste for me is very personal. it's subjective. You know, things that for me are good taste, my choice of shirt today, for other people would be very bad taste. and I think that makes it fiendishly difficult to sort of optimize something for taste because it

Whose taste? The other thing that I think is very interesting is that you start getting into questions of power dynamics as well. If you know to to optimize taste, you are standardizing something which is very diverse and you're you're taking you're privileging a certain taste above other tastes.

Welcome to Tastebench, I'm Bruno Marnette, my co-host is Philipp Zahn and this is a podcast about AI and taste. Our guest today is Austin Kilroy, an economist, facilitator and innovator whose work has taken him from the World Bank to governments and cities across the world. A Fulbright Scholar with a PhD from MIT, Austin has led major economic development programs, advised public institutions, and is writing a book exploring how better conversation can produce better decisions.

He is also developing an AI power tool designed to uncover blind spot and improve collaboration in meetings.

welcome Austin. so good to have you here today. I'd love to maybe start a bit by if you could tell us a bit about like your current work and

brief introduction to your journey.

Totally. Okay. Thank you. No, it's a massive pleasure to be here and I always have fun when I'm talking to you both. So I'm looking forward to it. So yeah, current currently there are two main projects I'm working on. One is a book, which I I don't know if that sounds a little bit pretentious, but it's it's based on where I've got to in my career so far, and I think there are some useful lessons to at least for myself to to kind of distill and encapsulate.

And the other project is called Whispers. It's a software tool for high stakes meetings and its job is to help teams think and collaborate better.

So it's not that I I imagine the there's a philosophy behind both projects which is quite quite similar. so yeah, maybe if you can yeah, feel free to tell us more about the about the book and because I imagine it's inspiring also the the the other project.

Yeah, well the let me give you the the the sort of medium to short version. So the book is tentatively titled The Catalytic Mindset and it's basically about my experience working for just over ten years in the World Bank. I was a development economist traveling around the world and learning the hard way how to be successful and how to fail. And the catalytic mindset essentially its main message is that all of my

learning that I did before in formal institutions and you know through PhDs and so on actually ended up counting for less than the relationship skills and the team management skills and the kind of informal sort of what people call soft skills, but that that really

Mm-hmm.

plays them down too much. So skills of knowing when to do the right thing at the right time, which is kind of what we

what a lot of us spend all of our lives learning and we still get it wrong in the end. So I think there's a lot to think about there. and do you want me to talk a little bit about whispers as well?

And I actually I'd love if you can remind me and people what what the World Bank does. and like

Yeah.

what what what kind of you know, just give a sense of what kind of social skills does this imply or or soft skills does this require.

Yeah, sorry. Sorry.

so the World Bank it is a bank and it operates worldwide. its main job is to lend to governments and it lends for development projects. so these could be new roads or reform to the educational institutions or new power stations. That that was sort of the traditional model and increasingly it's lending for

industrial development, competitiveness initiatives, and that was the kind of thing that I worked on, particularly in cities. So I think there's about 10,000 staff and consultants jetting around the world, flying into places and and sort of helping governments to formulate their projects and then lending money to do so.

Given that Philip has a background in econom as an economist, I'll let also let him ask a few questions on this.

I have actually a

Kind of opposing question. So what do people most often get wrong about the World Bank?

Hmm. So th th this is about perceptions from the outside.

Yes.

Well, I I think it's fair to say they get some things right. so I'm gonna give you a sort of warts and all viewpoint. So the stereotype of World Bank staff is that they are highly qualified, so would often have a PhD and or master's degree. you

Quite often would be economists, quite often would be arrogant and think that they probably know the answer, even though they've just landed in the country. and some of that is is right. in terms of what people get wrong about the World Bank, I would say there's a slightly outdated notion of it still being stuck in the time when I I don't know if you've you've seen the book

called is it called the economic hitman? Something like that. So it it encapsulates the World Bank. I think of the sort of 1970s as an instrument of keeping poor countries poor with various white elephant projects that they then have to spend decades paying back. And I think that is probably quite outdated now. There are a lot more

innovative projects and a lot more collaboration happening with with national governments. But obviously there's always more work to do and there are there are some parts of institutional memory which which are useful and others which don't fade fast enough.

How are employees of the World Bank different from employees on of the IMF?

that is quite a specific question. so it with my hat on as a former World Bank staff, I I'm sure I I should say completely different. There's no relationship between

Yes.

us. We're two different breeds. for example the stereotype about IMF staff is that they have largely written their reports and their aid memoir even before they've landed in the country because they're so sure of their analysis before they get

and then they spend their their week in a country having meetings and then issue the report that they've already written. and I I think a world banker would say, We would never do that, you know, it's all about building things from the ground up and working with local stakeholders and and so on. So that that's that's a little bit of the flavour. but I present the stereotype slightly.

Fine bars. We talked about this before. Meetings are an essential part of the work. I mean everyone does, but you highlighted as specifically important. And you have a lot of experience with meetings. So much so

Hmm.

that you're actually now betting also on your own tool that you are developing, your own product. Maybe you can briefly talk about it.

Yeah, I I this was I think the biggest learning for me in the sense that it I I became obsessed with it. So something which when I started at the World Bank was seemed like, you know, just part of the process, having a meeting, having a team meeting, meeting with a client, brainstorming, you know, it's something that you did in the course of making a project or implementing a project.

By

the end of my time at the World Bank, I was utterly obsessed with how meetings happen because it seemed to me all the important thinking or and especially all the important decisions didn't happen in print or in reports, they happen in discussion. And even in the meetings which feel a bit more like set pieces, it's the dynamic of the meeting that determines everything. So the way that people are reacting with each other, the way that maybe emotions get triggered.

the way that people present evidence or that they are just incapable of putting their point across in a clear way, and everyone kind of glazes over as soon as they start speaking. All of these kind of things that as soon as I say the word meeting, and everyone listening to this

Mm-hmm.

will immediately have their kind of mind association of feelings and and experiences

Mm-hmm.

with meetings. I realize that meetings are so difficult to get right, and I became so obsessed with

how to get meetings right that I decided to resign from the World Bank and focus on this software tool call called Whispers.

Before we talk a bit more about what the software does, what is your theory of a success film meeting?

so there are a number of different kind of meeting typologies which I think are useful because a w we have a single word meeting but it performs so many different functions.

Hm.

You know, in some cases we want a meeting to decide something, in other cases we want a meeting that connects people together, in other cases we want a meeting that informs people, in other cases we want a a meeting that

generates new ideas that ideates. We want to negotiate. You know, so there are and and most meetings have several objectives. So the the the idea of a meeting, it it's it's quite hard to simplify it. Sorry, just remind me how you phrase your question because I want to speak exactly to to that.

question was what is what is your theory of a successful meeting? And parts of that you already answered. The theory starts with a topology.

Right, so okay.

Mm.

So yes, I think it's important to recognise there are many different types of meeting. and obviously gauging success will depend on that particular type of meeting. What I found very interesting when I sat down to think about it was that there are d despite all of this sort of heterogeneity of meetings, there are some common themes which I think when when I thought about it.

are in common between all of those types of meetings. And I I I've simplified it into five five parts. So one is to do with the purpose of the meeting or establishing a kind of an outline of the meeting, an objective

Yeah.

for the meeting at the beginning. So the the sort of the big picture. Secondly is about the inputs that are voiced in the meeting, the ideas, the the the generation of of inputs, the

the evidence that people bring. Thirdly, it's about the the rigour of dealing with those ideas and the evidence. Fourthly is how people respond to and interact with each other. And fifth is about how the meeting finishes, so whether it's about deciding or closing or considering trade offs and risks, those kind of things.

So when I look at every single type of meeting, what all of them seem to have in common was some aspect of those five. And that gave me a really nice, as it were, toolbox, which is a term that everyone in development hates because it's so so overused. But I think I th that is an apt description of of this kind of vocabulary for thinking about successful meetings.

My experience, I mean everyone has some experience with meetings. my experience, the thing that you mentioned before, typology of meetings, this is something I can very much relate to in the sense of if you can ping bring people on board before, what is the purpose? Basically, what's the common understanding of what this meeting is? Basically, is it an

Mm-hmm.

exchange? Is it about actually we need to make a decision, etcetera, etcetera? All of the things you already alluded to, then that incre improves the quality, let's say, by fifty percent. if you had

to make one single suggestion to people how to make how to have better so that would be mine, basically. What would be your suggestion how to have better meetings right from the start? The single most important factor.

Well, I o obviously I would resist reducing anything to, you know, a a single most important

Okay.

and unl unless we really okay. And

No you have to. This time you have.

that's that that's still the that that that

Specifically to preparation, maybe like just before the meeting, right? That's the question.

that's that that's the legacy of of being trained as an economist. But yeah, in in terms of an im I I think a a hugely important thing is not to see the meeting as a performance, so to listen.

And it was very striking to me when I looked at the tools on offer to help people have better meetings, several of them seem to be coaching on communication skills, which is very useful. But it but it's there's a big difference between making a presentation in a meeting and having a conversation and and a successful meeting. And I think for me that that illustrates the difference between seeing success for the individual, which might be about presenting your ideas well, and seeing success for the group.

which is about joint thinking, it's about asking each other questions, it's about reaching a common understanding of the issue at hand, and ideally, you know, some kind of agreement on the way forward, depending on the style of meeting.

Can we so you well when you talked about clear goals and deliverables and so on, there there's a certain seriousness to it, which makes me think of a corporate meeting and maybe something a little dry potentially, if you see what I mean.

Mm. Yeah.

whereas I imagine but but you've already alluded to the fact that obviously, you know, emotions, a l a lot of things are important in the meeting, which are maybe more on the human, playful side. is it' is this part of the equation, essentially,

is my question.

Yeah, I so

so let's take for example a coaching meeting. So so you're meeting with a coach and it it's not going to be appropriate for the coach to say, look, before we start, you know, the objectives for the meeting is A, B, and C because that's not the style of meeting that you're having. What would what would be maybe more appropriate to hear from the coach is look, before we start, I want to say that everything that happens between us here just stays in the room between the two of us. You can feel safe.

you can, you know, speak what's on your mind. There's psychological safety here. So it so that's a very like not corporate necessarily meeting. and the the fr for me that the frame there is not the objective of the meeting, it's psychological safety. But but nevertheless, it is a frame for the meeting, right?

No,

no, it makes a of sense. can we maybe transition to your talking about the app and and and also the catalyst mindset. I really like this name you the you have for the book because I I guess it sets kind of a philosophy which is going to apply regardless of the type of meeting that we we or at least with a broad range. yeah, can you maybe give us a kind of a quick I guess introduction to

The app you're building and what is the sort of like catalyst mindset that might be promoted through it?

Right. Well, I think this this ties very neatly to c to some of the dilemmas that are surfacing around the use of AI in particular.

Mm.

Because one way to make this kind of app, which is essentially to help managers or leaders or people in high stakes meetings to know that they're thinking better and collaborating better. One way to make the app would be a very sort of engineering approach where there is a right way to have a meeting.

to assess a meeting against it and to and to feedback what's happening. And

And maybe just for clarity, we're talking about an app that is real time kind of companion during a meeting, right? Is that is that the setup? Yeah.

That's right. That

that's right, exactly. So joins your meeting but not as a bot, so just sort of listens in to the meeting, takes a transcript, analyses the transcript, and feeds back to you hopefully a helpful output. So

And

just to understand the kind of logistics, it's like people who discreet look at their screen or something to get kind of real time feedback while having a meeting, right? Is that the modality? Excellent. Yeah.

Yes. Yes. If you feel confident

enough to multitask, you can glance away at the screen while you're in the meeting. and in some meetings that might not be appropriate. You're having a one to one call or something, you shouldn't just look at it after the meeting. or or or just an interview. Yes. Exact exactly.

Like as as a feedback mechanism for the next me for the day. Yeah. Got it. Perfect. Yeah. Please go on. Yeah.

So so the the the question was about how the how the app kind of functions and I I

Yeah.

Yeah, I wanted to draw this parallel to the way that there are several dilemmas about the nature of recommendations from AI. Because one way to have the app would be a kind of recommendation engine to tell you

Mm-hmm.

what you should be thinking in the meeting. so you know, the the for example, the app may spot, look, there was a big gap in the analysis of X. You know, here's what the right

Mm.

answer would be. And that that is a way that some people might

like to have inputs in the meeting. The frame that I've decided to take instead is more like a facilitator. And that that's that that is sort of inspired by this idea of the catalytic mindset. So if it's about getting the best from people around the table who are in the meeting and recognizing that humans ultimately are at the moment still more sophisticated as deciders and as resources of

Taste as we're going to be talking about today and various other aspects that you want to kind of bring out in the meeting. The role of an AI tool would therefore be to kind of facilitate or help to choreograph or to to sort of nurture the group that is around the table. And that that's kind of the the paradigm that I'm working with in.

So it's not an assistant for you, the person who brought the app, it's really for the the group.

It is yes. The key elements of the app are one, it as you said, it happens in real time, not just after the meeting. So you it's up to you if you want to use it in real time, but as opposed to note takers where you just get notes after the meeting. The idea of Whispers is that you get live feedback in the meeting while you can still do something about it. secondly, I've tried to make it as evidence based as possible. So the whole structure of the output, which focuses on analytic rigor,

And the way that people think and the type of meetings and what success is for different types of meetings, the way that collaboration happens, all of that is based off evidence from multiple disciplines, which I've tried to distill, still not there yet, but I at some stage you have to kind of put out there what you've got. And and the third element that

makes the app distinctive is the focus on the group and that's the one I was talking about earlier that as opposed to just helping an individual perform better in their job, this is about helping groups and teams and projects perform better.

If

you had to describe a person that has this catalytic mindset and he's basically doing now or he's basically the facilitator in the group, how how would you describe such a person?

i a as in how would I name them or

No,

describe basically h what he does or she does, how

Right.

she acts, how what are the

Okay.

kind of things she does if certain things happen in a meeting, etcetera.

Right, w what does it look like?

Yes.

I think I mean I so I've I I I see sort of the

I see this as sort of three skills is too narrow, three kind of talents or or or almost ways of thinking. So one is about clarity, second is about rigor, and third is about collaboration. So or if you wanted to phrase it sort of as you might see

jobs if they the person will be part secretary as a you know to really get clear you know what people are saying, what the facts are, how this should be noted down. part c management consultant to kind of think laterally and and have a a good thinking process. And third a a sort of coach or facilitator to get the best from from the group. so if if if that makes it evocative in terms of what they will be doing.

Yeah, no, it's nice.

Yes.

Can you give an example of a situation, maybe you have been also part of the World Bank or otherwise, where you saw such a you know, a person that is has this mindset basically shifting the meeting from going in a certain direction that was bad and turning it around?

Totally. I mean that is the genius that we all try to emulate. And and even if, you know, we there are people who can do it more than others, you know, so some people can do it 80% of the time, some of us can only do it 20% of the time. And you you kind of you know it when you've when you've done it. And and

Yeah.

maybe this is a link to what we'll be talking about later in terms of taste. It's quite

Yes.

hard to define

exactly what the genius was because it's going to be different in every situation. So it's a it that it's finding that feeling of of how to unlock something that is holding people back or how to overcome a conflict or a friction or how to reframe something that you know throws entirely new light on it. those are the people I I think we would all like to be. When we see it, we're energized by it. and what I hope the app can do is

is bring those kind of skills to more of us.

How does it feel inside you when you're in that s in that set or when you're s when you're in this meeting and you're achieving this mindset basically? Or getting into this mindset, maybe more precisely.

Yeah, I I think I mean it's very interesting school of kind of work around this concept of flow, isn't it? the the the the the concept of flow which often is about creativity but I think y it's this in this instance is about social creativity. you know when you're able to s I was thinking this morning in a in another

sense, you know, some some days I wake up feeling like an ant where I'm sort of beavering away at a small task and I've you know, I'm very close to the ground. And other days I wake up feeling like an eagle that I can soar over topics and sort of zoom down on things and pick up and then have much more perspective. My th that sort of expansive feeling where you're really able to shape a situation, not just act within it.

I I think, you know, all of these kind of feelings are are maybe help us remember what it what it feels like. But but defining that I personally find very difficult because it depends so much on the specific situation, the specific meeting, the people in the room, the mix of personalities, there are endless permutations of meetings and teams. So responding to each one of those, I think that that's what makes you a real kind of virtuoso of of meetings.

You mentioned one important thing that I think is very much underappreciated and under utilised social creativity.

I mean just stating it out I can see of I can see many applications where this would be quite useful. why is it so hard for a group to get into this space?

Mm.

So there's quite an interesting book by a South African author called Something Like Dealing with People You Don't Like. And I think that work is you know, our work life is quite an interesting space because in contrast to our friendship circles where we you know there's a bit of path dependency, but we kind of pick and choose who we who we spend time with.

Work is maybe a bit more like family where we've been landed with, you know, several colleagues that we do like and others that we don't, and some people that we find very difficult and and so on. So I I think it's intrinsically very challenging to deal with people that we don't naturally have an affinity with. for me that's that's one big reason. What w what do you think? 'Cause I I th I I I sense in asking the question you you thought about this too.

I'm not sure. I would say I'm much more observing it. and your your comment or your your your answer basically made me think that it is surprising how l how how how few times we actually get into state as a group to really go into this let's do something together and be creative. Very often we sometimes there are these meetings in corporate c contexts where it's we are going through the motion, right? It's basically opposed from the top, let's have a meeting and let's try to ideate something. I mean

By basically if if it is announced by that you already know it's not gonna happen, right? And I I'm I was just struck by your comment basically that I think we know we're very much focused on the individual creativity, but this component

Mm-hmm.

on the social part is often missing.

Right.

and I think I mean for me it's for me one thing is clearly the mix of people. There are some there are some settings with people where it's just very hard. Then it's obviously also the size of the group, clearly. Too too big, no way. Or very hard. My my subjective

expression. And then there is basically I think also this almost like spark. When I had these sessions with people, I mean, coming out of academia, this this happens obviously, but it's very often that you start with something and then there is something happening in the meeting that pushes you is creating the spark basically where everyone is around, cool. Wow. You know, everyone is enthusiastic sorry, enthused about it

Yeah.

and takes it up from there.

Yeah. Yeah. I mean I I would build on that because w one thing that strikes me when you're when you were talking is also that when you find social creativity, if if you're lucky enough to find it in a meeting, that is not ne it's not necessarily what leads to the best thinking.

So

that you know, this is the just coming back to this idea of what makes meetings successful in these three lenses of clarity, rigor and collaboration. So let's say you have very good quality collaboration where everyone's talking to each other in a very enthused way and the spark has happened and people are very excited about it. But that could be exactly the situation in which you get complete groupthink where everyone's very excited about whatever idea that's

Mm-hmm.

Hundred percent, yeah, yeah.

been early convergence, people say, Yes, no, let's go ahead and do it

Mm-hmm.

have

Has everyone thought about the risks? What are the trade offs? Were there any better ideas? You know, have we decided on this too early? So

Mm-hmm.

all of those kind of th is a a brilliant illustration of why having the the best quality meeting is so difficult because these things are not

Yeah.

necessarily working in tandem, they might be going in opposite directions.

Yeah, you convinced me. I mean,

Yeah.

enthusiasm is maybe a necessary condition, but certainly not sufficient for reaching anything.

Yeah. You I

Yeah.

think this is a fantastic transition to towards the question of AI and taste because what you just highlighted is an example where I imagine if you took if you took a meeting where people just I don't know became friend and this is a lot of laughter and fun and everyone in the line, but there's a lot of group thing, it would be very easy to just you know copy this transcript to Chat GPT, ask Chat GPT how did the meeting go?

And Chat GPT being like, it's amazing. People are people were very aligned, very happy, and not complaining, right, about the group thing or something like that. So and and so that I I I get kind of like the the sense of what you're trying to do is getting AI to maybe borrowing some of your words, kind of optimize for quality and wisdom as opposed to this kind of more

Exactly.

sycophantic thumbs up that yeah, everyone was nice. You know, you you're trying to go for something deeper and yeah, I'd be curious to

Maybe here a a bit from your experience, like how hard how hard is it for you, for instance, when you're building this app to kind of get AI to really understand all the c all those kind of subtleties and to really get into the the nuance and kind of wisdom.

Yeah. well I think the first point is that I think there is a lot that we can do by asking the right questions of a transcript and a s and that is a large part of what Whispers does. So if I were to paste in a transcript, as you say, into Chat GPT and ask a few questions, and let's say my questions were, was the group collaborating well or was I, you know, w was it a good meeting? And and the flavour was about social creativity.

And as you say, the response comes back, great meeting. so if there's a much more highly structured question to Chat GPT about, you know, how was collaboration, but how did that weigh up against rigor, then you'll get a better quality answer. And then if rigor is defined in in a more precise way, there'll be an even better answer. And if rigor is defined

with reference to ten different frameworks, and each of those frameworks carries evidence and that which is essential essentially how I've built Whispers. You can see where I'm going with this, then you'll get the best quality answer. So that that is my hypothesis in a way. that the the it the AI in its current form is is already able to to push so far ahead of where we are as groups of humans working together. And that's what I'm

Kind of trying to break

I I I guess

yeah, I guess it's you're touching on the question of context, I guess. Maybe I I don't know if Philip wanted to also expand on that as well. But I guess one of the points you're making here is that okay, if you if you explain everything to the AI, if the goal is super clear, if it knows more about you know the people in the room and so on, it's more likely to help than if it doesn't. That that's I think that's quite intuitive.

But somehow my intuition is that it's not enough. Like that even with really good prompting, there will be moment today where AI is just not that good yet. You know, right? And I'm I'm talking about this because so obviously we in in this podcast we talk to variety of people, but you would have finds some writers who would talk about how, you know, even when they spend a lot of time explaining to an AI the their style, their way of thinking and so on, it's still the output is still very bland or still very like soulless, right? And

I I I you know, people especially people with I guess a philosophy or that's kind of unusual, you know, where when they try to just ask an AI to implement it, it's it's not always very good yet, right? So th that's where that's where I'm curious where you currently see kind of limits in in where AI is maybe very good at doing exactly what you want or where where it's even with the context kind of disappointing.

That's right. So

I think we're getting towards topics of taste now, right? So

Yes. Yeah.

yes, absolutely. I mean on the at a as it were superficial level, I've definitely found that there is a lot of calibration that has to happen in in the output. So stylistically, you know, how is what what what what what kind of register is used in the output? If you'll have a professional meeting, you don't necessarily want a chirpy kind of grok style

output from your large language model. the the kind of temperature of the suggestions that are given as well. In some meetings you will quite appreciate out-of-the-box thinking. In other meetings that's not so appropriate and you want to kind of delve deeper into and and and be precise. the but I think there may be a a kind of

Maybe

it maybe a deeper issue for me and one that I don't necessarily have a very good answer to is the relationship between taste and judgment or taste and expertise or even taste and bias. So I you know, in preparing for this podcast, I I started thinking, well, you know, if we're trying to get good taste from AI, that depends on experience and expertise and that

obviously is taken from a training data set. And then there's a sense of judgment about when to apply certain things and when not to apply them, which I've been talking about just now about calibration. But that's not necessarily taste. So you I you know I was thinking you can have instances of good taste but bad judgment or bad taste but

But it with good judgment. I maybe I should give ex I mean a v a very facile example of this. the the first one that came to mind actually is like an ugly sweater party at Christmas, which is all about bad taste, okay? But you show a good sense of judgment in wearing a bad taste sweater because that's the idea of the party. Okay. Do you are you following here? Or

But isn't there isn't there a level of meter taste on top? Be tasteful in the t bad taste of choosing. I'm su go sorry. Sorry. Sorry, go on. Go on. But I g we get it, yes.

That's that's quite a good get out, Clause. Yes. Indeed. Indeed. But

so in in matters of taste, I mean I yeah, I find it difficult to decouple the sense of taste and judgment. in I find it easy in those examples, but I find it more difficult in defining exactly what would be different. And so w when you when you're when you formulated the scope for this podcast as the taste bench,

How would you say it's different to the judgment bench or the the bias bench, for example?

mean speaking for myself first maybe I don't think there is at the moment we we do we do want to make a very strong distinction between these things because we believe there is a a cloud around taste and people obviously interpret and define taste in different ways. Some of them more or less equate them with judgment, maybe also depending on the kind of things they are evaluating and thinking about. but there is no point for us at that stage to say this is that or not. So we are broadly interested in the overall s surroundings of it. Now I think the distinctions you made

There is there are distinctions in the in the way you apply it for specific in specific domains as well. In some domains, judgment is closer to what we normally call taste. In others, it might be quite different, right? So for instance, a lawyer might have taste on a certain level, but judgment might be far away from it, right? It's not

Right.

necessarily the same thing or I mean, if you know what I mean, there is an aesthetic or a kind of a a knowing, almost like a chess player also has an an understanding of of what is going on, identifying patterns, whatever you want to refer to.

yeah, sorry, did I did I actually answer your question?

Well, it I mean i th i I think it's a it's a messy it's a messy kind of realm of of of of debate. So yeah, not not fully but yeah, it gives me some ideas.

Yeah. Domain, in a way. Yeah, yeah, yeah.

Domain. Yes. But I think this is this is the re the this is the reason we want to

If I c yeah. Yeah.

have this this debate exact or this kind of podcast because we want to exactly explore how people actually react to it and what their thinking is about these things. It's exactly that, that it's not it's not a skill like some people define it now. It's also not judgment alone. And we want to have

Yeah.

this discussion exactly for actually getting different perspectives of how different people in different contexts

Right.

react to it.

Right.

Yeah.

Or maybe to to add to that is maybe there's some element of skill to it. It's just like just not a skill. I

Mm.

I think that's the so for instance, let's take music, right? Some people have no taste in music, they don't listen to it, they don't care. The same way some people have no taste in football, they don't have a s favorite team or anything. But usually when they they start developing taste in a domain like this, there's there's many things that happen. There is

some aspect of preferences, just like op opinions if you want, about you know, what's a good song, what's not. there's usually an aspect of skill in the sense of, you know, the discernment aspect, you know, the being able to like recognize subtle differences between different things. there is a certain confidence as well, right? So there's basically there's a set of qualities that tend to go together when someone is typically described as having taste in a s specific domain, right? And it's I I I think in in in the case of your work

Maybe you might prefer word like expertise or you know the wisdom, you know, to to qualify someone who's a very good, I don't know, meeting facilitator. but but I think the the the shape of it is quite similar in the sense that you know for someone to be really, really good at I don't know facilitating a meeting, they need to have really this kind of fine grained discernment and they need to have this sort of clear kind of

line of deciding okay here I'm gonna give feedback because this is right. Here I'm not gonna give feedback

Yeah.

because this is because this is wrong. You see what I mean? That there's

Yeah.

almost some sort of agency and like confidence into like making judgment about things. And and one area where we feel AI is often very disappointing to today and we suspect it's gonna get better with time, is today it's very what's the word, jagged? Like it's it's kind of it's kind of a bit random about how sometimes it's just gonna like very strongly criticize something and sometime

not and you know it it seems to be missing this kind of grounding and this like a really clearly shaped opinions and and and skills. Does that that does that make sense?

Yeah, I mean so so for me, something that we haven't touched on yet, but I think is very important is that at least to me it seems taste is very personal, while the other concepts that we've talked about, judgment, expertise, experience and so on, are objective.

You know, so your your experience obviously is based on

Mm-hmm.

things happening, facts, knowledge and so on. your expertise is your ability to deploy that previous experience.

Your sense of judgment might, you know, that there might be a a a metric of success in a certain situation. Did you make a good judgment call? So those those kind of things are objective. Taste for me is very personal. it's subjective. You know, things that for me are good taste, my choice of shirt today, for other people would be very bad taste. and I think that makes it fiendishly difficult to sort of optimize something for taste because it

Whose taste? The other thing that I think is very interesting is that you start getting into questions of power dynamics as well. If you know to to optimize taste, you are standardizing something which is very diverse and you're you're taking you're privileging a certain taste above other tastes.

so I don't know if this is too goodness me. What happened there?

You're

mute sorry. You're muted.

It's almost just to check if we were paying attention.

No no no it's

Apple's automated

Reaction.

emojis, yes. You can ignore. Excellent. We'll see, we'll see.

amazing. Okay, well I I I definitely enjoyed the balloons. but just

I think maybe we we can cut this out later.

Well if not then cut me out when I say we cut it out.

but just

Okay. I'm

I'm I I know, go on.

but just but

just just coming back to institutional taste, which we you know we we we touched on before, to be more evocative about this. I mean I remember the World Bank. I there there could be people who are very experienced and have a good sense of judgment, but peop but their colleagues thought

w would do, you know, would make a project in bad taste or would have a bad way of running a meeting. there was also a sort of institutional taste it which you have to learn when you're in a large organization. So so for example, people were often told this project is too complicated. The the kind of metaphor was it's a Christmas tree project, there's too much going on. You need something which is much narrower and transformative. Well

We don't know which one is the right answer, but definitely the institution had a taste towards, you

Mm.

know, not to have Christmas tree projects or to have projects which are very fast dispersing and in the words of many colleagues, get money out of the door, which I think is a very poor taste thing to say when you're dealing with a loan of millions of dollars or hundreds of millions of dollars of taxpayers money that they have to pay back to talk about getting money out the door, I don't think is good taste. but

So I you know, it's i I I see it as quite a complicated area.

Yeah. No, I I find it's fantastic. I yeah, I'd love to yeah dig into that. let's stick to that definition of taste. I think it's very good.

I want to just connect two things here. and then see basically how this floats with you. Or three things actually. So the last the last thing is the institutional taste. Before you said taste is very personal. And then a couple of minutes before we talked about collaborative social creativity. and I would like to explore a bit this this kind of connection, how taste formation works for individuals and the social aspect of it. And let me start with

Mm-hmm.

two examples. So if you read a book for yourself, that's very insightful. But

What I learned, and I think this is quite universal, if you read a book and then have someone else who reads the same book and you talk about it, there is an amazing thing happening. That you learn something obviously that you totally oversaw. But I what I would say even more is that you are learning maybe even a different grammar of how to think about this and maybe also form your own taste about things. In a very concrete setting, also I mean, wine tastings are similar like that. I mean, I had this very, you know, long time ago, this experience of there was a note in a wine that I could not express.

And I basically gave it pepper. And then someone said basically casually, yeah, this wine has a la la la taste. Like cassis taste. And that moment is i actually clicked that yes, what I was referring to as pepper is more refined, actually more precise, is to think about it as as cassis because that's what it actually is, right? Black current. The third level is then the institutional connection. That was not a question, it's basically just an observation. How do you What's your thoughts on this?

goodness me. So on the See I'm not sure how to respond 'cause that's the that's it's it's very interesting. on the on the point of the sort of reading a book and appreciating things, w which I what I took from what you said was appreciating things in different ways. I is that sort of constitutive of

As well as learning, and this is again touching also a bit on judgment, learning

Mm.

different ways to think about it, maybe even different evaluative frameworks, like how people actually are even conceiving of thinking about books.

Mm.

And then you learn basically, this is a different way of looking at it. And then basically this might be part of your own process of thinking and evaluating things and judging, and then maybe also kind of teasing out what you find good and bad.

Yeah.

I mean the the the the the the the other example that I think is germane to this before I come on to a kind of more interpretive point. So there were I listened to a podcast the other day with Brian Eno and and Ezra Klein and talking about, you know, a a very easy observation but one that I think is very powerful listening to a piece of music, how two people can listen to the same piece of music and take away completely different feelings

Mm.

and sort of inner mental worlds caused by that same piece of music. So it I think that returns us to how subjective taste is and how our experience of, in my case, the same meeting or the same project or the same client relationship can be interpreted in in many, many different ways. How how do we deal with that in terms of navigating the world and sort of

doing better at things. Well, I guess that's the playoff between taste, which you know, some people might have liked and other people didn't like, and then judgment, which can be obsessed can be assessed objectively. So, you know, regardless of how you felt about that particular meeting, did it generate the outcome that you wanted, which was, you know, whatever, making the sale or

finding a good idea or, you know, connecting people together, whatever whatever the the objective was meant to be. so yeah, I d I don't know if that makes I I I think all of this implies to me that dealing with matters of taste is is horrendously complicated and and ultimately is only part of the picture, but nevertheless could be an important part.

is w we feel that taste is lacking probably more than other aspects of of AI in in in terms of gauging how how good is it.

Let me maybe I was maybe unclear a bit, sorry, Pruno.

Good.

what I see basically the the social collaboration and social creativity aspect. I think this is part where AI can support the process of forming your own your own and learning about yourself from other people is obviously still, you know, very subjective, but also this kind of could be leveraged. And that that's a connection that I see basically, that you know, p taste is not fully static. I mean, actually I would say it's it's quite the opposite, but it's developing over time. And

Y people are learning, right? And I think very often you can learn from other people who have taste.

No.

and then the question is in some sense technology sitting in between.

Is there a scope for it? And one scope I I see is is exactly this. if it makes us

Together here, exchanging the views and how we see, how we perceive it, and how we evaluate how what our framework of taste is, essentially, this would be going a long way.

Right.

w one one idea that I think is quite exciting for the role for AI in kind of social learning, which which I I sorry if I'm still not getting the way that you wanted to frame this, but that w

something that I think is a very exciting opportunity is the role of AI in interpreting between diverse human frames. So let's say that taste my experience of taste is very different from yours and

in some cases could be dramatically different to the extent where we find it difficult to talk to each other. We use different words, we we have

Yes.

different concepts, different conceptual frames, different backgrounds, maybe even different meanings associated with the same words that we're using. So it's sometimes quite hard to talk and reach common ground because we're simply talk talking past each other. I I've already seen that AI can be incredibly helpful in

in rephrasing or summarising those separate frames in a way that it allows

a much better interpretation across those frames. So kind of conceptual interpretation.

And understanding then from the other side, right? I see basically. I understand better

Right. And understand

how you think.

and even so when it comes to matters of taste, maybe that's also a useful role. So it sort of embracing the subjectivity of taste to the extent where the a role of AI can be to can help interpret things that o almost seem surpr I mean, that's when you know you will be dealing with a real expert AI system in matters of taste, where it can not only have good taste for you, but it can also interpret between different taste frames exactly.

Translate. Yes.

I think, yeah, we're getting very close to one of the crux, I think, which is I'd like to ask, I guess in somewhat possibly challenging way, to see kind of the success scenario and the failure for Whisper, right? So in my mind there is like basically possible failure mode and possible success mode, let's say. maybe if we're very optimistic about AI and like if we think the problem is simple, we could just say that.

Maybe there are some fairly universal rules about what makes a good meeting. And all it takes, we just need to give it to an AI. We give it like the five golden rules. And then AI will remind people of these rules and meetings will all be great. Right. That would be the kind of reductionist rule saying, you know, meetings are kind of simple, you can formalize it and so on. And then you could have a more pessimist view. I would say that would be the people I I guess, I guess, the bearish case, I guess, against Whisper, which would be to say,

You what, meetings are so complex, it's so hard to really know what the real goal is. Like it's so hard to formalize what everyone's preferences might be during a meeting, that you know, Chat GPT is just gonna speculate and it's just gonna be some random feedback from an AI. the only real thing that works is to have actually an expert from the World Bank in the room, someone with taste that has, you know, like really understand the nuance. What could be the what is closest to the truth, you think, or like what would you say to both?

Claims or cases.

I I think it depends where you're starting from. So, you know, if you if you're starting from a rather low level and your experience is that meetings are generally disasters, very awkward, difficult, uncomfortable, then

Yeah. Yeah, yeah.

probably some kind of recipe book and that there are plenty, could be helpful. So, you know, going back like eighty years, there's the

Is it called Rogers Rules, something like this, Roger's rules of order for meetings? So there's very strict rules about, you know, f even phrases to use in a meeting, which now obviously are very unfashionable to take that kind of approach. It seems quite outdated. But if you're starting from a status quo of total chaos in meetings, then something

Mm.

like that could be helpful. if like most of us, you're starting from meetings that have some good features and some very frustrating features.

But you can't quite put your finger on it and it's different in each meeting and so on. Then I think we get to the realm of knowledge, which I have always found very inspiring, called phronesis, as Aristotle titles it, so that which basically means practical wisdom. So rather than just thinking about scientific knowledge, a kind of recipe book, or even like an implementation guide, how to guide, which is technique, a second form of knowledge, talking about pharonesis, which is a

A kind of practical wisdom, essentially how to deploy your heterogeneous, very deep, rich knowledge experiences in a specific situation. And that's one of the most difficult things. Now, paradoxically, I think that's where modern AI excels because it's

no longer about you know a computer taking algorithms and spitting out, you know, going through a decision tree and spitting out a right answer. It if it can be about pattern recognition, you know, the way that machine learning works, then you get to the that's essentially how anyone becomes an expert or a real virtuoso is by being exposed to such a large number of situations that you then know how to

Bigger. Yeah.

how to you know what the feeling is.

of how to respond in this new situation that you've never come across before. And it that's definitely not in your instruction guide or your how-to book, but you know, drawing on that mix of experience before. And that that's what I what I think machine learning can do. So obviously that requires a very large training data set of meetings, or it requires a very sophisticated use of large language models. And that's what I'm experimenting with.

So in terms of whether Whispers will succeed or fail at that, let's see.

No, no, that's fantastic. No, I I think I completely agree with your intuition there, which is what we have today is already a step change compared to like what computers could do before when they were more logical. we are we're definitely in the realm of more flexible ways of learning and applying like you said, pattern recognition.

Mm.

and yeah, there's reason to believe it's gonna keep improving, right? That that machines will get even better at learning fast and so on.

I I guess it's the maybe just just one point yet that I'm still not I I I still hesitate about a lot, which is this question like the again like formalizing the goal of a meeting. Like I've I've been to so many meetings where you know people break out for lunch after the meeting and the just the opinions about the meeting are so different because you know people wanted different things or people you know the ultimate goal is different and so on. how can I I I'm still I'm still wondering how do you

or you address this without like I guess what's your experience in practice so far? Do you do for from your experiments with Whisper so far, do you feel like it it's been smart enough to figure out what the right goal or the right type of meeting was by just listening in? Or does it take a lot of effort to kind of explain?

in short, no. and but I you know, I I th I think it depends what we what we expect or or want to find. Do we want to end up with a perfect situation where we have a you know a hundred percent maximized success of a meeting every single time? Well, ideally, but it's probably not realistic. But even getting from a the current situation of fifty percent, you know.

And and surveys kind of suggest that that people in general think that about half of meetings were a total waste of time. So if we can get from fifty percent of meetings being worthwhile to eighty percent, that's already a massive step forward and would translate to billions of dollars every year or or more than that.

I see what you mean. It doesn't have to be even one percent improvement at scale is it's

Right. Right.

can be a big deal. thank you so much. We we have taken quite a bit of your time. We're getting close to the to the end maybe, but I I guess maybe just as as a closing question, I I'd love to if you could I don't know, share some happy experience you've had with Whisper so far. Like examples where you found that it was actually really useful and like it's

you know, proved to be really exciting, just to get finish on the on inspiring note.

Yeah, I think I mean

just just for a from a from a personal perspective, and this is a mix of inspiring and you know slightly depressing, I am endlessly

Ha ha ha.

amazed by my talent in having a complete blind spot in a meeting. So I can go through an entire discussion and then that feeling of afterwards thinking, you know, if only well,

yeah, I forget that. Yeah.

how do we not cover that? So

Trust us, we

know that very well.

Right.

Yes.

Now and I think this

Yes, yes.

this is the this is part of the human condition, you know, that that as we sort of decompress after a meeting, or even the next day, or we wake up and we just realize something should have been totally different. Well, I I have had experience already of whispers being able to identify a an a massive blind spot even within a meeting. so the the way that that happens is not

sort of mystery in a sense. It's just a question of going through almost like a checklist for for the ways that a meeting should hold up to scrutiny. so th those are in in summary the five things that I mentioned very early on about the frame

Mm.

of the meeting, the generation of new ideas, the way the evidence is is tackled, the challenges towards different ideas in between people, and then the end result of

the meeting in terms of a resolution, conclusion, trade-offs and so on. If you go through some of those and and there is a a similar set of of kind of frames for the way that people collaborate with each other, drawing on on different evidence and and research. so the way that groups come together, the way that you engage with people of different personalities, the way that you tackle, you know, a a kind of conflictual situation.

it you might be completely blind to something in a meeting because you're single mindedly focused on something else. But Whispers will give you well hang on a second, a little nudge in the meeting, you might not have realized, but A, B, and C. And and I've already had that that feeling of epiphany in the meeting of being like, Yeah, that's absolutely I don't know how we missed it. But you know, so when when I've been

Excellent. Mm-hmm.

working on something, when I'm doing a project, when I'm engaged with a particular team

when I'm discussing an idea, you know, that's and now unfortunately I didn't turn it on today because I can't have I can't I I can't have

I was gonna ask.

No

this recording software working at the same time of Whisper. I tried it last

yeah. Yeah.

time and something is mutually crashing each

Yeah.

other. so but I will be running it afterwards on the on the recording.

Excellent. And

We should actually ask

for trying it out on our recordings. And you should see

Yes. Yeah, yeah.

you should tell us basically what the blind spots are and what the things that we missed.

Okay.

That that that that would be a pleasure. Yes. so for anyone

Awesome.

and anyone who wants to look at the beta and test it themselves, go to whispers.tech and and if you click on try it now you can download a beta for Mac OS at the moment and I'm working on an iPhone releasing an iPhone version soon and thereafter other

Excellent.

Thank you so much for your time. Real pleasure.

Perfect. Thank you

very much, Austin. It was a pleasure.

Thank you guys.

Yes. Yeah, very, very stimulating and I've got a headache now.

I'm sorry. god.

No

Maybe this is the metric of our success, I'm not sure. Did our guests leave with a headache?

Yes, totally. Totally. Yes, but it but for my

particular taste that's probably a good thing, but for other people maybe not. Yeah.

Okay.

Excellent.

Thanks for watching. We've got more episodes in the oven, so please like and subscribe to get notified when they're out, and please check taste-bench.com to read about the taste benchmarks and the creative engines we're building towards.