The transformer based approach is seen as a highway to race down in terms of training and building better models with minimal organizational distractions allowed while every company puts its foot down on this highway. Exploratory research into other AI methods is highly discouraged for now and organizations are encouraging transformer based scale out and iteration as a top priority.
As an example Meta literally fired it's previously head of AI who held views that transformers won't scale. He wanted to veer off the highway and heavily fund exploratory new types of models. They got someone willing to scale the current method out. Essentially the messaging it "Foot down, there's a straight highway ahead don't divert".
I have the view this makes sense too. Current top models are mere (mere in big datacenter terms) trillions of parameters right now. We definitely haven't hit the ceiling and everyone's noticing "Hey this Opus 5.5 with the increased parameters seems much better than the last one". Well that's a trillion parameter model. We'll soon start hitting 10's of trillions and 100's of trillions of parameter models. Unless there's a ceiling which nobody expects they should keep scaling just as they have been. I really do agree that it'd be insane to veer off this highway with exploratory research at this particular point in time. Foot down, get to 10 trillion parameters, see if the scaling holds, if so off to 100 trillion parameters. If the scaling continues to hold we'll be asking the AI to do the exploratory research for us soon enough.
> As an example Meta literally fired it's previously head of AI who held views that transformers won't scale. He wanted to veer off the highway and heavily fund exploratory new types of models.
Almost all statements Yann LeCun made back then (LLMs won't be able to do this or that) turned out to be wrong. They were right to fire him.
Even if he was wrong about those statements, exploring other types of models isn't necessarily a bad thing. Especially if we can find ones that perform well with less demanding hardware requirements.
Its not a bad thing, but exploratory research as opposed to reaearch that finetunes along a path already paved by exploratory research is something that doesn’t usually happen much in for-profit product-oriented firms when there is such a path available; it happens in academia and corporate labs pick it up once it reaches the point where the basics show commercial promise. CS is a little bit of an exception because the capital cost of basic research isn’t, proportionally, as high, and because the promise-to-product path is often short enough that having done the basic work in house can be a meaningful advantage in getting to market first (though, see Google on much of modern AI for a counterexample).
But even in computing, when there is a promising way forward that the basic reaearch has been done on, its hard to justify side quests.
The issue isn't the exploration itself, it's the opportunity cost. When a figure of Yann LeCun's influence pushes a specific direction, it pulls key talent and resources away from mainstream LLM work. That kind of organizational distraction has a massive cost, and would've stopped Meta from delivering a polished product like Muse sooner.
The mainstream is wrong and in a lot of ways hazardous. He had every right to say he wasn't willing to sign is name on the wrong technology when he new better. When you hire smart people you listen to them. Elsewise you get what you deserve.
Absolutely agree fwiw. Scale out over LeCunns desired exploratory research was absolutely the right call. Called out because it seems like MS is making a similar call.
The author states "I consider the scaling-only focus to be both misdirected and not fun.". I get the point but also totally get the focus on scale out for now.
I empathize with LeCunn but he is a prime example of why scientists often make poor leaders. It’s too tempting to always do the cool and interesting thing instead of drilling down on the boring but reliable thing that actually creates value. There is of course a place for this kind of eternal curiosity and I hope he’s found it, but hyper-competitive industry may not be it.
I'm becoming more sympathetic to LeCun's views, but being able to train something that acts the tiniest bit like those who made the training data (authors, professors, researchers, etc) is an absolute juggernaut and will be for some time.
TPS (and Lean, which it played a role in inspiring) is largely about finding what works empirically and ruthlesslessly optimizing around it.
It does (Lean at least) incorporate controlled experimentation, but tightly controlled with a both a laser focus on what the goal is and tight resource constraints. And Lean (at least several writers on Lean) favors taking multiple paths with a a quick and low risk and a longer-payoff, higher-risk, higher reward path for some efforts—one source used the US Navy’s two-path approach to SLBMs with Polaris and Poseidon as a model for this—but that's different than “maybe there is something down this path” basic research, which is outside of the concern a production model like TPS/Lean and often completely outside the model of a product-oriented firm; basic research is something firms consume and incorporate more than generate, because until the basics have been proven out, the rationally perceived chance of any particular avenue paying off is near zero for any firm.
as fast as possible implies spending the resources that way.
Going close to as fast as possible while saving a stream for the other approaches makes sense as an approach, but let's not ignore that there's a compromise/tradeoff here. Might not matter in orgs with near infinite resources
Except doing both..is..so..hard when the future payoff of one is more unclear and you are faced with quarterly earnings reports.
Look at what happened to Meta where they squandered a lead, had some mixed results, and reorged to more product certainty and away from fundamental research.
This is a dumb question since I know very little about LLMs and transformers but I keep hearing similar things such as "more GPUs/parameters/compute" leads to a predictable scaling in intelligence.
Is that observed scaling part of some equation or law? Is there a limit to how far that scaling goes before a ceiling or diminishing returns hit? Or is it simply based on more GPUs/parameters/compute plus how much human knowledge is in existence and will scale predictably forever?
Yes, the Chinchilla scaling laws from 2022: https://arxiv.org/abs/2203.15556 More parameters directly and mechanically means less prediction error for each token.
(This moves the argument to "is being really good at predicting the next token the same as being intelligent?" which has happily diverted many minds for years now)
I'm seeing the same pattern at other multinationals with huge R&D departments. Mostly colleagues with academic pedigree in traditional ai research who have been for years in denial that transformers are actually - in combination with recent advanced patterns in software development - the key to ai. it's really simple as that! except they weren't part of the revolution and still don't want to accept the reality that their research has become obsolete.
Is it the key to AI? It probably depends on the squishy definition of AI and what different people want from it. If your definition includes self awareness and sentience I’m not sure it is the key (I’m also not entirely convinced it isn’t, either), but it’s not clear either way in my opinion.
The one thing that is clear is if it can become more cost effective it’s world changing. Even at its subsidized standards now it’s world changing. And it’s reasonably classified as AI IMO, but it’s not self aware.
I think a lot of your general traditional researchers are fundamentally looking to understand ourselves through machines as proxies in a sense. Something self aware that senses, learns, adapts, so on. The current path isn’t there or necessarily clearly the key to achieving that. That may be the disconnect. I think LLMs may give us some hints at that but not the full picture.
Still incredibly powerful for things we once considered nearly impossible.
is scaling working? It is also my understanding that Opus 5.5 is probably a well under 10T parameter model. Probably a similar size class to Fable 5 earlier. And supposedly even GPT-4 was a 1T model. Clearly most of the gains are coming from something other than scaling up the model size. Yes, scaling up RL counts as scaling too, but the fact that labs have pivoted to a different axis of scaling is evidence that it's not just a "straight highway" to improve LLMs.
It's worth it to keep exploiting transformers while doing just a bit of exploration of alternatives because they work so well. Heck, just exploiting transformers more will accelerate exploration of alternatives...
100 trillion parameters doesn't make any sense if your dataset is ~30 trillion tokens. My uneducated guess is 1/10 compression ratio is good and that point to ~300 to ~1000 trillion tokens, which again, doesn't make any practical sense.
Depending on how you count, the human brain is between 100 trillion and 1 quadrillion parameters, which is one guess as to why we're so much more sample efficient at learning than LLMs are.
Unless quantum effects do infact play a role in consciousness. It’s not clear they do by any means, but if it did that could shift the understanding of our own minds and consciousness drastically. Otherwise we are just a dense complex network we can reproduce with enough money.
I mean we literally do this every few seconds already with natural birth processes.
When a company pays billions to OpenAI, any activity that does not do a glorious dance in praise and support of that investment is taken to be an act of disloyalty.
This doesn't engage with the article in the precise way that transformer-based improvements and fundamental research that way are not being incorporated into models, at Microsoft AI, which is concerning.
It seems extremely likely to me that scaling produces benefits, it will get better at particular things things continue but scaling by itself isn't heading to general intelligence.
I should also say that I use ChatGpt daily and enjoy it for researching algorithms. I tried to set up Claude last week and it/he was so overwhelming as to be unusable. So I think frontier models can be "unwieldy" or something for average use.
> Foot down, get to 10 trillion parameters, see if the scaling holds, if so off to 100 trillion parameters. If the scaling continues to hold we'll be asking the AI to do the exploratory research for us soon enough.
Where's the money coming for those 10X and 100X ventures?
> Current top models are mere (mere in big datacenter terms) trillions of parameters right now. ... We'll soon start hitting 10's of trillions and 100's of trillions of parameter models.
Chinchilla Law says no, we wont. We're likely well past the point of diminishing returns but MOE hides a fair bit of it.
We already know that power laws apply to scaling so 10x the size is going to result in minor gains - where costs become a real factor (as if they weren't already).
FTA "The core philosophy there is scaling, and aggressively not research."
This is where the real gains for AI come from - good enough and CHEAP is going to win the day here (brute force, mixture of models). It's the engineers displacing the academics - and that will mean a bloodbath for revenue (see DeepInfer for what the end game looks like).
I'm not an AI booster (in a "should" not "can" sense), but in terms of "can" we are going to see next generation hardware (i.e. Vera Rubin GPU architecture) and improvements in model density (i.e. loosely improvements in intelligence per millions of weights), so the models will get bigger, the models will get more dense, and we may see more "expert models" i.e. in the sense that for a while there were human generalists that knew all of science and mathematics until the 1800s, after a certain point we had to specialize since there was too much to know. The models will need to partially specialize after hitting the hardware ceiling.
Bioinspired algorithms may eventually lead to vastly improved intelligence per a weight too.
I don't think we should do this, but I feel doomed in that I suspect that we "can".
> Hey this Opus 5.5 with the increased parameters seems much better than the last one"
Is this true?
I barely notice any improvements ever since... 4.6? At least for software development.
Opus makes the same sort of mistakes and has the same sort of strengths ever since that point in my perception. My impressions is that increasing the number of parameters clearly has diminishing returns.
In fact, the best improvement I got from Claude was Haiku 5.5. Good cheap workhorse to implement plans.
As in your statement to me indicates that you clearly see the scaling working over longer timespans. You chose a relatively recent 2026 model as your baseline after all, not a 2025 model or earlier. You clearly see that the 2026 model is better than ones in the past.
Might i suggest that you perhaps just don't see the improvement as clearly when comparing iterations that are fairly close together? If you're not anecdotally seeing improvement between an early 2026 model and one from ~6 months later perhaps just give it another iteration? Personally i do see 5.5 as much better but i can understand not seeing a lot of difference to 4.6 since they literally aren't that far apart in releases.
This refers specifically to "AI Frontiers", a research group within Microsoft Research that focuses on fundamental advances in AI. There is no end in sight to real AI frontiers.
Perhaps "The End of Microsoft Research's AI Frontiers Group"
would be a less ambiguous title.
Yeah, I went down the rabbit hole scratching my head the whole time. This is about the group at Microsoft Research that was disbanded or reorged into what is now Microsoft Research.
To the author: 100% need this kind of thinking. It’s a balancing act. Historically though, dedicated labs haven’t had the best track record at commercializing their tech. The skillset differs vastly from running a product-focused business.
That said, even within this narrow scope - there are areas ripe for research.
Some of my particular interests include:
(1) Latent thinking/reasoning, which could arguably be more efficient but also opaque is one
(2) Separation of “intelligence” from knowledge - ie. keep the ability to reason like a frontier model and make tool calls (calling the ‘oracle’ or searching the web) but remove all parameters associated with niche knowledge/facts.
These are a couple off the top of my head that are very much aligned to scaling.
That statement is broad enough to be unfalsifiable, but if you narrow it down just a bit to scaling LLMs, maybe there is an end in sight. At least financial markets are skeptical of there being no end in sight.
If scaling LLMs really was the road to God in a box, then compute demand would be infinite, and nobody building to meet that demand would have a junk debt rating. Did those money people just not get it?
Companies with billions of $, stopping R&D on the most revolutionizing technology/ breakthrough is something I can’t understand. I do understand there’s an AI Race to win, and that is with transformer and current LLMs architectures, but if research could find a better, more efficient approach they would re-shape this race, lead it, or even dominate it.
But is it a race to zero? Maybe AI will make very expensive things (eg software engineering) very cheap, but there will be enough competition that the margins are tiny and don't actually justify putting crazy dollar amounts into research
That's exactly what's happening. Software -> 0 marginal cost (obviously still need engineers to actually make the architecture). LLMs -> Price of Energy / Amortized Cost of GPUs. "AI Labs" -> Owned by Creditors
To me, all signs point to US over investment in hardware capacity to support the "frontier" when in reality the next major breakthroughs are likely radical optimizations that end in increased consumer training and inference efficiency.
I'm not sure how that, combined with open models inching up on frontier intelligence, is compatible with these massive AI IPO valuations?
It seems like closed model AI demand has a lot of contingencies that are ultimately fragile and possibly short sighted.
Yeah, I'm speaking to "scaling vs research" and where investment is going. Which is relevant to the article. Accelerating scale requires demand, closed frontier demand is what I'm questioning.
And gp's comment is relevant to the particular problems the guy (John Langford) cites in his comment about leaving. IE, that research first winds being only about scaling and then winds-up ignored because "just scale" can be an easier answer.
He didn't say they were not doing transformers research. In fact, he specifically mentioned a type of transformer they were working on. And labs like Qwen and DeepSeek (at least) have shown how incredible results can be when you innovate in the area of LLMs.
I think the real point of the post is that office politics and poor management got in the way of innovation due to infighting and poor execution in transfer of technology from research to product.
Almost as fascinating is that the article references Digg as the source - genuinely hadn't heard of Digg for maybe a decade? I thought the DVD encryption fiasco + reddit had ended it?
The headline makes good bait for AI skeptics. But this is a team of 50 people with a grandiose name.
The more interesting idea is that MS has abandoned AI research entirely, presumably because their zillions invested in other companies are a better bet in that area.
If MS abandoned a promising idea, pursuing it outside makes perfect sense. I am sure there are investors willing to fund this. This could even be an opening to attack the moat of the current AI frontier model companies.
This seems to track with a recent blog post from MAI regarding their approach to pre-training their first Frontier level model which seems promising but does seem to have a focus on scaling.
(In the context of Microsoft Research) Man, the history of how large corps fumble major wins should be studied.
I remember during my thesis how Microsoft Academic (a Google scholar/Scopus alternative) was ridiculously good at its job.
I'm talking, "input the key words and filter out the wrong fields and you've got 20 highly relevant papers that no other search term has found" level.
...Of course, they proceeded to close it down.
I can't even express in words how mad and disappointed I was and still am at Microsoft for this alone. It felt the closest to a genie or an oracle in being amazing. (The old Swype vs modern Gboard is probably a close example for those who'd used both.) It's just... crazy, frustrating, and crazy frustrating.
If you value the freedom to pursue your own research direction and share your work freely, then choose academia. All the perks of working in an industry research role are provided in exchange for this freedom.
If your research aligns with the incentives of industry, you can get the best of both worlds in industry. This was the case for machine learning researchers in the post-AlexNet, pre-ChatGPT era. The good times can't last forever, though.
>If you value the freedom to pursue your own research direction and share your work freely, then choose academia
Perhaps it is more free, but my understanding has always been that professional academia still restricts freedom of research. You have to apply for funding, convince the board that your direction will be fruitul.
An archaeologist can't just dig wherever. They compete with their peers for a limitted funding pool. If the school's board is more interested in the bronze age than the paleolithic, you're probably not going to get your dino dig approved.
The same is true for CS, I assume. You can't just "i would like spend university resources developing brainfuck 2.0" because the other teams are more likely to get their funding over you getting yours.
Both corporations and university's have finite budgets, both with bottom lines and margins. The uni wants their research done in a popular field that will attract students because students generate income that funds research that attracts students that generate income that funds research that...
Academic research is free in the same sense as running your own business. You can try whatever you want if you can find the funding, but you still have to deal with people and regulations.
Universities are mostly just administrative organizations. They have little control over your research, they rarely fund it, and they are usually not even interested in it.
That’s not the issue here, it’s not like AIF was doing purely academic work.
They just think it makes sense to explore things, rather than just rinse and repeat with more cash.
The good news for John is that when 99.99% of the competitors are doing all the same, it’s when the rewards for looking around can be the biggest. Those are the real moments for disruption, and companies disregard them at their peril.
>
If you value the freedom to pursue your own research direction and share your work freely, then choose academia.
The working environment in academia is often precarious :-( , so such people often have no alternatives to going into industry, even though they still deeply appreciate and are influenced by the values of academia.
Real research at most big companies is dead. Unfortunately, what has replaced it is mostly venture-backed startups, where the timeline to a return is still expected to be quite short.
As a result, you end up with companies doing basic research under the guise of developing a product, announcing timelines which are never going to be met. Fusion energy and quantum computing startups are some of the most egregious examples of this.
Agreed. I used to be an industrial researcher at one of the last companies that didn’t require their researchers to think of short-term productization goals. However, things changed at my former company, and I ended up leaving for a tenure-track position at a community college.
I don’t know what industrial research in CS is like in the era of agentic coding. On one hand I suppose building prototypes is faster than ever, but on the other hand I genuinely don’t know how well LLMs do with harder programming tasks. It’s one thing to have an agent work on a CRUD app, but I wonder how effective agents are at handling things like parallelism, distributed computing, load-balancing, and other topics that are harder to write than the average CRUD app. I am also aware of efforts such as Sakana AI’s automated scientific discovery research, and I wonder if many of the incremental, short-term research topics many companies pursue can be automated.
I’m a bit pessimistic about the state of research these days, from funding to publish-or-perish to the effects of generative AI (automation, AI slop publications, AI slop reviews). Then again, I’m a bit of a curmudgeon, and maybe things are better than I think.
Remember Bell labs, XEROX parc? Ask yourself why that existed and why research came out of them.
Monopoly.
So - many people don’t want to admit this but monopolies can be good from an R&D perspective, but without strong oversight by regulators - money will instead spent on R&D projects that specifically raise entry barriers.
I can show you many more examples of monopolies which did not produce PARC or Bell Labs than those which did. It's possible such research happened in spite of dominating the market, not because of it. Anyway, you cannot have both monopolies and strong regulators. These are incompatible situations. Monopolies occur because regulation doesn't exist or is captured.
I have been shorting most of the Quantum Computing startups for about a year. They have been printing. Missed out on Fusion! (?) These were one of few examples of the market leaving money on the table. Others examples have been Nikola. Usually the scammyiest companies don't IPO.
They've offered voluntary redundancy to a quite large proportion of the staff (a quarter?) so they'll also soon be losing anyone with employment prospects elsewhere.
I see a lot of news about waves of layoffs in large tech companies. Is MS the only shitty place now? I think all of them are, maybe except OpenAI & Anthropic, since they're building something new and are kind of on the "good" side right now.
> If the research products were truly innovative and useful, he wouldn't need to force Microsoft to adopt them.
That's just patently not true. History is littered with products that were very innovative and useful (and often far superior to competing products), but were canned, bought by competetors, or shelved in other way - often because they threatened the profits being made by some entity (though that's not the only reason).
Autogen is open source. I don't know the availability of other projects.
Generally engineers in megacorps aren't shy of picking up the latest AI tools. I suppose Microsoft may have barriers around this though, curious to discover more details.
Yeah, I agree. What he describes reflects the exact challenges I've experienced literally everywhere. He is basically saying work/life is hard so he looking for a better playground. I do not know his contributions to AI so maybe he deserves to be better listened to but I find that never reflects corpo life.
The transformer based approach is seen as a highway to race down in terms of training and building better models with minimal organizational distractions allowed while every company puts its foot down on this highway. Exploratory research into other AI methods is highly discouraged for now and organizations are encouraging transformer based scale out and iteration as a top priority.
As an example Meta literally fired it's previously head of AI who held views that transformers won't scale. He wanted to veer off the highway and heavily fund exploratory new types of models. They got someone willing to scale the current method out. Essentially the messaging it "Foot down, there's a straight highway ahead don't divert".
I have the view this makes sense too. Current top models are mere (mere in big datacenter terms) trillions of parameters right now. We definitely haven't hit the ceiling and everyone's noticing "Hey this Opus 5.5 with the increased parameters seems much better than the last one". Well that's a trillion parameter model. We'll soon start hitting 10's of trillions and 100's of trillions of parameter models. Unless there's a ceiling which nobody expects they should keep scaling just as they have been. I really do agree that it'd be insane to veer off this highway with exploratory research at this particular point in time. Foot down, get to 10 trillion parameters, see if the scaling holds, if so off to 100 trillion parameters. If the scaling continues to hold we'll be asking the AI to do the exploratory research for us soon enough.
Almost all statements Yann LeCun made back then (LLMs won't be able to do this or that) turned out to be wrong. They were right to fire him.
But even in computing, when there is a promising way forward that the basic reaearch has been done on, its hard to justify side quests.
10% chance to be market leader doesn’t sound as good as 60% second place to them for example
Once you get close to the top, the risks don't look worth it.
The author states "I consider the scaling-only focus to be both misdirected and not fun.". I get the point but also totally get the focus on scale out for now.
Which is superior, the big f*king monstrosity or the elegant on-demand, small batches, reduce waste?
Meta will go in the shitter and Deepseek will make the small batches.
It does (Lean at least) incorporate controlled experimentation, but tightly controlled with a both a laser focus on what the goal is and tight resource constraints. And Lean (at least several writers on Lean) favors taking multiple paths with a a quick and low risk and a longer-payoff, higher-risk, higher reward path for some efforts—one source used the US Navy’s two-path approach to SLBMs with Polaris and Poseidon as a model for this—but that's different than “maybe there is something down this path” basic research, which is outside of the concern a production model like TPS/Lean and often completely outside the model of a product-oriented firm; basic research is something firms consume and incorporate more than generate, because until the basics have been proven out, the rationally perceived chance of any particular avenue paying off is near zero for any firm.
Race down the transformer highway as fast as possible and explore other AI methods.
One need not preclude the other.
Going close to as fast as possible while saving a stream for the other approaches makes sense as an approach, but let's not ignore that there's a compromise/tradeoff here. Might not matter in orgs with near infinite resources
Look at what happened to Meta where they squandered a lead, had some mixed results, and reorged to more product certainty and away from fundamental research.
Is that observed scaling part of some equation or law? Is there a limit to how far that scaling goes before a ceiling or diminishing returns hit? Or is it simply based on more GPUs/parameters/compute plus how much human knowledge is in existence and will scale predictably forever?
And you can in practice often see a large network follow an expected training trajectory for loss/entropy.
In practice data is getting more sparse than compute, so you are usually overtraining using more tokens and being suboptimal.
(This moves the argument to "is being really good at predicting the next token the same as being intelligent?" which has happily diverted many minds for years now)
I see it that most of legacy research is not obsolete but can be significantly amplified by AI.
The one thing that is clear is if it can become more cost effective it’s world changing. Even at its subsidized standards now it’s world changing. And it’s reasonably classified as AI IMO, but it’s not self aware.
I think a lot of your general traditional researchers are fundamentally looking to understand ourselves through machines as proxies in a sense. Something self aware that senses, learns, adapts, so on. The current path isn’t there or necessarily clearly the key to achieving that. That may be the disconnect. I think LLMs may give us some hints at that but not the full picture.
Still incredibly powerful for things we once considered nearly impossible.
I mean we literally do this every few seconds already with natural birth processes.
Already happening: https://news.ycombinator.com/item?id=50397994
Obviously we'll see how it pans out, but other approaches feel a bit like trying to ignore the bitter lesson.
I should also say that I use ChatGpt daily and enjoy it for researching algorithms. I tried to set up Claude last week and it/he was so overwhelming as to be unusable. So I think frontier models can be "unwieldy" or something for average use.
Where's the money coming for those 10X and 100X ventures?
Chinchilla Law says no, we wont. We're likely well past the point of diminishing returns but MOE hides a fair bit of it.
We already know that power laws apply to scaling so 10x the size is going to result in minor gains - where costs become a real factor (as if they weren't already).
FTA "The core philosophy there is scaling, and aggressively not research."
This is where the real gains for AI come from - good enough and CHEAP is going to win the day here (brute force, mixture of models). It's the engineers displacing the academics - and that will mean a bloodbath for revenue (see DeepInfer for what the end game looks like).
Bioinspired algorithms may eventually lead to vastly improved intelligence per a weight too.
I don't think we should do this, but I feel doomed in that I suspect that we "can".
Is this true?
I barely notice any improvements ever since... 4.6? At least for software development.
Opus makes the same sort of mistakes and has the same sort of strengths ever since that point in my perception. My impressions is that increasing the number of parameters clearly has diminishing returns.
In fact, the best improvement I got from Claude was Haiku 5.5. Good cheap workhorse to implement plans.
As in your statement to me indicates that you clearly see the scaling working over longer timespans. You chose a relatively recent 2026 model as your baseline after all, not a 2025 model or earlier. You clearly see that the 2026 model is better than ones in the past.
Might i suggest that you perhaps just don't see the improvement as clearly when comparing iterations that are fairly close together? If you're not anecdotally seeing improvement between an early 2026 model and one from ~6 months later perhaps just give it another iteration? Personally i do see 5.5 as much better but i can understand not seeing a lot of difference to 4.6 since they literally aren't that far apart in releases.
Perhaps "The End of Microsoft Research's AI Frontiers Group" would be a less ambiguous title.
To the author: 100% need this kind of thinking. It’s a balancing act. Historically though, dedicated labs haven’t had the best track record at commercializing their tech. The skillset differs vastly from running a product-focused business.
That said, even within this narrow scope - there are areas ripe for research.
Some of my particular interests include:
(1) Latent thinking/reasoning, which could arguably be more efficient but also opaque is one
(2) Separation of “intelligence” from knowledge - ie. keep the ability to reason like a frontier model and make tool calls (calling the ‘oracle’ or searching the web) but remove all parameters associated with niche knowledge/facts.
These are a couple off the top of my head that are very much aligned to scaling.
That statement is broad enough to be unfalsifiable, but if you narrow it down just a bit to scaling LLMs, maybe there is an end in sight. At least financial markets are skeptical of there being no end in sight.
If scaling LLMs really was the road to God in a box, then compute demand would be infinite, and nobody building to meet that demand would have a junk debt rating. Did those money people just not get it?
I'm not sure how that, combined with open models inching up on frontier intelligence, is compatible with these massive AI IPO valuations?
It seems like closed model AI demand has a lot of contingencies that are ultimately fragile and possibly short sighted.
I think the real point of the post is that office politics and poor management got in the way of innovation due to infighting and poor execution in transfer of technology from research to product.
https://en.wikipedia.org/wiki/Digg#2025%E2%80%93present:_The...
Seems like it quickly pivoted quickly to cash in on "AI" hype.
It was my introduction to feature hashing.
The more interesting idea is that MS has abandoned AI research entirely, presumably because their zillions invested in other companies are a better bet in that area.
https://microsoft.ai/news/frontier-pre-training-at-microsoft...
I remember during my thesis how Microsoft Academic (a Google scholar/Scopus alternative) was ridiculously good at its job.
I'm talking, "input the key words and filter out the wrong fields and you've got 20 highly relevant papers that no other search term has found" level.
...Of course, they proceeded to close it down.
I can't even express in words how mad and disappointed I was and still am at Microsoft for this alone. It felt the closest to a genie or an oracle in being amazing. (The old Swype vs modern Gboard is probably a close example for those who'd used both.) It's just... crazy, frustrating, and crazy frustrating.
If your research aligns with the incentives of industry, you can get the best of both worlds in industry. This was the case for machine learning researchers in the post-AlexNet, pre-ChatGPT era. The good times can't last forever, though.
Perhaps it is more free, but my understanding has always been that professional academia still restricts freedom of research. You have to apply for funding, convince the board that your direction will be fruitul.
An archaeologist can't just dig wherever. They compete with their peers for a limitted funding pool. If the school's board is more interested in the bronze age than the paleolithic, you're probably not going to get your dino dig approved.
The same is true for CS, I assume. You can't just "i would like spend university resources developing brainfuck 2.0" because the other teams are more likely to get their funding over you getting yours.
Both corporations and university's have finite budgets, both with bottom lines and margins. The uni wants their research done in a popular field that will attract students because students generate income that funds research that attracts students that generate income that funds research that...
Universities are mostly just administrative organizations. They have little control over your research, they rarely fund it, and they are usually not even interested in it.
They just think it makes sense to explore things, rather than just rinse and repeat with more cash.
The good news for John is that when 99.99% of the competitors are doing all the same, it’s when the rewards for looking around can be the biggest. Those are the real moments for disruption, and companies disregard them at their peril.
The working environment in academia is often precarious :-( , so such people often have no alternatives to going into industry, even though they still deeply appreciate and are influenced by the values of academia.
This is certainly an interesting timeline in this section of the light cone
As a result, you end up with companies doing basic research under the guise of developing a product, announcing timelines which are never going to be met. Fusion energy and quantum computing startups are some of the most egregious examples of this.
I don’t know what industrial research in CS is like in the era of agentic coding. On one hand I suppose building prototypes is faster than ever, but on the other hand I genuinely don’t know how well LLMs do with harder programming tasks. It’s one thing to have an agent work on a CRUD app, but I wonder how effective agents are at handling things like parallelism, distributed computing, load-balancing, and other topics that are harder to write than the average CRUD app. I am also aware of efforts such as Sakana AI’s automated scientific discovery research, and I wonder if many of the incremental, short-term research topics many companies pursue can be automated.
I’m a bit pessimistic about the state of research these days, from funding to publish-or-perish to the effects of generative AI (automation, AI slop publications, AI slop reviews). Then again, I’m a bit of a curmudgeon, and maybe things are better than I think.
Remember Bell labs, XEROX parc? Ask yourself why that existed and why research came out of them.
Monopoly.
So - many people don’t want to admit this but monopolies can be good from an R&D perspective, but without strong oversight by regulators - money will instead spent on R&D projects that specifically raise entry barriers.
> Monopolies occur because regulation doesn't exist or is captured.
IMO, the stronger the regulator, the stronger the incentive to capture it.
Unless by "strong" you mean "competent"?
They look more like complete organizational shit shows.
https://archive.is/AaY5O
(Original https://hunch.net/?p=13764006 )
That's just patently not true. History is littered with products that were very innovative and useful (and often far superior to competing products), but were canned, bought by competetors, or shelved in other way - often because they threatened the profits being made by some entity (though that's not the only reason).
Some are, some aren't. Some try, but botch it.
Edit: Fara is open weight as well. Seems like a great match for https://www.microsoft.com/en-us/windows-365/agents ?