Collecting user feedback on Duolingo’s Birdbrain

[Numenorian22]

Hello! I’m a senior at my university studying computer science and am collecting data for my final project in my AI in Education course. I am performing an evaluation on Duolingo’s birdbrain AI, specifically how well it calibrates difficulty of exercises to individual users over time.
If you are interested, I am collecting data through google forms here: https://forms.gle/8FvTwMKYaWnguhPz5

P.S. to avoid confusion, Birdbrain is not the LLM used by Duolingo, and the LLM is not what I am reviewing.

[John Little]

Done but I"m not aware of any AI making allowances for my progress. My wife and I have both been doing the same course for the same time and are both about the same distance along the path with the same questions in the same order

[Numenorian22]

That’s actually expected for the most part. The main thing that BirdBrain personalizes are the reviews. The lesson plans are standardized across Duolingo, but as you learn new words and complete lessons, Birdbrain uses that information to construct a learner model. The learner model, a 40 dimensional vector, is updated based on words you get right, lessons you complete, words you get wrong, and the time since learning a new word. Taking the time since you learned a new word it uses a probability function call half life regression to predict when that word leaves your working memory and needs to be reintroduced for it to enter your long term memory. Using that it chooses which words to make you review.

Knowing that, it’s not super surprising that two people who are introduced to new words at relatively the same time would have similar half life regression and learner models.

If you’d like to learn more, you can read here: https://blog.duolingo.com/learning-how ... birdbrain/
And here: https://spectrum.ieee.org/duolingo
Some of it is pretty technical but it’s a good read.

[John Little]

It might be more noticable in the practice hub but thats only recently been available to the free version so I'm not in the habit of using it. Maybe I should.

[PtolemysXX]

Knowing that, it’s not super surprising that two people who are introduced to new words at relatively the same time would have similar half life regression and learner models.

Eventually they should start drifting apart because every learner will develop a different profile.

In your questionnaire you have just two questions about the learning experience (the last two). How do you want to make any useful conclusions out of that?

First of all, as @John Little indicated it is not clear where BirdBrain is implemented at all. You may want to clarify it in order to prevent misleading / useless responses referring to features that have nothing to do with BirdBrain. Secondly it might be good to give some guidance as to how a learner could notice this AI engine being at work and what would be the indication that it actually does any good.

Judging by the articles that you quoted I would qualitatively expect something like this:

I learn a new word or phrase – then after some time it pops up in the review. If, at that point I still remember the phrase but I am just about to forget it then I'd assume that it was the best time for a refresher (unless I misunderstand the theory). Now, if this pattern applies to most, if not all phrases that I learn then I would consider the engine to be doing great work. In contrast, if a portion of words that I am served in the review are either:

• too easy (I remember them well and don't have a feeling I am going to forget them)
• too difficult (I lost them completely)
• they do not come back at all

that would mean for me that something with the engine is not working. Having few such cases would indicate a well designed engine – the higher the percentage of such „missed targets“ the worse the tool's quality.

The ieee paper that you quoted is more than three years old by now. That should be sufficient to collect enough data to verify the efficacy of the solution. Are you aware of any such studies?

[Numenorian22]

This may or may not be satisfactory but I agree that those questions would have been better and I wish that my reviewer would’ve pointed that out as well, but they didn’t.

To answer your last question, there have been no studies about the efficacy of birdbrain itself that I could find. Duolingo has a page where they post studies about the app here: duolingo.com/efficacy/studies

Will this change anything? Probably not, this is an undergrad final project for a class, not a masters thesis, so I’m aware that I’m going to probably make some mistakes and that this won’t be published.

[buho]

done! hope you do good in your studies!

[PtolemysXX]

there have been no studies about the efficacy of birdbrain itself that I could find.

It could be an interesting topic for a master thesis (maybe not strictly computer science) to measure - based on real data - if / how Birdbrain accelerates learning. It could be seen as the next step after rota learning and spaced repetition. But it is hard to say up front. A good spaced repetition algorithm is supposed to do exactly that: remind you of vocabulary at optimal intervals and remind you only of vocabulary that needs reminders. You don't need AI for everything.

[Vlot Vlaams]

The name BirdBrain makes me think of a trick used in marketing for milk.
They called it "Groeimelk" which in Dutch just means "milk that makes you grow".

Imho all milk, any milk, does that, so nothing new under the sun.

Imo, as shown in another post how bad the "answer evaluation logic" of DL is in the app,
nothing new behind BirdBrain either. Just a new marketing slogan.

I am waiting to see a first screenshot of a DL screen where it helps a user having made an error.
Where it explains what the rule is, and why the given answer wasn't "acceptable" (enough)

Till then, nothing new.

---

That said, so many of the more recent and modern apps, being commercial or not, meanwhile have fully adopted the AI possibilitities, and implemented them on the user side.

And if it needs to be free for some of us, a simple chatGPT or Deepseek does roleplay too, with error corection and explanation.

And for those who prefer google, try the free language apps from their Labs.
Or use AI built-in into google translate.

BUT beware: i read an article stating that AI based translations are intentionally made "worse" (enshittification, emmerdement) so you're likely to pay for a subscription; But my experience says otherwise. Translations by AI are amazingly good and fully context aware.

---