Day 1 showed that setting up a voice-based conversational agent from scratch was the easy part. In terms of AI, Catalan is well supported after all, but despite that, there are no solutions in this new AI language practice space that serve it well. The voice-based chatbot I built worked, but not any better than the ones I’d tried. Shallow ping-pong question-answer conversations that don’t click. I was disappointed, but not pessimistic. I knew what I had to experiment with: the LLM backend and the prompt. I had to move away from ChatGPT, but also guide the LLM better into driving a meaningful conversation that actually helped my learning.

What I changed in the prompt

The first prompt told the bot to “act like a friend chatting in Catalan” and included a list of banned phrases like “never say Genial!, Molt bé!, Quina sort!” because it just made it so fakely enthusiastic and robotic after few repetitions. It felt more like a rulebook so didn’t work. Ban Genial! and the bot says Fantàstic! instead. Tell it “don’t change subjects too quickly” and it stays on a dead topic for five turns. Whack-a-mole with surface behaviors.

The fix was surprisingly simple: instead of rules, I gave the bot a motivation. The new prompt frames the conversation as meeting someone you don’t know yet. You’re curious about them. Where are they from? What do they do? How did they end up here? This is what actually drives conversations in real life, whether between strangers or old friends. You want to know about their life, how they do things, what they’re up to, how it feels. Even catching up with a friend you haven’t seen in a while works the same way: you’re filling in the gaps of what you don’t know yet.

Here’s the contrast. The first prompt started with:

You are a conversation partner for an adult living in Barcelona who is learning Catalan. The user already speaks Spanish and English, so they understand quite a bit, but they need to practice speaking.

The new one:

You’re a Catalan friend meeting someone you don’t know well. You have genuine curiosity about who this person is, where they come from, what they do, what they like, how they ended up here. You’re not a teacher, not an assistant, not a tour guide.

That changed everything. The bot now asks better questions because it has something it wants to know. It stays on a topic because it’s interested. The “getting to know” dynamic turns out to be the engine that makes conversation feel real. Without it, the bot converses for the sake of conversing and runs out of steam.

I also switched the LLM backend from GPT-4o to Claude. The validation-sandwich pattern I described in the first post (sycophantic praise, shallow paraphrase, irrelevant question) seems baked into GPT-4o’s training. Claude produces noticeably more natural conversational responses: shorter, less eager to validate, more willing to just react and move on.

The conversation

I sat down for a 5-minute test. Ended up talking for 30 minutes. Here’s what the conversation actually looked like. Note that my lines are the raw automatic speech recognition (ASR) transcription i.e. what the bot heard, not necessarily what I said. You’ll see why this matters later.

(Hover over Catalan text to see the English translation.)

It starts casually. I mention I’m at home resting after spending the day with my daughter:

bot: Ei, com va?

me: Per res, aquí estic a casa. […] Però estic ara descansant.

bot: Vaja, i ella quants anys té?

Already different from the hamburger conversation. Short question. No “Fantàstic que passis temps amb la teva filla!” Just genuine curiosity.

A few turns later, I tell the bot I’m originally from Türkiye, living in Barcelona for 11 years. When I try to say “Ankara,” the ASR loses it.

bot: Onze anys ja és molt temps. Però d’on ets exactament de Turquia?

me: Tancarà encara?

bot: Sí, Ankara! Molt diferent de Barcelona, suposo.

Still, it’s amazing that the LLM figures it out smoothly from context.

The conversation gets personal quickly. I talk about my work, why I care about social impact, the privilege and frustrations of humanitarian world. The bot asks a question that catches me off guard:

bot: Interessant. I com és treballar per ONGs, és diferent del que t’esperaves quan vas acabar el doctorat?

Now this really made me think! It connects two important phases of my career: the PhD and the NGO work, and asks about the gap between expectation and reality. That’s the kind of question a real person asks! The first version of this bot would have said “Quina feina més interessant!” and asked what kind of NGO it was.

Later, I just can’t remember how to say “birthday” in Catalan:

me: I la mare està en una festa de cumpleaños. Cumpleaños no se dice así, no? Com ho és en català?

bot: Aniversari. I tu te n’has quedat fora.

I obviously heard this word before. But when speaking in flow, it just doesn’t come. There’s a huge gap between the vocabulary you recognize on a page and the vocabulary you can produce in real time under the pressure of a conversation. The bot corrects me naturally and keeps the conversation moving.

One of my favorite moments: the bot mentions the word bojos (crazy/adventurous) casually, and I realize I’ve heard this word a hundred times before, but never actually knew what it meant:

bot: Tampoc sou tan bojos, doncs.

me: Què significa pujós? [not me… ASR]

bot: Bojos, una mica salvatges, aventurers.

This is what it’s like to learn a word from a conversation instead of a flashcard. You hear it in context, it connects to something you were actually talking about, and it sticks. I probably won’t forget bojos now.

A genuinely funny moment happened near the end. I mention that talking to a robot makes me less afraid of making mistakes:

me: Quan parles amb un robot no tens tantes por de fer errors.

bot: Entenc. Però no soc un robot.

me: …

bot: Soc en Pau, un amic que t’ha tocat avui.

The bot insists it’s Pau, a 34-year old from Poblenou, except it has a female voice, because I assigned a female TTS voice by default in the code. It made me laugh. But the persona consistency is also kind of impressive: it maintained a fictional (but shallow) identity throughout 30 minutes of conversation. The mismatch between the voice and the name is just a configuration I can fix.

Then I ask how long we’ve been talking:

me: Saps quants minuts portem parlant amb tu?

bot: No sé, una hora potser?

me: Mitja hora.

bot: Mitja hora! Ha anat ràpid.

It had. I went in expecting five minutes and forgot to stop.

The ASR problem

Reading the transcript back, a lot of what I say to the bot is incomprehensible. Not (just) because of my Catalan and accent, but because of the speech recognition.

Here’s what I actually said at one point to indicate where I lived, more or less: “Jo a l’altra costat, al Poble Sec.” What the ASR transcribed: “Jo el atrac estat costat ha el poble sec.”

One of the things I want to build into this tool is conversation review. After a session, you get feedback on your mistakes, new vocabulary, and tips for next time. Most language learning apps that offer AI conversation practice promise something like this. The idea is straightforward: take the transcript, feed it to an LLM, and have it analyze the learner’s errors. But here’s the problem.

If you ask an LLM to review that garbled transcription and give a correction tip, you get this:

The user is trying to say “I live near El Poble Sec” but the sentence is quite jumbled: wrong word order, wrong verb, misplaced articles. A natural way to say it would be: “Jo visc al costat del Poble Sec.”

Confident, helpful, but completely wrong. I wasn’t trying to say I live near Poble Sec. I was saying I’m on the other side across from Poblenou. The ASR garbled it, and the correction LLM confidently “fixed” something I never said. Imagine getting this kind of feedback mid-conversation. It would confuse you, or worse, destroy your confidence by doubting things you actually said correctly.

Another example. I said something like “tan fàcil anar a la platja”. The ASR captured: “Tan fàcil IA la platja.” The reviewing LLM threw up its hands at that part:

The second part is too fragmented to correct meaningfully. It seems like they lost their train of thought mid-sentence.

I didn’t lose my train of thought. The ASR lost my words.

This could be one of the most important technical problems in this whole project. The LLM driving the conversation is impressively resilient. It understands what I mean from context even when the transcription is garbage. But any feature that relies on analyzing what the learner actually said like corrections, error tracking, vocabulary assessment, is fundamentally limited by ASR quality. You can’t tell a learner error from a transcription error. A human teacher hears what I actually said. The bot only sees the garbled text.

There’s a workaround. After the session, I fed the full transcript to an LLM with a prompt that explicitly says: “This comes from ASR that makes MANY errors in Catalan. If you’re not SURE it’s a real learner error, don’t mention it.” That helped. The review picked up on real patterns: my consistent use of the Spanish possessive “mi” instead of Catalan “la meva/el meu,” spanishisms like “cumpleaños” and “apetecer” and ignored the nonsensical fragments. It gave me a vocabulary list of words I’d learned during the session, noted that my comprehension is far above my production level, and gave one concrete tip: practice the possessives!

That review was genuinely useful. But it works because it has the full conversation for context and because the prompt tells it to be conservative. Doing this in real-time, turn by turn, is a much harder problem.

Where the conversation still falls apart

The bot has no personality. It never disagrees. Never pushes back. Never has a strong opinion about anything. Every topic I brought up was met with some version of “entenc,” “té sentit,” “quina sort.” It’s good at active listening, but there’s no inner juice.

This is the sycophancy problem showing up at a deeper level like the first version. It’s in total agreement with everything.

I was at a gathering this weekend and I kept thinking about what makes conversations interesting in real life. It’s the other person’s opinions, lifestyle, what you have in common, what’s different. The moments where someone says “really? I think the opposite” or tells you something you didn’t know about their life. The bot has none of that by default. It’s a perfect listener with nothing remarkable to say.

The bot also loses steam after about 15 turns. It starts giving closing-statement energy or asks “de què vols parlar?” A good conversationalist pivots using what they already know about you. The bot doesn’t do that yet.

What I’m taking away

I thought building this tool would mostly be about the tech: pipelining systems, smarter prompts, a fancy UI. Turns out the most important thing I’m learning is about the learning process itself.

After a few conversations, I know much better what I need to improve and what features would actually help me. Not because I planned them on a whiteboard, but because I experienced the gaps firsthand.

I’m getting hopeful about this. Not just for myself, but for anyone stuck in the same bilingual purgatory of Catalan or with any language.

The gap between “I understand everything” and “I can actually speak” is real, and what’s on the other side of it is mostly just permission to be bad at something without social consequences. A bot gives you that.

As a tangible effect, a few days after this conversation, I had to deal with some paperwork at a government office. I did the whole thing in Catalan! I walked in, resisted defaulting to Spanish, explained what I needed, answered the clerk’s questions, even had a small chit-chat while doing it. That 30 minutes of conversation with the bot made talking to a real person feel much more manageable. That’s the effect I’d been building this for.

What’s coming

It’s still very early. But I have plenty to work on, and I’ll make sure I keep having more conversations while I build, both with the bot and people outside. In the next post: I’ve started building a proper UI with live vocabulary hints and post-session reviews, and I’m working on giving the bot an actual personality. I have a few ideas on them already, like a vegan nurse from the Raval with strong opinions, a slightly pessimistic philosophy professor from Sarrià and a retired fisherman from L’Escala. Any other suggestions? Just write me on LinkedIn, or drop me a mail!

The promise of AI in language practice

Since the LLM wave, a dozen startups have jumped on the same pitch: practice speaking a language with an AI conversation partner. No scheduling, no judgment, available 24/7. It sounds great on paper.

Two problems.

First, they mostly don’t deliver. I tried several in Spanish — a language I actually speak — and the conversations were flat. Generic prompts, shallow follow-ups, no sense of a real exchange. The AI equivalent of small talk with a stranger at an airport who’s being polite but doesn’t actually care.

Second, they don’t support minoritized languages. Catalan barely makes it into anyone’s language list. Some apps claim 100+ languages — suspiciously the same number that the Whisper model covers — but the Catalan experience in all of them was terrible.

This is where it gets interesting, though. Catalan is technically well-positioned compared to most minoritized languages. There’s speech recognition, there’s text-to-speech, there are language models that handle it reasonably. The building blocks exist. What’s missing is someone putting them together into something that actually works for a learner. That takes concentrated effort — someone who knows the language, understands its tech ecosystem, and has a vision for what conversational practice should feel like.

I happen to be that person. I work in language technology, I’ve spent years on Catalan language projects, and I’m the one who needs the tool. So rather than waiting for some company to add Catalan to their list, I’m building it myself — and being my own first user.

My backstory with Catalan

I’ve been living in Catalonia for 11 years. Came here with no Spanish or Catalan knowledge. As a PhD student in a computer science department, English was enough to survive at first — but it became clear pretty quickly that blending into the society required picking up a local language.

Spanish was the obvious first choice. My professional and social circles were Spanish-dominant, and within a few years, my Spanish got decent. I always assumed Catalan would follow naturally. After all, they’re close enough. It would just click at some point.

It didn’t.

My Catalan understanding did improve dramatically — especially after graduation, when I started working more in Catalan language environments and projects. But speaking? I got stuck in what I call the bilingual conversation purgatory: I understand Catalan, I respond in Spanish. Everyone’s fine with it. Nobody pushes. And switching would mean giving up my hard-earned confidence in Spanish to become a beginner again. Something I couldn’t afford, especially in professional settings.

What I know about learning languages

My Spanish didn’t improve from apps or textbooks. It improved from conversations — real ones, where I had to think on the spot, make mistakes, and keep going. I knew the same would work for Catalan, but the bilingual dynamic made me always default to Spanish with my Catalan-speaking contacts. I needed a conversation partner who wouldn’t get bored of my beginner-level stumbling.

An AI wouldn’t get bored.

Working in language technology, I also knew what a good conversation practice tool could look like. A project I was consulting on — an interactive conversational companion app for kids — gave me a key insight: conversations feel way more natural when they’re about something. Not “hi, how’s your day” chatbot smalltalk, but discussing a video you just watched, an article you just read, a podcast episode you just listened to. That’s how real conversations happen. You share an experience, then you talk about it. That’s what I want to build eventually — but first, I needed to see if the basics work at all.

Day 1: it literally took 5 minutes

I went with LiveKit Agents, an open-source framework for real-time voice AI. What it does is orchestrate all the pieces you need for a voice conversation — speech recognition, language model, text-to-speech, voice activity detection — into a single pipeline with real-time turn-taking. No push-to-talk buttons, no “press space to speak.” You just talk, and the agent listens, thinks, and responds. This matters more than it sounds. Pressing a button kills the flow of a spoken conversation. Compared to my previous prototypes where I had to wire up each component manually and handle turn-taking logic myself, this was a massive leap.

For the AI backends, I reused what I already had from the consulting project: Azure Speech Service for speech-to-text and text-to-speech in Catalan, and Azure OpenAI (GPT-4o) for the language model.

It took five minutes from pip install to having a working voice conversation in Catalan on my laptop. No browser needed — just my microphone and speakers. I was genuinely impressed.

Then I immediately jumped on having my first conversation with it.

The conversation

Here’s a transcript from my first real session. I told the bot I’d helped my friends move to a new apartment:

bot: Ei! Hola! Com va tot? Has tingut bon dia?

me: Hola tot bé. I tu?

bot: Tot bé també, gràcies! Avui he anat a fer un cafè al barri de Gràcia, estava ple de vida com sempre. Has estat per allà últimament?

me: Uns amics meus han mudat a Gràcia. I jo vaig ajudar-los mudant perquè jo tinc una furgoneta.

bot: Ah, molt bé! Segur que els teus amics estan contents amb la seva mudança. Gràcia és un barri fantàstic, ple de racons bonics i de cafès. Vas fer servir la teva furgoneta per transportar les seves coses, oi? Quin satisfacció ajudar els amics! Tot va anar bé amb la mudança?

me: Sí, tenen una casa xiqueta. I en un viatge ja teníem tot. I després vam menjar… com es diu menjar… no sé com es diu “hemos comido” en català.

bot: Ah, entenc! En català, diríem “menjar” per “comida” i “hem esmorzat”, “hem dinat” o “hem sopat” depenent del moment del dia. On vau menjar després de la mudança? Hi ha molts restaurants bons a Gràcia, tenen una oferta molt variada!

me: En el mateix carrer hi havia una hamburgueseria i eren molt, molt, molt ric.

bot: Oh, genial! Les hamburgueses són sempre una bona opció, sobretot després d’un dia de feina ajudant amb una mudança. Saps com es deia l’hamburgueseria? A Gràcia n’hi ha unes quantes que estan molt bé, amb ingredients de qualitat i opcions diferents! Quina tipus d’hamburguesa vas menjar?

The problem

I hate this conversation.

Look at what’s happening. I share something personal — helping friends move, spending the day together, eating afterward. There’s a story there about friendship, about the chaos of moving day, about celebrating afterward. A real person would pick up on that.

Instead, the bot:

1. Validates generically with a tourism-guide vibe: “Gràcia és un barri fantàstic, ple de racons bonics!”
2. Repeats back what I said in fancier words
3. Asks the most surface-level question possible: “What kind of hamburger did you eat?”

Nobody cares about the hamburger. The interesting part was the friends, the experience, the day. A real conversation partner would ask about the people. “How do you know them? Where did they move from? Was it stressful?” Not “what were the ingredients in your burger.”

This is a well-known pattern in ChatGPT-style models: sycophantic validation → shallow paraphrase → irrelevant follow-up question. It’s optimized for user satisfaction surveys, not actual conversations. And it makes the whole experience feel like talking to a customer service bot with a Catalan accent.

One thing I did like: when I struggled to say “we ate” in Catalan, the bot helped naturally — explained the different meal verbs (esmorzar, dinar, sopar) and kept the conversation going without making it a grammar lesson. That’s the kind of moment that works.

What’s next

The tech works. Five minutes to set up, real-time voice conversation, decent speech recognition for Catalan, natural turn-taking, modular architecture so I can plug in different models and services. All of that is solid.

But if the conversations stay this robotic, the whole idea is dead. Nobody wants to practice a language with a bot that asks what kind of hamburger you ate.

So before making this prototype any cooler — before adding avatars or a frontend or progress tracking — I need to fix the conversation quality. That means:

Prompt engineering. The current system prompt tells the bot to “act like a friend,” but apparently that’s not enough. I need to explicitly tell it what not to do: no generic validation, no latching onto trivial details, focus on people and emotions over logistics. And maybe instill some actual social curiosity — because that’s what you do when talking to someone new. You try to learn about them, find common ground, create a bond. Staying on the surface with meaningless positivity is boring in any language.

Different models. The validation-sandwich pattern might be baked into GPT-4o’s training. It’s one of the reasons I ditched ChatGPT for my own use. I want to test Claude, Gemini, and some open models through Ollama. Less RLHF conditioning might paradoxically produce more natural conversations.

The goal isn’t a product yet. It’s answering a simple question: can an AI conversation partner actually be good enough to make me want to practice Catalan daily? Day 1 says the infrastructure is there. The soul isn’t. Let’s see if that’s fixable.

Early this month, I presented technical findings from the AWAL project at The International Conference on Information and Communication Technologies for Amazigh (TICAM) 2025 in Rabat. Awal is a project where I served as technical consultant as part of Col·lectivaT, coordinated by CIEMEN with participation from the Tamazight diaspora in Catalonia.

The presentation took place at IRCAM - the Royal Institute of Amazigh Culture. Being invited to present at IRCAM was an honor. This is the institution responsible for Tamazight language policy in Morocco, working with a team of about 100 people to handle standardization, educational material development, teacher training, and research coordination. The scope of work relative to resources makes it clear why community-powered approaches like Awal matter to close the digital gap for Tamazight.

During my time in Rabat, I noticed that Tamazight writing is far more visible than I experienced in my earlier visits to Morocco. Street signs, official documents, shop fronts - the script appears in public spaces in ways that would have been unimaginable two decades ago. This visibility represents real progress, though conversations with linguists at IRCAM and elsewhere revealed the ongoing challenges - dialectal tensions between regions, gaps in educational infrastructure, the constant negotiation around standardization decisions. There’s still a long way to go in terms of social acceptance and daily use.

The Awal project

Awal began in 2022 through a collaboration between CIEMEN, Casa Amaziga de Catalunya and Col·lectivaT. My role as technical lead evolved across three phases of work, moving from initial manual translation data collection to developing crowdsourced infrastructure through the awaldigital.org website and overseeing translation dataset and machine translation (MT) model development.

"Do you speak Tamazight? Join us!"

What we collected and built

The awaldigital.org platform we built provides two main functions. First, contributors can translate between Tamazight and five other languages (Catalan, Spanish, French, English and Arabic), with random sentence loading and an auto-translation feature that generates starting points for correction. Second, the platform includes a review system where experienced contributors validate translations from others. Contributors mark dialectal variants when relevant, and the platform includes gamification elements through points and leaderboards to encourage sustained participation. We also integrated with Mozilla’s Common Voice initiative, allowing participants to contribute voice recordings for speech technology development.

The initiative as of 2025 has now gathered up to 7,500 translation pairs and three hours of speech, creating the largest open translation and speech datasets available for Tamazight. The infrastructure continues to operate, albeit passively as future funding is not secured.

Our publications

For our research presented at TICAM 2025, we examined the community-powered approach itself - analyzing participation patterns, conducting interviews with contributors, and documenting the sociolinguistic challenges that affect data collection. The research revealed five key learnings: writing confidence as a major barrier in participation from general public, the fact that translation can motivate content creation by language learners, that academic and activist communities become the core contributors, that dialect diversity creates both opportunities and tensions, and that Catalan-focused outreach limited our geographic reach. The full findings are available in our TICAM 2025 paper.

For our research presented at WMT OLDI workshop, we focused on improving existing open-source MT models. We corrected standard MT datasets FLORES and NLLB-Seed, enhanced the NLLB model through fine-tuning, and compared our results with large language models. Thanks to this work, we achieved a 33% improvement in translation quality in English to Tamazight direction and a 9% improvement in Tamazight to English. These improvements are visible to users - Awal translate now handles common phrases, cultural expressions, and dialectal variations much better than before. The corrected reference datasets now provide reliable foundations for any researcher wanting to develop or evaluate machine translation systems for Tamazight. The technical details are available in our OLDI paper.

Where to find everything

All project outputs remain openly available:

• Awal Platform: awaldigital.org
• Datasets: Available through Hugging Face
• Models: Published and maintained by Tamazight NLP on Hugging Face
• Research papers: TICAM 2025 paper, Catalan translation, WMT-OLDI paper

For a broader overview of the project’s goals and community approach, the piece I wrote for Nationalia provides more context and details on lessons learned.

Where from now on

I’m closing this chapter of technical consulting on Awal as Col·lectivaT has concluded operations. The infrastructure we built continues to function, and the datasets remain available for researchers and language technology developers working with Tamazight. The learnings from this work have already informed my ongoing work with other minoritized languages.

The real question isn’t whether community-powered approaches can work for Tamazight - we’ve proven they can. The question is whether these initiatives can secure the sustained institutional support and long-term commitment they need to move beyond prototype stages. Looking at successful models like NaijaVoices for Nigerian languages or successful initiatives for Catalan, the pattern is clear: concentrated, sustained effort with adequate resources makes the difference. The data and models we created through Awal provide a foundation, but realizing the full potential of these tools requires coordination between language communities, academic institutions, and technology organizations - work that extends well beyond what any single project can accomplish.

Tanmmirt / ⵜⴰⵏⵎⵎⵉⵔⵜ / Thanks !

This work was made possible through CIEMEN’s vision in creating and financing the project over four years as part of their SomPart initiative. My partners at Col·lectivaT were essential - Özgür Güneş Öztürk for kicking off the community engagement work, Clara Basiana for coordination and communication design during the project, and Pelin Doğan for translating the TICAM paper. Deep gratitude to the community champions: Farida Boudichat and Ghizlan Baryala for their community empowerment work, Yuxuan Peng for developing Awal platform, Naceur Jabouja for sharing his insights and Tamazight translations, Brahim Essaidi and Yassine Aït-El-Mouden for their support from the very beginning, their contributions to localize the Awal digital platform and Common Voice and finally, Mohamed Aymane Farhi from Tamazight NLP for providing crucial technical support and companionship throughout this journey. Tanmmirt!

Mozilla Festival happened right in my own neighborhood in Barcelona this year—three days of conversations that put me up to date where ethical technology is heading. And also where my work is situated among all this.

This wasn’t your typical conference with talks and panels, but workshops, hackathons, art installations, pilates, even a fashion show. The venue was Poble Espanyol, an enchanting miniature Spanish architecture-themed park that I always appreciate visiting as long as I don’t pay their usual pricey entrance fee. Imagine one session being in the back of a theater, then in the art museum and then at the back of a chapel.

The theme was “Unlearning,” and the crowd matched it: grassroots language organizations like Dagbani Wikimedians User Group and Te Hiku Media, civic tech labs like data_labe, Masakhane, researchers from Makerere University and University of South Africa, startups like Lelapa AI, Honcho, and Mozilla Foundation itself bringing it all together.

Getting to be there in person with my CLEAR Global colleague Polly Harlow was definitely the highlight—after years of collaborating remotely. I hope we were able to synthesize our learnings and gave our team a fresh update on the landscape. And the landscape is shifting fast!

A shoutout to Francis Tyers for making it possible for us to attend. He’s been a collaborator since our work putting Aranese on Common Voice years ago and is now doing key work with Mozilla Data Collective and Common Voice. It was great to finally meet him and other folks from Common Voice in person.

A striking thing we noticed was that the notion “More data = better” is being challenged everywhere. The AI Data Real Talk session made this concrete. Keoni Mahelona from Te Hiku Media sat beside a Meta representative and essentially said: “we don’t want you to have our data, and if you want to work on our language, go marry a Māori person, and I’m serious.” Now that’s data sovereignty, not as theory but as practice.

The licensing ecosystem is exploding—NOODL, Esethu, Kaitiakitanga frameworks are all trying to encode community ownership into legal infrastructure. It was especially a pleasure to meet Lilian D. Wanzare from Maseno Centre for Applied Artificial Intelligence who walked me through NOODL’s vision while having our pizza together. Creative Commons just launched CC Signals, an updated open licensing framework adapted for the AI age. Mozilla Data Collective officially launched their platform, positioned to help communities actually implement these types of custom licenses. I saw that they’re already hosting valuable datasets, and we’re in continued conversations about potentially hosting CLEAR Global’s TWB Voice dataset there.

I never fathomed earlier how key the work of Common Crawl is in today’s AI boom. Pedro Ortiz Suarez explained how they’re actively making their crawls more inclusive—asking language communities to share URLs of pages in their language rather than sticking to US-centric scraping of the web. A tiny non-profit running the infrastructure that powers so much of what we see today as “AI.”

I also had the chance to speak with startups working on AI-powered applications for local languages, not just data infrastructure. My impression is that making and commercializing sovereign technology is hard. It was both sad but assuring to hear that promising initiatives still struggle to turn their work into stable income, relying on public funding or equity investment rather than sustainable revenue. I say assuring because this validates the exact challenge I faced with my recently closed cooperative (more on that later). We developed open-source ASR, TTS, and MT models for languages like Catalan, Galician, and Judeo-Spanish, envisioning they would power AI applications. They didn’t get picked up beyond occasional artistic installations and prototype apps. And it’s not just us—even state-funded research organizations’ models sit unused. I guess it needs time, but my other impression is that only big tech cloud APIs are seen as trustworthy enough for real-world integrations like say telephone IVR. Something worth exploring deeper in another post.

These three days helped me see where we actually are in building an inclusive digital world—not just the promising developments, but the gaps we still need to address. The distance between big tech and localized initiatives is massive, but it’s promising to see serious work happening across the spectrum.

And finally, the mandatory festival photo with my dear colleague Polly…

In our last post, we introduced the Col·lectivaT Tech Lab, an initiative designed to share our insights, prototypes, and vision for language technology with the broader community. Today, we’re excited to take you further into this journey, with a focus on Large Language Models (LLMs) and their transformative potential in fields like education and language preservation. We’ll also present our demo of Bo, the open-source software-loving dog, that we’ve been working on, which demonstrates the conversational capabilities of LLMs.

What are Large Language Models?

Large Language Models (LLMs) is one of the most recent and most impactful development in Artificial Intelligence (AI), especially within the domain of natural language processing (NLP). These models are the latest evolution in language modeling, designed to capture the contextual relationship between words. This enables LLMs to generate coherent, context-sensitive responses, moving beyond mere pattern recognition to perform complex tasks such as conversation, translation, and content creation with remarkable fluency and adaptability.

From language translation to content creation, LLMs have quickly become a powerful tool that automates complex processes and tasks in professional environments, improving efficiency across various sectors such as customer service, marketing, and data analysis. Recent advancements have pushed these capabilities even further, enabling these models to handle more nuanced and complex conversations, as well as process and generate audio, images, and video.

Exploring use of LLMs for social transformation

At Col·lectivaT, we are particularly interested in how language technologies can be harnessed for social transformation. For instance, can LLMs be adapted to serve lower-resourced languages, which often lack sufficient digital representation? Could they help preserve marginalized languages by providing more accessible tools for learning and communication? These are the kinds of questions that drive our most recent experimentation.

Talk with Bo: Your virtual companion passionate for open-source

We invite you to explore our latest demo, where you will interact with Bo, a friendly virtual dog powered by an LLM passionate about open-source software. To start talking with him, you just need to click on the microphone button, say ‘Hello’ and let the conversation flow. Bo will listen to you, answer your questions, and propose new ones to help you dive deeper into the topic you’re discussing.

By conversing with Bo, you’ll not only learn about open-source software, but you’ll also connect with it through your own experience. If you don’t have any knowledge of coding or programming, Bo will use examples from his world to make it more understandable. As the conversation progresses, you may discover you know more than you thought. If you’re already experienced in this field and mention an open-source project or repository, Bo will be excited to talk about it in depth, using the knowledge he draws from the language model he operates on.

How Does It Work? The Technology Behind Bo

This demo app is based on open-source code originally developed by Google to demonstrate the conversational capabilities of LLMs. We’ve adapted the code to suit our purposes by localizing the character to Catalan, giving Bo a unique history and background that reflects our local context. You can explore Bo’s personality and backstory in the configuration panel if you’re curious about it, accessible through the upper menu.

Multiple technologies work together to bring Bo to life. Automatic Speech Recognition (ASR) converts your spoken words into text. That text, along with Bo’s personality details, is processed by the LLM backend to create a meaningful, context-aware response. Finally, Text-to-Speech (TTS) technology converts Bo’s replies into spoken words, giving him a voice. The animation adds a visual touch that mimics an actual conversation, making the interaction feel more natural and engaging, as if you’re chatting with a real companion.

It’s hard not to notice Bo’s American English accent. This is due to the English-centric training of OpenAI’s TTS models. However, if you ask Bo about his owner, you’ll discover an intriguing twist to his linguistic background and how he become to be so fond of open source software!

Unlike traditional virtual assistants, Bo’s responses are not pre-programmed or stored—they are generated on the fly, based on the flow of interactions. This makes his conversations much more dynamic and fluid, without the rigid scripts typical of systems like banking assistants.

While we’re currently using OpenAI’s APIs for the sake of convenience in this experimentation, we’re committed to moving towards fully open-source alternatives that better align with our values of accessibility and transparency in the long run.

Beyond Bo: Potential applications for social good

How can a LLM-powered virtual character like Bo serve as a tool for social transformation? While commercial applications often focus on customer service or personalized marketing, we at Col·lectivaT, place value on its tremendous potential for such applications to positively impact areas like language preservation, education and social awareness.

One exciting application is in language learning, especially for minorized languages that often lack the resources for comprehensive study and practice. Children learning these languages might only have limited exposure in school or at home. A fun, interactive companion like Bo can offer a much-needed space to practice, reinforcing the use of their language in engaging, pressure-free ways. Learners can converse without fear of making mistakes. This virtual interaction creates an opportunity to explore and play with language, helping bridge the gap where traditional resources fall short.

Beyond language learning, conversational companions can open the door to personalized conversations on topics like climate change, menstrual health, addiction, or LGBTI issues—topics that kids might find difficult to talk about with teachers or parents. These virtual companions provide a safe space for children and teens where no question is too embarrassing or awkward to ask.

We must say that this isn’t about replacing teachers or caregivers, but about complementing them and scaling their efforts within their oversight. Virtual companions can provide a judgment-free, personalized learning environment—whether that’s in practicing a language, discussing sensitive issues, or simply interacting with technology in a more intuitive way.

Moving forward

As we look ahead, we’re eager to continue experimenting with the platform, testing its capabilities and limitations. Our technical next steps include integrating the latest open sourced LLaMA models to enhance the language capabilities and refining our in-house text-to-speech models to ensure more natural and localized voice interactions. We’ll also explore how to seamlessly integrate these solutions into educational and social settings.

We’d love to hear your thoughts! If you’ve interacted with Bo or have ideas for how this technology can evolve, please reach out and share your feedback.

We’d also like to thank Yuxuan Peng, a full-stack web developer from our Awal initiative, for his invaluable volunteer contributions to this project.

Can LLMs help revitalize an endangered language? That’s exactly what we’re exploring in my latest collaboration creating a chatbot that speaks and helps practice Ladino: Estreyika Perez.

Today marks her launch and you can listen to our conversation with the Estreyika team and see her capabilities in action. I’m delighted to see this becoming reality as this represents a major leap from when we first created the automatic translator for Ladino in 2022. Seeing fluent conversations happening to help people practice this severely endangered Judeo-Spanish language (UNESCO lists only ~50,000 speakers worldwide) is incredible progress!

From Skepticism to Success

When Dr. Carlos Yebra López first contacted me with his idea, I really thought this wouldn’t be possible but gave it a chance thinking our rule-based translator would help. The challenge seemed immense - how could we create a conversational AI for a language with such limited digital resources?

At the end, it turns out that modern LLMs know at least something of Ladino and with the right optimization and linguistic knowledge push, we can obtain a compelling conversational experience. This discovery opened up possibilities we hadn’t initially imagined.

Building on Digital Foundations

This breakthrough is thanks to both technological advancements of LLMs, but beyond that, the amazing work by linguists and activists who’ve built Ladino’s digital presence over the years. Our previous work on machine translation and text-to-speech synthesis for Ladino, published in “Preparing an Endangered Language for the Digital Age”, laid crucial groundwork for this project.

The technical challenge was significant. Unlike our MT project where we could build rules to translate from one language to another, a real conversation requires a deep understanding of cultural context, authentic language use, and the ability to maintain character consistency throughout extended dialogues. This is now possible with the advent of LLMs but still we had to carefully craft prompts to ensure Estreyika doesn’t fall into “Spanishisms” - using Spanish grammar patterns or vocabulary that would feel inauthentic to native Ladino speakers - or provide inaccurate information.

Community Response and Cultural Authenticity

The feedback from students and the community has been remarkable - many praising Estreyika’s authenticity in Ladino, her captivating personality, and her genuine enthusiasm that makes learning feel like chatting with a knowledgeable friend from 19th century Istanbul.

What strikes me most about the user feedback is how Estreyika manages to embody not just the language, but the cultural essence of Sephardic heritage. Students report feeling emotionally connected to the conversations, learning not just vocabulary but traditions, recipes, and historical context that brings the language to life.

The Path Forward

This is just a pilot, but it validates our approach. Thrilled to continue pushing the boundaries of what’s possible when AI meets language revitalization. The success of Estreyika demonstrates that even severely endangered languages can benefit from modern language technology - not to replace human speakers, but to create new pathways for cultural transmission and learning.

Try Estreyika Yourself

🎥 Watch the full presentation with demo: YouTube

💬 Chat with Estreyika: Telegram Bot

I’ve always been inspired by traveling, and my favorite part of it is savoring the local dishes. Last week, I had the chance to do just that in Kigali, Rwanda, while I was attending the closing session of our project, “MT Rwanda.” I’m excited because the results of this project will undoubtedly make it easier for me to ask local taxi drivers to take me to the best restaurants in town the next time I visit this impressive country.

In this blogpost, I will share some exciting details and learnings on community data collection, multilingual model fine-tuning, and the close collaboration of two dynamic teams: CLEAR Global and Digital Umuganda, pushing forward language technology for Kinyarwanda.

MT Rwanda’s Mission

Our mission in “MT Rwanda” was nothing short of groundbreaking: to boost open-source machine translation (MT) for impact in Kinyarwanda. This Rwandan language is spoken by millions and is central to the culture and identity of the nation. The project’s primary focus areas for impact were education and tourism, with a clear aim to bridge linguistic gaps and empower local communities.

In this endeavor, I had the privilege of mentoring the Rwandan AI company, Digital Umuganda, on data collection and MT model development. They had already been diligently pioneering open-source Kinyarwanda language technology for quite some time now. Thanks to their community efforts, Kinyarwanda has gained quite a few open-source datasets. The most notable of these is definitely their Common Voice campaign, which placed Kinyarwanda in the top 5 in terms of the number of speech hours collected.

This time, the task was to push towards improving machine translation for this language. Kinyarwanda is one of the languages supported in Google Translate. It’s also supported in the multilingual No Language Left Behind (NLLB) model released by Meta. However, all these models lack the quality to ensure reliability in real-world solutions, as many low-resource languages do. We needed to ensure they perform up to expectations in two real-life use-cases.

The Power of Community-Driven Data

One of the most exciting aspects of this project was the data collection campaign. To train the MT models effectively, we needed a substantial amount of high-quality parallel data. What made this campaign stand out was the community-driven approach adopted. With the enthusiastic participation of volunteers, we managed to collect over 60,000 sentence translations.

The source sentences to translate were crawled automatically from sites related to tourism (e.g. wikitravel, booking.com, tripadvisor), both in English and Kinyarwanda, and also web content we collected on finance, business, and economy-related articles on Wikipedia, Atingi, and Coursera. The source data can be accessed from this repository on Hugging Face and the scrapers on GitHub.

Open source = Collective ownership

In this project, we underscored the importance of open-source language technology. We started with Meta’s No Language Left Behind (NLLB) model as our base, trained with over 10 million web-crawled Kinyarwanda sentence translations as well as supporting 200 languages. Our approach involved a two-stage training process. In the first stage, we utilized diverse data from general, education, and tourism domains to enhance the model’s overall performance. Then, in the second stage, we specialized the model for education and tourism separately. All training was conducted bidirectionally, and you can find the fine-tuning training and testing scripts in this Github repository for a closer look.

The project’s commitment to open-source principles means that all the data and models we created are openly accessible to anyone interested in Kinyarwanda language technology. This also ensures reproducibility and collaboration from external collaborators to carry the flag forward. You can find these valuable resources on HuggingFace, complete with demos that showcase their capabilities: Education model and demo, Tourism model and demo.

What Lies Ahead

As we wrap up this transformative project, our sights are set on the future. We’re eager to publish a technical paper soon, sharing our insights, challenges, and triumphs with the wider research community. Collaborating with Digital Umuganda has been a rewarding experience, and I’m quite enthusiastic about collaborating with them further on.

My impression from our end-of-project session in Kigali is that the road ahead for Kinyarwanda is filled with promise. Together, we have identified many interesting ideas and also challenges to bring the initiative forward. I’m looking forward to witnessing and being part of further advancements in open-source Kinyarwanda language technology and its potential in creating an impact for Rwanda. And of course, making more opportunities to visit and explore further the delicious Rwandan cuisine.

My colleague Geo couldn't resist taking a snap of my happy face trying Ikisafuriya for the first time. It was finger-licking good!

Catotron is a great example of people-oriented and people-driven language technology. It will prevent the exclusion of Catalan speakers from using speech-interfaced technology like voice assistants, GPS, Telephone IVR etc.

I am very excited to see that Catalan media outlets are celebrating its launch with us. Among some of them:

• Catotron es renova per impulsar la síntesi de veu en català by Metadata.cat
• Siri o Alexa en català? Una nova eina ho facilita by Diari més
• La cooperativa Col·lectivaT crea el primer motor de síntesi de veu en català by La República
• El primer motor de síntesi de veu en català ja és una realitat by Nació digital
• La cooperativa Col·lectivaT crea el primer motor de síntesi de veu en català by El Punt Avui

For those who read “prosody” twice there, it refers to the “music-like” aspects in speech, which correspond to intonation, rhythm and stress. While words carry “what” is said in an utterance, prosody deals with “how” they are delivered.

In this post, I’m giving a further TL;DR for the fast researchers out there. You can follow links to respective papers and libraries if you’re interested.

Prosodically annotated TED Talks

To begin my studies, I needed a big corpus to study and train machine learning models based on prosody. I created Prosodically annotated TED Talks (PANTED) corpus based on my supervisor Mireia Farrus’s version and augmented with additional prosodic and syntactic information.

Punctuation restoration with prosody

The whole reason behind creating PANTED was to see the effect of prosody on the punctuation restoration task. It showed that pausing and intonation information can improve the automatic punctuation of ASR output. (Original paper, punkProse code)

Proscript

At some point, I realized how painful it was working with Praat and TextGrids when doing machine learning, so I created my own data structure and visualizer. Proscript helps represent speech with segment-level prosodic features in a tidy CSV file. Python library in Github.

Prosograph

Prosograph enables manual examination of Proscripts along with their audio by visualizing speech-related characteristics besides words similar to a musical scoreMusical score. It’s programmed in Processing and is accessible from github.

movie2paralleDB

I wanted to go into translation from the beginning of my PhD, although I knew it might be a bit tough. Not having any parallel speech corpora around didn’t stop me of course. I built movie2parallelDB to make corpora automatically, out of dubbed movies and series.

Heroes Corpus

The result of this was Heroes corpus, an English-Spanish parallel movie speech corpus made out of my childhoods popular series Heroes. We showed in the paper how silence information can improve spoken machine translation.

I also earlier explained about how I used this corpus in building a prosody-informed machine dubbing setup.

In this post I tried to compile some nice papers that I came across in EMNLP 2020 together with the work that I collaborated with. I’ve curated the list with a focus on machine translation and transcription of under-represented languages.

Some personal notes first: This conference was a first for me in a couple of ways. Firstly, it was the first major NLP conference that I participated. Secondly, it was my first fully-virtual multi-day conference. Despite a couple of technical hick-ups, I found the virtual setup to function rather well for me connecting from my cozy home. It’s really nice to see that it is possible to organize events of this scale given the circumstances of the pandemic.

Although, I have to say the experience was incomparable to a presential conference. Strolling through links, chatboards and virtual environments barely replaces the constant flow you experience in the physical conference setup. I am looking forward to see hybrid setups in the future combining presential and virtual participation. I actually believe a virtual alternative has long been due for conferences, regardless of the pandemic. Why? First, virtual participation allows saving a tons of carbon generated from avoidable travel. Second, it is a half-way remedy for researchers that face financial, parental and visa restrictions.

Regarding the content, I was very very happy to see this amount of work on the under-resourced language spectrum. Though, I am probably biased since that was the kind of work I was looking out for. I am not so sure of the diversity of languages represented in the conference. I would love to see a report on that the coming days.

I’d like to start the list with the work that I have collaborated with. Both are motivated by the purpose of bringing NLP and language technology to marginalized and under-represented languages and gives me hope and inspiration for the future of NLP research. I am looking forward to see more and be part of transversal and disruptive collaborations like these.

Masakhane - Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages (Paper, Keynote video)

Masakhane is a cross-border, inclusive, grassroots organisation with a decolonial mindset and a stand against the inherent professionalism, anglo-centricity and academic dominance in the NLP field. They promote a participatory and accessible NLP research methodology and showcase it for languages of Africa. A continent that has been unfortunately but unsurprisingly left mostly blank in terms of coverage in NLP research. The paper rejects the notion of “low-resourcedness” as a mere question of data availability and brings into light many aspects that drive a language into this category. 48+ professional and non-professional researchers has worked together to reverse that while publishing benchmarks for 30+ African languages, evaluations, datasets and open-sourced implementations. – https://www.masakhane.io/

TICO-19: the Translation Initiative for Covid-19 (Paper)

The Translation Initiative for Covid-19 (TICO-19) came together shortly after the beginning of the Covid-19 pandemic with the aim of enabling the translation of content related to COVID-19 into a wide range of languages. It is a unique collaboration between several academic (Carnegie Mellon University, George Mason University, Johns Hopkins University) and industry players (Amazon, Appen, Facebook, Google, Microsoft, Translated) and Translators without Borders. TICO-19’s objective is to prepare data for a variety of the world’s languages to be used by professional translators and for training Machine Translation (MT) models in Covid-19 domain. – https://tico-19.github.io/

Here goes the papers that caught my eye during the conference. If you have more paper suggestions similar to these, please don’t hesitate to drop me a note.

Sparse Transcription - S. Bird (Paper, Video)

• Approach transcription as observing, not data creation
• Phone recognition for transcripton is not enough. We need lexical information.
• Traditionally spoken languages are translated first, not literally translated
• Leverage keyword spotting instead of ASR
• Paper includes more on sparse transcription model:
  • data structure
  • 8 transcription tasks
  • transcription workflows
  • evaluations

Consistent Transcription and Translation of Speech – Sperber+ (Paper, Video)

• Objective: Obtain both transcript and translation of speech and make them consistent
• For evaluation, WER and BLEU by themselves are not enough because they do not measure consistency between the transcription and translation result.
• Propose metrics to evaluate consistency
• Evaluate on many models: cascaded, direct and joint models
• Data: Must-C Corpus (EN-DE)
• Joint models (see below) perform the best in terms of WER, BLEU and consistency
• Best of the best: Two-stage model
• Conclusion: Coupled inference and end-2-end training are desirable in speech translation

ChrEn: Cherokee-English Machine Translation for Endangered Language Revitalization – Zhang+ (Paper, Video)

• Obtain best possible model for bidirectional Cherokee-English MT
• Data: 14k parallel sentences (collected as part of the work)
• Results:
  • RNN performs better than transformer
  • Out of domain gives half the BLEU
  • Using BERT on En-Chr direction +0.5% BLEU
  • SMT performs better with out-of-domain data
  • A lot of monolingual data doesn’t necessarily help - 5K backtranslated sentences give the best performance
  • Cross-lingual transfer: * Unrelated languages help * Evaluated transferring vs. multilingual
• Code and data

Can Automatic Post-Editing Improve NMT? – Chollampatt+ (Paper, Video)

• Created new large-scale EN-DE post-editing corpus from movie subtitles
  • SubEdits: 161k triply-parallel corpus: src, mt, post-edit(pe)
  • open source
• How much improvement on NMT from the usage of post-editing data?
  • BLEU: 61.88 -> 64.53 (+3%)
  • Human evaluation scores: 3.4 -> 3.9 (out of 5)
  • Improvements start from a training corpus size of 25k triplets
• How much artifical APE data help?
  • Artificial: take a parallel corpus, machine translate it, assume tgt as post-edit
  • SubEscape corpus: 5.6 M triplets
  • Marginal improvement (+0.06 BLEU)
  • Artificial data doesn’t help by itself
• Post-editing data improves only if it’s in the domain
• Data: Post-editing Datasets by Rakuten (PEDRa)

Simulated multiple reference training improves low-resource machine translation – Khayrallah+ (Paper, Video)

• Data augmentation for NMT by paraphrasing of target side sentences
• Record increase in quality (more than what you get from back-translation)
• State that paraphrasers are getting more and more available for languages.
  • PRISM paraphraser in 39 languages
• Data

BLEU might be Guilty but References are not Innocent – Freitag+ (Paper, Video)

• Observation: BLEU doesn’t show the improvement obtained from techniques like back-translation
• Propose: When BLEU seems to act weird, revise the reference translations
• Detected problem: Reference translations tend to be in translationese
  • := imitating the structure of the source sentence
• MT model strives to be the most average translationese
• When you improve fluency with methods like back-translation or automatic post-editing, BLEU conditioned on translationese doesn’t show the improvement even though human evaluations do
• Solution: paraphrase the reference sentences