The process of building language features for Galaxy AI involved much time and effort as each language presents a unique structure and culture. Samsung’s Researchers from around the world — in Brazil, China, India, Indonesia, Japan, Jordan, Poland and Vietnam — shared the challenges and triumphs behind the development of Galaxy AI. Samsung Newsroom compiled a recap of their stories below.

Developing a Translation Model

Galaxy AI features such as Live Translate perform three core processes: automatic speech recognition (ASR), neural machine translation (NMT) and text-to-speech (TTS).

▲ Automatic speech recognition (ASR), neural machine translation (NMT) and text-to-speech (TTS) each require distinct sets of information for training

Samsung R&D Institute Vietnam (SRV) faced obstacles with automatic speech recognition (ASR) models because Vietnamese is a language with six distinct tones. Tonal languages can be difficult for AI to recognize because of the complexity tones add to linguistic nuances. SRV responded to the challenge with a model that differentiates between shorter audio frames of around 20 milliseconds.

Samsung R&D Institute Poland (SRPOL) had the mammoth hurdle of training neural machine translation (NMT) models for a continent as diverse as Europe. Leveraging its rich pool of experience in projects spanning more than 30 languages across four time zones, SRPOL was able to navigate the untranslatability of certain phrases and handle idiomatic expressions that may not have direct equivalents in other languages.

Samsung R&D Institute Jordan (SRJO) adapted Arabic — a language spoken across more than 20 countries in about 30 dialects — for Galaxy AI. Creating a text-to-speech (TTS) model was no small endeavor since diacritics and guides for pronunciation are widely understood by native Arabic speakers but absent in writing. Based on a sophisticated prediction model for missing diacritics, SRJO was able to publish a language model that understands dialects and can answer in standard Arabic.

The Importance of Data

Throughout the process of training Galaxy AI in each language, an overarching theme was the importance of open collaboration with local institutions. The quality of data used directly affects the accuracy of ASR, NMT and TTS. So Samsung worked with various partners to obtain and review data that reflected each region’s jargon, dialects and other variations.

▲ Each language has a distinct set of qualities that pose challenges in creating an AI language model for it. Tones add to the complexity for tonal languages such as Vietnamese.

Samsung R&D Institute India-Bangalore (SRI-B) collaborated with the Vellore Institute of Technology to secure almost a million lines of segmented and curated audio data on conversational speech, words and commands. The students got hands-on experience on a real-life project as well as mentorship from Samsung experts; the rich store of data helped SRI-B train Galaxy AI in Hindi, covering more than 20 regional dialects and their respective tonal inflections, punctuation and colloquialisms

Local linguistic insights were imperative for the Latin American Spanish model because the diversity within the language is mirrored by the diversity of its user base. For example, the word for swimming pool could be alberca (Mexico), piscina (Colombia, Bolivia, Venezuela) or pileta (Argentina, Paraguay, Uruguay) based on which region you’re from. Samsung R&D Institute Brazil (SRBR) worked with science and technology institutes SiDi and Sidia to collect and manage massive amounts of data as well as refine and improve upon audio and text sources for Galaxy AI’s Latin American Spanish model.

Samsung R&D Institute China-Beijing (SRC-B) and Samsung R&D Institute China-Guangzhou (SRC-G) partnered with Chinese companies Baidu and Meitu to leverage their expertise from developing large language models (LLM) such as ERNIE Bot and MiracleVision, respectively. As a result, Galaxy AI supports both main modes of Mandarin Chinese and Cantonese.

In addition to external cooperation, due diligence and internal resources were also essential.

Bahasa Indonesia is a language notorious for its extensive use of contextual and implicit meanings that rely on social and situational cues. Samsung R&D Institute Indonesia (SRIN) researchers went out into the field to record conversations in coffee shops and working environments to capture authentic ambient noises that could distort input. This helped the model learn to recognize the necessary information from verbal input, ultimately improving the accuracy of speech recognition.

There are many homonyms in Japanese as the number of sounds is limited in the language. So many words must be determined based on the context. Samsung R&D Institute Japan (SRJ) used Samsung Gauss, the company’s internal LLM, structure contextual sentences with words or phrases relevant to each scenario to help the AI model differentiate between homonyms.

Samsung’s Global Research Network

The professionals across various Samsung R&D Institutes made full use of Samsung’s global research network.

Before tackling Hindi, SRI-B collaborated with teams around the world to develop AI language models for British, Indian and Australian English as well as Thai, Vietnamese and Indonesian. Engineers from other Samsung research centers visited Bangalore, India, to bring Vietnamese, Thai and Indonesian to Galaxy AI.

▲ Staff and collaborators pose in front of Samsung R&D Institute India-Bangalore (SRI-B)

SRPOL had extensive experience developing ASR, NMT and TTS models for a multitude of languages. A key player in Galaxy AI’s language expansion, SRPOL collaborated across continents to support SRJO with Arabic dialects and SRBR with Brazilian Portuguese and Latin American Spanish.

Samsung developers at each of these locations learned to collaborate across borders and time zones. Developers from SRIN even observed the local fasting customs in India when meeting their SRI-B colleagues. Many reflected on their work with pride and gratitude — realizing the lasting implications this project has on language, culture, heritage and identity.

Ongoing Efforts as the Journey Continues

Samsung recently introduced Galaxy AI to its latest foldables and wearables. Since its release earlier this year, Galaxy AI has already been used on more than 100 million devices. “We’re expecting to reach 200 million devices by the end of 2024,” said Won-joon Choi, EVP and Head of the Mobile R&D Office, Mobile eXperience Business at Samsung Electronics at a recent panel discussion.

Amidst this mission to democratize AI, it is important to look back and celebrate the accomplishments and progress that have led to providing this safe and inclusive technology that will benefit humanity and improve lives. By building up the Galaxy AI ecosystem with even more features, languages and regional variations, Samsung is facilitating cross-cultural exchanges in unprecedented ways to realize its vision of AI for All.

Samsung R&D Institute Japan (SRJ) was established as an R&D center focused on hardware such as home appliances and displays. With the demand for AI innovation ramping up globally, SRJ in Yokohama has also been operating a software development lab to create Galaxy AI’s Live Translate, which automatically translates voice calls in real time, since the end of last year.

“Live Translate is particularly efficient for travel scenarios such as visitors to this year’s Olympic Games in Paris,” says Takayuki Akasako, the Head of Artificial Intelligence at SRJ. “We are currently developing a speech recognition program for people who are both sightseeing and watching the Paris Olympic Games; by training the speech recognition program to learn about the games and locations of stadiums for Paris 2024.”

Understanding Context in Voice Recognition

For those already using the translation features of Galaxy AI, such functionalities may seem very useful. But for developers who have made the features come to life, they know that being able to communicate while traveling abroad isn’t something that can be taken for granted.

One thing the team noted was that there are more homonyms in Japanese than some other languages. For instance, ‘chopsticks’ (Hashi,箸) and ‘bridge’ (Hashi,橋) are relatively easy to distinguish due to the difference in intonation, but words like ‘sightseeing’(Kankō,観光), ‘customs’(Kankō,慣行), ‘public’ (Kōkyō,公共) and ‘prosperity’ (Kōkyō,好況) must be judged based on the context.

“Judgement becomes more difficult when the context is ambiguous, such as names of locale and people, proper nouns, dialects and numbers,” says Akasako. “So in order to improve the accuracy of speech recognition, a lot of data is needed.”

“We always look for ways to fine-tune the AI model for key events and moments in a timely manner,” continues Akasako. “With a lot of new combinations of place names and activities, it’s important that the context is still clear when people are using Galaxy AI.”

Challenges in Collecting Efficient Data

While recognizing the types of data needed is also important, collecting the data in and of itself is a challenge in its own right.

Previously, the SRJ team used human-recorded data to train the speech recognition engine for Live Translate, which didn’t result in sufficient data collection.

Samsung Gauss, the company’s Large Language Model (LLM), uses scripts to structure sentences with words or phrases that are relevant to each scenario. The data collected with Samsung Gauss is not only recorded by humans, but also generated by a speech synthesis text-to-speech (TTS) data, through which human resources do the final check on the quality. Using this method, the team has seen a dramatic improvement in data collection efficiency.

“Every time a problem is identified and solved, the accuracy of speech recognition improves significantly,” says Akasako. “Regardless of where people are, our goal is connecting people with each other, and the tools powered by Galaxy AI will ensure more fun and efficient communication.”

There’s a saying at the Samsung R&D Institute Poland (SRPOL): “A day at SRPOL lasts 96 hours”. It refers to the center’s global role as one of the largest and fastest-growing R&D centers in the region, often working across four different time zones. Sitting at the heart of Europe while covering many European and global markets, SRPOL has worked on automatic speech recognition, neural machine translation and text-to-speech models for more than 30 languages. When it came to bringing 10 languages to Galaxy AI, this expertise meant the team was well suited to seamlessly blend cultural perspectives with Samsung’s global technology.

SRPOL has years of experience in Natural Language Processing. What makes it unique is its adaptability to work on any language thanks to the passionate team and their tools, such as a crowdsourcing platform that enables fast and agile development.

“Collaboration across the continent means relentless data collection, annotation and research, which has become something we really enjoy,” says Kornel Jankowski, Head of Speech Decoding at SRPOL. “We’ve dealt with so many languages that our team developed universal, language-agnostic skills. When we’re asked to support a new language model, everybody’s attitude is: Oh wow, we get to learn another one, that’s going to be fun!”

A European Center for AI Language Development

Language is a cornerstone of culture and communication across Europe regardless of whether it’s incorporated into technology. However, it presents unique challenges for the team at SRPOL, who develop AI models for European languages.

“Each language and the culture that it is part of, comes with hurdles that make us reevaluate how we perceive a specific issue,” explains Adam Ros, Head of Artificial Intelligence at SRPOL. These hurdles include navigating the untranslatability of certain phrases and handling idiomatic expressions that may not have direct equivalents in other languages.

The team saw these challenges as an opportunity to make SRPOL a European center for AI language development. The biggest benefit of this is that it shortens the communication path between different departments and crucially, the decision-making path. Whether it is a matter of automatic speech recognition, neural machine translation or text-to-speech, teams could simply walk over to colleagues in Mobile Quality Assurance and efficiently solve problems together.

While this has helped, it hasn’t overcome all AI challenges. Inevitably, there are limitations in AI models when dealing with multiple European languages, such as translating without context or variations in intonation. However, the team saw these as an opportunity to keep learning and innovating.

“My team never stops at just one example when handling a new word or topic. Some European languages are harder than others,” adds Ros. If you’ve ever been to Spain, you know that Spanish is often spoken at blazing-fast speeds and we need to train AI well to handle that.”

Galaxy AI’s expansion required novel cross-continent collaboration, but the work soon grew beyond European borders. SRPOL supported the Jordan team’s efforts to teach Galaxy AI Arabic’s myriad of dialects, as well as the Brazil team’s work on Latin American languages.

The importance of language and cultural difference subtleties are all on the radar of SRPOL product developers because they can all be noticed by the target — the end users.

“There are subtle differences between European cultures that impact whether something feels natural to the end user. For example, people in some countries expect to read prices with the euro symbol (€), while others are accustomed to seeing it spelled out, e-u-r-o-s,” says Agata Maria Rozycka, Head of Voice Intelligence Research at SRPOL. “If this cultural nuance is not reflected translated text, the interface might seem less intuitive to a user. Implementing these micro-level insights into interface design can make technology feel more natural across diverse cultures.”

“The team has been remotely communicating and collaborating across different countries for many years, building up numerous effective communication channels,” says Marcin Mrugala, Head of Mobile Quality Assurance at SRPOL. “We were ready to do our part in enabling Galaxy AI to lower language barriers around the world.”

Technology for Bridging Cultures

Managing and integrating diverse linguistic and cultural insights is a challenging task, but it is essential for Samsung’s vision for Galaxy AI — lowering the barriers that divide people based on language and culture, and enabling them to create deeper connections.

“We’re not just building technology of the future, we’re building teams of the future too. Our best practices are designed to refine products based on differences across countries, but we fundamentally believe our similarities far outweigh our differences and our technology can unite cultures,” says Mrugala.

“Our goal is to bring people together, to make their lives easier, and to simplify their daily tasks. We’re seeing our families using the Voice Recorder in new ways, and we can now call our friends and different countries and talk with them in their own language. It is magical to see this change in the world and to be part of it. Galaxy AI brought SRPOL people together and now we are bringing together the world,” concludes Rozycka.

Hidden inside the Vellore Institute of Technology in Chennai, India, is a lab filled with futuristic audio equipment. One will find mannequins — known in the industry as head and torso simulators — as well as binaural microphones and hearing devices. They are stored in special chambers treated with an advanced sound absorption system, making this lab the first of its kind in India. Imagine such a facility is used to develop the latest high-end high fidelity (Hi-Fi) equipment.

This is where the Vellore Institute of Technology collaborates with Samsung to produce and develop data and insights that power the latest AI models for Galaxy AI’s language capabilities. The facility was developed as part of Samsung SEED (Students Ecosystem for Engineered Data) Labs — an initiative that enables university staff, students and interns in India to work on projects requested by Samsung since 2021. This is just one of several university programs funded by Samsung in which students have the opportunity to work on projects with technical experts from the company.

“As a student, I love being able to work on multiple projects with a well-known and respected company such as Samsung,” says Yashika Ilanchezhiyan, a Samsung SEED student. “I’m given the confidence to learn new skills in a practical way and feel like I’m making a real difference in current and future products.”

“This kind of collaboration is a win-win situation,” says Giridhar Jakki, Head of Language AI at Samsung R&D Institute India – Bangalore (SRI-B). “Thanks to our projects with universities, we are able to access additional expertise and custom datasets. Partnering universities receive investment, financial incentives and expert mentorship from Samsung as a result.”

Lowering Language Barriers

SRI-B has collaborated with teams around the world to develop AI language models for British, Indian and Australian English as well as Thai, Vietnamese and Indonesian. Recently, core engineers from other Samsung Research centers visited Bangalore, India — where the SRI-B team helped ramp up the technology to bring Vietnamese, Thai and Indonesian to Galaxy AI. SRI-B was therefore ideally positioned to develop the Hindi language for Galaxy AI.

“Every language has its challenges,” says Jakki. “But when you consider the end goal of bringing people the ability to communicate in other languages, it’s worth every ounce of effort. We couldn’t wait to bring Hindi to Galaxy AI.”

Developing the Hindi AI model wasn’t simple. The team had to ensure more than 20 regional dialects, tonal inflections, punctuation and colloquialisms were covered. Additionally, it is common for Hindi speakers to mix English words in their conversations. This required the team to carry out multiple rounds of AI model training with a combination of translated and transliterated data.

“Hindi has a complex phonetic structure that includes retroflex sounds — sounds made by curling the tongue back in the mouth — which are not present in many other languages,” says Jakki. “To build the speech synthesis element of the AI solution, we carefully reviewed data with native linguists to understand all the unique sounds and created a special set of phenomes to support specific dialects of the language.”

Collaborative efforts between Samsung and academic partners were instrumental in developing the AI language model that reflected the cultural nuances of the India’s regions. The Vellore Institute of Technology helped secure almost a million lines of segmented and curated audio data on conversational speech, words and commands. Data was a crucial component for a task as critical as incorporating the fourth most spoken language in the world into Galaxy AI. Working with universities ensured Samsung was using the highest quality data.

Global Connections Deliver Big Impacts

This project perfectly encapsulates Samsung’s philosophy of open collaboration and the company’s belief that sharing expertise and perspectives ensures meaningful innovation. In the case of SRI-B, this not only includes working with academia but also sharing insights and best practices with other Samsung research centers around the world.

“I’m extremely proud of what we’ve achieved with the help of our partners,” says Jakki. “AI innovation through collaboration is a big part of what we do. We will continue to better understand, collect and analyze language data so more people can have access to AI tools in the future.”

A diverse country with more than 203 million people embodying a wide range of cultures and traditions, Brazil uses Brazilian Portuguese as its official language. Meanwhile, 22 neighboring countries use Latin American Spanish.

Although Brazilian Portuguese and Latin American Spanish are widely spoken, intricate variations in both languages presented various challenges when teaching Galaxy AI to discern and distinguish regional differences. That’s why Samsung R&D Institute Brazil (SRBR) collaborated with Samsung experts from Mexico — as well as third-party partners such as the science and technology institutes SiDi and Sidia — to assemble a multidisciplinary and highly skilled team that could tackle the task.

Lower Barriers, Higher Understanding

The team used thousands of sources and a combination of machine learning and language processing tools to improve the AI model’s recognition of speech, written texts and regional variations. But local jargon and names of famous figures — including sports teams, celebrities and bands — vary widely between regions. Also, the same meaning can be expressed in many different words. While language models need localized data to gain a comprehensive understanding of the different languages to be translated, such variations inevitably present obstacles.

For example, swimming pool is “alberca” in Mexico — but it is “pileta” in Argentina, Paraguay and Uruguay. Meanwhile, in Colombia, Bolivia and Venezuela, swimming pool is “piscina”, which is also used in Brazil but with a slight tonal difference. And while Colombians might say “chévere” to refer to something cool, Mexicans instead say “padre.”

These differences represent huge challenges for AI language understanding and learning, but the team overcame them by building larger language models, refining processing tools — and collaborating across borders and time zones.

“We had to consider local slang and different ways of speaking before adapting and testing the model accordingly, which required close collaboration between the SRBR quality assurance (QA) team and development teams,” says Mateus Pedroso, Senior Manager and Head of Software Quality Lab at SRBR. “Since SRBR is located three hours ahead of the QA team in Mexico and 12 hours behind the management team in Korea, we had to create new communication channels and processes to align results and share progress. This multicultural collaboration generated a fiesta of ideas and solutions for Galaxy AI.”

Communicating Success

Samsung’s philosophy of open collaboration came to life during this regional project as it was an iterative process that leveraged evolving technology on a global scale. To overcome linguistic and cultural barriers, the SRBR team needed to collect and manage massive amounts of data — continually refining and improving upon audio and text sources.

The teams carved out key areas of responsibility to ensure everyone could benefit from the collective skill sets across the company’s Latin American offices. The SRBR development team served as the intermediate stakeholder of the project, receiving directions from Samsung’s headquarters and developing new updates to improve the AI model while carrying out tests for numerous use cases.

“The testing phase required extensive communication and collaboration with QA teams to optimize the user experience, and each adjustment needed further testing and review,” says Leandro Flores de Moura, Software Development Manager at SiDi. “The success of Galaxy AI’s language capabilities is built on communication and collaboration as much as it is on technical expertise” adds Nathan Castro, QA Test Developer at SiDi.

A Roadmap for Culture

What makes Galaxy AI particularly interesting for everyone involved is the fact that this wasn’t merely a language project. To them, language is a cultural guide that provides valuable insight into people’s heritage and identity.

“For SiDi’s QA team, this was an endeavor that will change the world by enabling cultures to come together and overcome the difficulty of communicating in different languages,” adds Estefanía Castro Suárez, Test Developer at SiDi. “Knowing we were part of this fills us with pride and motivation.”

“The way the SRBR team collaborated exemplifies what Galaxy AI sets out to achieve — making the world a smaller place through communicating, sharing and interacting with people, even those who speak different languages,” concludes Pedroso. “This capability will only grow as more languages come on board with Galaxy AI.”

As Samsung continues to pioneer premium mobile AI experiences, we visit Samsung Research centers around the world to learn how Galaxy AI is enabling more users to maximize their potential. Galaxy AI now supports 16 languages, so more people can expand their language capabilities, even when offline, thanks to on-device translation in features such as Live Translate, Interpreter, Note Assist and Browsing Assist. But what does AI language development involve? Last time, we visited Vietnam to learn about preparing the data that is used to train AI models. This time, we’re seeing how teams made Galaxy AI a unique offering for both the Chinese mainland and Hong Kong.

The rapid growth in AI tools that use large language models (LLM) has been seen worldwide, and China is no exception. With Baidu’s ERNIE Bot and Meitu’s MiracleVision emerging as popular choices in China, Samsung R&D Institute China partnered with both companies to help build Galaxy AI features for the country.

Samsung R&D Institute China in Guangzhou (SRC-G) and Beijing (SRC-B) worked to ensure Mandarin speakers in China had the same Galaxy AI experience as other users around the world, despite the back-end technology looking very different. The team took advantage of the dedicated resources of Chinese dialects from third-party partners and built a unique Galaxy AI solution for China.

“We have the advantage of blending global best practices with China’s local practices, as well as creating new features and constantly improving them through daily communication with Chinese consumers,” says Hairong Zhang, Software Innovation Group Leader at SRC-G. “With rich development experience from the Galaxy S24, I’m proud of how our team cooperated with local Chinese AI companies such as Baidu and Meitu to provide a solution that resonates in China.”

At the beginning, the teams had to acclimate to each other’s working styles and iron out the initial kinks of information asymmetry. Daijun Zhang, Head of SRC-B, established a task force to ensure the project followed the development schedule and moved quickly toward its goals.

Thanks to the Beijing team’s experience in generating large-scale models and successful collaboration with third-party partners, all the generative AI features were successfully launched in China. The result is a solution that has local relevance and market-specific features such as Touch to Search.

Expanding on Chinese To Develop for the Cantonese Dialect

Chinese for mainland China (Mandarin) arrived on Galaxy AI with the launch of the Galaxy S24 in January 2024. But the job for Samsung R&D Institute China was far from finished. The team was also tasked with developing the AI model for Chinese in Hong Kong (Cantonese), a dialect that builds on the work already carried out for Mandarin but brings an entirely new set of language features to address.

In developing for Cantonese, the China R&D team faced major cultural challenges that it needed to respond to in order to fully support localization for the market. The first cultural phenomenon is the two sets of systems for writing and speech. Hong Kong locals use grammar and expressions similar to Mandarin when writing but adopt a completely different colloquial grammar when communicating daily. Also, Cantonese has nine tones for pronunciation, whereas Mandarin has four.

Another cultural phenomenon is that the Cantonese dialect itself develops with the times. Add to that the fact that people often blend Cantonese and English into conversations, and it’s clear to see why it was complicated to create test cases and validate language packs.

“Cantonese is a very unique dialect that varies in different Cantonese-speaking regions,” says Jing Li, who leads the operation for testing the Cantonese AI solution. “Some of the slang, phrases, vocabulary and even the tones are varied from place to place. Therefore, we conducted a large amount of work in verifying the Hong Kong-specific data, as well as proofreading tens of thousands of relevant test cases.”

With these complexities in mind, SRC-G and SRC-B worked together to support a deep code mix using a mixture of Cantonese and English for speech recognition, simultaneously supporting both written and spoken expressions in machine translation and reflecting current pronunciations in speech synthesis.

Cultural Impact of Communication

When Galaxy AI launched the Chinese (Hong Kong) language option, the customer feedback showed that the hard work of the Samsung R&D team was justified.

For both the Chinese mainland and Hong Kong, Samsung’s Galaxy AI activities show the importance of a global brand having a local presence and expertise, as well as the power of open collaboration with other organizations. In Hong Kong, Cantonese is a key part of the cultural identity of those who live there. That’s why it was so important for the team to get the AI language model right.

“Language and communication are crucial in every region and in all walks of life,” says Henry Wat, Heads of Engineering Group at Samsung Electronics Hong Kong. “No matter the language, any tool that helps people communicate is invaluable. I believe our work is meaningful.”

In the next episode of The Learning Curve, we will head to Brazil to see how a team works across cultures and borders to bring Galaxy AI to more people.

Samsung is pioneering premium mobile AI experiences. To learn how Galaxy AI is maximizing the potential of its users, we are visiting Samsung Research centers around the world. Now supporting 16 languages, Galaxy AI is enabling more people to expand their language capabilities, even when offline, thanks to on-device translation in features such as Live Translate, Interpreter, Note Assist and Browsing Assist. We recently visited Jordan to learn the complexities of developing an AI model for Arabic, a language with many dialects. This time, we’re going to Vietnam to explore how data is prepared to train AI models.

What is the difference between a ghost, grave and mother in Vietnamese? For a language spoken by 97 million people worldwide, very little. Each word translates to “ma,” “mả” and “má,” respectively — and can only be distinguished by tone. This illustrates how difficult it can be for AI models to learn a language, considering they cannot recognize firsthand the context and emotions of conversations nor the intentions of those speaking.

Samsung R&D Institute Vietnam (SRV) used finely refined data to help its AI model properly recognize even the most subtle differences in language.

The quality of data used directly affects the accuracy of automatic speech recognition (ASR), neural machine translation (NMT) and text-to-speech (TTS) — processes that help Galaxy AI features such as Live Translate, Interpreter, Chat Assist and Browsing Assist break down language barriers.

A Typhoon of Challenges

“Vietnamese is a complex and diverse language with rich expressions, many of which are challenging to capture,” says Ngô Hồng Thái, NMT lead at SRV. Of the 16 languages that Galaxy AI supports, Vietnamese was particularly difficult to develop.

“Personally, creating an AI model for Vietnamese was more daunting than our typhoons!” he adds before explaining the hurdles faced during the development process.

Vietnamese is a tonal language with six distinct tones. As evident in the “ma” example above, small nuances in vocalization can drastically alter the meanings of words. Therefore, a meticulous and detailed approach was necessary.

“When similar sounding words are broken down, one word consists of several short segments, or ‘frame sets’,” says Bui Ngoc Tung, ASR lead at SRV. “The AI model differentiates between the short audio frames of around 20 milliseconds to recognize what words correspond to a certain set of consecutive frames. As such, it is critical to put great effort into the early stages of the AI learning process.”

Furthermore, homophones and homonyms are common in Vietnamese. People can normally rely on context and nonverbal elements in conversations to differentiate between words that sound the same or are written the same but have different meanings. However, AI models need to be taught to accurately identify and differentiate between tones and similar words.

“This isn’t a straightforward task,” Thái explains. “Apart from the amount, the data needs to be accurate to ensure it is capable of recognizing the linguistic nuances that exist in Vietnamese.”

Rigorous Preparation

The data refinement process consists of three steps. First, the audio and text used to train the AI model must be reviewed and corrected. Then, this dataset goes through random checks for overall quality. Finally, the dataset is normalized and cleaned before use in training.

“We thoroughly performed a series of tests to check the accuracy of our dataset,” says Nguyen Manh Duy, TTS lead at SRV who oversees database creation. “We faced a number of unexpected problems including misspelled words in scripts and background noise or incorrect pronunciation during audio recordings. We spent significant time refining and improving our training data.”

A vital part of the data refinement process and the journey of taking AI data from good to great is the work of the Software Quality Engineering (SQE) team. The team plays an important role in testing and improving AI language data quality and they work closely with the AI language development project team to make it happen.

In addition to the unique linguistic challenges in Vietnamese, there is a lack of universally accessible data compared to more widely spoken languages. “This is another reason why the data refinement stage is so important,” he adds. “Since we had limited sources, every piece of data had to be fully reliable. There was no margin for error.”

Moreover, the AI model for Vietnamese must consider both tonal and regional differences. To improve the AI model’s accuracy, the team collected vast amounts of data with Vietnam’s northern, central and southern accents — resulting in an enormous amount of information to refine and verify.

Continued Improvement

Developers at SRV completed the project after months of hard work, and Vietnamese became one of the first languages to be supported by Galaxy AI. Despite this success, the team is ceaselessly working to improve the Vietnamese Galaxy AI experience.

“We’re continuing to enhance the AI model by incorporating user feedback about the relevance of words and phrases in Galaxy AI,” says Tran Tuan Minh, leader of the AI language development project at SRV. “We have just taken our first steps into a more open world  —  and we have so much more to explore together.”

In the next episode of The Learning Curve, we will head to China to dig into how AI models are trained and fine-tuned.

Teaching a language to an AI model is a complex process, but what if it isn’t a singular language, but a collection of diverse dialects? That was the challenge faced by the team at Samsung R&D Institute Jordan (SRJO). While Arabic was added as a language option for Galaxy AI features such as Live Translate, the team had to cater to the various Arabic dialects that span the Middle East and North Africa, with each varying in pronunciation, vocabulary and grammar.

Arabic is one of the top six most widely spoken languages around the world, used daily by more than 400 million people.¹ The language is categorized into two forms: Fus’ha (Modern Standard Arabic) and Ammiya (the dialects of Arabic). Fus’ha is typically used in public and official events, as well as in news broadcasts, while Ammiya is more commonly used for day-to-day conversations. Over 20 countries use Arabic, and there are currently around 30 dialects in the region.

Unwritten Rules

Recognizing the variation presented by these dialects, the team at SRJO employed a range of techniques to discern and process the unique linguistic features inherent in each. This approach was crucial in ensuring that Galaxy AI could understand and respond in a way that accurately reflects the regional nuances.

“Unlike other languages, the pronunciation of the object in Arabic varies depending on the subject and verb in the sentence,” says Mohammad Hamdan, project leader of the Arabic language development team. “Our goal is to develop a model that understands all these dialects and can answer in standard Arabic.”

TTS is the component of Galaxy AI’s Live Translate feature that lets users interact with speakers of different languages by translating spoken words into written text, and then vocally reproducing them. The TTS team faced a unique challenge, caused by the quirk of working with Arabic.

Arabic uses diacritics, which are guides for the pronunciation of words in some contexts, such as religious texts, poetry and books for language learners. Diacritics are widely understood by native speakers but absent in everyday writing. This makes it difficult for a machine to convert raw text into phonemes, the basic units of sound that are the building blocks of speech.

“There is a shortage of high-quality and reliable datasets that accurately represent how diacritics are correctly used,” explains Haweeleh. “We had to design a neural model that can predict and restore those missing diacritics with high accuracy.”

Neural models work similarly to human brains. To predict diacritics, a model needs to study lots of Arabic text, learn the language’s rules and understand how words are used in different contexts. For instance, the pronunciation of a word can vary greatly depending on the action or gender it describes. Extensive training from the team was the key to enhancing the Arabic TTS model’s accuracy.

Enhancing Understanding

The SRJO team also had to collect diverse audio recordings of the dialects from various sources, which had to be transcribed, focusing on unique sounds, words and phrases. “We assembled a team of native speakers in the dialects who were well-versed in the nuances and variations,” says Ayah Hasan, whose team was responsible for database creation. “They listened to the recordings and manually converted the spoken words into text.”

This work was crucial for enhancing the Automatic Speech Recognition (ASR) process so that Galaxy AI could handle the rich tapestry of Arabic dialects. ASR is pivotal in enabling Galaxy AI’s real-time understanding and response capabilities.

“Building an ASR system that supports multiple dialects in a single model is a complex undertaking,” says Mohammad Hamdan, ASR lead for the project. “It demands a thorough understanding of the language’s intricacies, careful data selection and advanced modeling techniques.”

The Culmination of Innovation

After months of planning, building and testing, the team was ready to release Arabic as a language option for Galaxy AI, enabling many more people to communicate across borders. This single team has made Galaxy AI services accessible to Arabic speakers, lowering the language and cultural barriers between them and people all over the world. In doing so, they have established new best practices that can be rolled out globally. This success is only the beginning: the team continues to refine their models and enhance the quality of Galaxy AI’s language capabilities.

In the next episode, we go to Vietnam to see how the team makes language data better. Plus, what does it take to train an effective AI model?

Arabic is just one part of the languages and dialects newly supported by Galaxy AI and available for download from the Settings app. Galaxy AI’s language features such as Live Translate and Interpreter are available on Galaxy devices running Samsung’s One UI 6.1 update.²

¹ UNESCO, World Arabic Language Day 2023, https://www.unesco.org/en/world-arabic-language-day
² One UI 6.1 was first released on Galaxy S24 series devices with a wider roll out to other Galaxy devices including S23 series, S23 FE, S22 series, S21 series, Z Fold5, Z Fold4, Z Fold3, Z Flip5, Z Flip4, Z Flip3, Tab S9 series and Tab S8 series

As Samsung continues to pioneer premium mobile AI experiences, we visit Samsung Research centers around the world to learn how Galaxy AI is enabling more users to maximize their potential. Galaxy AI now supports 16 languages, so more people can expand their language capabilities, even when offline, thanks to on-device translation in features such as Live Translate, Interpreter, Note Assist and Browsing Assist. But what does AI language development involve? This series examines the challenges of working with mobile AI and how we overcame them. First up, we head to Indonesia to learn where one begins teaching AI to speak a new language.

The first step is establishing targets, according to the team at Samsung R&D Institute Indonesia (SRIN). “Great AI begins with good quality and relevant data. Each language demands a different way to process this, so we dive deep to understand the linguistic needs and the unique conditions of our country,” says Junaidillah Fadlil, Head of AI at SRIN, whose team recently added Bahasa Indonesia (Indonesian language) support to Galaxy AI. “Local language development has to be led by insight and science, so every process for adding languages to Galaxy AI starts with us planning what information we need and can legally and ethically obtain.”

Galaxy AI features such as Live Translate perform three core processes: automatic speech recognition (ASR), neural machine translation (NMT) and text-to-speech (TTS). Each process needs a distinct set of information.

ASR, for instance, needs extensive recordings of speech in numerous environments, each paired with an accurate text transcription. Varying background noise levels help account for different environments. “It’s not enough just to add noises to recordings,” explains Muchlisin Adi Saputra, the team’s ASR lead. “In addition to the language data we obtained from authorized third-party partners, we must go out into coffee shops or working environments to record our own voices. This allows us to authentically capture unique sounds from real life, like people calling out or the clattering of keyboards.”

The ever-changing nature of languages must also be considered. Saputra adds, “We need to keep up to date with the latest slang and how it is used, and mostly we find it on social media!”

Next, NMT requires translation training data. “Translating Bahasa Indonesia is challenging,” says Muhamad Faisal, the team’s NMT lead. “Its extensive use of contextual and implicit meanings relies on social and situational cues, so we need numerous translated texts that the AI could reference for new words, foreign words, proper nouns and idioms – any information that helps AI understand the context and rules of communication.”

TTS then requires recordings that cover a range of voices and tones, with additional context on how parts of words sound in different circumstances. “Good voice recordings could do half the job and cover all the required phonemes (units of sound in speech) for the AI model,” adds Harits Abdurrohman, TTS lead. “If a voice actor did a great job in the earlier phase, the focus shifts to refining the AI model to clearly pronounce specific words.”

Stronger Together

It takes vast resources to plan for much data, and SRIN worked closely with linguistics experts. “This challenge requires creativity, resourcefulness and expertise in both Bahasa Indonesia and machine learning,” Fadlil reflects. “Samsung’s philosophy of open collaboration played a big part in getting the job done, as did our scale of operations and history of AI development.”

Working with other Samsung Research centers around the world, the SRIN team was able to quickly adopt best practices and overcome the complexities of establishing data targets. Furthermore, collaboration was good for advancing not only technology but also culture. When the SRIN team joined their counterparts in Bangalore, India, they observed the local fasting customs, creating deeper connections and expanding their understanding of different cultures.

For the team, Galaxy AI’s language expansion project took on a new significance. “We are particularly proud of our achievements here as this was our first AI project, and it won’t be our last as we continue to refine our models and improve the quality of output,” Fadlil concludes. “This expansion not only reflects our values of openness but also respects and incorporates our cultural identities through language.”

In the next episode of The Learning Curve, we will head to Samsung R&D Institute Jordan to speak to the team who led Galaxy AI’s Arabic language project. Tune in to learn about the complexities of building and training an AI model for a language with diverse dialects.