The process of building language features for Galaxy AI involved much time and effort as each language presents a unique structure and culture. Samsung’s Researchers from around the world — in Brazil, China, India, Indonesia, Japan, Jordan, Poland and Vietnam — shared the challenges and triumphs behind the development of Galaxy AI. Samsung Newsroom compiled a recap of their stories below.

Developing a Translation Model

Galaxy AI features such as Live Translate perform three core processes: automatic speech recognition (ASR), neural machine translation (NMT) and text-to-speech (TTS).

▲ Automatic speech recognition (ASR), neural machine translation (NMT) and text-to-speech (TTS) each require distinct sets of information for training

Samsung R&D Institute Vietnam (SRV) faced obstacles with automatic speech recognition (ASR) models because Vietnamese is a language with six distinct tones. Tonal languages can be difficult for AI to recognize because of the complexity tones add to linguistic nuances. SRV responded to the challenge with a model that differentiates between shorter audio frames of around 20 milliseconds.

Samsung R&D Institute Poland (SRPOL) had the mammoth hurdle of training neural machine translation (NMT) models for a continent as diverse as Europe. Leveraging its rich pool of experience in projects spanning more than 30 languages across four time zones, SRPOL was able to navigate the untranslatability of certain phrases and handle idiomatic expressions that may not have direct equivalents in other languages.

Samsung R&D Institute Jordan (SRJO) adapted Arabic — a language spoken across more than 20 countries in about 30 dialects — for Galaxy AI. Creating a text-to-speech (TTS) model was no small endeavor since diacritics and guides for pronunciation are widely understood by native Arabic speakers but absent in writing. Based on a sophisticated prediction model for missing diacritics, SRJO was able to publish a language model that understands dialects and can answer in standard Arabic.

The Importance of Data

Throughout the process of training Galaxy AI in each language, an overarching theme was the importance of open collaboration with local institutions. The quality of data used directly affects the accuracy of ASR, NMT and TTS. So Samsung worked with various partners to obtain and review data that reflected each region’s jargon, dialects and other variations.

Samsung R&D Institute India-Bangalore (SRI-B) collaborated with the Vellore Institute of Technology to secure almost a million lines of segmented and curated audio data on conversational speech, words and commands. The students got hands-on experience on a real-life project as well as mentorship from Samsung experts; the rich store of data helped SRI-B train Galaxy AI in Hindi, covering more than 20 regional dialects and their respective tonal inflections, punctuation and colloquialisms

Local linguistic insights were imperative for the Latin American Spanish model because the diversity within the language is mirrored by the diversity of its user base. For example, the word for swimming pool could be alberca (Mexico), piscina (Colombia, Bolivia, Venezuela) or pileta (Argentina, Paraguay, Uruguay) based on which region you’re from. Samsung R&D Institute Brazil (SRBR) worked with science and technology institutes SiDi and Sidia to collect and manage massive amounts of data as well as refine and improve upon audio and text sources for Galaxy AI’s Latin American Spanish model.

Samsung R&D Institute China-Beijing (SRC-B) and Samsung R&D Institute China-Guangzhou (SRC-G) partnered with Chinese companies Baidu and Meitu to leverage their expertise from developing large language models (LLM) such as ERNIE Bot and MiracleVision, respectively. As a result, Galaxy AI supports both main modes of Mandarin Chinese and Cantonese.

In addition to external cooperation, due diligence and internal resources were also essential.

Bahasa Indonesia is a language notorious for its extensive use of contextual and implicit meanings that rely on social and situational cues. Samsung R&D Institute Indonesia (SRIN) researchers went out into the field to record conversations in coffee shops and working environments to capture authentic ambient noises that could distort input. This helped the model learn to recognize the necessary information from verbal input, ultimately improving the accuracy of speech recognition.

There are many homonyms in Japanese as the number of sounds is limited in the language. So many words must be determined based on the context. Samsung R&D Institute Japan (SRJ) used Samsung Gauss, the company’s internal LLM, structure contextual sentences with words or phrases relevant to each scenario to help the AI model differentiate between homonyms.

Samsung’s Global Research Network

The professionals across various Samsung R&D Institutes made full use of Samsung’s global research network.

Before tackling Hindi, SRI-B collaborated with teams around the world to develop AI language models for British, Indian and Australian English as well as Thai, Vietnamese and Indonesian. Engineers from other Samsung research centers visited Bangalore, India, to bring Vietnamese, Thai and Indonesian to Galaxy AI.

▲ Staff and collaborators pose in front of Samsung R&D Institute India-Bangalore (SRI-B)

SRPOL had extensive experience developing ASR, NMT and TTS models for a multitude of languages. A key player in Galaxy AI’s language expansion, SRPOL collaborated across continents to support SRJO with Arabic dialects and SRBR with Brazilian Portuguese and Latin American Spanish.

Samsung developers at each of these locations learned to collaborate across borders and time zones. Developers from SRIN even observed the local fasting customs in India when meeting their SRI-B colleagues. Many reflected on their work with pride and gratitude — realizing the lasting implications this project has on language, culture, heritage and identity.

Ongoing Efforts as the Journey Continues

Samsung recently introduced Galaxy AI to its latest foldables and wearables. Since its release earlier this year, Galaxy AI has already been used on more than 100 million devices. “We’re expecting to reach 200 million devices by the end of 2024,” said Won-joon Choi, EVP and Head of the Mobile R&D Office, Mobile eXperience Business at Samsung Electronics at a recent panel discussion.

Amidst this mission to democratize AI, it is important to look back and celebrate the accomplishments and progress that have led to providing this safe and inclusive technology that will benefit humanity and improve lives. By building up the Galaxy AI ecosystem with even more features, languages and regional variations, Samsung is facilitating cross-cultural exchanges in unprecedented ways to realize its vision of AI for All.

Hidden inside the Vellore Institute of Technology in Chennai, India, is a lab filled with futuristic audio equipment. One will find mannequins — known in the industry as head and torso simulators — as well as binaural microphones and hearing devices. They are stored in special chambers treated with an advanced sound absorption system, making this lab the first of its kind in India. Imagine such a facility is used to develop the latest high-end high fidelity (Hi-Fi) equipment.

This is where the Vellore Institute of Technology collaborates with Samsung to produce and develop data and insights that power the latest AI models for Galaxy AI’s language capabilities. The facility was developed as part of Samsung SEED (Students Ecosystem for Engineered Data) Labs — an initiative that enables university staff, students and interns in India to work on projects requested by Samsung since 2021. This is just one of several university programs funded by Samsung in which students have the opportunity to work on projects with technical experts from the company.

“As a student, I love being able to work on multiple projects with a well-known and respected company such as Samsung,” says Yashika Ilanchezhiyan, a Samsung SEED student. “I’m given the confidence to learn new skills in a practical way and feel like I’m making a real difference in current and future products.”

“This kind of collaboration is a win-win situation,” says Giridhar Jakki, Head of Language AI at Samsung R&D Institute India – Bangalore (SRI-B). “Thanks to our projects with universities, we are able to access additional expertise and custom datasets. Partnering universities receive investment, financial incentives and expert mentorship from Samsung as a result.”

Lowering Language Barriers

SRI-B has collaborated with teams around the world to develop AI language models for British, Indian and Australian English as well as Thai, Vietnamese and Indonesian. Recently, core engineers from other Samsung Research centers visited Bangalore, India — where the SRI-B team helped ramp up the technology to bring Vietnamese, Thai and Indonesian to Galaxy AI. SRI-B was therefore ideally positioned to develop the Hindi language for Galaxy AI.

“Every language has its challenges,” says Jakki. “But when you consider the end goal of bringing people the ability to communicate in other languages, it’s worth every ounce of effort. We couldn’t wait to bring Hindi to Galaxy AI.”

Developing the Hindi AI model wasn’t simple. The team had to ensure more than 20 regional dialects, tonal inflections, punctuation and colloquialisms were covered. Additionally, it is common for Hindi speakers to mix English words in their conversations. This required the team to carry out multiple rounds of AI model training with a combination of translated and transliterated data.

“Hindi has a complex phonetic structure that includes retroflex sounds — sounds made by curling the tongue back in the mouth — which are not present in many other languages,” says Jakki. “To build the speech synthesis element of the AI solution, we carefully reviewed data with native linguists to understand all the unique sounds and created a special set of phenomes to support specific dialects of the language.”

Collaborative efforts between Samsung and academic partners were instrumental in developing the AI language model that reflected the cultural nuances of the India’s regions. The Vellore Institute of Technology helped secure almost a million lines of segmented and curated audio data on conversational speech, words and commands. Data was a crucial component for a task as critical as incorporating the fourth most spoken language in the world into Galaxy AI. Working with universities ensured Samsung was using the highest quality data.

Global Connections Deliver Big Impacts

This project perfectly encapsulates Samsung’s philosophy of open collaboration and the company’s belief that sharing expertise and perspectives ensures meaningful innovation. In the case of SRI-B, this not only includes working with academia but also sharing insights and best practices with other Samsung research centers around the world.

“I’m extremely proud of what we’ve achieved with the help of our partners,” says Jakki. “AI innovation through collaboration is a big part of what we do. We will continue to better understand, collect and analyze language data so more people can have access to AI tools in the future.”

We are at the beginning of the most exciting and probably one of the most historic moments in technology. The era of mobile AI is here — which for Samsung opened up by the introduction of Galaxy AI features on the Galaxy S24 series, our first AI phone.

Revolutionizing Mobile Experiences With a Hybrid AI Approach

To bring the real-life benefits of generative AI technology to the Galaxy S24 series, we took a hybrid approach to our AI integration. AI offers so many possibilities, and we believe mobile devices are the primary access point for unlocking its potential as users around the globe continue to rely on their phones for most of their day-to-day needs.

The essential role that mobile devices play in everyday life means they are very personal items that people use during many critical and memorable moments. We understand these behaviors and believe smartphones should work harder so that users can truly make the most of important moments in a simpler, more intuitive and more epic way. We also recognize the importance of privacy which is why we give users full control over what they share and what they keep private.

We believe our hybrid approach is the most practical and reliable solution to meet all these needs and puts Samsung ahead of the curve. We are providing users with a balance between the instant responsiveness and extra privacy assurance of on-device AI and the versatility of cloud-based AI through open collaborations with industry-leading partners in offering a variety of functions they need for daily life.

Expanding Possibilities With On-Device AI

We made the Live Translate feature on-device AI based because voice calls are the most basic feature of a smartphone, yet also an intimate and private means of communication. Samsung has committed to empowering users to communicate without language barriers, while at the same time ensuring every communication is safe and private.

To make this feature a reality, our R&D team worked day and night as part of a collaborative process that truly brought out the best in our teams. From determining the right size of AI language models to training and testing in real-life scenarios, our R&D organizations within the MX business challenged themselves and pushed beyond the limits of what we thought was possible to get this feature running entirely on-device.

Samsung’s global R&D networks around the world, which are in charge of growing the best talents and developing core technologies in each region, have also played a pivotal role. Samsung Research Centers around the globe, including Poland, China, India and Vietnam, have dedicated themselves to developing and expanding the languages supported by Galaxy AI. Languages are very cultural, contemporary and localized, so to help people break down language barriers and communicate more naturally, the role of our local R&D offices is even more crucial in opening up new possibilities for users in more regions than ever before.

All these efforts have enabled Samsung to offer something completely new which I’m excited to share. Soon, Samsung is extending the power of Galaxy AI beyond Samsung’s own native calling app by expanding Live Translate to other third-party message apps to support voice calls. So you can stay in contact with friends or colleagues, communicating on your favorite apps in multiple languages.

And since this feature has been integrated into our on-device AI language translation model, users will be able to experience barrier-free communication without worrying about privacy issues like personal data being shared outside their phone when using Live Translate.

I expect that more and more mobile AI features will be incorporated inside mobile devices as the intelligent computing power of the latest chips — particularly NPUs — advances at a blazing fast speed, along with other technologies. This will enable more people to embrace AI and make every day more convenient while offering greater peace of mind.

Growing Galaxy AI for Tomorrow’s Needs, Today

This exciting news is all part of the extension of our commitment to Galaxy AI and entering the next phase of the mobile AI era. The introduction of Galaxy AI on the Galaxy S24 series was only the start. To deliver a completely new and unique AI experience, we will further optimize the Galaxy AI experience for the upcoming foldable devices. Our foldables are the most versatile and flexible form factor in Samsung Galaxy and when combined with Galaxy AI, these two complementary technologies will together unlock all new possibilities.

Samsung is also committed to expanding Galaxy AI experiences across the broader Galaxy ecosystem in a way that only Samsung can. As the mobile AI era moves forward at a rapid pace, Samsung is accelerating mobile AI innovations to meet the needs of not only today, but tomorrow.

Samsung Research combined data and cutting-edge technology. Together with the Mobile eXperience (MX) Business R&D Office, they further honed this technology to develop the translation features powered by on-device AI — which can be used for real-time translation during calls and across various applications. Samsung Newsroom met with Yoonjung Choi and Yonghyun Ryu from Samsung Research’s Global AI Center to learn more about these ambitious features.

▲ (From left) Yoonjung Choi and Yonghyun Ryu of Samsung Research’s Global AI Center

Smooth Communication and Strong Security With On-Device AI

On-device AI is the key differentiator to Galaxy AI’s Interpreter and Live Translate features — introduced to users through the release of the Galaxy S24 series earlier this year. Leveraging the advanced computing resources built into the devices themselves, smartphones with on-device AI can provide services without relying on servers or the cloud. Users can be assured that their data will remain private and secure since information is not shared with external sources.

Samsung Research’s Global AI Center contributed to integrating proprietary technology into its AI translation model to commercialize these features for widespread use.

Envisioning a wide range of applications, the Samsung Research team and MX R&D Office obtained an expansive amount of data. “We collected colloquial data for real-time translation during calls in Live Translate and travel-related data for Interpreter,” explained Yoonjung Choi, who led the project. “To provide the most accurate translation, we studied and incorporated casual language used in chatrooms and HTML tags used in web browsers.”

▲ Use cases for on-device AI translation include ① Live Translate, ② text message and conversation translation, ③ translation in Samsung Notes and ④ web browser translation.

How Samsung Research Trained Its AI Model

The Samsung Research team’s AI translation model is based on deep learning technology that learns from its own data. Yonghyun Ryu, who is in charge of AI research and development, likened this process to raising a child. “Similar to how a child needs excellent educational resources and caregivers to grow and thrive, good language data and talented researchers are required when developing a high-performance AI translation model,” he described.

Samsung Research has both — since 2013, the company has been providing in-house translation services, conducting R&D related to AI translation and accumulating high-quality data.

Samsung Research’s team of deep learning experts played an important role in training the AI model. “If incorrect translations occur during the research and development process, it is necessary to identify the problem and make improvements. However, this can be challenging and time-consuming for researchers without sufficient capabilities and experience,” he explained. “Our researchers used their expertise and know-how to quickly analyze the cause of the issue and come up with a solution to enhance the AI translation model.”

▲ Yonghyun Ryu from Samsung Research’s Global AI Center

To assess the performance of Galaxy AI’s translations, the Samsung Research team used quantitative metrics based on test sets as well as qualitative evaluations by human translators and the MX R&D Office.

In addition, the team gained credibility by competing in global machine translation competitions. “Although participation requires time and effort, good performance in competitions provides momentum for research and development,” Ryu emphasized. “We were able to achieve strong results because our team members could freely discuss new ideas and put them to the test.”

Politeness and Punctuation: Understanding the Quirks of Each Language

Each language has characteristics that are unique to its culture — these can include honorifics, tonal inflections and distinct punctuation symbols. To make translations as accurate as possible, the AI translation model considers all these linguistic idiosyncrasies. For example, in Korean and Japanese, honorifics are translated to maintain a respectful tone.

Samsung collaborated with regional R&D centers to fully understand languages in their cultural context. “By working closely with researchers and linguists in other countries, we were able to offer a more accurate and complete translation,” said Choi.

At the same time, dealing with different languages often involves trial and error. “Vietnamese, for instance, is a tonal language. However, we realized during the research process that Vietnamese users often omit tones in casual conversations when chatting,” she explained. “We needed additional data to help the features translate sentences without tones.” For Thai, Samsung Research developed a special sentence separator because the language does not use punctuation.

Why Lightweight Technology Is the Key to Effective On-Device AI Models

Samsung Research began developing its on-device AI translation model in 2019. “Unlike server-based AI models, on-device AI models must be driven only using users’ devices,” stated Ryu. “Developing lightweight technology that uses minimal resources is key.” To make the model lighter, the team used “knowledge distillation” and “quantization” technology.

Knowledge distillation is a method that extracts knowledge from a large, high-performing teacher model and delivers it to a smaller student model. This is similar to how a teacher summarizes a topic so that a student can digest the information more easily and efficiently.

Quantization simplifies AI algorithms to reduce model size and streamlines the process to increase response speed.

Ryu compared quantization to drawing strawberries. “You need a wide range of colors to paint lifelike strawberries — but strawberries can also be made with just red and green,” he explained. “Quantization is the process of minimizing the number of colors needed to draw strawberries while trying to make it as close as possible to the real thing.”

▲ Yoonjung Choi from Samsung Research’s Global AI Center

In the AI field, knowledge distillation and quantization are well-known approaches to making models lighter. Yet, implementing them on a commercial scale is not easy due to differences in each researcher’s detailed experimental methods and factors. Samsung Research developed proprietary technology by discovering an efficient quantization technique and creating a accelerated algorithm based on it. “Through constant experimentation, we found an optimal way to make the model lighter,” commented Choi.

By combining the high-quality AI translation model with algorithms that make models lighter and speed up response time, lightweight and fast on-device AI features were born.

The Culture Behind the Language: What Would the Perfect AI Translation Model Look Like?

The researchers at Samsung Research’s Global AI Center have bright goals as they lead the field of on-device AI. “My ultimate goal is to help users communicate smoothly and conveniently with people who speak other languages,” said Choi. Ryu revealed his vision to build the perfect translator. “One day, we want to create a translator that truly understands the cultures behind the languages it is translating, equipped with an extensive pool of knowledge,” he expressed. “I want to challenge myself to develop a translator the world has never seen before.”

▲ Samsung Research’s Global AI Center team developed the on-device AI translation feature.

Samsung’s on-device AI translation features allow anyone with an enabled mobile device to communicate freely — without worrying about internet connection or information leakage. Going forward, the Samsung Research team will continue to spearhead innovations in the rapidly evolving field of AI and bring new levels of convenience to users.

It’s AI that tags along throughout your workday. On your subway commute, it suggests an eloquent response to your boss’s email. In the afternoon, it live translates a call with your global team and provides a summary and transcript immediately after. At night, it helps you archive all the resources and materials you gathered during another dizzying day of multitasking – in seconds.

For all the talk about AI transforming work, the game changer for the workforce is AI that goes everywhere you go. In other words, AI that’s as mobile as your mobile device.

Enter Galaxy S24, the first AI-powered phone.

With Galaxy S24, Samsung is harnessing AI to deliver new dimensions of personal convenience, productivity and communication into our daily lives. The impact will be profound in the world of work, where being productive on the go has become part of the job.

Here’s how several of the device’s features, powered by Galaxy AI, transform how we work:

Removing Communications Barriers

Businesses today routinely link up across borders and time zones to pursue new market opportunities. Accuracy and speed are a must, but international communication can create misunderstanding, swirl and delays. Galaxy S24 helps professionals collaborate more effectively whether they’re at the office, working from home or somewhere in between.

Real-time, two-way translation on conference calls: You’re setting up a conference call with a potential business partner in an overseas company. You want it to go as smoothly as possible, but you’re concerned about the language barriers.

Galaxy S24’s Live Translate translates phone conversations in real time, ensuring you’ll be as clear, creative and professional with overseas colleagues or clients as you’d be in your own language. Offered in 13 languages at launch, Live Translate provides two-way translation on both devices, even if one party isn’t using a Galaxy – or even a smartphone.

Translated and tailored tone for professional emails or texts: It’s your first day in a new job, and you want to make a good impression in your first email to your colleagues. With Galaxy S24’s Chat Assist, you’re offered different AI-generated options on how to restructure your message and writing style, so you get the tone just right. Whether it’s drafting a catchy social media post or a formal email to your boss, this feature is accessible right from the Samsung Keyboard, so you can use it on any application and make sure you hit the appropriate note every time. And if you’re working with a colleague overseas, Chat Assist’s availability in 13 languages allows you to do all this in a language you don’t even speak.

Converse face-to-face without language barriers: As a small business owner attending an expo on emerging market trends, you spot a renowned expert in your niche from a popular YouTube business page. You want to pick her brain, but you know from the show that she speaks Italian, and you don’t.

Interpreter takes live translation out of the call app and into the physical world. With this feature, Galaxy S24 becomes a real-time, in-person translator. Because the tool splits the phone screen into two horizontally, each person can show instantly translated text to the other, along with accompanying audio. It also lets you select the language you need and communicate smoothly and professionally with anyone around you.

Organize Your Phone and Your Thoughts

Although our phones provide a handy way to capture ideas, research projects and arrange meetings, organizing all that on a handheld device can be a challenge. Using responsive Galaxy AI capabilities, however, Galaxy S24 intelligently organizes your phone, so you stay focused on the big picture.

Keep your documents and plans organized: Deep into a work project, you realize the relevant resources you need are scattered everywhere: long emails, detailed articles, PDFs and more. If you could pull everything together into one cohesive format, it would save you hours of laborious notetaking and file conversion.

Thanks to the Note Assist feature, you can generate, read and save abridged versions of all your materials in a few taps of your finger, transforming the daily churn of documents into a well-structured archive. By pulling up a document, Note Assist prompts you to save its content straight to your Samsung Notes app. Once there, the feature can auto format the text and correct spelling mistakes before creating a simple or detailed bullet point summary of the whole document above the main text so you can see all the key points at a glance whenever you need to.

The feature can also extract and summarize text straight from physical books and papers and can translate notes into other languages, while its Browsing Assist capabilities can turn long, detailed articles into succinct summaries.

Accurately document everything: It’s a classic dilemma: You’re in an important work meeting, and you need to take detailed notes while at the same time staying focused and making contributions. With Galaxy S24, you can have it both ways.

Transcript Assist creates full, accurate records of meetings and conversations. Simply record the meeting using the native Samsung voice recording app, and afterwards Transcript Assist will offer to convert it to text, with sentences separated according to individual speakers. Once you’ve got your transcript, you can share it with your colleagues as is, or send it to Samsung Notes and add an AI-generated bullet summary to it using Note Assist.

If you have preexisting audio files recorded without Transcript Assist, the feature can convert these files to notes using Text to Audio.

When it comes to work, the power of having AI on our mobile devices goes beyond automating mundane tasks. Galaxy S24 illustrates how technology can dramatically improve how we work, think and create – by being indispensable, mobile and close at hand.