There’s a saying at the Samsung R&D Institute Poland (SRPOL): “A day at SRPOL lasts 96 hours”. It refers to the center’s global role as one of the largest and fastest-growing R&D centers in the region, often working across four different time zones. Sitting at the heart of Europe while covering many European and global markets, SRPOL has worked on automatic speech recognition, neural machine translation and text-to-speech models for more than 30 languages. When it came to bringing 10 languages to Galaxy AI, this expertise meant the team was well suited to seamlessly blend cultural perspectives with Samsung’s global technology.

SRPOL has years of experience in Natural Language Processing. What makes it unique is its adaptability to work on any language thanks to the passionate team and their tools, such as a crowdsourcing platform that enables fast and agile development.

“Collaboration across the continent means relentless data collection, annotation and research, which has become something we really enjoy,” says Kornel Jankowski, Head of Speech Decoding at SRPOL. “We’ve dealt with so many languages that our team developed universal, language-agnostic skills. When we’re asked to support a new language model, everybody’s attitude is: Oh wow, we get to learn another one, that’s going to be fun!”

A European Center for AI Language Development

Language is a cornerstone of culture and communication across Europe regardless of whether it’s incorporated into technology. However, it presents unique challenges for the team at SRPOL, who develop AI models for European languages.

“Each language and the culture that it is part of, comes with hurdles that make us reevaluate how we perceive a specific issue,” explains Adam Ros, Head of Artificial Intelligence at SRPOL. These hurdles include navigating the untranslatability of certain phrases and handling idiomatic expressions that may not have direct equivalents in other languages.

The team saw these challenges as an opportunity to make SRPOL a European center for AI language development. The biggest benefit of this is that it shortens the communication path between different departments and crucially, the decision-making path. Whether it is a matter of automatic speech recognition, neural machine translation or text-to-speech, teams could simply walk over to colleagues in Mobile Quality Assurance and efficiently solve problems together.

While this has helped, it hasn’t overcome all AI challenges. Inevitably, there are limitations in AI models when dealing with multiple European languages, such as translating without context or variations in intonation. However, the team saw these as an opportunity to keep learning and innovating.

“My team never stops at just one example when handling a new word or topic. Some European languages are harder than others,” adds Ros. If you’ve ever been to Spain, you know that Spanish is often spoken at blazing-fast speeds and we need to train AI well to handle that.”

Galaxy AI’s expansion required novel cross-continent collaboration, but the work soon grew beyond European borders. SRPOL supported the Jordan team’s efforts to teach Galaxy AI Arabic’s myriad of dialects, as well as the Brazil team’s work on Latin American languages.

The importance of language and cultural difference subtleties are all on the radar of SRPOL product developers because they can all be noticed by the target — the end users.

“There are subtle differences between European cultures that impact whether something feels natural to the end user. For example, people in some countries expect to read prices with the euro symbol (€), while others are accustomed to seeing it spelled out, e-u-r-o-s,” says Agata Maria Rozycka, Head of Voice Intelligence Research at SRPOL. “If this cultural nuance is not reflected translated text, the interface might seem less intuitive to a user. Implementing these micro-level insights into interface design can make technology feel more natural across diverse cultures.”

“The team has been remotely communicating and collaborating across different countries for many years, building up numerous effective communication channels,” says Marcin Mrugala, Head of Mobile Quality Assurance at SRPOL. “We were ready to do our part in enabling Galaxy AI to lower language barriers around the world.”

Technology for Bridging Cultures

Managing and integrating diverse linguistic and cultural insights is a challenging task, but it is essential for Samsung’s vision for Galaxy AI — lowering the barriers that divide people based on language and culture, and enabling them to create deeper connections.

“We’re not just building technology of the future, we’re building teams of the future too. Our best practices are designed to refine products based on differences across countries, but we fundamentally believe our similarities far outweigh our differences and our technology can unite cultures,” says Mrugala.

“Our goal is to bring people together, to make their lives easier, and to simplify their daily tasks. We’re seeing our families using the Voice Recorder in new ways, and we can now call our friends and different countries and talk with them in their own language. It is magical to see this change in the world and to be part of it. Galaxy AI brought SRPOL people together and now we are bringing together the world,” concludes Rozycka.

Samsung Research and Samsung R&D Institute Poland (SRPOL) participated in a competition between scientific groups and laboratories to compare the quality of their translation tools. Teams from all over the world participated in the eight machine translation task competitions, from those representing widely known companies to research groups from various universities.

Samsung Research Global AI Center’s Language Lab participated in the Biomedical Translation task, which aims to evaluate systems for translating sentences from the biomedical domain. The task addressed a total of 14 language pairs, including English, French, German and Spanish. The team won first prize for effectively translating two language pairs: English → Spanish and Spanish → English. This was a particularly impressive feat due to the biomedical field’s frequent use of domain terminology.

In the case of domain-specific translation, one of the big factors that determine translation quality is terminology translation. Even with the same word, the translated word may vary depending on the domain, and compared to general terms, technical terms are used less frequently, making it difficult to learn. Considering these limitations, Samsung Research Global AI Center’s Language Lab improved domain-specific translation performance by incorporating soft-constrained terminology translation, which provides the terminology constraints of the target language as input with source sentences like a hint, and improved the domain terminology to be reflected in translation results as much as possible. Currently, Samsung Research is conducting research on domain-specific translation, including providing patent translation service (Korean—English) on Samsung Research’s translation service “SR Translate” (https://translate.samsung.com).

▲ Samsung Research Global AI Center’s Language Lab

SRPOL also participated in two General Machine Translation tasks, achieving high ranks by placing second for English → Russian and English → Croatian.

During the competitions, WMT only provides teams with a limited amount of corpora, collections of structured texts, to be analyzed for their translation model. Therefore, the SRPOL team attributed their success to focusing on improving the quality of corpora through processes like data preprocessing and filtering. In addition, the team focused on optimizing their model’s architecture and AI training process.

Using the improved corpus, SRPOL’s Machine Translation Team built a classifier using a machine learning framework called BERT (Bidirectional Encoder Representations from Transformers). This classifier successfully categorized millions of sentences from the corpus into different domains. As a result, SRPOL was able to create models for not only general translation but also medical and legal.

SRPOL has been performing well in the field of machine translation, winning the challenges at the International Workshop on Spoken Language Translation (IWSLT), one of the world’s longest-running workshops on automatic language translation, for four consecutive years from 2017 to 2020.

Now more than ever, the goal of attaining a human-like level of language understanding seems to be within our grasp. As machine translation and language understanding slowly become an integral part of our everyday lives, Samsung will stay at the forefront of this technology to design the tools to overcome language barriers and improve your daily life.

▲ Samsung R&D Institute Poland

In this relay series, Samsung Newsroom is introducing tech experts from Samsung’s R&D centers around the globe to hear more about the work they do and the ways in which it is directly improving the lives of consumers.

The second expert in the series is Lukasz Slabinski, Head of the Artificial Intelligence Team at Samsung R&D Institute Poland (SRPOL). Slabinski joined SRPOL in 2013 as a Senior Engineer, and following 8 years of dedicated work, now leads the AI Team at SRPOL. Read on to hear more about the exciting innovation Slabinski and his team are involved with at SRPOL.

Q: Designing solutions for the speech recognition field is known to be highly intricate. When working on language-related technologies, what challenges have you encountered and how have you been overcoming them?

In my opinion, language-related technologies are far more complex than any other ones. Humankind communicates in almost 7000 constantly evolving languages, sub-divided into endless accents and dialects. Moreover, human language is far less objective than, for example, a picture, which can be described in mathematical formulas. People encode their thoughts as a set of sounds or characters into a message, which then needs to be decoded and interpreted by others. Because each phase of this process is personal, creative and non-deterministic, language-based human communication is very complex and ambiguous. Thus, on the one hand, we can enjoy beautiful poetry and funny jokes, and on the other, occasionally suffer from misunderstandings.

The R&D people who work on natural language processing (NLP) often reach their own, innately human, limitations. Even we encounter issues communicating clearly with colleagues at work, or family at home. So how, for example, can an engineer who speaks 2 languages design and code a machine translation system for 40 different languages? We solve this paradox using machine learning technologies.

During the process known as ‘training’, we automatically extract general patterns based on examples from our datasets and memorize them in the form of a model. To build a machine translation system, we train a neural network to map a sentence in different languages based on millions of examples, all carefully collected and cleaned beforehand. It sounds easy, but we deal here with 3 fundamental challenges.

The first challenge is the design of an appropriate machine learning model architecture capable of memorizing and generalizing enough language patterns for given problems such as machine translation, sentiment analysis, text summarization and others.

The second challenge is the preparation of sufficient amount of training data, as machine learning systems can recognize and memorize only those patterns presented in the training dataset.

The final challenge is the deployment of an already-trained machine learning model onto a dedicated Cloud or on-device platform.

We address these challenges by harnessing the vast expertise of our engineers, sophisticated approaches to collecting data and through endless experimentation with the state-of-the-art machine learning architectures.

Q: Can you please briefly introduce your AI Team, the Samsung R&D Institute Poland (SRPOL) and the kind of work that goes on there?

SRPOL is one of the largest international software R&D centers in Poland. It is located in two cities: Warsaw, the capital city of Poland and Cracow which is a major technology hub in its region. We closely collaborate with local start-ups, universities and research institutions.

The mission of the AI Team at SRPOL is the creation of the AI-based features, tools and services capable of facilitating and enriching human lives. We mainly focus on the NLP and Audio Intelligence areas, but we also possess expertise across many different specialties, including recommendation systems, indoor positioning, visual analytics and AR.

Q: As the head of the Polish Institute’s AI Team since 2018, you have overseen a myriad of projects both with and without the NLP focus. What are you and your team working on now?

Regarding the NLP area, we have been continuing our journey that began over 10 years ago by the development of systems such as Machine Translation, Dialogue Systems including Question Answering and Text Analytics. We work both on scalable, powerful cloud-based services as well as on fast and offline working on-device applications.

Audio Intelligence is a newer area for us. We began to focus our research capabilities on it around several years ago as the area had been gaining importance. Currently, we work on sound recognition, separation, enhancement and analysis. During our work, we take all levels of audio processing into consideration, from acoustic scene understanding to the fine-tuning of the embedded audio algorithms on devices with very limited hardware resources, such as wireless earbuds.

Q: Your technological focuses include NLP, text & data mining, audio intelligence and more. Has your research directly affected the development of any specific Samsung product or service, and what benefit has your team’s contribution offered to users?

SRPOL has a long record of commercializing AI technologies, but we did not do it alone. We are proud to be a part of a bigger picture, wherein SRPOL works closely with other Samsung R&D centers and contributes to commercialization.

For example, we contributed to the development of several intelligent text entry features for Samsung’s mobile devices, including the on-screen keyboard, hashtag feature, Samsung Note title recommendation and smart text replies on smartwatches.

We also contributed to the Galaxy Store’s Recommendation System, which suggests the most interesting games to a user based on their preferences.

Q: As an advocate for the new AI fields such as audio intelligence, what do you see as the main trends within your industry right now? How will this technology affect people’s daily lives?

I do believe that audio intelligence will be the next game-changer for all consumer electronic devices. Working on audio analytics is extremely important, as it is the missing part in advanced, truly human-centered AI-based systems.

Powerful NLP systems analyze the user’s intent as expressed by text and speech. Computer vision algorithms are behind almost every camera and visual content’s output. For most of us, it is hard to imagine driving a car without navigation, typing a message without spelling correctors, or searching for information without the Internet. But, except for a few professional applications, so far, we very rarely use intelligent audio technology to enhance our hearing. In my opinion, this is set to change soon.

Let’s imagine that we have a commonly available technology that allows people to select what and how they want to hear. For example, during a lunch with a friend in a park located in a busy city center, someone could choose to hear only the sounds of nature and the person they are speaking with. Or, let’s imagine an advanced VR or AR system, recently referred to as Metaverse that creates an immersive 3D audio experience directly in people’s heads. Just these two concepts generate hundreds of new possible use cases, but let’s go further. How about hearing things that are currently inaudible to people? Now humans can hear only a narrow spectrum of different sounds. Our world is full of meaningful sounds which, for the most part, the current AI technologies are not involved in. With the development of the audio intelligence technologies, I believe that all of this is going to affect people’s lives hugely.

▲ Researchers at Samsung R&D Institute Poland work on Active Noise Cancellation (ANC) technology development with a Head & Torso Simulator (HATS) in an anechoic room.

Q: How have you been incorporating the current trends into the research you do at Samsung R&D Institute Poland?

Aside from NLP and Audio, we are also working to find the most effective ways to build truly multimodal systems. To do that, we proceed with research and analyzing use cases from different perspectives. Such analysis is made possible thanks to our diverse and interdisciplinary team that consists of engineers, linguists, data scientists and more.

Q: What has been your most important achievement at SRPOL so far?

That would be our Machine Translation solution. Our solution has garnered wins at various competitions for five years straight: the International Workshop on Spoken Language Translation (IWSLT) from 2017 to 2020; the Workshop on Machine Translation (WMT) in 2020; and the Workshop on Asian Translation (WAT) in 2021. These are among the most prestigious international competitions in our field.

Winning recognition at WAT this year was a particularly satisfying milestone, as developing our solution for the Asian languages was originally a difficult feat for us as Polish engineers – but this achievement has proven the true power of our technology that goes beyond a mere demo showcase.

Another achievement that I am very proud of is the speed of growth that the audio intelligence team and its technology development have achieved. In just a few years, after starting pretty much from scratch, we were able to stand on the podium of the workshop on Detection and Classification of Acoustic Scenes and Events for two consecutive years, 2019 and 2020. We have also published several scientific papers and patents in this area. I am sure this is just the beginning of our prolific activities in this field.

An interview with Bin Dai, a machine learning expert from Samsung Research Institute China-Beijing can be found in the following episode.

Spearheading Speech Translation Research

Samsung R&D Institute Poland and Samsung R&D Institute China-Beijing competed with some of the world’s top universities and research labs to win first place in two separate challenges at the International Workshop on Spoken Language Translation (IWSLT), one of the world’s longest-running workshops on automatic language translation. This year, IWSLT joined the Association for Computational Linguistics conference (ACL), a premier conference in the field of computational linguistics, to cover a broad spectrum of research areas that are concerned with computational approaches to natural language.

For the Offline Speech Translation task, which assesses the translation of TED talks from English to German, Samsung R&D Institute Poland won first place for the second time with its own research capabilities in audio to text translation. The conferral of this award marks the fourth consecutive year that teams from Samsung R&D Institute Poland have taken first prize in IWSLT challenges, including previous years’ text translation tasks.

This year’s Offline Speech Translation task allowed participants to submit systems based on either the traditional speech translation pipeline system composed of an automatic speech recognition (ASR) and a machine translation (MT) or an End-to-End (E2E) system. Samsung R&D Institute Poland’s system is based on a single encoder-decoder deep neural network – an E2E system – capable of both English and German texts.

In computational linguistics, E2E systems are harnessed to solve the common problem of error accumulation, wherein, in a traditional pipeline, an error in the speech recognition phase can lead to a nonsensical translation. However, research from over the past three years has shown that traditional systems have constantly been outperforming E2E speech translation systems. The Samsung team’s system not only placed first in the E2E category, but also outscored all traditional pipeline system entrants, a remarkable achievement that puts Samsung R&D Institute Poland at the forefront of speech translation research.

The team from Samsung R&D Institute Poland Team who participated in this year’s IWSLT challenges

Innovative Approaches in the Field of Computational Linguistics AI

Samsung R&D Institute China-Beijing took part in a second challenge, the Open Domain Translation task evaluating Japanese to Chinese translation capability, ultimately taking first place. The main goals of this task were the promotion of research into translation between Asian language, the exploitation of noisy parallel web corpora for machine translation and the thoughtful handling of data provenance.

Samsung R&D Institute China-Beijing submitted a system based on Transformer model architecture and adopted the relative position attention. The team focused on improving the Transform baseline system with elaborate data preprocessing and managed to achieve significant improvements. The team also tried shared and exclusive word embedding and compared different granularity of tokens, approaching the process at a sub-word level, including Byte Pair Encoding (BPE) and Sentence Piece. Large-scale back translation on monolingual corpus was used to improve the Neural Machine Translation (NMT) performance.

Members of the team from Samsung R&D Institute China-Beijing Team who participated in this year’s IWSLT challenges

Achievements in AI Audio Signal Interpretation

In addition to their first-place finish in the IWSLT challenge, Samsung R&D Institute Poland was also recognized as one of the leading teams at the Detection and Classification of Acoustic Scenes and Events (DCASE) 2020 challenge, held by IEEE (Institute of Electrical and Electronics Engineers), which aims to use state-of-the-art AI technology to understand and interpret audio signals.

Engineers from Samsung R&D Institute Poland, who possess previous experience in Acoustic Scene Understanding and Sound Sources Localization tasks (having ranked first place in two tasks in 2019), set their focus on Task 2: Unsupervised Detection of Anomalous Sounds for Machine Condition Monitoring. The goal of this task was to identify whether the sound emitted from a target machine was normal or anomalous. The main challenge was detecting unknown anomalous sounds under a condition within which only normal sound samples have been provided as training data. The engineers scored second place out of 40 teams.

The team from Samsung R&D Institute Poland Team who participated in this year’s DCASE challenge

Envisaging the Future of Computer Vision and Pattern Recognition

In June, Samsung R&D Institute China-Beijing also participated in three challenges hosted by the 2020 Conference on Computer Vision and Pattern Recognition (CVPR 2020): the Embodied AI Challenge, the VizWiz-Captions Challenge and the VATEX Video Captioning Challenge. The team claimed second place in the challenges.

The Embodied AI Challenge aimed to enable robots to understand human commands and perform correct actions within a virtual environment, while the VizWiz-Captions Challenge involved predicting an accurate caption when given an image taken by a visually impaired person and the VATEX Video Captioning Challenge aimed to benchmark progress towards models that can describe videos in various languages including English and Chinese.

Members of the team from Samsung R&D Institute China-Beijing Team who participated in this year’s CVPR challenges

Samsung R&D Institute Poland at IWSLT 2019

2019 marks the third year in a row that Samsung R&D Institute Poland, in partnership with the U.K.’s University of Edinburgh (UEDIN), has received accolades at the International Workshop on Spoken Language Translation (IWSLT), one of the top two global workshops on automatic language translation, along with the Workshop on Machine Translation (WMT). This year, Samsung R&D Institute Poland won first place in two categories, the first being text-to-text translation from English to Czech and the second – an end-to-end system translating English speech into German text.

For the text-to-text translation category, researchers worked to develop a model to translate the transcript of a spoken English-language TED Talk into Czech. Developing their winning model required the Samsung team to develop large, filtered corpora from which to work and generate as much synthetic data as possible. The work done by the Samsung R&D Institute Poland team, together with additional modeling help from UEDIN, was selected as the best in the challenge by human evaluators. This means that the translations produced by Samsung R&D Institute Poland’s system scored the highest both in fluency and adequacy.

Samsung R&D Institute Poland’s participation in their second winning category this year, the end-to-end translation system from English to German, was a first for the team. The task was to produce a German-language transcription of an English-language TED Talk audio recording. This task required the development of a single model that could take an audio file input and subsequently produce a translated transcription. It was made more difficult by the deficiency of the provided audio sources, compared to typical speech recognition task. Samsung R&D Institute Poland proposed several innovative methods for end-to-end speech translation that mitigated this source paucity, obtaining a state-of-the-art result with their final system that won them first place in the challenge.

Samsung Research America at ICCV 2019

This October, researchers from Samsung Research America’s AI Center received first place in the International Conference on Computer Vision (ICCV)’s challenge: Linguistic Meets Image and Video Retrieval (Fashion IQ). ICCV is a premier international computer vision conference that took place in Seoul, Korea, this year.

The challenge Samsung Research America AI Center took part in, sponsored by IBM research, aims to develop conversational shopping assistants that are more natural and real-world applicable. The task given to Samsung Research America AI Center‘s team, the ‘Superraptors’, belonged to the domain of image retrieval. In the task, an input query was specified in the form of a candidate image as well as in two natural language expressions that describe the visual differences of the search target. The goal of this challenge was to gather opinions and experience from researchers on the emerging space of visual content retrieval with a natural language interface.

Samsung Research America AI Center’s submission to the challenge, “Multimodal Ensemble of Diverse Models for Image Retrieval Using Natural Language Feedback”, blended the given data in different modalities with multiple deep learning models. The team’s win marks the first time a Samsung Research team has won a multimodal (language and vision) challenge; previously, Samsung AI Center Moscow, Samsung R&D Institute Poland and Samsung R&D Institute China-Beijing have received awards in single modality challenges.