In this relay series, Samsung Newsroom is introducing tech experts from Samsung’s R&D centers around the globe to hear more about the work they do and the ways in which it is directly improving the lives of consumers.

The sixth and final expert in the series is Brian Harms, a Research Engineer at Samsung Research America (SRA). Following 8 years of exploration into advanced robotics research at SRA, Harms and his team now employ an innovative array of methods in order to work towards changing the way robots are made and perceived. Read on to learn more about the fascinating research Harms and his team are undertaking at SRA.

Q. Can you please briefly introduce the kind of work you undertake at Samsung Research America?

In addition to developing innovative technologies, SRA is conducting research into various fields including AI, 5G/6G communication, Digital Health, AR and Robotics for Samsung’s future innovation.

When I joined SRA, I was drawn to one in particular of my team’s key areas of focus, which was imagining how robots will affect the future of our homes and everyday lives. A lot of my work at SRA focuses on prototyping experiences as rapidly as possible so that we can make decisions about how certain devices or products should or shouldn’t work.

Our projects usually start very organically, and individuals are encouraged to pursue their ideas and then bring them to the team for feedback and creative input. Thanks to our strong relationships with different divisions within Samsung, our team is empowered to think about a really wide variety of ways we can improve people’s lives, and that freedom and support is a really cool aspect of what we do in the Think Tank Team at SRA.

Q. Following the recent accolades you have received for your work in advanced robotics, what are you and your team working on at the moment?

At any one time we may have approximately 10 to 20 projects that are happening simultaneously, but that operate on different time scales and with different resources. In past years our team’s goal was to have the majority of those projects aim to be ‘productizable’ within 3 to 5 years if successful, but in more recent years we have shifted our goal towards 1 to 3 years, as we are striving to make a strong impact on the user-facing market as quickly as possible.

In order to achieve this, we are working on several projects within the umbrella of practical robotics whose scopes are mindfully constrained so that we can work with different teams to transform these prototypes into products. Our goal is to find a balance where we provide a great deal of user value while still constraining the problem space within realistic bounds. We also pride ourselves in being optimistic about finding room for innovation, even in products that have largely remained the same over many years.

Our team is also currently working on many projects that are outside the realm of robotics, including new apps, phone features, connectivity devices and improved appliances with the goal of empowering users and keeping them connected.

Q. Practical robotics is a field that provides innovative and convenient user experiences and is primed to change the way we think about robots. Can you elaborate on this further?

I think that it is important for people to rethink what they consider to be “robots” because the way they are defined tends to vary greatly. Many common definitions might clash with each other or reject some actual robotic devices from the category. Personally, I lean towards an extremely inclusive definition along the lines of: if a machine actuates automatically in response to stimuli, you might as well call it a robot.

The reason I think it is important for people to take a moment to consider what robots really are is that so-called “practical robots” are all around us and affect us every day of our lives in impactful ways. Consider a mattress with sensors that measure sleep quality or temperature, adjust the mattress’ position with actuators and cool the user by pumping fluid through a network of tubes. I think by almost any definition of robot, this is a robot – but perhaps the owner of such a mattress might not actively consider it one.

From automatic doors at grocery stores to cars that measure their distance to other cars and adjust speed accordingly and even to coffee makers that brew a fresh pot of coffee for you in the morning through sensor detection – these are all robots, and if you were to accept this idea of what a robot is, you’ll start seeing them much more frequently in your day-to-day life.

Q. What do you see as the main user benefits brought about by the implementation of novel robot capabilities into consumer-facing technological devices?

The main user benefit brought by the inclusion of robotic technologies in a device will of course vary by device and the problem it solves for the user, but if I had to generalize, I think that the benefits boil down to making an activity or experience faster, easier, safer or more rewarding. Automation is a powerful mechanism in affecting these four criteria, whether it is in an industrial manufacturing plant or someone’s living room.

Q. Your team is made up of a unique range of researchers from a diverse range of backgrounds. Can you give an example of a time when this ability to ideate in an interdisciplinary manner resulted in the development of an innovative new robotics approach or technology?

Occasionally we hold brainstorming sessions where 1 or 2 people have an idea they want to turn into a project. Those people come up with a series of questions or prompts for the participants, and then every person in the room takes a stack of sticky notes and fills as many as they can with ideas and sketches for the new project and puts them all up on the wall. The cool thing about this is that when the prompts are about potential industrial design ideas, for example, we have not only industrial designers, but also programmers, scientists, electrical engineers and more, all responding to the same prompts in different ways.

Through this kind of multidisciplinary collaboration, designers on our team benefit from developing an improved understanding of what is technically possible, and engineers get a better understanding of what constraints good design might add to the project. What this results in is a team made up of designers who speak the language of engineering, and engineers that can speak the language of design. This kind of collaboration was critical for a project like Samsung Bot Chef, where both the aesthetic and engineering elements were highly dependent on one another.

Q. What would you designate as the latest trends in robotics technology right now? How are you incorporating them into your research at SRA?

Automation and robotics are evergreen fields that are growing exponentially at the moment. The main way we approach projects is to first identify a need or possible method of improving some aspect of daily life, and then consider the mechanisms for executing the idea. Fortunately, automation and robotics are effective tools that lend themselves well to addressing and solving some of these problems.

Our future product concept Samsung Bot Chef was one result of us monitoring the latest trends. Our then-team head noticed that there was a huge gap between the kinds of low-cost, low-performance robot arms you might see on crowdfunding platforms and the high-cost, high-performance of industrial robot arms, and had a strong intuition that there was opportunity in the consumer market for a robot offering in between the two. The goal there was to minimize end user cost while maximizing performance and capability, which took us down a long road towards designing our own servo mechanisms from scratch. What we created is one of the best-looking robot arms that I’ve seen on or off the market, that is tailor made for interacting with the same everyday objects that we use at home.

Q. When you envisage a future powered by innovative robotics technologies, what does it look like?

When I picture the future, I try to imagine “what might a typical day look like for me.” I would hope that, in the future, robotics and automation provide opportunities for me to preserve more time for myself to do the activities that I love. Between maintaining relationships, work, hobbies, errands, finding time to rest and unexpected events in life, I constantly feel that I lack the time or energy to engage with each of these activities in balance. I believe that automation might be one mechanism that will help me preserve more of my time so that I can spend it in ways that I choose in order to feel more fulfilled.

Q. What has been your most important achievement at SRA so far, or the one that you are most proud of?

I was really proud of our team achieving our Samsung Bot Chef demonstration in Berlin, Germany at IFA 2019. It truly took a monumental amount of effort for us to design, manufacture and assemble completely new versions of Samsung Bot Chef from the ground up, by hand. We also had to plan a complex demo, program all of the interactions, and test everything repeatedly, not to mention transport our demo robots to Germany, work around the construction of the demo kitchen and collaborate with the host chefs. It was a really challenging but rewarding experience that not only brought our team closer together, but also reminded us that when we are united in pursuit of a single goal we can achieve amazing things.

In this series, Samsung Newsroom has introduced six tech leaders from Samsung R&D Institutes around the world who are actively involved in advanced technology development. Through the consolidation of the research and development capabilities of experts in Samsung’s R&D institutes, just a few of whom have been showcased in this series, Samsung is able to bring next-level technologies and experiences to users in their devices. Samsung Research currently sees collaboration between the experts in its 14 R&D institutes in 12 different countries around the world.

In the future, collaboration will be a key factor towards advancing research into advanced technology. Samsung will continue to work towards a better future powered by innovation, inspired by daily routines and designed for users.

In this relay series, Samsung Newsroom is introducing tech experts from Samsung’s R&D centers around the globe to hear more about the work they do and the ways in which it is directly improving the lives of consumers.

The fifth expert interviewed for the series is Ratnakar Rao V R, who heads the Beyond 5G Team at Samsung R&D Institute India – Bangalore (SRI-B). Rao is soon to complete a decade at SRI-B, and the bulk of his experience has been in the research and development of wireless communication technologies like 4G and 5G. Check out the interview below to find out more about the promising technologies Rao and his team have been working on.

Q: Advanced system software plays a crucial role in activating all kinds of technologies developed to provide better user experiences. How has research into applied AI been factoring into your work within the communications field?

Traditionally, all cellular communication systems were implemented using mathematical models and were strictly rule-based. However, this is now changing in the 5G era due to a few key factors.

Firstly, since a single network has to cater to a number of diverse use cases simultaneously, such systems cannot operate to the best of their capabilities if implemented based only on traditional modeling. Secondly, advances in computation algorithms and processor architectures are making it easier to run AI and machine learning models on a wider range of devices. Thirdly, wireless networks are being virtualized and are getting split into micro-services that run in the cloud. On-Device AI capabilities are also being added to wireless terminals. From 5G onwards, networks will be closely integrated with applications, making it necessary for them to be more contextually aware of users and applications in order to deliver personalized network experiences to all.

All these factors enable and necessitate broader use of AI and machine learning in next-generation wireless networks and terminals.

Q: Can you please briefly introduce SRI-B and the kind of work that goes on there?

The Samsung R&D Institute in Bangalore (SRI-B) has established five Centers of Excellence (CoEs) with the focus areas of Communication, Camera and Multimedia, On-Device AI, IoT and Services. SRI-B has experience executing projects from the research to market stage in each of these areas, and makes impactful contributions to Samsung product lines on the backs of these CoEs every year.

At the Communication CoE, SRI-B has dedicated teams working on mobile terminals, network RAN/core development and wireless standards. Strong synergy between these teams has resulted in the establishment of end-to-end domain expertise. In addition to this, we have recently seeded advanced communication research in a bid to make impactful contributions to Beyond 5G and 6G evolution.

Q: What kind of communication-related work are you and your team engaged in now?

Firstly, our team specializes in radio, data networking protocols and embedded modem system software. We craft the 5G radio experience for different markets around the world. Our team is engaged in the product development of 5G mobile terminals for a range of world-wide markets.

Secondly, we are engaged in advanced research and development surrounding communication protocols. Some of this work makes it into Samsung products as differentiating features and solutions. The rest of our work is aimed at creating standards and implementation IP (intellectual property) pertaining to Beyond 5G and 6G systems.

Q: How do you expect the Beyond 5G era to change the way users interact with technology in their day-to-day lives?

The early launches of non-standalone 5G have unlocked a lot of hitherto unused spectrum in mid and high bands and also enabled the re-farming of the existing 4G spectrum for use with 5G. Thus, the transition to a very high-capacity communication system is underway.

This massive addition of capacity will enable more users to connect more devices to the internet and bring the benefits of connectivity to the masses who live in rural areas. For regular users, these benefits will be evident in terms of very high-resolution streaming, faster downloads and uploads and real-time interactive gaming. They will also see mixed-reality experiences in video streaming and video calls.

The subsequent enhancements to 5G in the coming years will unlock consumer infotainment and a lot of other use cases. For example, the low-power, low-bandwidth features of IoT devices will help public services, the agricultural industry and factories automate for better efficiency. Likewise, satellite-based 5G will provide ubiquitous coverage all over the globe. The highly reliable, low-latency optimizations applied to 5G networks will also enable better remote delivery of services like healthcare and education.

Q: Which of SRI-B’s achievements in the communications field are you most proud of?

We at SRI-B are proud to have played a critical role in the launch of the world’s first 3G, 4G and 5G smartphones. Recently, SRI-B has enabled standalone 5G and 5G carrier aggregation in mobile terminals for commercial use, developed 4G and 5G network software and helped establish MC-PTT (Mission Critical Push to Talk) capabilities.

SRI-B has also been an IP powerhouse for several years. Every year, a number of IPs are created by SRI-B engineers from across the various domains. We have created more than 200 implementation IPs in the area of wireless communication, and more than 100 standard essential IPs in the areas of 4G and 5G.

Q: How does collaborating with other institutes like Samsung Research America, Samsung R&D Institute UK and Samsung Research in Korea complement your work and research capabilities?

We have worked closely with Samsung Research on early technology development and realization of 5G, and are now collaborating on nascent 6G technology development. I strongly believe that a lot of potentials can be unlocked by further collaboration between SRI-B and the teams at Samsung Research America and Samsung R&D Institute UK.

SRI-B has a very large pool of communication engineers, including innovators and domain experts. It is therefore possible to build high-quality teams and execute research promptly. We are actively exploring these possibilities by interacting with R&D leaders at global research centers to enable breakthrough innovations.

Q: How are AI and machine learning being applied to Beyond 5G and 6G wireless communication technology? How do you expect these technology combinations to evolve going forward?

It is widely agreed that AI and machine learning will have a significant influence on network management and radio resource management for Beyond 5G and 6G networks. We envisage AI and machine learning applications being present in block-level AI, procedural AI and system software AI, and are actively researching along these lines.

• • Block-level AI: A specific block in the terminal or network could be added with AI/machine learning without impacting the rest of the system, resulting in performance improvements and/or computation savings. For example, a channel decoder could decide to terminate the decoder iterations early if it is able to predict whether the block decoding will eventually pass or fail.
  • Procedural AI: This is where at least two entities in the end-to-end system exchange information to enable accurate use of AI and machine learning techniques. For example, meta-data needs to be exchanged between the terminal and network for an auto encoder or decoder to work within a margin of error. Another example is mobility management for terminals.
  • System software AI: Most entities in next-gen communication systems will have to operate in several modes. The embedded system software should be able to scale up or scale down system resources very dynamically. AI-assisted embedded system software is expected to learn context-specific requirements and adapt accordingly.

Q: You are a senior member of the Institute of Electrical and Electronics Engineers (IEEE). What kind of activities are you involved with in this role? How does your position in the IEEE inform your other work?

In this role, I deliver invited lectures and talks on various technology topics for the student communities and teach fraternities at regional engineering colleges. The role has also enabled me to seed new study items internally, allowing us to initiate new collaborations with student communities and faculties from reputed universities.

The aim is to influence as many people as possible to improve their domain expertise and pursue advanced research in the communications field. I also represent Samsung in various talks and discussions with industry, government and academia. My interactions help me stay in touch with the latest trends in various areas adjacent to my area of expertise.

I also encourage my team members to publish their results in outstanding conferences and journals. This year and last year, we have published more than 20 papers in various outstanding forums.

In this relay series, Samsung Newsroom is introducing tech experts from Samsung’s R&D centers around the globe to hear more about the work they do and the ways in which it is directly improving the lives of consumers.

The fourth expert in the series is Evgeny Pavlov, Head of the Advanced System Software Lab at Samsung R&D Institute Russia (SRR). Following 9 years of dedicated work into advanced techniques for program analysis at SRR, Pavlov was made the head of his laboratory in 2020.

The systems Pavlov works on, System software (SW), is software that has been designed to provide a basis for other software, such as the Operating system (OS) you use in your smartphone, the frameworks for AI-based applications, tools for developers and more. System SW is responsible for the communication between applied software and hardware. Read on to learn more about the crucial research Pavlov and his team undertake at SRR.

Q: The results of AI and machine learning research are of key importance to designing and optimizing all kinds of technologies. What role does system software research play in further activating these technologies? 

System SW research now plays a very important role in machine learning, although this may not always be visible to the end user. First of all, machine learning frameworks do not always work optimally on general-purpose hardware and processors, so they need to be optimized in ways that take into account various hardware features and use additional central processing unit (CPU) extensions.

Furthermore, the latest trends in the artificial intelligence (AI) industry include the integration of specialized processing units for neural network acceleration. Many companies have been developing specialized accelerators for neural networks called neural processor units (NPU) recently. For the optimal processing of a machine learning model, it is necessary to transform the neural network model into a set of instructions for this accelerator.

These neural network model conversions are usually automated using a neural network compiler since a deep understanding of NPU architecture is required for the development of these compilers. This means that us system SW developers are involved in their development since we have a deep understanding of how computer hardware works.

In other words, thanks to this change in industry requirements, the focus of System SW engineers is moving from the optimization of general-purpose programs towards the optimization of AI- and machine learning-based programs.

Q: Can you please briefly introduce Samsung R&D Russia Institute (SRR) and the kind of work that goes on there?

These days, we at SRR are focusing on developing our expertise and capabilities in three main R&D areas: Sensor Solution, AI Imaging and System SW. SRR has end-to-end experience in sensor R&D, which includes hardware and algorithm development as well as commercialization specifically for biometric and life care solutions. SRR has been deeply involved in the development of iris, face and fingerprint biometry as well as body composition estimation for smartwatches. SRR has also contributed to the strengthening of the well-known Super Slow Motion and Night Mode features on smartphone cameras through consistently developing the synergy between optics and AI within the AI Imaging area.

I believe that System SW is one of the most promising areas of research happening in SRR right now. Based on our deep understanding of various hardware and operating systems (OS), as well as strong engineering manpower, we do our best to be a System SW core tech provider for the entire business.

Q: Following your accomplishments within the Advanced System SW Lab at SRR, what are you working on at the moment?

We are conducting extensive research into potential new directions for our System SW team in order to understand the latest trends in System SW that may well replace traditional System SW techniques in the near future.

Our lab is also currently working on a project related to enabling the 5G scalable vRAN infrastructure to support multiple network types, as well as other projects related to compiler technologies for the Android and Tizen OS, advanced OS developing and Software Development Kit (SDK) development for On-Device AI.

Besides leading the Advanced System SW lab, I am also currently leading an SRR project for On-Device AI platform called ONE, or On-Device Neural Engine. This project is being developed in collaboration with the On-Device Lab at Samsung Research, and a major aspect of this project is being maintained by Samsung as an open-source project located on github.com.

Q: On-device AI and advanced System SW technologies are crucial to providing users with robust, innovative mobile experiences. Could you explain a bit more about why this is, and the direction of research you and the Advanced System SW Lab have been taking?

System SW plays a key role in application operation and user experience. System SW is the lower layer that sits between a device’s hardware and user applications – meaning that it is the foundation for all other software. Users may not see System SW in action, since their interactions with their mobile applications are relegated to simply engaging with the interface, but under the hood of their favorite apps are many layers of program logic – for example, managing the recognition of a tap to the screen in the system kernel and then drawing a corresponding window through the graphics library. If there is a delay at any one of these levels of recognition, the entire system performance is affected and a user’s experience can be affected, too. Therefore, System SW includes special requirements for memory consumption and latency.

The ability to integrate specialized hardware accelerators into mobile devices has already been greatly influencing the development of AI-based applications. This integration improves image quality, biometric device locking, predictive keyboards, and more – technologies that users are these days so accustomed to that it would be difficult to imagine a mobile device that does not feature them. The further development of accelerators is set to make our mobile devices even smarter, easier to use, and will open up new possibilities for AI applications that, previously, might only have been dreamt up in sci-fi films.

System SW also can be improved by utilizing these AI-based technologies for the customization of a mobile device for a specific user, by, for example, providing adaptive settings depending on the user’s location, behavior and device use patterns. Our team is actively involved in such research into the improvement of System SW through the utilization of On-Device AI technologies.

Q: What do you see as the main user benefits brought about by the incorporation of On-Device AI technologies into mobile devices?

On-Device AI is a relatively new technology, and is closely related to the growing popularity of AI-based applications. Initially, such applications were executed using a high-performance cloud server where all complex calculations were undertaken, but both the growth of mobile processor performance and the integration of specialized hardware accelerators mean that AI applications can now be developed to run directly on a mobile device, not a server.

Running neural networks on-device for AI applications has a number of advantages for users. Firstly, the response time for users enjoying their application is reduced, since there is no longer any need to send data to the server and then to wait for the result; secondly, the privacy of user data is maintained as all processing occurs on-device; and thirdly, these applications can run even without an Internet connection.

▲ Researchers at Samsung R&D Institute Russia

Q: How does your idea development process, both internally and with national companies and universities, serve to ultimately provide users with better experiences?

Here at SRR, we are proactive in monitoring the latest trends in relevant areas, conducting internal seminars, exchanging experiences, interacting with other teams and developing our proof-of-concepts. This experience exchange takes place mainly at informal events, at lunches or in the kitchen, and often brings about very interesting results. We also regularly conduct brainstorms to generate new ideas. One of the last brainstorm sessions we did was related to the future development of an open-source low level virtual machine (LLVM) project, wherein we generated about 30 different ideas, and after filtering, we chose 3 of the most promising areas that I am confident are set to expand our competence and will be useful further down the line for Samsung’s business.

In addition to interactions with other teams within SRR, our Research center organizes external seminars and joint workshops in which we share experiences, discuss current trends and share ideas for existing technological challenges. Here in Russia, we are lucky to have a very strong set of system programmers thanks to the emphasis placed on System SW development at the university stage.

Q: What do you see as being the main trends within your industry right now? How have you been incorporating them into the research you do at SRR?

I believe that System SW will become more and more optimized through the adoption of machine learning. This will allow us to focus on more complex tasks and get rid of routine optimization tasks. Smart System SW will allow us to achieve the best performance in information processing.

Additionally, On-Device AI will not only make our mobile devices smarter, but also our wearable devices, which will ultimately lead to the widespread use of AI across all kinds of devices. Connecting these smart devices will require high-speed communication methods that harness communication technologies such as 5G and beyond that have the ability to dynamically balance the load between the computing nodes of the network. This direction of research is also currently being actively explored in our laboratory.

An interview with Ratnakar Rao, an Advanced Communications Systems Expert from Samsung R&D India – Bangalore can be found in the following episode.

In this relay series, Samsung Newsroom is introducing tech experts from Samsung’s R&D centers around the globe to hear more about the work they do and the ways in which it is directly improving the lives of consumers.

The third expert in the series to be introduced is Bin Dai, Staff Engineer at the Artificial Intelligence (AI) Lab in Samsung R&D Institute China – Beijing (SRC-B). Dai joined SRC-B in 2020 to join his colleagues in working on network compression and on-device model design and research. Read on to learn more about the groundbreaking technologies Dai and his team are developing at SRC-B.

Q: AI-based technologies, including NLP (Natural Language Processing) and acoustic intelligence, are cutting-edge research areas that are constantly breaking new ground. But what role does the core research offering provided by machine learning play as a background for these innovations?

Machine learning plays a crucial role into bringing all kinds of technologies directly to users. Computer vision and speech recognition are two of the most successful areas currently utilizing AI. However, existing AI algorithms require huge computation resources, making it difficult to deploy state-of-the-art algorithms on mobile devices. In order to fix this issue, our AI Lab is working on producing tiny models with powerful performance from both a theoretical and a practical perspective. In this way, our core research is set to innovate all kinds of AI-based technologies.

Q: Can you please briefly introduce the Beijing Research Institute, and the kind of work that goes on there?

SRC-B is one of Samsung’s Electronics’ advanced R&D centers and was established in 2000, the first Samsung R&D center to be established in China. SRC-B focuses on groundbreaking technologies and specializes in artificial intelligence (AI) and next-generation telecommunications, from machine learning, computer vision, language processing and voice intelligence through to 3GPP standardization and more. We also promote tight industrial-academic partnerships. In April 2019, the AI Lab was established to focus on fundamental research into machine learning, and we are continuously looking for ways to apply our research results to Samsung products.

Q: Following the success of your major research thesis and other accomplishments, what are you working on at the moment?

SRC-B is currently aiming to find the best possible way to enhance the accuracy of an AI algorithm while reducing the computation complexity and resources used to do so. In order to achieve these goals, we are currently working on two research topics that enable accurate predictions with less data required: equivariant networks, part of the broader topic of geometric deep learning, and dynamic inference. There are many kinds of symmetries in computer vision datasets which are able to provide accurate depth measurement like human eyes can, such as image and LiDAR point clouds. With an equivariant network, these symmetries are taken into consideration when designing the network. It is thus able to achieve better performance with fewer resources since we have specifically considered the intrinsic structure of the dataset.

Dynamic inference is also a very interesting research direction. Unlike conventional methods which harness a fixed architecture for all data samples, dynamic inference can adaptively decide how many resources to use for each data sample. Accordingly, it will use fewer computational resources for simple samples and more resources for difficult ones. By doing so, the average computation resource used can be significantly reduced.

Q: Fundamental research into AI has been empowering all kinds of user-forward application fields, from computer vision to speech recognition. Could you explain a bit more about why this is, and the direction of research you and the AI Lab have been taking in order to optimize mobile experiences?

In this era of the internet, data is flooding everywhere around us. Where there is data, there is knowledge. AI algorithms are the very best tool for uncovering the knowledge hidden behind the data and make use of this knowledge to make all of our lives better.

We have developed a network compression algorithm based on the information bottleneck theory – which posits that extraneous details can be removed from noisy input data as if squeezed through a bottleneck – which has been applied to multiple tasks including video recognition, image segmentation and machine translation. We also actively collaborate with other labs in SRC-B in order to develop more powerful AI algorithms, including the Neural Architecture Search (NAS) and Once-For-All (OFA) solutions.

Q: What do you see as the main user benefits from incorporating all base mobile technologies with machine learning-based AI technologies?

Machine learning-based AI technologies can dramatically improve users’ lives in three key ways. Firstly, there are many convenient functions that simply cannot work without AI technologies. For example, the automatic question and answering system on mobile devices has to be powered by AI algorithms. Other more traditional methods are only able to handle very limited, pre-defined questions.

Secondly, AI techniques can significantly improve the performance of many applications compared to their performance when harnessing conventional technologies only. For example, after applying deep neural networks to a camera’s neural image signal processing (ISP) function, the quality of photos taken on that camera becomes significantly better.

Thirdly, AI technologies are capable of providing services that users previously didn’t even know they needed. For example, AI is capable of developing a user-specific software based on that user’s specific preferences, meaning that the user’s device experience can continuously be improved.

▲ Researchers at Samsung R&D Institute China – Beijing

Q: How does the work you do synergize with the work undertaken by the rest of Samsung R&D Institute China – Beijing, or perhaps even other R&D Institutes around the world? How does it come together to make users’ lives more convenient?

We are constantly collaborating with the other teams within SRC-B. We have been collaborating recently with our Visual Computing team in order to apply our information bottleneck-based compression algorithm to video recognition tasks and human segmentation tasks, resulting in the significant reduction of model sizes without any performance drop. In 2021, we participated in the Conference on Computer Vision and Pattern Recognition (CVPR)’s Neural Architecture Search (NAS) competition as one team with this solution, and won 1^st place.

We have also been working with our Language Intelligence team to compress their machine translation model, which facilitates the commercialization of their application.

We also believe that we can produce better research and application results by further communication, discussion and collaboration with AI centers globally.

Q: What do you see as being the main trends within your industry right now? How have you been incorporating them into the research you do at Samsung R&D Institute China – Beijing?

There are a lot of trending topics within our field at this time. Efficient network architecture design, self-supervised learning and graph neural networks are just a few examples.

Our focus is on network compression and tiny model design, which is ultimately useful for applications on mobile devices. There are a lot of mobile devices, such as smartphones, that possess very limited computational resources, meaning that it is impossible to deploy the huge models designed for services to these devices. Therefore, my team is focused on designing models suitable for these devices.

There are different ways to achieve these kinds of light yet powerful models. For instance, network pruning, quantization, knowledge distillation, neural network architecture search and dynamic inference are just a few industry areas that we are focusing on right now to achieve this.

Q: What has been the achievement at Samsung R&D Institute China – Beijing that you are most proud of so far?

Developed together in collaboration with our Communication Research team, we engineered AI algorithms for wireless communication. This solution achieved first place at the Wireless Communication AI Competition (WAIC) this year, which is the official competition for 5G+AI in China with over 600 teams enter from around the world and is held by the China Academy of Information and Communication Technology (CAICT). I am proud of this achievement and feel that it validates my belief that 5G combined with AI is a research direction with great potential.

An interview with Evgeny Pavlov, a system software expert from Samsung R&D Institute Russia (SRR) can be found in the following episode.

In this relay series, Samsung Newsroom is introducing tech experts from Samsung’s R&D centers around the globe to hear more about the work they do and the ways in which it is directly improving the lives of consumers.

The second expert in the series is Lukasz Slabinski, Head of the Artificial Intelligence Team at Samsung R&D Institute Poland (SRPOL). Slabinski joined SRPOL in 2013 as a Senior Engineer, and following 8 years of dedicated work, now leads the AI Team at SRPOL. Read on to hear more about the exciting innovation Slabinski and his team are involved with at SRPOL.

Q: Designing solutions for the speech recognition field is known to be highly intricate. When working on language-related technologies, what challenges have you encountered and how have you been overcoming them?

In my opinion, language-related technologies are far more complex than any other ones. Humankind communicates in almost 7000 constantly evolving languages, sub-divided into endless accents and dialects. Moreover, human language is far less objective than, for example, a picture, which can be described in mathematical formulas. People encode their thoughts as a set of sounds or characters into a message, which then needs to be decoded and interpreted by others. Because each phase of this process is personal, creative and non-deterministic, language-based human communication is very complex and ambiguous. Thus, on the one hand, we can enjoy beautiful poetry and funny jokes, and on the other, occasionally suffer from misunderstandings.

The R&D people who work on natural language processing (NLP) often reach their own, innately human, limitations. Even we encounter issues communicating clearly with colleagues at work, or family at home. So how, for example, can an engineer who speaks 2 languages design and code a machine translation system for 40 different languages? We solve this paradox using machine learning technologies.

During the process known as ‘training’, we automatically extract general patterns based on examples from our datasets and memorize them in the form of a model. To build a machine translation system, we train a neural network to map a sentence in different languages based on millions of examples, all carefully collected and cleaned beforehand. It sounds easy, but we deal here with 3 fundamental challenges.

The first challenge is the design of an appropriate machine learning model architecture capable of memorizing and generalizing enough language patterns for given problems such as machine translation, sentiment analysis, text summarization and others.

The second challenge is the preparation of sufficient amount of training data, as machine learning systems can recognize and memorize only those patterns presented in the training dataset.

The final challenge is the deployment of an already-trained machine learning model onto a dedicated Cloud or on-device platform.

We address these challenges by harnessing the vast expertise of our engineers, sophisticated approaches to collecting data and through endless experimentation with the state-of-the-art machine learning architectures.

Q: Can you please briefly introduce your AI Team, the Samsung R&D Institute Poland (SRPOL) and the kind of work that goes on there?

SRPOL is one of the largest international software R&D centers in Poland. It is located in two cities: Warsaw, the capital city of Poland and Cracow which is a major technology hub in its region. We closely collaborate with local start-ups, universities and research institutions.

The mission of the AI Team at SRPOL is the creation of the AI-based features, tools and services capable of facilitating and enriching human lives. We mainly focus on the NLP and Audio Intelligence areas, but we also possess expertise across many different specialties, including recommendation systems, indoor positioning, visual analytics and AR.

Q: As the head of the Polish Institute’s AI Team since 2018, you have overseen a myriad of projects both with and without the NLP focus. What are you and your team working on now?

Regarding the NLP area, we have been continuing our journey that began over 10 years ago by the development of systems such as Machine Translation, Dialogue Systems including Question Answering and Text Analytics. We work both on scalable, powerful cloud-based services as well as on fast and offline working on-device applications.

Audio Intelligence is a newer area for us. We began to focus our research capabilities on it around several years ago as the area had been gaining importance. Currently, we work on sound recognition, separation, enhancement and analysis. During our work, we take all levels of audio processing into consideration, from acoustic scene understanding to the fine-tuning of the embedded audio algorithms on devices with very limited hardware resources, such as wireless earbuds.

Q: Your technological focuses include NLP, text & data mining, audio intelligence and more. Has your research directly affected the development of any specific Samsung product or service, and what benefit has your team’s contribution offered to users?

SRPOL has a long record of commercializing AI technologies, but we did not do it alone. We are proud to be a part of a bigger picture, wherein SRPOL works closely with other Samsung R&D centers and contributes to commercialization.

For example, we contributed to the development of several intelligent text entry features for Samsung’s mobile devices, including the on-screen keyboard, hashtag feature, Samsung Note title recommendation and smart text replies on smartwatches.

We also contributed to the Galaxy Store’s Recommendation System, which suggests the most interesting games to a user based on their preferences.

Q: As an advocate for the new AI fields such as audio intelligence, what do you see as the main trends within your industry right now? How will this technology affect people’s daily lives?

I do believe that audio intelligence will be the next game-changer for all consumer electronic devices. Working on audio analytics is extremely important, as it is the missing part in advanced, truly human-centered AI-based systems.

Powerful NLP systems analyze the user’s intent as expressed by text and speech. Computer vision algorithms are behind almost every camera and visual content’s output. For most of us, it is hard to imagine driving a car without navigation, typing a message without spelling correctors, or searching for information without the Internet. But, except for a few professional applications, so far, we very rarely use intelligent audio technology to enhance our hearing. In my opinion, this is set to change soon.

Let’s imagine that we have a commonly available technology that allows people to select what and how they want to hear. For example, during a lunch with a friend in a park located in a busy city center, someone could choose to hear only the sounds of nature and the person they are speaking with. Or, let’s imagine an advanced VR or AR system, recently referred to as Metaverse that creates an immersive 3D audio experience directly in people’s heads. Just these two concepts generate hundreds of new possible use cases, but let’s go further. How about hearing things that are currently inaudible to people? Now humans can hear only a narrow spectrum of different sounds. Our world is full of meaningful sounds which, for the most part, the current AI technologies are not involved in. With the development of the audio intelligence technologies, I believe that all of this is going to affect people’s lives hugely.

▲ Researchers at Samsung R&D Institute Poland work on Active Noise Cancellation (ANC) technology development with a Head & Torso Simulator (HATS) in an anechoic room.

Q: How have you been incorporating the current trends into the research you do at Samsung R&D Institute Poland?

Aside from NLP and Audio, we are also working to find the most effective ways to build truly multimodal systems. To do that, we proceed with research and analyzing use cases from different perspectives. Such analysis is made possible thanks to our diverse and interdisciplinary team that consists of engineers, linguists, data scientists and more.

Q: What has been your most important achievement at SRPOL so far?

That would be our Machine Translation solution. Our solution has garnered wins at various competitions for five years straight: the International Workshop on Spoken Language Translation (IWSLT) from 2017 to 2020; the Workshop on Machine Translation (WMT) in 2020; and the Workshop on Asian Translation (WAT) in 2021. These are among the most prestigious international competitions in our field.

Winning recognition at WAT this year was a particularly satisfying milestone, as developing our solution for the Asian languages was originally a difficult feat for us as Polish engineers – but this achievement has proven the true power of our technology that goes beyond a mere demo showcase.

Another achievement that I am very proud of is the speed of growth that the audio intelligence team and its technology development have achieved. In just a few years, after starting pretty much from scratch, we were able to stand on the podium of the workshop on Detection and Classification of Acoustic Scenes and Events for two consecutive years, 2019 and 2020. We have also published several scientific papers and patents in this area. I am sure this is just the beginning of our prolific activities in this field.

An interview with Bin Dai, a machine learning expert from Samsung Research Institute China-Beijing can be found in the following episode.

Samsung Electronics has long recognized the significance of these advanced technologies, and has been actively pursuing innovation in these fields. Expert researchers are working hard at Samsung Research’s¹ 14 R&D centers and 7 global AI centers all over the world in order to prepare for the future, innovate for users and create the next generation of cutting-edge technologies and services that the Samsung Electronics’ legacy is built on.

In this relay series, Samsung Newsroom is introducing tech experts from Samsung’s R&D centers around the globe to hear more about the work they do and the ways in which it is directly improving the lives of consumers.

The first expert in the series to be introduced is Sergii Lytvynenko, Head of the Visual Intelligence Team at Samsung R&D Institute Ukraine (SRK). Sergii has been working for SRK for more than decade since he first joined as a SW Engineer. Read on to hear more about the groundbreaking work Lytvynenko and his team undertake at SRK.

Q: Can you please briefly introduce the Samsung R&D Institute Ukraine and the kind of work that goes on there?

Our R&D center is located in Kyiv, situated in the heart of Ukraine. Since its inception in 2009, SRK has been focused on and has deep expertise in the AI, Augmented Reality (AR) / Virtual Reality (VR) and Security domains. SRK is composed of prominent industry professionals and is currently working on the study of intelligence security, computer vision, context-aware intelligent services, and more. Also, as part of industrial-educational cooperation initiatives, SRK actively co-operates with local universities and schools.

Q: What are you and the Visual Intelligence Team working on at the moment?

Our team is currently conducting fundamental research into the AI, Computer Vision and Computer Graphics domains. The main mission of our team is to transform research advancements into holistic user experiences, thereby enhancing the quality of people’s lives, simplifying their daily routines and delivering positive emotions and immersive experiences.

To do so, we are collaborating closely with various teams in other countries by conducting advanced research in our focal domains and working with different business units by contributing our core technologies to Samsung products.

Q: Your team covers two major technological domains – Computer Vision and Computer Graphics. How do these technologies contribute to innovating new user experiences?

Last year, we undertook extensive work on the Smart Trainer solution, which enables a totally new level of home fitness experiences. Through the USB camera connected to the Samsung Smart TV, the system can track your activities, keep track of the exercises you do and even offer recommendations on your form accuracy, all thanks to AI. We are now very happy that Samsung TV users can enjoy this feature in their homes.

Q: How are you incorporating the key technologies from your focal domains into your current projects, such as AR Glasses?

These days we are performing advanced R&D to tackle major challenges in the computer vision and graphics areas for AR Glasses. On the vision side, we are working on the essential solutions required for AR, including Simultaneous Localization and Mapping (SLAM), Depth Estimation, Environment Understanding and Human Computer Interaction (HCI). On the graphics side, we are conducting research into low-latency rendering for AR and Game Performance optimization.

▲ Visual Intelligence Team at Samsung R&D Institute Ukraine

Q: As well as AR, your team contributes to S Pen technology development. Can you give us a bit of background into the development of this technology?

One of our focal R&D areas and core solutions is handwriting recognition technology for S Pen enabled devices, which is being deployed and spread to Galaxy line up. While working on our handwriting recognition solution, we also developed a rich patent portfolio, thus contributing to Samsung’s core technology development.

Q: In what ways do you think the optimized S Pen technologies your team created for the Galaxy Z Fold3 will complement users’ experience of the device?

Galaxy Z Fold3 is a really unique product. Its large, flexible display expands the boundaries and opens up new possibilities for users to serve as a true productivity companion for daily business and education. Within this context, S Pen and handwriting recognition and low latency become of crucial importance and we are taking the very best of the conventional pen and paper to deliver those same types of experiences to the digital screen.

Q: In what ways do the technologies your team contributed to Galaxy Z Fold3 set to enhance the quality of users’ lives and simplify their routines?

We deployed our AI Based Point Prediction solution to minimize the latency input of S Pen in order to make the writing and drawing experience feel more like that of pen and paper. Furthermore, handwriting recognition technologies make digital writing smarter, easier and more enjoyable. Users can transform their notes to printed documents, recognize tables, diagrams, embed links, undertake math problems and more, simpler than ever before. Experiences like this are what make a real difference in our daily lives.

Q: What do you see as the main technology trends right now?

These days, we recognize Visual Modality as the next big thing: how to transform a note into a smart note, how to make a video into a smart video, and how much useful context information we can extract from these processes. For this technology, AR opens up tons of possibilities, as well as challenges to be resolved. For example, “Digital Eyes” that would fully explore an environment for a user and provide well-organized contextual information could totally change our lives.

Another big trend right now is HCI. Here we think multi-modal interaction, which is a crucial part of HCI, would be essential. Multi-modal interactions are user-machine interactions that encapsulate vision, language and knowledge, and this technology can help a Samsung device understand the world in which it’s situated.

Q: What has been your most memorable achievement at SRK so far?

June 2021 was a really special month for us as we won the CVPR (Conference on Computer Vision and Pattern Recognition) 2021 Chart Question Answering Challenge. CVPR is the world’s biggest conference on computer vision and AI. We are really proud of what we achieved.

Q: Visual intelligence technologies are crucial when it comes to innovating new mobile experiences for users. In what ways do language-related technologies also contribute to these experiences?

Natural Language Processing (NLP) is one of the most challenging research areas. We really wish that every single person around the world were able to use and experience our solutions, and to achieve this, language expansion and support are of crucial importance. In S Pen Handwriting recognition, we are continuously working to extend the language coverage. Our solution now supports more than 80 languages, and more are on the way.

An interview with Lukasz, a natural language processing expert from Samsung Research Institute Poland can be found in the following episode.

¹ Samsung Research is the advanced research and development (R&D) hub of Samsung’s Consumer Electronics (CE) Division and IT & Mobile Communications (IM) Division.