[Into the Future With Samsung Research ④] Samsung R&D Institute Russia: Optimizing User Experience and More With Intelligent System Software Solutions

Following Episode 3

In this relay series, Samsung Newsroom is introducing tech experts from Samsung’s R&D centers around the globe to hear more about the work they do and the ways in which it is directly improving the lives of consumers.

The fourth expert in the series is Evgeny Pavlov, Head of the Advanced System Software Lab at Samsung R&D Institute Russia (SRR). Following 9 years of dedicated work into advanced techniques for program analysis at SRR, Pavlov was made the head of his laboratory in 2020.

The systems Pavlov works on, System software (SW), is software that has been designed to provide a basis for other software, such as the Operating system (OS) you use in your smartphone, the frameworks for AI-based applications, tools for developers and more. System SW is responsible for the communication between applied software and hardware. Read on to learn more about the crucial research Pavlov and his team undertake at SRR.

Q: The results of AI and machine learning research are of key importance to designing and optimizing all kinds of technologies. What role does system software research play in further activating these technologies? 

System SW research now plays a very important role in machine learning, although this may not always be visible to the end user. First of all, machine learning frameworks do not always work optimally on general-purpose hardware and processors, so they need to be optimized in ways that take into account various hardware features and use additional central processing unit (CPU) extensions.

Furthermore, the latest trends in the artificial intelligence (AI) industry include the integration of specialized processing units for neural network acceleration. Many companies have been developing specialized accelerators for neural networks called neural processor units (NPU) recently. For the optimal processing of a machine learning model, it is necessary to transform the neural network model into a set of instructions for this accelerator.

These neural network model conversions are usually automated using a neural network compiler since a deep understanding of NPU architecture is required for the development of these compilers. This means that us system SW developers are involved in their development since we have a deep understanding of how computer hardware works.

In other words, thanks to this change in industry requirements, the focus of System SW engineers is moving from the optimization of general-purpose programs towards the optimization of AI- and machine learning-based programs.

Q: Can you please briefly introduce Samsung R&D Russia Institute (SRR) and the kind of work that goes on there?

These days, we at SRR are focusing on developing our expertise and capabilities in three main R&D areas: Sensor Solution, AI Imaging and System SW. SRR has end-to-end experience in sensor R&D, which includes hardware and algorithm development as well as commercialization specifically for biometric and life care solutions. SRR has been deeply involved in the development of iris, face and fingerprint biometry as well as body composition estimation for smartwatches. SRR has also contributed to the strengthening of the well-known Super Slow Motion and Night Mode features on smartphone cameras through consistently developing the synergy between optics and AI within the AI Imaging area.

I believe that System SW is one of the most promising areas of research happening in SRR right now. Based on our deep understanding of various hardware and operating systems (OS), as well as strong engineering manpower, we do our best to be a System SW core tech provider for the entire business.

Q: Following your accomplishments within the Advanced System SW Lab at SRR, what are you working on at the moment?

We are conducting extensive research into potential new directions for our System SW team in order to understand the latest trends in System SW that may well replace traditional System SW techniques in the near future.

Our lab is also currently working on a project related to enabling the 5G scalable vRAN infrastructure to support multiple network types, as well as other projects related to compiler technologies for the Android and Tizen OS, advanced OS developing and Software Development Kit (SDK) development for On-Device AI.

Besides leading the Advanced System SW lab, I am also currently leading an SRR project for On-Device AI platform called ONE, or On-Device Neural Engine. This project is being developed in collaboration with the On-Device Lab at Samsung Research, and a major aspect of this project is being maintained by Samsung as an open-source project located on github.com.

Q: On-device AI and advanced System SW technologies are crucial to providing users with robust, innovative mobile experiences. Could you explain a bit more about why this is, and the direction of research you and the Advanced System SW Lab have been taking?

System SW plays a key role in application operation and user experience. System SW is the lower layer that sits between a device’s hardware and user applications – meaning that it is the foundation for all other software. Users may not see System SW in action, since their interactions with their mobile applications are relegated to simply engaging with the interface, but under the hood of their favorite apps are many layers of program logic – for example, managing the recognition of a tap to the screen in the system kernel and then drawing a corresponding window through the graphics library. If there is a delay at any one of these levels of recognition, the entire system performance is affected and a user’s experience can be affected, too. Therefore, System SW includes special requirements for memory consumption and latency.

The ability to integrate specialized hardware accelerators into mobile devices has already been greatly influencing the development of AI-based applications. This integration improves image quality, biometric device locking, predictive keyboards, and more – technologies that users are these days so accustomed to that it would be difficult to imagine a mobile device that does not feature them. The further development of accelerators is set to make our mobile devices even smarter, easier to use, and will open up new possibilities for AI applications that, previously, might only have been dreamt up in sci-fi films.

System SW also can be improved by utilizing these AI-based technologies for the customization of a mobile device for a specific user, by, for example, providing adaptive settings depending on the user’s location, behavior and device use patterns. Our team is actively involved in such research into the improvement of System SW through the utilization of On-Device AI technologies.

Q: What do you see as the main user benefits brought about by the incorporation of On-Device AI technologies into mobile devices?

On-Device AI is a relatively new technology, and is closely related to the growing popularity of AI-based applications. Initially, such applications were executed using a high-performance cloud server where all complex calculations were undertaken, but both the growth of mobile processor performance and the integration of specialized hardware accelerators mean that AI applications can now be developed to run directly on a mobile device, not a server.

Running neural networks on-device for AI applications has a number of advantages for users. Firstly, the response time for users enjoying their application is reduced, since there is no longer any need to send data to the server and then to wait for the result; secondly, the privacy of user data is maintained as all processing occurs on-device; and thirdly, these applications can run even without an Internet connection.

▲ Researchers at Samsung R&D Institute Russia

Q: How does your idea development process, both internally and with national companies and universities, serve to ultimately provide users with better experiences?

Here at SRR, we are proactive in monitoring the latest trends in relevant areas, conducting internal seminars, exchanging experiences, interacting with other teams and developing our proof-of-concepts. This experience exchange takes place mainly at informal events, at lunches or in the kitchen, and often brings about very interesting results. We also regularly conduct brainstorms to generate new ideas. One of the last brainstorm sessions we did was related to the future development of an open-source low level virtual machine (LLVM) project, wherein we generated about 30 different ideas, and after filtering, we chose 3 of the most promising areas that I am confident are set to expand our competence and will be useful further down the line for Samsung’s business.

In addition to interactions with other teams within SRR, our Research center organizes external seminars and joint workshops in which we share experiences, discuss current trends and share ideas for existing technological challenges. Here in Russia, we are lucky to have a very strong set of system programmers thanks to the emphasis placed on System SW development at the university stage.

Q: What do you see as being the main trends within your industry right now? How have you been incorporating them into the research you do at SRR?

I believe that System SW will become more and more optimized through the adoption of machine learning. This will allow us to focus on more complex tasks and get rid of routine optimization tasks. Smart System SW will allow us to achieve the best performance in information processing.

Additionally, On-Device AI will not only make our mobile devices smarter, but also our wearable devices, which will ultimately lead to the widespread use of AI across all kinds of devices. Connecting these smart devices will require high-speed communication methods that harness communication technologies such as 5G and beyond that have the ability to dynamically balance the load between the computing nodes of the network. This direction of research is also currently being actively explored in our laboratory.

An interview with Ratnakar Rao, an Advanced Communications Systems Expert from Samsung R&D India – Bangalore can be found in the following episode.