But Samsung’s cutting-edge new foldable solutions couldn’t have been realized without the hard work and collaborative efforts of Samsung’s global network of engineers. Among other accomplishments, these employees were able to incorporate advanced technologies like AI to enable drawing and writing on the Galaxy Z Fold3’s large screen, and to allow the device to convert handwriting in a range of languages into on-screen text with S Pen.

To find out more about just what went into the development of the innovative new S Pen, Newsroom spoke to Olga Radyvonenko from the Visual Intelligence Team at Samsung Research Ukraine.

Q: Can you briefly introduce the Ukrainian Research Institute and the kind of work that goes on there?

Artificial intelligence (AI) is one of the key technologies of our R&D center. Our lab is focused on bringing Samsung users human-centric AI technologies that make input and interaction easy and enjoyable.

Q: The S Pen has now been applied to the Galaxy Z Fold3, making it the first of its kind among foldable smartphones. What role did you play in this process?

The major contributions of our lab to the Z Fold3 were the development of on-device deep learning technologies for handwritten content recognition and the technology for reducing S Pen input latency that combines advances in graphics and AI.

Q: What is the S Pen Software Development Kit? What particular advantages does it bring?

The S Pen Software Development Kit (SDK) is being developed by Samsung researchers and developers from many countries around the world, and we are very proud to be part of this international team. The S Pen SDK provides ways to speed up rendering and to reduce and optimize S Pen input latency. In addition, it comprises smart AI-powered features, such as recognition of handwritten text, shapes, diagrams and handwriting straightening.

Q: Given the unique function of the S Pen on the Z Fold3, was there any function of the S Pen that was developed or expanded specifically for the foldable smartphone?

As the new Z Fold3 has a massive and unique foldable display, we’ve focused on using our AI technology to optimize the S Pen experience as well as improve the latency. Thanks to our point prediction technology powered by deep learning, we are able to predict the next movement of the S Pen to minimize the delay between input and output on the screen of the Z Fold3. This, in combination with the front buffer rendering technology and unique large screen, means that our users can enjoy the most natural handwriting, note-taking and drawing on their Z Fold3.

Q: How many languages does the S Pen now support? What were some of the difficulties of expanding the number of recognized languages for the Z Fold3’s S Pen?

At this moment, the S Pen supports 88 languages in total, 14 of which were newly added in the past year.

Adding a new language for handwriting recognition requires a lot of effort. In order to achieve the world’s best level of recognition quality and speed, we have developed an advanced AI technology. This technology is based on deep learning across millions of users’ samples and language modeling for natural language processing (NLP).

Q: Do you have any special tips for making the most out of the S Pen on the Z Fold3 to share with users?

For those who love writing by hand, the S Pen To Text feature now supports writing with the S Pen directly into text fields. The handwriting is then automatically converted to text. Also, you can edit any printed text with your S Pen – for example, when correcting spacing or inserting words.

One feature that will also interest many users is Neat Shape mode. It can help you to not only draw shapes, but also build a flowchart.

Make sure to check the Tips application and not skip usage tips – there you will discover the newest S Pen and input features, and also learn how to use many other Samsung apps to their fullest potential.

Q: Looking forward, what do you think the future of the S Pen looks like? How might it evolve?

We hope to introduce more ways to support better user experiences by providing even more personalization in the near future. Our mission is to make S Pen input like writing in your notepad with a pen, but better and smarter.

Most of all, I want for as many people as possible to experience the power and convenience of the S Pen and its handwritten input capabilities. It has the potential to enhance input on a wide range of devices. We view the S Pen not only as a cool gadget, but first and foremost as an accessible and useful tool. I hope I will see the S Pen and its features casually used in education, in the workplace and in everyday life. I do believe we are set to see a revolution in input technology.

The full interview video can be viewed here:

Samsung Electronics has long recognized the significance of these advanced technologies, and has been actively pursuing innovation in these fields. Expert researchers are working hard at Samsung Research’s¹ 14 R&D centers and 7 global AI centers all over the world in order to prepare for the future, innovate for users and create the next generation of cutting-edge technologies and services that the Samsung Electronics’ legacy is built on.

In this relay series, Samsung Newsroom is introducing tech experts from Samsung’s R&D centers around the globe to hear more about the work they do and the ways in which it is directly improving the lives of consumers.

The first expert in the series to be introduced is Sergii Lytvynenko, Head of the Visual Intelligence Team at Samsung R&D Institute Ukraine (SRK). Sergii has been working for SRK for more than decade since he first joined as a SW Engineer. Read on to hear more about the groundbreaking work Lytvynenko and his team undertake at SRK.

Q: Can you please briefly introduce the Samsung R&D Institute Ukraine and the kind of work that goes on there?

Our R&D center is located in Kyiv, situated in the heart of Ukraine. Since its inception in 2009, SRK has been focused on and has deep expertise in the AI, Augmented Reality (AR) / Virtual Reality (VR) and Security domains. SRK is composed of prominent industry professionals and is currently working on the study of intelligence security, computer vision, context-aware intelligent services, and more. Also, as part of industrial-educational cooperation initiatives, SRK actively co-operates with local universities and schools.

Q: What are you and the Visual Intelligence Team working on at the moment?

Our team is currently conducting fundamental research into the AI, Computer Vision and Computer Graphics domains. The main mission of our team is to transform research advancements into holistic user experiences, thereby enhancing the quality of people’s lives, simplifying their daily routines and delivering positive emotions and immersive experiences.

To do so, we are collaborating closely with various teams in other countries by conducting advanced research in our focal domains and working with different business units by contributing our core technologies to Samsung products.

Q: Your team covers two major technological domains – Computer Vision and Computer Graphics. How do these technologies contribute to innovating new user experiences?

Last year, we undertook extensive work on the Smart Trainer solution, which enables a totally new level of home fitness experiences. Through the USB camera connected to the Samsung Smart TV, the system can track your activities, keep track of the exercises you do and even offer recommendations on your form accuracy, all thanks to AI. We are now very happy that Samsung TV users can enjoy this feature in their homes.

Q: How are you incorporating the key technologies from your focal domains into your current projects, such as AR Glasses?

These days we are performing advanced R&D to tackle major challenges in the computer vision and graphics areas for AR Glasses. On the vision side, we are working on the essential solutions required for AR, including Simultaneous Localization and Mapping (SLAM), Depth Estimation, Environment Understanding and Human Computer Interaction (HCI). On the graphics side, we are conducting research into low-latency rendering for AR and Game Performance optimization.

▲ Visual Intelligence Team at Samsung R&D Institute Ukraine

Q: As well as AR, your team contributes to S Pen technology development. Can you give us a bit of background into the development of this technology?

One of our focal R&D areas and core solutions is handwriting recognition technology for S Pen enabled devices, which is being deployed and spread to Galaxy line up. While working on our handwriting recognition solution, we also developed a rich patent portfolio, thus contributing to Samsung’s core technology development.

Q: In what ways do you think the optimized S Pen technologies your team created for the Galaxy Z Fold3 will complement users’ experience of the device?

Galaxy Z Fold3 is a really unique product. Its large, flexible display expands the boundaries and opens up new possibilities for users to serve as a true productivity companion for daily business and education. Within this context, S Pen and handwriting recognition and low latency become of crucial importance and we are taking the very best of the conventional pen and paper to deliver those same types of experiences to the digital screen.

Q: In what ways do the technologies your team contributed to Galaxy Z Fold3 set to enhance the quality of users’ lives and simplify their routines?

We deployed our AI Based Point Prediction solution to minimize the latency input of S Pen in order to make the writing and drawing experience feel more like that of pen and paper. Furthermore, handwriting recognition technologies make digital writing smarter, easier and more enjoyable. Users can transform their notes to printed documents, recognize tables, diagrams, embed links, undertake math problems and more, simpler than ever before. Experiences like this are what make a real difference in our daily lives.

Q: What do you see as the main technology trends right now?

These days, we recognize Visual Modality as the next big thing: how to transform a note into a smart note, how to make a video into a smart video, and how much useful context information we can extract from these processes. For this technology, AR opens up tons of possibilities, as well as challenges to be resolved. For example, “Digital Eyes” that would fully explore an environment for a user and provide well-organized contextual information could totally change our lives.

Another big trend right now is HCI. Here we think multi-modal interaction, which is a crucial part of HCI, would be essential. Multi-modal interactions are user-machine interactions that encapsulate vision, language and knowledge, and this technology can help a Samsung device understand the world in which it’s situated.

Q: What has been your most memorable achievement at SRK so far?

June 2021 was a really special month for us as we won the CVPR (Conference on Computer Vision and Pattern Recognition) 2021 Chart Question Answering Challenge. CVPR is the world’s biggest conference on computer vision and AI. We are really proud of what we achieved.

Q: Visual intelligence technologies are crucial when it comes to innovating new mobile experiences for users. In what ways do language-related technologies also contribute to these experiences?

Natural Language Processing (NLP) is one of the most challenging research areas. We really wish that every single person around the world were able to use and experience our solutions, and to achieve this, language expansion and support are of crucial importance. In S Pen Handwriting recognition, we are continuously working to extend the language coverage. Our solution now supports more than 80 languages, and more are on the way.

An interview with Lukasz, a natural language processing expert from Samsung Research Institute Poland can be found in the following episode.

¹ Samsung Research is the advanced research and development (R&D) hub of Samsung’s Consumer Electronics (CE) Division and IT & Mobile Communications (IM) Division.