But Samsung’s cutting-edge new foldable solutions couldn’t have been realized without the hard work and collaborative efforts of Samsung’s global network of engineers. Among other accomplishments, these employees were able to incorporate advanced technologies like AI to enable drawing and writing on the Galaxy Z Fold3’s large screen, and to allow the device to convert handwriting in a range of languages into on-screen text with S Pen.

To find out more about just what went into the development of the innovative new S Pen, Newsroom spoke to Olga Radyvonenko from the Visual Intelligence Team at Samsung Research Ukraine.

Q: Can you briefly introduce the Ukrainian Research Institute and the kind of work that goes on there?

Artificial intelligence (AI) is one of the key technologies of our R&D center. Our lab is focused on bringing Samsung users human-centric AI technologies that make input and interaction easy and enjoyable.

Q: The S Pen has now been applied to the Galaxy Z Fold3, making it the first of its kind among foldable smartphones. What role did you play in this process?

The major contributions of our lab to the Z Fold3 were the development of on-device deep learning technologies for handwritten content recognition and the technology for reducing S Pen input latency that combines advances in graphics and AI.

Q: What is the S Pen Software Development Kit? What particular advantages does it bring?

The S Pen Software Development Kit (SDK) is being developed by Samsung researchers and developers from many countries around the world, and we are very proud to be part of this international team. The S Pen SDK provides ways to speed up rendering and to reduce and optimize S Pen input latency. In addition, it comprises smart AI-powered features, such as recognition of handwritten text, shapes, diagrams and handwriting straightening.

Q: Given the unique function of the S Pen on the Z Fold3, was there any function of the S Pen that was developed or expanded specifically for the foldable smartphone?

As the new Z Fold3 has a massive and unique foldable display, we’ve focused on using our AI technology to optimize the S Pen experience as well as improve the latency. Thanks to our point prediction technology powered by deep learning, we are able to predict the next movement of the S Pen to minimize the delay between input and output on the screen of the Z Fold3. This, in combination with the front buffer rendering technology and unique large screen, means that our users can enjoy the most natural handwriting, note-taking and drawing on their Z Fold3.

Q: How many languages does the S Pen now support? What were some of the difficulties of expanding the number of recognized languages for the Z Fold3’s S Pen?

At this moment, the S Pen supports 88 languages in total, 14 of which were newly added in the past year.

Adding a new language for handwriting recognition requires a lot of effort. In order to achieve the world’s best level of recognition quality and speed, we have developed an advanced AI technology. This technology is based on deep learning across millions of users’ samples and language modeling for natural language processing (NLP).

Q: Do you have any special tips for making the most out of the S Pen on the Z Fold3 to share with users?

For those who love writing by hand, the S Pen To Text feature now supports writing with the S Pen directly into text fields. The handwriting is then automatically converted to text. Also, you can edit any printed text with your S Pen – for example, when correcting spacing or inserting words.

One feature that will also interest many users is Neat Shape mode. It can help you to not only draw shapes, but also build a flowchart.

Make sure to check the Tips application and not skip usage tips – there you will discover the newest S Pen and input features, and also learn how to use many other Samsung apps to their fullest potential.

Q: Looking forward, what do you think the future of the S Pen looks like? How might it evolve?

We hope to introduce more ways to support better user experiences by providing even more personalization in the near future. Our mission is to make S Pen input like writing in your notepad with a pen, but better and smarter.

Most of all, I want for as many people as possible to experience the power and convenience of the S Pen and its handwritten input capabilities. It has the potential to enhance input on a wide range of devices. We view the S Pen not only as a cool gadget, but first and foremost as an accessible and useful tool. I hope I will see the S Pen and its features casually used in education, in the workplace and in everyday life. I do believe we are set to see a revolution in input technology.

The full interview video can be viewed here:

Samsung Electronics last month announced its goal to strengthen its leadership in the global system semiconductor industry by 2030 through expanding its proprietary NPU technology development. The company recently delivered an update to this goal at the conference on Computer Vision and Pattern Recognition (CVPR), one of the top academic conferences in computer vision fields.

This update is the company’s development of its On-Device AI lightweight algorithm, introduced at CVPR with a paper titled “Learning to Quantize Deep Networks by Optimizing Quantization Intervals With Task Loss”. On-Device AI technologies directly compute and process data from within the device itself. Over 4 times lighter and 8 times faster than existing algorithms, Samsung’s latest algorithm solution is dramatically improved from previous solutions and has been evaluated to be key to solving potential issues for low-power, high-speed computations.

Streamlining the Deep Learning Process

Samsung Advanced Institute of Technology (SAIT) has announced that they have successfully developed On-Device AI lightweight technology that performs computations 8 times faster than the existing 32-bit deep learning data for servers. By adjusting the data into groups of under 4 bits while maintaining accurate data recognition, this method of deep learning algorithm processing is simultaneously much faster and much more energy efficient than existing solutions.

Samsung’s new On-Device AI processing technology determines the intervals of the significant data that influence overall deep learning performance through ‘learning’. This ‘Quantization¹ Interval Learning (QIL)’ retains data accuracy by re-organizing the data to be presented in bits smaller than their existing size. SAIT ran experiments that successfully demonstrated how the quantization of an in-server deep learning algorithm in 32 bit intervals provided higher accuracy than other existing solutions when computed into levels of less than 4 bits.

When the data of a deep learning computation is presented in bit groups lower than 4 bits, computations of ‘and’ and ‘or’ are allowed, on top of the simpler arithmetic calculations of addition and multiplication. This means that the computation results using the QIL process can achieve the same results as existing processes can while using 1/40 to 1/120 fewer transistors².

As this system therefore requires less hardware and less electricity, it can be mounted directly in-device at the place where the data for an image or fingerprint sensor is being obtained, ahead of transmitting the processed data on to the necessary end points.

The Future of AI Processing and Deep Learning

This technology will help develop Samsung’s system semiconductor capacity as well as strengthening one of the core technologies of the AI era – On-Device AI processing. Differing from AI services that use cloud servers, On-Device AI technologies directly compute data all from within the device itself.

On-Device AI technology can reduce the cost of cloud construction for AI operations since it operates on its own and provides quick and stable performance for use cases such as virtual reality and autonomous driving. Furthermore, On-Device AI technology can save personal biometric information used for device authentication, such as fingerprint, iris and face scans, onto mobile devices safely.

“Ultimately, in the future we will live in a world where all devices and sensor-based technologies are powered by AI,” noted Chang-Kyu Choi, Vice President and head of Computer Vision Lab of SAIT. “Samsung’s On-Device AI technologies are lower-power, higher-speed solutions for deep learning that will pave the way to this future. They are set to expand the memory, processor and sensor market, as well as other next-generation system semiconductor markets.”

A core feature of On-Device AI technology is its ability to compute large amounts of data at a high speed without consuming excessive amounts of electricity. Samsung’s first solution to this end was the Exynos 9 (9820), introduced last year, which featured a proprietary Samsung NPU inside the mobile System on Chip (SoC). This product allows mobile devices to perform AI computations independent of any external cloud server.

Many companies are turning their attention to On-Device AI technology. Samsung Electronics plans to enhance and extend its AI technology leadership by applying this algorithm not only to mobile SoC, but also to memory and sensor solutions in the near future.

Four individuals who played key roles in developing Samsung’s On-Device AI Lightweight Algorithm. From Left to right; Jae-Joon Han, Chang-Young Son, Sang-Il Jung, Chang-Kyu Choi of Samsung Advanced Institute of Technology

1 Quantization is the process of decreasing the number of bits in data by binning the given data into sections of limited number levels, which can be represented in certain bit values and are regarded as having the same value per section

² Transistors are devices that control the flow of current or voltage in a semiconductor by acting as amplifiers or switches

The AI Lab is located in Mila – Montreal Institute for Learning Algorithms – in Montreal, Canada. Founded by Professor Yoshua Bengio at the University of Montreal, Mila is one of the greatest research centers in the field of deep learning and has a partnership with the University of Montreal and McGill University. SAIT AI Lab Montreal has an open workspace with the aim of working closely with the AI research communities in Mila.

SAIT AI Lab Montreal will focus on unsupervised learning and Generative Adversarial Networks (GANs) research to develop disruptive innovation and breakthrough technologies, including new deep learning algorithms and next generation of on-device AI.

To drive the effort, this AI Lab has actively recruited leaders in deep learning research, including Simon Lacoste-Julien, Professor at the University of Montreal, who recently joined as the leader of the lab. In addition, Samsung is planning to dispatch R&D personnel in its Device Solutions Business to Montreal over time and utilize AI Labs as a base for training AI researchers and collaborating with other advanced AI research institutes.

On the other side, SAIT AI Lab Montreal continues to build a strong relationship with Yoshua Bengio, one of the world’s greatest experts on deep learning, machine learning, and AI. SAIT and Professor Bengio collaborated on deep learning algorithm research since 2014, successfully publishing three papers on academic journals.

Professor Yoshua Bengio said, “Samsung’s collaboration with Mila is well established already and has been productive and built strong trust on both sides. With a new SAIT lab in the midst of the recently inaugurated Mila building and many exciting research challenges ahead of us in AI, I expect even more mutually positive outcomes in the future.”

SAIT has actively pursued research collaboration with other top authorities in the field. In addition to Professor Bengio, SAIT has worked with Yann LeCun, Professor at New York University and Richard Zemel, Professor at University of Toronto. Yoshua Bengio and Yann LeCun, along with computer scientist Geoffrey Everest Hinton won the 2018 Turing Award which is deemed the ‘Nobel Prize in computer science.’

“SAIT focuses on research and development – not only in next generation semiconductor but also innovative AI as a seed technology in system semiconductors. SAIT AI Lab Montreal will play a key role within Samsung to redefine AI theory and deep learning algorithm for the next 10 years,” said Sungwoo Hwang, Executive Vice President and Deputy Head of SAIT.