CVPR is a world-renowned international Artificial Intelligence (AI) conference co-hosted by the Institute of Electrical and Electronics Engineers (IEEE) and the Computer Vision Foundation (CVF) which has been running since 1983. CVPR is widely considered to be one of the three most significant international conferences in the field of computer vision, alongside the International Conference on Computer Vision (ICCV) and the European Conference on Computer Vision (ECCV). CVPR 2022 will be held as a hybrid event, both in-person and online, from June 19 to 24 in New Orleans, Louisiana, U.S..

Of the thesis papers submitted by Samsung Research, two papers submitted by its Toronto AI Center were selected for oral presentations. Opportunities to give oral presentations at CVPR 2022 are extended to the top 4-5% of the total number of papers submitted. For Samsung’s Toronto AI Center, this is the second time in two years they have earned such a chance, as they were also selected for oral presentation in 2020.

The first of these two oral presentations from the Toronto AI Center will focus on their paper “P³IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervision”, a study done on how to build next-level AI systems capable of analyzing and mimicking human behavior. Procedure planning is gaining attention in the field, as it could lead to technologies capable of assisting humans in solving goal-directed problems, such as cooking food or installing and repairing devices.

The research team’s approach undercuts the previous requirement of costly data annotations that the start and end times of each intermediate instructional step were labeled with. Instead, the new approach allows AI to learn from natural language instructions, sourced from the internet for example, and predict the intermediate steps. Additionally, the model is enhanced with a probabilistic generative module to handle the uncertainty inherent to procedural planning.

▲ A section from the presentation for “P³IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervision” by the Toronto AI Center

The second oral presentation to be given by the Toronto AI Center is a study on “Day-to-Night Image Synthesis for Training Nighttime Neural ISPs”. This study is focused on how to synthesize the nighttime image data needed to train Night Mode using neural Image Signal Processors (ISPs) on smartphone cameras. This technology converts clear daytime images — which are much easier to capture than nighttime images — into nighttime image pairs, a strategy that demonstrates performance on par with training on real data captured at night.

▲ A visual from “Day-to-Night Image Synthesis for Training Nighttime Neural ISPs” by the Toronto AI Center

Innovative Approaches Spanning Samsung Research’s Global AI Centers

As well as the two thesis papers submitted by the Toronto AI Center, other global Samsung AI centers — such as the Moscow AI Center, the Cambridge AI Center and the New York AI Center — have also attracted academic attention in anticipation of the conference.

Two papers submitted by the Moscow AI Center were accepted into the conference. The first is a study on what is currently the world’s most competitive Single-View Depth Estimation (SVDE). This study on depth estimation — a research area that concerns many forms of image manipulation, generation and analysis — has gained attention due to its high accuracy. Unlike its predecessors which require resource-intensive post-processing, the proposed GP2 (General-Purpose and Geometry-Preserving) SVDE approach demonstrates outstanding capabilities without the need for this post-processing.

The second paper, “Stereo Magnification with Multi-Layer Images”, is a study of a novel method of 3D photo synthesis. Unlike existing methods of 3D photo synthesis, which necessitate high-capacity memory and processing abilities, the method championed in this paper can be applied to mobile devices as well, thanks to a drastically increased memory efficiency which has not caused accuracy or processing effectiveness to suffer.

The Cambridge AI Center’s paper on “Gaussian Process Modeling of Approximate Inference Errors for Variational Autoencoders” achieves state-of-the-art performance by proposing a novel Gaussian Process (GP) modeling method. This enables test time inference using a single feed forward pass in Variational Autoencoder (VAE).

They also introduced the paper titled “Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Difference”. In this paper, the research team proposes a novel neural network based on innovative transformer architecture for the few-shot learning which is a representative method for dealing with situations where data has scarce labelling.

These achievements, among many others, help emphasize Samsung Research’s position in the world of AI research and in the field of computer vision. Other papers that were accepted for CVPR 2022 include works submitted by Samsung Research’s Platform Team and Samsung R&D Institute India’s Virtual Intelligence Team.

Samsung Research operates AI centers in seven different regions; Korea (Seoul), the U.S. (Silicon Valley and New York), Canada (Toronto and Montreal), the U.K. (Cambridge) and Russia (Moscow). Going forward, Samsung will continue to conduct advanced research and actively push innovation in the AI field.

¹ Samsung Research, acting as Samsung Electronics’ advanced R&D hub, leads the development of future technologies for the company’s Device eXperience (DX) Division.

Spearheading Speech Translation Research

Samsung R&D Institute Poland and Samsung R&D Institute China-Beijing competed with some of the world’s top universities and research labs to win first place in two separate challenges at the International Workshop on Spoken Language Translation (IWSLT), one of the world’s longest-running workshops on automatic language translation. This year, IWSLT joined the Association for Computational Linguistics conference (ACL), a premier conference in the field of computational linguistics, to cover a broad spectrum of research areas that are concerned with computational approaches to natural language.

For the Offline Speech Translation task, which assesses the translation of TED talks from English to German, Samsung R&D Institute Poland won first place for the second time with its own research capabilities in audio to text translation. The conferral of this award marks the fourth consecutive year that teams from Samsung R&D Institute Poland have taken first prize in IWSLT challenges, including previous years’ text translation tasks.

This year’s Offline Speech Translation task allowed participants to submit systems based on either the traditional speech translation pipeline system composed of an automatic speech recognition (ASR) and a machine translation (MT) or an End-to-End (E2E) system. Samsung R&D Institute Poland’s system is based on a single encoder-decoder deep neural network – an E2E system – capable of both English and German texts.

In computational linguistics, E2E systems are harnessed to solve the common problem of error accumulation, wherein, in a traditional pipeline, an error in the speech recognition phase can lead to a nonsensical translation. However, research from over the past three years has shown that traditional systems have constantly been outperforming E2E speech translation systems. The Samsung team’s system not only placed first in the E2E category, but also outscored all traditional pipeline system entrants, a remarkable achievement that puts Samsung R&D Institute Poland at the forefront of speech translation research.

The team from Samsung R&D Institute Poland Team who participated in this year’s IWSLT challenges

Innovative Approaches in the Field of Computational Linguistics AI

Samsung R&D Institute China-Beijing took part in a second challenge, the Open Domain Translation task evaluating Japanese to Chinese translation capability, ultimately taking first place. The main goals of this task were the promotion of research into translation between Asian language, the exploitation of noisy parallel web corpora for machine translation and the thoughtful handling of data provenance.

Samsung R&D Institute China-Beijing submitted a system based on Transformer model architecture and adopted the relative position attention. The team focused on improving the Transform baseline system with elaborate data preprocessing and managed to achieve significant improvements. The team also tried shared and exclusive word embedding and compared different granularity of tokens, approaching the process at a sub-word level, including Byte Pair Encoding (BPE) and Sentence Piece. Large-scale back translation on monolingual corpus was used to improve the Neural Machine Translation (NMT) performance.

Members of the team from Samsung R&D Institute China-Beijing Team who participated in this year’s IWSLT challenges

Achievements in AI Audio Signal Interpretation

In addition to their first-place finish in the IWSLT challenge, Samsung R&D Institute Poland was also recognized as one of the leading teams at the Detection and Classification of Acoustic Scenes and Events (DCASE) 2020 challenge, held by IEEE (Institute of Electrical and Electronics Engineers), which aims to use state-of-the-art AI technology to understand and interpret audio signals.

Engineers from Samsung R&D Institute Poland, who possess previous experience in Acoustic Scene Understanding and Sound Sources Localization tasks (having ranked first place in two tasks in 2019), set their focus on Task 2: Unsupervised Detection of Anomalous Sounds for Machine Condition Monitoring. The goal of this task was to identify whether the sound emitted from a target machine was normal or anomalous. The main challenge was detecting unknown anomalous sounds under a condition within which only normal sound samples have been provided as training data. The engineers scored second place out of 40 teams.

The team from Samsung R&D Institute Poland Team who participated in this year’s DCASE challenge

Envisaging the Future of Computer Vision and Pattern Recognition

In June, Samsung R&D Institute China-Beijing also participated in three challenges hosted by the 2020 Conference on Computer Vision and Pattern Recognition (CVPR 2020): the Embodied AI Challenge, the VizWiz-Captions Challenge and the VATEX Video Captioning Challenge. The team claimed second place in the challenges.

The Embodied AI Challenge aimed to enable robots to understand human commands and perform correct actions within a virtual environment, while the VizWiz-Captions Challenge involved predicting an accurate caption when given an image taken by a visually impaired person and the VATEX Video Captioning Challenge aimed to benchmark progress towards models that can describe videos in various languages including English and Chinese.

Members of the team from Samsung R&D Institute China-Beijing Team who participated in this year’s CVPR challenges

Samsung Electronics’ Global Research & Development (R&D) Centers have presented their studies to the CVPR (Computer Vision and Pattern Recognition) introducing new computer vision, deep learning and AI related technical researches.

CVPR is the world’s biggest conference on computer engineering and AI. At this year’s conference, held online from June 14 to 19, Samsung Research, an advanced R&D hub within Samsung Electronics’ SET Business and its advanced R&D centers gave presentations on a total of 11 thesis papers. Researchers from Samsung Moscow AI center and Samsung Toronto AI center were invited to oral presentations, an opportunity given to only 5% of the entire attendees.

At the oral presentation, Pavel Solovev of Samsung Moscow AI Center introduced ‘High Resolution Daytime Translation without Domain Labels’, which is a technology that changes a high resolution landscape photograph into scenes from various times of the day using data without domain label. Konstantin Sofiiuk also introduced ‘f-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation’, which is a technology that allows a user to simply click an object in a photograph to precisely select and separate it.

‘High Resolution Daytime Translation without Domain Labels’

Joining from the Toronto AI Center, researcher Michael Brown and his team introduced the paper titled ‘Deep White-Balance Editing’, which was also selected for an oral presentation. This AI technology corrects white-balance mistakes made in a captured photograph much more accurately than existing photo editing programs. This technology also allows users to accurately adjust the photo’s white-balance color temperature.

Deep White-Balance Editing

Researchers from Samsung Research America also presented interesting findings at the conference. Eric Luo’s study titled ‘Wavelet Synthesis Net: An Efficient Architecture for Disparity Estimation to Synthesize DSLR Calibre Bokeh Effect on Smartphones’ focused on key enablers to narrow the gap between DSLR and smartphone camera in terms of bokeh, the narrow depth of field (DoF).

Yilin Shen from Samsung Research America’s AI Center introduced a study on out-of-distribution (OoD) benchmarks for deep neural networks research. Shen’s study titled ‘Generalized ODIN: Detecting Out-Of-Distribution Image Without Learning From Out-Of-Distribution Data’ proposed the key machine learning algorithm of drastically improving the detection rate, one of major challenges in AI technology.

Additionally, the studies proposed by researchers from the Samsung Research’s Visual Technology team and Samsung R&D Institute India-Bangalore were also selected by CVPR.

Samsung Electronics last month announced its goal to strengthen its leadership in the global system semiconductor industry by 2030 through expanding its proprietary NPU technology development. The company recently delivered an update to this goal at the conference on Computer Vision and Pattern Recognition (CVPR), one of the top academic conferences in computer vision fields.

This update is the company’s development of its On-Device AI lightweight algorithm, introduced at CVPR with a paper titled “Learning to Quantize Deep Networks by Optimizing Quantization Intervals With Task Loss”. On-Device AI technologies directly compute and process data from within the device itself. Over 4 times lighter and 8 times faster than existing algorithms, Samsung’s latest algorithm solution is dramatically improved from previous solutions and has been evaluated to be key to solving potential issues for low-power, high-speed computations.

Streamlining the Deep Learning Process

Samsung Advanced Institute of Technology (SAIT) has announced that they have successfully developed On-Device AI lightweight technology that performs computations 8 times faster than the existing 32-bit deep learning data for servers. By adjusting the data into groups of under 4 bits while maintaining accurate data recognition, this method of deep learning algorithm processing is simultaneously much faster and much more energy efficient than existing solutions.

Samsung’s new On-Device AI processing technology determines the intervals of the significant data that influence overall deep learning performance through ‘learning’. This ‘Quantization¹ Interval Learning (QIL)’ retains data accuracy by re-organizing the data to be presented in bits smaller than their existing size. SAIT ran experiments that successfully demonstrated how the quantization of an in-server deep learning algorithm in 32 bit intervals provided higher accuracy than other existing solutions when computed into levels of less than 4 bits.

When the data of a deep learning computation is presented in bit groups lower than 4 bits, computations of ‘and’ and ‘or’ are allowed, on top of the simpler arithmetic calculations of addition and multiplication. This means that the computation results using the QIL process can achieve the same results as existing processes can while using 1/40 to 1/120 fewer transistors².

As this system therefore requires less hardware and less electricity, it can be mounted directly in-device at the place where the data for an image or fingerprint sensor is being obtained, ahead of transmitting the processed data on to the necessary end points.

The Future of AI Processing and Deep Learning

This technology will help develop Samsung’s system semiconductor capacity as well as strengthening one of the core technologies of the AI era – On-Device AI processing. Differing from AI services that use cloud servers, On-Device AI technologies directly compute data all from within the device itself.

On-Device AI technology can reduce the cost of cloud construction for AI operations since it operates on its own and provides quick and stable performance for use cases such as virtual reality and autonomous driving. Furthermore, On-Device AI technology can save personal biometric information used for device authentication, such as fingerprint, iris and face scans, onto mobile devices safely.

“Ultimately, in the future we will live in a world where all devices and sensor-based technologies are powered by AI,” noted Chang-Kyu Choi, Vice President and head of Computer Vision Lab of SAIT. “Samsung’s On-Device AI technologies are lower-power, higher-speed solutions for deep learning that will pave the way to this future. They are set to expand the memory, processor and sensor market, as well as other next-generation system semiconductor markets.”

A core feature of On-Device AI technology is its ability to compute large amounts of data at a high speed without consuming excessive amounts of electricity. Samsung’s first solution to this end was the Exynos 9 (9820), introduced last year, which featured a proprietary Samsung NPU inside the mobile System on Chip (SoC). This product allows mobile devices to perform AI computations independent of any external cloud server.

Many companies are turning their attention to On-Device AI technology. Samsung Electronics plans to enhance and extend its AI technology leadership by applying this algorithm not only to mobile SoC, but also to memory and sensor solutions in the near future.

Four individuals who played key roles in developing Samsung’s On-Device AI Lightweight Algorithm. From Left to right; Jae-Joon Han, Chang-Young Son, Sang-Il Jung, Chang-Kyu Choi of Samsung Advanced Institute of Technology

1 Quantization is the process of decreasing the number of bits in data by binning the given data into sections of limited number levels, which can be represented in certain bit values and are regarded as having the same value per section

² Transistors are devices that control the flow of current or voltage in a semiconductor by acting as amplifiers or switches