[All About Exynos] ① Meet the GPU · ISP Development Leaders

Exynos: The Technology Intensive System Semiconductor

In the first installment of this series, Samsung Newsroom shares the importance of the graphics processing unit (GPU) and image signal processor (ISP) in the Exynos mobile processor (SoC). Before meeting with the seven IP development leaders, Samsung Newsroom first met with the SoC Development Head and Senior Vice President Mingoo Kim, who oversees system-on-chip (SoC) design in the System LSI Business.

Kim first explained the fundamental reason behind the concept of SoC, in which various features are all incorporated into a single chip. “When chips are divided, it becomes difficult to manage their power consumption completely,” Kim said. “When each feature consumes power separately, the battery efficiency also decreases. In addition, the bandwidth limit and transfer time latency will affect data transmission between chips, which can lead to reduction in performance.” Consuming minimal power is very important for smartphones because, unlike desktop computers, smartphones are not continuously supplied with power.

“SoC has high efficiency due to comprehensive power control. And as a single chip, it also takes up less space within a smartphone,” Kim explained. “When all functions are performed by a single chip, performance is significantly improved.” Modern mobile phones go beyond simply sending and receiving calls and texts and have evolved to perform a multitude of advanced functions, including taking videos, mobile gaming and accessing financial services. The SoC, which is smaller than a thumb nail, has played a big role in making this possible.

As the SoC is the epitome of all main types of information technology that exist today, Kim described it as the “flower of system semiconductors.” “SoC is not an easy field to work in, but it’s an optimistic field that any engineer would aspire to become a part of,” Kim said. “The role of SoC will be limitless in future industries, such as the metaverse, autonomous driving and 6G.”

Samsung will continue to focus on developing proprietary IP, including GPUs, NPUs, ISPs, modems, RF and more. Samsung strives to take another step forward in chip design to become a platform solution company. “With our competitiveness in SoC, we aim to see Exynos recognized as the best mobile processor brand available,” Kim said. “Through this feature series, we hope the role and importance of SoC, the characteristics and advantages of Exynos as well as its developmental direction will be shared with more people.”

Expanding the Possibilities of Mobile Gaming: GPU for More Enhanced Graphics

In general, graphics processing requires large-scale computation, and it is faster and more efficient if processed in parallel (parallel processing). However, the structure of a central processing unit (CPU) is specialized for fast, serial processing. As such, graphics processing with a CPU would cause important computations to be delayed due to this constant computational performance by CPU. One example of this can be seen when playing a game. It’s possible that your character may not be able to avoid an enemy due to the touch input being delayed while graphics are being rendered on the display.

To address these issues, GPU was developed. Before there were GPUs, CPUs took care of everything. However, GPU was conceived to increase efficiency as a separate accelerator for similar computations that are frequently used. In short, CPU is like a general-purpose calculator while GPU is a large-scale parallel calculator specializing in graphics processing. This is how GPU came to be, and it is one of the key components in graphics processing. It receives commands from CPU and displays an object’s shape, location, color and texture on the monitor.

The Xclipse 920 equipped in the Exynos 2200 is the first mobile GPU Samsung jointly developed with AMD, a U.S. based global company that specializes in semiconductor products like GPU for PC and consoles. The name ‘Xclipse’ is a combination of the ‘X’ from Exynos and the word ‘eclipse.’ The name represents Samsung’s goal to go beyond the limits of mobile gaming and carry out performance on the level of console games, ushering in a new era of gaming.

▲Vice President Sungboem Park, a mobile processor design expert who oversees GPU development

In line with this, Samsung collaborated with AMD to develop a power efficient, console-level GPU for mobile devices. AMD’s GPUs are designed specifically for PCs or consoles, meaning they had to be redesigned to fit the mobile environment. Specifically, it was redesigned to accommodate the memory bandwidth of mobile, which is relatively limited, in addition to managing heat dissipation. “Based on our rich knowledge of low-power design, acquired from developing mobile SoCs, we were able to successfully achieve power efficiency and miniaturization in the first-generation product,” said Vice President Sungboem Park, a mobile processor design expert who oversees GPU development. “We focused heavily on minimizing heat, because mobile devices do not have fans like gaming consoles, while also maintaining performance so that the frames do not lag.”

The main role of the GPU in Exynos is to display objects in a 3D virtual space on a 2D smartphone screen, a role that is especially important when playing graphically demanding games on mobile devices. In particular, the Xclipse 920 is the first mobile GPU to support hardware-accelerated ray tracing (RT). Ray tracing is a technology which produces realistic lighting effects by simulating light rays reflected from 3D objects. By being accelerated by hardware rather than software, Xclipse 920 is able to perform real-time computations faster. In addition, variable rate shading (VRS) technology adjusts the amount of GPU computation depending on changes in color, shade, movement and other variables of the objects on screen to reduce GPU overload.

▲The Xclipse 920 has its own hardware support for ray tracing (RT) technology, which produces realistic lighting effects by simulating light rays reflected from 3D objects.

As the number of game users grows and graphics become more detailed, the development direction of GPU is becoming increasingly more important. This includes increasing the performance level to that of high-performance consoles while lowering battery consumption. By improving these two crucial areas, users are able to experience high fidelity graphics on mobile devices similarly to how they are experienced on consoles. “In general, mobile tends to lag around five years or so behind consoles when it comes to graphics technology, however, we were able to incorporate the latest console technologies in the Exynos 2200 mobile processor quickly through our collaboration with AMD. The SoC is applied to the Galaxy S22,” Park said. “We plan to continue to implement other features in the RDNA series by working closely with AMD going forward.”

“As the performance of smartphones has become enhanced overall, it’s true that it’s not easy to create a performance edge that is significant enough for consumers to notice,” Park answered when asked about the future direction of mobile GPUs. “From now on, the reason for flagship smartphone users to purchase the latest smartphones will most likely be dependent on gaming performance,” he said. This means that the possibility of the further advancement of GPUs, which determines a smartphone’s gaming performance, is high.

He also predicted that mobile GPUs will become even more important in the fields of AR and VR as well. “For AR, GPUs will need to be equipped in lightweight devices, such as glasses, so a low-power design is very important,” Park said. “Performance requirements are much more demanding in the field of VR, as the entire visible virtual world needs to be rendered instantly. As such, we need to satisfy various development requirements while also generating more realistic images at a much faster rate than we currently can. So, the advancement of mobile GPU is crucial going ahead and will have limitless applications,” he emphasized.

Elevating the Level of Satisfaction in Camera Performance: ISP for More Natural and Vivid Images

An ISP corrects raw data transferred from an image sensor and creates a photo or video to a form preferred by the user. ISP also corrects the potential physical limitations of a camera module — comprised of an optical system and an image sensor — interpolates red, green and blue (R/G/B) and removes noise. In addition, it carries out post processing, such as adjusting the brightness of a video and emphasizing detailed areas. Simply put, through fine tuning and post processing, ISP generates the picture or video that is most desirable by the users.

▲ Whole process of ISP in Mobile Camera

In earlier smartphone models, ISP was equipped as a separate chip. However, due to market demand, embedded ISP soon became the norm. “At first, we cooperated with research institutes overseas to develop a high-performing ISP that can be used for digital cameras,” said Jongseong Choi, a project leader (PL) in the Multimedia Development Team who has been working in the video processing field for more than 20 years. “As a result, our first embedded ISP solution was utilized in the Galaxy S4’s main camera.” Since 2012, the Exynos mobile processor has been contributing to a substantial advancement in smartphone cameras’ video image quality through the internalization of ISP, enabling performance at the level of digital single-lens reflex (DSLR) cameras.

▲Jongseong Choi, Project Leader of the Multimedia Development Team, has been working in the video processing field for more than 20 years.

With a high-performing ISP, consumers are able to enjoy higher video image quality as well as faster processing speeds. “It’s difficult to explain in specific numbers how great a photo is because photography is subjective, but we are conducting various research studies such as video evaluation using deep learning and ISP tuning technique to create natural and sharp photos and videos,” Choi explained. In addition, ISP also determines the speed of the burst rate in continuous shooting and is responsible for the fast-processing of high-resolution photos and videos.

The high-performing ISP equipped in the latest Exynos can process up to 200 million pixels. Up to seven image sensors are supported, and videos and images from up to four image sensors can be processed simultaneously. Also included is the ability to apply different parameters per each element, such as sky, bushes, skin and so on by incorporating semantic segmentation technology that recognizes the scenes being captured with the help of NPU. The AI feature detects and marks a person’s face and adjusts the brightness, focus and color of the video based on the face’s coordinates.

▲ Example of contents aware image processing using semantic segmentation

When asked about the future direction of ISP, Choi answered that development will be focused on maintaining low-power usage and improving video image quality. “The data transferred from the image sensor is processed in real-time by ISP, but the amount of data is increasing exponentially,” Choi said. “So, a significant amount of power is consumed in writing the data into memory and then in retrieving it later. ISP in Exynos, however, is designed to save data in memory only once, thereby minimizing power consumption,” he emphasized.

“The era of videos has arrived faster than expected and, because of this, we’re focusing on improving the video image quality,” Choi said. “In particular, we’re making efforts to increase our competitiveness by improving the video image quality, even those taken in dark, low-light environments.”

* All images shown are provided for illustrative purposes only and may not be an exact representation of the product or images captured with the product. All images are digitally edited, modified or enhanced.