Helping users say goodnight to washed-out, blurry night images was a group effort driven by Samsung’s experts. To learn more about this groundbreaking camera feature for the Galaxy S23 series, Samsung Newsroom spoke with the experts at Samsung Research’s¹ global R&D centers, who collaborated on the technology.

Read on to see what John Seok-Jun Lee, Head of Computational Camera Part at Samsung Research America (SRA), and Narasimha Gopalakrishna Pai, Head of AI Computational Imaging Part at Samsung R&D Institute India-Bangalore (SRI-B), had to say about their joint development of Nightography, the best way to take crisp photos and videos from dusk until dawn.

Like so many great innovations, the latest UDC was a group effort — driven by Samsung’s experts and consumers’ valuable feedback. To learn more about the enhanced UDC for Galaxy Z Fold4, Samsung Newsroom spoke with the experts at Samsung Research,¹ who collaborated on the technology.

Read on to see what John Seok-Jun Lee, Head of Computational Camera Part at Samsung Research America (SRA), and Alok Shukla, Head of AI Vision Solutions Part at Samsung R&D Institute India-Bangalore (SRI-B), had to say about their collaboration behind bringing enhanced UDC technology.

Q: What is an Under Display Camera (UDC)?

Seok-Jun: An Under Display Camera (UDC) is an innovative camera solution that enables an uninterrupted viewing experience on a mobile device by hiding the camera under the display and dedicating the whole screen to users while applications are running. It not only requires hardware innovation by placing a camera under a display panel but also requires algorithm innovation for restoring image quality — one of the most complex image restoration problems.

Q: How does the UDC work?

Alok: In UDC, a front-facing camera sits underneath the display. The screen pixel density is reduced in this area, allowing the screen to display imagery unhindered — but at the same time, not block light from reaching the camera when it’s in use. As the camera is placed underneath the display, UDC can suffer from poor image quality caused by diffraction artifacts, which results in flare, saturation, blur and haze. Therefore, while the UDC brings a better display experience, it also affects camera image quality and other downstream vision tasks. These complex and diverse distortions make restoring UDC images extremely challenging.

Q: Why is the UDC an important technology when it comes to Galaxy Z Fold4? What value does it provide to customers?

Seok-Jun: Samsung’s foldables are designed to be new and different, combining smartphone and immersive viewing experiences into one mobile device via breakthrough display technology. When folded, it provides smartphone-like use, and when unfolded, the display provides a more immersive and productive experience. Our consumer research showed that Galaxy Z Fold series users prefer the Cover Screen camera for capturing photos and selfies and the immersive Main Display for more productive tasks like watching videos, playing games and working on documents. Therefore, we thought UDC technology would be the perfect fit to meet their needs for an uninterrupted screen while still being capable of tasks like opening video calls. Samsung has launched the first-ever foldable UDC solution in Galaxy Z Fold3 and enhanced its quality further in the Galaxy Z Fold4.

Alok: Galaxy Z Fold4 truly unfolds a world of immersive display. UDC technology is at the core of the pursuit of a true full-screen display. By combining the leading innovation in display technology with advanced AI techniques for image restoration, Galaxy Z Fold4 provides the perfect balance between screen quality and camera image quality.

Q: How has the camera’s image quality improved since Galaxy Z Fold3?

Alok: Samsung Galaxy Z Fold3 showcased the world’s first UDC on a foldable. After the market release of Galaxy Z Fold3, one of the key feedbacks from consumers was to make the UDC less visible. On the Galaxy Z Fold4, sub-pixel arrangement was optimized to further hide the UDC.

As mentioned previously, when the camera is placed underneath the display, this can result in poor image quality caused by diffraction artifacts, which cause flare, saturation, blur and haze.

That’s why, in Galaxy Z Fold4, we have leveraged the power of AI, backed with multi-frame processing, to develop a novel computational imaging pipeline for UDC image restoration.

SRI-B, along with Visual SW R&D Group at MX Business, focused on developing AI Image Restoration for photo capture and preview to compensate for optical diffraction in UDC. At the same time, SRA’s Mobile Processor Innovation (MPI) Lab improved the multi-frame computational photography algorithm. Galaxy Z Fold4’s UDC solves key technical challenges in image quality to bring a unique experience to end customers.

Q: Walk us through the journey of the SRA-SRI partnership project. Why did your two teams collaborate?

Seok-Jun: The SRA MPI Lab has been working with SRI-B throughout MPI’s entire history at Samsung, from 2014’s Galaxy Note4 camera commercialization all the way to 2022’s Galaxy Z Fold4 and Z Flip4. SRA MPI has been leading MX multi-frame processing (MFP) technology R&D for UDC, Night Mode, Expert RAW and other technology. SRI-B has contributed to all commercialized SRA MPI’s MFP by providing amazing camera software, image quality tuning and resources toward commercialization as well as AI camera R&D. Our 3-way collaboration with MX R&D group, SRA MPI and SRI-B has been a success — especially when the worldwide team used time zone differences effectively to complete milestones on time for such a complicated project. It’s always a pleasure to work with them and deliver Galaxy devices’ camera successes together to accomplish our mutual goal of being the world’s best smartphone camera experience.

Q: Please explain your job role and your team’s contributions to the UDC. How did you contribute to making the best possible user experience?

Alok: Samsung Galaxy Z Fold4 UDC was a complex R&D project, and SRI-B’s role was to think in terms of end-to-end for the best consumer experience. This required multi-domain expertise and deep collaboration among multiple teams. Experts from SRI-B, SRA-MPI Lab and MX Visual SW R&D Group worked seamlessly to realize an end-to-end UDC experience.

SRI-B also played a key role in utilizing a Neural Processing Unit (NPU) to execute the AI model, achieving real-time performance in previews and significantly faster capture processing time than in Samsung Galaxy Z Fold3 UDC. Camera previews, video recordings and video calling scenarios need real-time performance without losing any frames. AI-based UDC restoration accelerated with NPU for preview processing was applied for the first time in Samsung Galaxy Z Fold4. Although real-time processing was not required, as camera capture requires higher image quality than a preview, achieving better image quality with less processing time was desired for user experience.

Q: What were some challenges you faced while developing this technology, and how did you overcome them?

Seok-Jun: Low light transmittance and light diffraction are the main challenges for UDC camera technology. Still, one more challenge, which is different from a punch hole camera image pipeline, is that UDC capture quality is affected by not only the camera module but also the display panel. SRA MPI’s AI-based 3D ISP algorithm was designed with the needed flexibility to tune the output image quality with the different display characteristics. Having such flexibility in 3D ISP helped to overcome such challenges.

Alok: SRI-B’s key focus was developing an AI image restoration module for Samsung Galaxy Z Fold4 UDC. These restorations are required for photo capture mode as well as video mode. Video recording and video calling scenarios have strict requirements for real-time processing within the constraints of current consumption.

AI-based UDC restoration was applied for the first time in Samsung Galaxy Z Fold4 UDC for video. To achieve a better video quality experience than Galaxy Z Fold3, we had to make the AI model lighter while still achieving good restoration quality. Multiple experiments were conducted in network design and training to achieve good quality in real-time. SRI-B extensively utilized NPU and advanced quantization techniques to prepare a lighter, highly optimized AI model that achieves real-time performance with low current consumption.

Q: What is your vision for future UDC iterations at Samsung?

Alok: Samsung Galaxy Z Fold4’s UDC has opened up a new dimension to the smartphone camera form factor. UDCs in the future will enable placing larger camera sensors and multiple cameras at flexible positions without compromising the viewing experience. Another big challenge is towards mass adoption of UDC technology in more products as the main front camera. For this to happen, there will need to be a significant improvement in camera image quality, processing time and display experience.

Seok-Jun: Since the Galaxy Z Fold4 UDC provides a much better experience in terms of hiding the camera under the display and improving image quality, the R&D direction for the upcoming flagship devices will work towards making the UDC completely invisible. This will provide the best uninterrupted immersive mobile display experience while achieving the same image quality as a punch-hole camera.

¹ Samsung Research, acting as Samsung Electronics’ advanced R&D hub, leads the development of future technologies for the company’s Device eXperience (DX) Division.

Like so many great inventions, Expert RAW was a group effort — driven by Samsung’s experts and Galaxy users’ valuable feedback. To learn more about the Expert RAW experience, Samsung Newsroom spoke with the experts at Samsung Research² who collaborated on the app.

Read on to see what Hamid Sheikh, Head of the Intelligent Imaging Team at Samsung Research America’s (SRA) Mobile Processor Innovation (MPI) Lab, and Girish Kulkarni, Head of the Camera Solutions Group at the Samsung R&D Institute India-Bangalore (SRI-B), had to say about this innovative camera collaboration.

Q: What does the Expert RAW app do? What are the main benefits for users?

Hamid: Expert RAW really gives users the ability to take full control of their mobile photography experience much like a professional photographer would. The app captures much more comprehensive data for each image, which can be used to both improve photos and artistically enhance them with editing software like Adobe Lightroom — the go-to app for professional photographers.

To improve the overall camera experience, we worked closely with Adobe to tightly couple Expert RAW with Lightroom. The result is that photographers have full control over their creative experience. Novice photographers can also utilize automatic set points for camera controls, and then use the Samsung-specific default values in Lightroom. In both cases, the tight integration of the two apps makes for an enjoyable experience.

Girish: Expert RAW is an epic leap forward for photography. It essentially places a professional studio in your pocket. Whether you are a pro photographer or you’re just a user who wants to capture amazing content, anyone can take full advantage of the Galaxy S22’s pro-grade camera with Expert RAW. It captures high-quality, high dynamic range pictures in the multi-frame RAW format that enables precise post-capture editing and professional results.

With Expert RAW, people can capture much more information in a single picture — from dark details to bright highlights. It also drastically reduces noise and increases sharpness and detail. If you’re looking for the ultimate creative controls, you can choose your own settings for aspects like ISO, shutter speed, white balance, EV and focus, and even choose which lenses you want to use for the subject and the scene. Finally, because both JPEG and DNG files are saved when shooting, you can hop right into the Adobe Lightroom app on your smartphone or PC and edit with ease.

Q: What was the original inspiration for Expert RAW?

Hamid: Samsung camera technologies have always relied on sophisticated computational photography algorithms to produce the best picture quality. However, the standard camera app makes many decisions about how the image will be rendered in terms of color, tone, detail and noise. The user has very limited say in these decisions.

Our inspiration came from the confluence of user experience limitations and the leaps in computational and AI capabilities of today’s mobile processors. We asked ourselves, “How can we make the photography experience more inspiring for users?” So really, our inspiration came from the expert and enthusiast photography community, which yearned for an easier mobile photography experience, without sacrificing the creative control that DSLR and post-capture editing allows. Expert RAW was born out of a desire to transform that inspiration into reality.

Girish: As we heard the input of professional photographers who were mostly utilizing DSLR cameras, they wanted to achieve a pro-grade experience with smartphones. Our consumer research also showed that after using their smartphone’s auto camera for a long time, typical users were very passionate about exploring Samsung’s pro-grade camera capabilities in-depth to become better photographers.

The more we talked to professional and amateur photographers, the more we understood their needs in terms of RAW image capturing, camera controls, the post-editing experience, creative capabilities and more. That’s when we set out to create a new pro-grade experience based around RAW capturing so that every level of photographer could merge our camera controls and their creativity to enjoy professional-looking photos.

Q: What makes Expert RAW such an important release for mobile photography in general?

Hamid: The camera experience is one of the most important experiences that a user has with his or her smartphone. Samsung has been at the cutting edge of camera quality, and the gap between consumers and professional/enthusiast photographers has always been a tempting one to bridge.

I think with the release of Expert RAW, users will find that their photography experience will take a quantum leap in terms of creative control and artistic expression. I expect more and more people will find this new experience rewarding because it lets them personalize their camera based on their tastes — exactly how they want them — and not based on what some other engineers chose for them.

Girish: As an app, this platform has simplified the RAW capturing process with precise camera and lens controls and post-editing with Adobe Lightroom. It’s for every user who is willing to go beyond — to unleash the true power of an Expert RAW camera, and explore their creative side.

Q: What role did the MPI Lab play in Expert RAW’s development?

Hamid: Any time we are creating a brand-new user experience, we have to consider and overcome many tough engineering challenges. The MPI Lab developed the main idea behind Expert RAW and its technologies, and we knew it was challenge worth pursuing.

We developed a brand-new computational photography pipeline that can produce both DNG as well as processed JPEG files. Developing a complex computational photography pipeline from scratch was not easy. We no longer had the benefit of an older, mature technology base to build upon. But our team successfully developed this new, AI-based 3DISP technology to produce raw data that looks very natural when users load it in editing apps like Adobe Lightroom.

Q: What role did SRI-B play in Expert RAW’s development? What were the team’s main areas of focus?

Girish: Expert RAW was a complex R&D project, and SRI-B’s role was to think in terms of end-to-end for the best consumer experience. This required multi-domain expertise and deep collaboration among multiple teams.

SRI-B is equipped with strong multi-domain expertise teams, so we formed a task force of talented experts. They all worked together as a single team within SRI-B and also worked with a larger team of experts based in Korea and at the MPI Lab.

The multi-domain expertise and close collaboration with all key stakeholders helped SRI-B continuously focus on improving Expert RAW’s product experience, end-to-end pipeline, etc.

Q: Could you offer more details on your collaboration with one another?

Hamid: The MPI Lab has been working with SRI-B throughout our entire history at Samsung. SRI-B contributed its amazing skillset and resources towards the development of the software technologies, as well as key AI blocks for the Expert RAW pipeline. They also owned the UI piece of the Expert RAW application and provided many innovation points on the Expert RAW experience. It’s always fun working with them.

Girish: The MPI Lab has world-class expertise in computational imaging domain, and has a proven record of developing successful IPs for Samsung flagship products. That expertise was one of the key requirements for Expert RAW’s development. The MPI Lab came up with the multi-frame pipeline building blocks implementation, which helped SRI-B integrate and enhance the same for the end-to-end architecture and pipeline.

One of the major expectations for Expert RAW was that it would enable the best image quality in different lighting conditions, which requires the tuning of the multi-frame processing pipeline. MPI’s strong expertise in this area helped us continuously improve image quality before the commercial launch. In summary, MPI’s expertise in computational imaging and SRI-B’s breadth of knowledge in multiple subdomains helped SRI-B prepare Expert RAW’s end-to-end experience for commercial launch.

SRI-B and the MPI Lab used their 12-hour time zone difference effectively to plan and collaborate on Expert RAW. SRI-B’s day was MPI’s night, and vice versa. It helped to have 24/7 development to finish this complex feature in record time. We are proud of Samsung’s culture that enables respectful collaboration among different overseas R&D centers and we always enjoy working with MPI Lab.

Q: How do the MPI Lab and SRI-B plan to build on this collaboration to develop even more innovative camera technologies?

Hamid: Our journey with the new RAW camera experience is just beginning. I am excited to think about what new improvements and enhancements we can make. One of the best parts of the Expert RAW experience is that the photographer is front and center. We’ll preserve that as we actively look into the experiences of different users and continuously implement improvements to make Expert RAW even better than before.

Girish: The MPI Lab and SRI-B have been working together for the last few years developing the world’s best night mode for Samsung flagship smartphones. This collaboration helped lead us to the success story that is Expert RAW on the Galaxy S22. We’re excited to continue that momentum for the next innovative products to come.

In the future, we would like to continuously enhance Expert RAW with a focus on creating a new ecosystem for pro photography that fully utilizes pro-grade camera capabilities.

Collaborating for Creativity 

Features like Expert RAW underline Samsung’s commitment to developing tools that empower users to unleash their creativity.

Going forward, the company will continue to strengthen collaborations that leverage the knowledge and capabilities of its world-class network of R&D centers. Doing so will help unlock technologies that change the way we live, work and create.

¹ Expert RAW must be downloaded separately from the Galaxy Store, for free, before use. Expert RAW app availability is currently limited to the Galaxy S21 Ultra and S22 series. Samsung Members notified Expert RAW will become available on Galaxy Z Fold3 in April 2022 and Galaxy S20 Ultra, Galaxy Note20 Ultra and Galaxy Z Fold2 later this year.

² Samsung Electronics’ advanced R&D hub, which leads the development of future technologies for its Device eXperience division

“AI and machine learning are major strategic imperatives for Samsung. There’s no doubt that, in the coming years, these technologies will completely change the way people interact with every device in their lives, from phones and wearables to home appliances and cars,” said Young Sohn, President and Chief Strategy Officer at Samsung Electronics, and Chairman of the Board at HARMAN. “For this to happen, we believe AI must be open—so we can harness the talent and expertise of a vast ecosystem of companies, and ultimately give consumers smarter, simpler, more meaningful interactions with the devices they love. This Summit underscores our commitment to an open, collaborative AI ecosystem, and serves as a call to action for all researchers and entrepreneurs who share our vision for a more connected world.”

Samsung’s AI Summit opened with a keynote by the company’s Senior Vice President of AI Research, Dr. Larry Heck, a world-renowned leader in AI and voice recognition who has led AI teams at Microsoft, Google and Nuance. During his talk, Dr. Heck outlined the need for increased industry collaboration to drive broader adoption and consumer confidence in AI technology. He also advocated for the importance of making AI open to 3^rd parties.

“This is an incredible time in our industry. Artificial Intelligence and machine learning are becoming mainstream, used by millions each year. Yet, we need more collaboration in the industry—from companies to academics and researchers—to let AI flourish,” said Dr. Heck. “It’s time for us to find new ways to work together to ensure AI serves the needs of people first and foremost.” As a part of this collaborative effort, Samsung plans to host many more AI-related events around the globe this year.

Dr. Heck also shared Samsung’s plans to tap its massive device ecosystem to learn user preferences and needs. “Under a common architecture, Samsung’s AI platform will not only scale quickly, but will also provide the deepest understanding of usage context and behaviors, making AI more relevant and useful,” he said. Attendees at the event also heard from experts at Google, MIT, Carnegie Mellon and the University of California at Berkeley.

In November 2017, Samsung announced the creation of a new AI Research Center dedicated to applied research and development in the space. Samsung’s new AI research center plans to open additional AI research labs in Canada, the UK and Russia in 2018 to strengthen the company’s AI research efforts.

Dr. Larry Heck (pictured second from the left), Senior Vice President of AI Research at SRA, Dick Lyon (pictured second from the right), a Director at Google Research, and Ian Lane (pictured on the far right), a Assistant Professor at Carnegie Mellon University, discuss open collaboration in the AI industry at the Artificial Intelligence (AI) Summit on January 16, hosted by Samsung Research America at Mountain View.