Nearly every smartphone has a camera today. It is one of the very important parts of a phone. A lot of phone users, myself included cannot do without a one on a phone. Taking photos today when compared to twenty years ago seems so simple today. You basically take out your phone and point and shoot. When you want to take a photo, you open the app, aim it at the subject and click on the shutter button. In a few seconds, the photo appears on your screen and you can choose to either keep it or get rid of it and take another one.
Within those few seconds between taking the photo and the subsequent picture appearing on your screen, there is a lot of computing and processing that takes place and these are all handled by the part of the SoC known as the ISP (Image Signal Processor).
An ISP also known as an Image Processor (IP), Digital Signal Processor (DSP) or Image Processing Unit (IPU) is a type of specialized microprocessor used for image processing.
The ISP helps to transform the light image captured by the pixels on the camera sensor into a RAW image format and then processes the photo further into the compressed final jpg format that you see.
The ISP has a series of computation processes which are referred to collectively as a Pipeline. When a photo is taken, light bouncing off the subject(s) enter into the camera and are captured by the pixels on the image sensor. The pixels then convert the light energy into electrical energy and then sends the data to the ISP. The ISP then carries out demosaicing on the photo and assigns each pixel a colour value after which it gives out an uncompressed RAW image.
Significant processing is required to convert a RAW image into the final jpg product. Different companies use different software to try to achieve this but the process remains the same. They include:
i. Denoising (noise reduction)
ii. Auto white balance and color correction
iii. Lens shading (improves brightness and color non-uniformity)
iv. Pixel correction (corrects defective pixel output)
v. Tone mapping (for increased contrast) and HDR
vi. Focus correction and image sharpening
vii. Auto exposure adjustment
As said earlier, different companies approach photo processing differently. Some may choose to go with warmer colours and a high level of exposure and saturation, others could go with cooler colours and a moderate level of exposure as well as saturation. Some could go for a really sharp image or a dull smooth one.
TYPES OF ISP
There are a lot of ISP products in the market but for this post we’re only going to be looking at the ISP used in smartphones by SoC manufacturers as well as OEMs (phone manufacturers).
• Kirin ISP: manufactured by HiSilicon and used in Kirin SoCs exclusively for Huawei phones. It is very capable and in combination with the cameras used on such phones have made Huawei a photography giant in the smartphone market.
• Qualcomm Spectra (and Hexagon DSP): Qualcomm spectra is the ISP that ships with Snapdragon chipsets. To boost its performance, an extra module for AI and machine learning called Hexagon DSP is added as well making them very capable, fast and able to handle cameras of very high resolutions.
• Exynos Dual ISP: is made by Samsung for their Exynos chipsets which are used on Exynos variants of the flagships as well as other Samsung phones shipping with as Exynos SoC.
• Google Visual Core: Google initially used Qualcomm’s ISP for their phones but with the release of Pixel 2, Google designed the Pixel Visual core as the main ISP for the device and continue to use it till it was replaced by the upgraded Pixel Neural core used in the Google Pixel 4. Both versions of Google’s ISP are very good and combined with Google’s Computational Photography have made Pixel phones one of the best in terms photography when smartphones are being discussed.
• Mediatek Imagiq: is Mediatek’s response in the camera arms race among SoC and Smartphone makers. It also offers support for multiple cameras and good pixel correction as well as zoomed in images with little distortion.
• Sony Bionz X: BIONZ X for mobile is an image-processing engine based on Sony’s digital imaging technologies used in Alpha and Cyber-shot cameras. BIONZ X includes RAW noise reduction, a technology which offers a better than ever low light shooting performance. It reduces the noise of the image before the compression, so that the final picture is smoother and clearer.
• HTC Image sense: used on HTC devices and helps with image processing and fine tuning.
The grade of the SoC used (i.e. Flagship, mid-range or budget etc.) dictates the level of the ISP that comes with it and that translates into the resolution of camera that it can support. A low end SoC can only support poor resolution camera sensors because it may lack the processing power needed to handle high resolution photos. A high end processor should come with an excellent ISP and should have zero problems processing neat photos. So low end SoC = weak ISP = low resolution cameras = average or poor photos. Mid-range SoCs should equal = decent ISP = decent resolution cameras = good photos. Flagship SoCs…well I don’t need to explain that do I?