What the ISP does
Image sensors don’t produce pictures — they produce raw Bayer data: one color sample per pixel, offset by a black level, speckled with defective pixels, and shaded darker toward the lens edges. An image signal processor (ISP) is the pipeline that turns that raw stream into video the rest of the system can actually use. Running it in the FPGA keeps the whole chain deterministic and low-latency, on the same device that already handles capture — with no dependency on a specific SoC’s built-in ISP or its driver stack.
The processing pipeline
- Video input. Raw video enters over AXI4-Stream from whatever front end the system uses — a MIPI CSI-2 receiver, a GMSL2/FPD-Link deserializer, a parallel or LVDS sensor interface, or our MIPI Aggregator IP core — single sensor or aggregated multi-camera streams.
- Sensor corrections. Black level subtraction, defective pixel correction, and lens shading correction clean up the raw data before any color processing — fixing the artifacts every real sensor and lens introduces.
- Debayer (demosaic). Full RGB is reconstructed at every pixel from the Bayer mosaic, with edge-aware interpolation to avoid the zipper and false-color artifacts of naive approaches.
- White balance. Per-channel gains neutralize the scene illuminant, with statistics collection to support automatic white balance control loops in software.
- Color correction and gamma. A color correction matrix maps the sensor’s color response to the target color space, and gamma encoding matches the output to displays and downstream consumers.
- Statistics for tuning. Per-stage histograms and counters exposed over registers, so image quality tuning and bring-up are driven by data.
- Video output. Processed video leaves over AXI4-Stream, feeding an on-FPGA vision pipeline or encoder directly — or a MIPI CSI-2 transmitter, display path, or SoC capture port, depending on where the stream goes next.
Integration and delivery
- Standard AXI interfaces. Video moves over AXI4-Stream, and every stage is configured and monitored through an AXI4-Lite register interface — so the core drops into existing Vivado/Platform Designer block designs and is controlled from software like any other memory-mapped peripheral.
- Bare-metal, Zephyr, or Linux (V4L2). The control software runs on whatever stack your system uses: a bare-metal or RTOS application driving the registers directly, a Zephyr-based firmware, or Linux with the ISP exposed through the V4L2 media framework so standard capture pipelines (GStreamer, OpenCV, libcamera-style applications) work unmodified.
- Modular pipeline. Stages are sized and selected against your sensor, resolution, and frame rate — you don’t pay in logic for processing you don’t need.
- Vendor-portable RTL. Targeted at AMD/Xilinx devices and portable to Intel/Altera, Lattice, Microchip, and Efinix families.
- Sensor experience. We work with image sensors from Sony, OmniVision, Aptina, and onsemi — bring-up and tuning follow the register map, mode tables, and Bayer specifics of the part you’re actually using.
- Delivered as a working system. RTL, verification environment, integration documentation, and engineering support — including sensor bring-up and image quality tuning. Our image and video pipeline services cover the rest of the chain, from capture to edge AI inference on the processed streams.
Applications
- Embedded and machine vision. Clean, repeatable color video for inspection, measurement, and classification pipelines.
- Automotive and robotics cameras. Deterministic raw-to-color processing for perception stacks, viewing systems, and driver monitoring.
- Custom camera products. Camera modules where the FPGA is the camera — ISP included, no external processor required, whatever the sensor and output interface.
- Multi-camera systems. Combined with the MIPI Aggregator, one ISP pipeline serving several aggregated sensors.
Frequently asked questions
What does an ISP do, and why put it in an FPGA?
An image signal processor turns the raw Bayer data a sensor actually produces into usable color video: correcting sensor artifacts, reconstructing full color at every pixel, and balancing color for the scene. Running it in an FPGA gives deterministic, low-latency processing at the full sensor rate, on the same device that already handles capture — without depending on a specific SoC’s hardwired ISP or its driver stack.
Which processing stages are included?
The pipeline covers the core raw-to-color chain: black level subtraction, defective pixel correction, lens shading correction, debayer (demosaic), white balance, color correction matrix, and gamma, with per-stage statistics for tuning and bring-up. Stages are configurable at run time over AXI4-Lite, and the pipeline is sized to your sensor and application during evaluation.
What software stack controls the ISP?
Bare-metal, Zephyr, or Linux. The core is a memory-mapped AXI4-Lite peripheral, so a bare-metal or RTOS application can drive it directly, and Zephyr firmware integrates the same way. On Linux, the ISP is exposed through the V4L2 media framework, so standard camera pipelines — GStreamer, OpenCV, libcamera-style applications — capture from it like any other video device.
Does the ISP work with the MIPI Aggregator and GMSL2 cameras?
Yes. The ISP consumes and produces standard streaming video over AXI4-Stream, so it drops in behind a MIPI CSI-2 receiver, a GMSL2 or FPD-Link deserializer, a parallel or LVDS sensor interface, or our MIPI Aggregator IP core — processing aggregated multi-camera streams as well as a single sensor.
Building a camera product? Tell us your sensor, target device, and image quality requirements — we’ll come back with the product brief and a concrete assessment of fit.