Opportunity
Machine vision is fundamental to advancements in autonomous vehicles, drones, and robotics. Traditional frame-based cameras, which capture entire scenes at fixed intervals, generate massive amounts of data, creating significant challenges for transmission speed and real-time processing, especially in resource-constrained environments. Event cameras, or dynamic vision sensors (DVS), offer a promising alternative by asynchronously recording only pixel-level brightness changes, resulting in high dynamic range, low latency, and sparse data output. However, efficiently processing this event data for critical tasks like corner detection on extreme-edge Internet of Things (IoT) devices remains a major challenge. Existing corner detection methods adapted for event data, such as eHarris (Harris-based) and Asynchronous Corner Detection (ACD), often struggle with balancing throughput, accuracy, and energy consumption. More recent approaches like LuvHarris improve speed and accuracy but introduce high energy costs during data structure updates, making them unsuitable for low-power, resource-constrained Application-Specific Integrated Circuits (ASICs) commonly used in edge IoT applications. Therefore, there is a pressing need for an energy-efficient, high-throughput, and accurate corner detection technique specifically optimized for deployment on extreme-edge devices.
Technology
The present invention provides an adaptive corner detection method and system specifically designed for efficient deployment on edge devices using data from dynamic vision sensors (DVS). The core innovation lies in a novel processing pipeline that transforms sparse, asynchronous event data into a structured format conducive to fast and accurate corner detection while minimizing memory usage and computational overhead. The method begins by obtaining event data, where each event contains pixel coordinates (x, y), a timestamp (t), and polarity (p). These recorded events are captured and organized into a compact 2D array. Crucially, instead of storing full timestamps, the array stores primarily spatial coordinates (x, y) and uses the array's structure itself to implicitly maintain the global temporal order of events, significantly reducing memory footprint. This 2D array can be organized in rows with a fixed number of events (e.g., 25 to 100), facilitating predictable processing. Optionally, a Spatial-Temporal Correlation Filter (STCF) can be applied beforehand to reduce noise.
The key transformation step involves converting this 2D array into one or more Ordered Surface (OS) matrices. This is done by populating an empty Image Matrix (IM) with the recorded events based on their coordinates. Each position in the OS matrix is assigned an "order value" (an index) that reflects the temporal sequence of the events, effectively restoring explicit spatial ordering while retaining relative temporal information without the burden of large timestamp storage. This creates a coherent, fixed-size representation ideal for subsequent analysis. Corner detection is then performed on the OS matrix. The method typically extracts a local patch (e.g., 7x7 to 11x11 pixels) centered on an event of interest from the OS matrix. Before applying a corner detector like the Harris detector, a computationally efficient sort normalization may be applied to the patch values, bypassing expensive division operations. The Harris score is calculated for the patch and compared against a threshold to determine if a corner is present. The entire pipeline—from the efficient 2D array structure to the OS transformation and optimized patch processing—is designed to minimize memory writes, streamline computations, and enable effective implementation on power-constrained edge processors like microcontrollers or ASICs.
Advantages
- Achieves high corner detection accuracy comparable to or better than state-of-the-art methods (e.g., LuvHarris) while being optimized for resource-constrained environments.
- Significantly reduces energy consumption and memory usage by employing a compact 2D array for event storage and an efficient Ordered Surface (OS) transformation, avoiding costly per-event timestamp updates.
- Enables high-throughput processing suitable for real-time applications by using a fixed-size data representation (OS matrix) and streamlined operations like sort normalization.
- Offers adaptability through tunable parameters, such as the row size in the 2D array (C) and the patch size (MxN) for Harris detection, allowing performance tuning for different event camera resolutions and application needs.
- The processing pipeline is well-suited for parallelization and potential acceleration using in-memory computing (IMC) techniques, further enhancing speed and efficiency.
- Maintains both spatial and crucial temporal information from event data through the order values in the OS matrix, which is essential for accurate dynamic scene analysis.
Applications
- Autonomous Vehicles and Advanced Driver-Assistance Systems (ADAS): For real-time feature detection and tracking in dynamic environments using event cameras.
- Robotics and Drones: Enabling agile navigation, obstacle avoidance, and SLAM (Simultaneous Localization and Mapping) on robots with limited onboard processing power.
- Extreme-Edge IoT and Smart Surveillance: Deploying intelligent, low-power vision capabilities in sensors for motion analysis, activity recognition, or anomaly detection.
- Wearable and Augmented/Virtual Reality (AR/VR) Devices: Providing efficient, low-latency visual feature detection for gesture recognition or environment interaction.
- Industrial Automation and Machine Vision: For high-speed inspection, part tracking, or quality control in settings where traditional cameras may struggle with lighting variations or high motion.
