Jonas Kühne

Last Name

Kühne

First Name

Jonas

ORCID

0000-0003-4632-8856

Organisational unit

01225 - D-ITET Zentr. f. projektbasiertes Lernen / D-ITET Center for Project-Based Learning

Show all metadata

Search Results

Publications 1 - 10 of 10

Parallelizing Optical Flow Estimation on an Ultra-Low Power RISC-V Cluster for Nano-UAV Navigation
Kühne, Jonas; Magno, Michele; Benini, Luca (2022)
2022 IEEE International Symposium on Circuits and Systems (ISCAS)
Optical flow estimation is crucial for autonomous navigation and localization of unmanned aerial vehicles (UAV). On micro and nano UAVs, real-time calculation of the optical flow is run on low power and resource-constrained microcontroller units (MCUs). Thus, lightweight algorithms for optical flow have been proposed targeting real-time execution on traditional single-core MCUs. This paper introduces an efficient parallelization strategy for optical flow computation targeting new-generation multicore low power RISC-V based microcontroller units. Our approach enables higher frame rates at lower clock speeds. It has been implemented and evaluated on the eight-core cluster of a commercial octa-core MCU (GAP8) reaching a parallelization speedup factor of 7.21 allowing for a frame rate of 500 frames per second when running on a 50MHz clock frequency. The proposed parallel algorithm significantly boosts the camera frame rate on micro unmanned aerial vehicles, which enables higher flight speeds: the maximum flight speed can be doubled, while using less than a third of the clock frequency of previous singlecore implementations.
CR3DT: Camera-RADAR Fusion for 3D Detection and Tracking
Baumann, Nicolas; Baumgartner, Michael; Ghignone, Edoardo; et al. (2024)
2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
To enable self-driving vehicles accurate detection and tracking of surrounding objects is essential. While Light Detection and Ranging (LiDAR) sensors have set the benchmark for high-performance systems, the appeal of camera-only solutions lies in their cost-effectiveness. Notably, despite the prevalent use of Radio Detection and Ranging (RADAR) sensors in automotive systems, their potential in 3D detection and tracking has been largely disregarded due to data sparsity and measurement noise. As a recent development, the combination of RADARs and cameras is emerging as a promising solution. This paper presents Camera-RADAR 3D Detection and Tracking (CR3DT), a camera-RADAR fusion model for 3D object detection, and Multi-Object Tracking (MOT). Building upon the foundations of the State-of-the-Art (SotA) camera-only BEVDet architecture, CR3DT demonstrates substantial improvements in both detection and tracking capabilities, by incorporating the spatial and velocity information of the RADAR sensor. Experimental results demonstrate an absolute improvement in detection performance of 5.3% in mean Average Precision (mAP) and a 14.9% increase in Average Multi-Object Tracking Accuracy (AMOTA) on the nuScenes dataset when leveraging both modalities. CR3DT bridges the gap between high-performance and cost-effective perception systems in autonomous driving, by capitalizing on the ubiquitous presence of RADAR in automotive applications. The code is available at: https://github.com/ETH-PBL/CR3DT.
LEVIO: Lightweight Embedded Visual Inertial Odometry for Resource-Constrained Devices
Kühne, Jonas; Vogt, Christian; Magno, Michele; et al. (2026)
IEEE Sensors Journal
Accurate, infrastructure-less sensor systems for motion tracking are essential for mobile robotics and augmented reality (AR) applications. The most popular state-of-the-art visual-inertial odometry (VIO) systems, however, are too computationally demanding for resource-constrained hardware, such as micro-drones and smart glasses. This work presents LEVIO, a fully featured VIO pipeline optimized for ultra-low-power compute platforms, allowing six-degrees-of-freedom (DoF) real-time sensing. LEVIO incorporates established VIO components such as Oriented FAST and Rotated BRIEF (ORB) feature tracking and bundle adjustment, while emphasizing a computationally efficient architecture with parallelization and low memory usage to suit embedded microcontrollers and low-power systems-on-chip (SoCs). The paper proposes and details the algorithmic design choices and the hardware-software co-optimization approach, and presents real-time performance on resource-constrained hardware. LEVIO is validated on a parallel-processing ultra-low-power RISC-V SoC, achieving 20 FPS while consuming less than 100 mW, and benchmarked against public VIO datasets, offering a compelling balance between efficiency and accuracy. To facilitate reproducibility and adoption, the complete implementation is released as open-source.
Towards Real-Time Fast Unmanned Aerial Vehicle Detection Using Dynamic Vision Sensors
Mandula, Jakub; Kühne, Jonas; Pascarella, Luca; et al. (2024)
2024 IEEE International Instrumentation and Measurement Technology Conference (I2MTC)
Unmanned Aerial Vehicles (UAVs) are gaining popularity in civil and military applications. However, uncontrolled access to restricted areas threatens privacy and security. Thus, prevention and detection of UAVs are pivotal to guarantee confidentiality and safety. Although active scanning, mainly based on radars, is one of the most accurate technologies, it can be expensive and less versatile than passive inspections, e.g., object recognition. Dynamic vision sensors (DVS) are bio-inspired event-based vision models that leverage timestamped pixel-level brightness changes in fast-moving scenes that adapt well to low-latency object detection. This paper presents F-UAV-D (Fast Unmanned Aerial Vehicle Detector), an embedded system that enables fast-moving drone detection. In particular, we propose a setup to exploit DVS as an alternative to RGB cameras in a real-time and low-power configuration. Our approach leverages the high-dynamic range (HDR) and background suppression of DVS and, when trained with various fast-moving drones, outperforms RGB input in suboptimal ambient conditions such as low illumination and fast-moving scenes. Our results show that F- UAV- D can (i) detect drones by using less than <15W on average and (ii) perform real-time inference (i.e., <50 ms) by leveraging the CPU and GPU nodes of our edge computer.
A Fast and Accurate Optical Flow Camera for Resource-Constrained Edge Applications
Kühne, Jonas; Magno, Michele; Benini, Luca (2023)
2023 9th International Workshop on Advances in Sensors and Interfaces (IWASI)
Optical Flow (OF) is the movement pattern of pixels or edges that is caused in a visual scene by the relative motion between an agent and a scene. OF is used in a wide range of computer vision algorithms and robotics applications. While the calculation of OF is a resource-demanding task in terms of computational load and memory footprint, it needs to be executed at low latency, especially in robotics applications. Therefore, OF estimation is today performed on powerful CPUs or GPUs to satisfy the stringent requirements in terms of execution speed for control and actuation. On-sensor hardware acceleration is a promising approach to enable low latency OF calculations and fast execution even on resource-constrained devices such as nano drones and AR/VR glasses and headsets. This paper analyzes the achievable accuracy, frame rate, and power consumption when using a novel optical flow sensor consisting of a global shutter camera with an Application Specific Integrated Circuit (ASIC) for optical flow computation. The paper characterizes the optical flow sensor in high frame-rate, low-latency settings, with a frame rate of up to 88 fps at the full resolution of 1124 by 1364 pixels and up to 240 fps at a reduced camera resolution of 280 by 336, for both classical camera images and optical flow data.
ForzaETH Race Stack - Scaled Autonomous Head-to-Head Racing on Fully Commercial Off-the-Shelf Hardware
Baumann, Nicolas; Ghignone, Edoardo; Kühne, Jonas; et al. (2025)
Journal of Field Robotics
Autonomous racing in robotics combines high-speed dynamics with the necessity for reliability and real-time decision-making. While such racing pushes software and hardware to their limits, many existing full-system solutions necessitate complex, custom hardware and software, and usually focus on Time-TrIals rather than full unrestricted Head-to-head racing, due to financial and safety constraints. This limits their reproducibility, making advancements and replication feasible mostly for well-resourced laboratories with comprehensive expertise in mechanical, electrical, and robotics fields. Researchers interested in the autonomy domain but with only partial experience in one of these fields, need to spend significant time with familiarization and integration. The ForzaETH Race Stack addresses this gap by providing an autonomous racing software platform designed for F1TENTH, a 1:10 scaled Head-to-Head autonomous racing competition, which simplifies replication by using commercial off-the-shelf hardware. This approach enhances the competitive aspect of autonomous racing and provides an accessible platform for research and development in the field. The ForzaETH Race Stack is designed with modularity and operational ease of use in mind, allowing customization and adaptability to various environmental conditions, such as track friction and layout, which is exemplified by the various modularly implemented state estimation and control systems. Capable of handling both Time-Trials and Head-to-Head racing, the stack has demonstrated its effectiveness, robustness, and adaptability in the field by winning the official F1TENTH international competition multiple times. Furthermore, the stack demonstrated its reliability and performance at unprecedented scales, up to over 10m s-1 $10\,{\text{m s}}<^>{-1}$ on tracks up to 150 m in length.
Planar Velocity Estimation for Fast-Moving Mobile Robots Using Event-Based Optical Flow
Boyle, Liam; Kühne, Jonas; Baumann, Nicolas; et al. (2025)
IEEE Robotics and Automation Letters
Accurate velocity estimation is critical in mobile robotics, particularly for driver assistance systems and autonomous driving. Wheel odometry fused with Inertial Measurement Unit (IMU) data is a widely used method for velocity estimation, however, it typically requires strong assumptions, such as non-slip steering, or complex vehicle dynamics models that do not hold under varying environmental conditions, like slippery surfaces. We introduce an approach to velocity estimation that is decoupled from wheel-to-surface traction assumptions by leveraging planar kinematics in combination with optical flow from event cameras pointed perpendicularly at the ground. The asynchronous mu -second latency and high dynamic range of event cameras make them highly robust to motion blur, a common challenge in vision-based perception techniques for autonomous driving. The proposed method is evaluated through in-field experiments on a 1:10 scale autonomous racing platform and compared to precise motion capture data demonstrating not only performance on par with the State-of-the-Art Event-VIO method but also a 38.3% improvement in lateral error. Qualitative experiments at highway speeds of up to 32 m/s further confirm the effectiveness of our approach, indicating significant potential for real-world deployment.
Low Latency Visual Inertial Odometry with On-Sensor Accelerated Optical Flow for Resource-Constrained UAVs
Kühne, Jonas; Magno, Michele; Benini, Luca (2025)
IEEE Sensors Journal
Visual Inertial Odometry (VIO) is the task of estimating the movement trajectory of an agent from an onboard camera stream fused with additional Inertial Measurement Unit (IMU) measurements. A crucial subtask within VIO is the tracking of features, which can be achieved through Optical Flow (OF). As the calculation of OF is a resource-demanding task in terms of computational load and memory footprint, which needs to be executed at low latency, especially in robotic applications, OF estimation is today performed on powerful CPUs or GPUs. This restricts its use in a broad spectrum of applications where the deployment of such powerful, power-hungry processors is unfeasible due to constraints related to cost, size, and power consumption. On-sensor hardware acceleration is a promising approach to enable low latency VIO even on resource-constrained devices such as nano drones. This paper assesses the speed-up in a VIO sensor system exploiting a compact OF sensor consisting of a global shutter camera and an Application Specific Integrated Circuit (ASIC). By replacing the feature tracking logic of the VINS-Mono pipeline with data from this OF camera, we demonstrate a 49.4% reduction in latency and a 53.7% reduction of compute load of the VIO pipeline over the original VINS-Mono implementation, allowing VINS-Mono operation up to 50 FPS instead of 20 FPS on the quad-core ARM Cortex-A72 processor of a Raspberry Pi Compute Module 4.
Efficient Visual-Inertial Perception on Resource-Constrained Systems
Kühne, Jonas (2025)
Autonomous vehicles such as drones, self-driving cars, and mobile robots depend on robust perception to understand and navigate their environments safely and efficiently. Among the various sensing modalities, visual perception, enabled by cameras, is particularly attractive due to its low cost, wide availability, and rich information content. Despite its advantages, computer vision is inherently computationally intensive, and the high data rate of visual sensors imposes significant processing demands. This thesis investigates the design and implementation of lightweight, efficient Visual-Inertial Odometry (VIO) systems tailored for highly resource-constrained mobile platforms. With the growing demand for autonomous perception in applications such as micro aerial vehicles and wearable devices, this work bridges the gap between state-of-the-art VIO performance and the strict power and computational limits of embedded systems. This research gap is explored by tackling the trade-off between computational efficiency and estimation accuracy, thus advancing the respective Pareto frontier in embedded VIO. This work explores four complementary designs, all placed in the gap between high-accuracy VIO pipelines traditionally run on computationally powerful systems and lightweight implementations suitable for microcontrollers with a power envelope of less than 100 mW. The first part of the thesis investigates the use of on-sensor acceleration for calculating optical flow as part of a VIO pipeline. This project demonstrates that the power requirement of an established and accurate VIO pipeline (VINS-Mono) can be reduced to 3.8 W through the use of on-sensor acceleration and a Raspberry Pi Compute Module. This system illustrates the potential of tightly coupled vision pipelines that minimize processing overhead while maintaining real-time performance. The second part of the thesis targets GAP9, a parallel, ultra-low-power system-on-chip (SoC) that utilizes RISC-V cores, developed by GreenWaves Technologies. It assesses the suitability for accurate low-power VIO by implementing and optimizing three different feature trackers (ORB, SuperPoint, and PX4FLOW), and comparing them in a common downfacing VIO pipeline. Furthermore, the proposed pipeline reduces estimation errors by employing a rigid body motion model, achieving accurate planar pose tracking with minimal resource usage. The optimized pipeline demonstrates an average reduction in Root Mean Squared Error (RMSE) of up to a factor of 3.65 compared to the baseline pipeline when using the ORB feature tracker. The third and primary contribution, LEVIO (Lightweight Embedded Visual-Inertial Odometry), presents a full six-degree-of-freedom VIO pipeline on GAP9, incorporating feature tracking, perspective solvers (EPnP and 8-point), and sliding-window bundle adjustment. Through algorithmic simplification and hardware-aware design, LEVIO achieves a carefully crafted balance between estimation accuracy and computational tractability using a sub-100 mW compute platform. The fourth contribution investigates event-based vision as an alternative sensor modality, demonstrating planar velocity estimation from event-based optical flow on an Intel NUC. This proof-of-concept highlights the promise of asynchronous vision for high-speed motion scenarios and informs future integration strategies for mixed sensor modalities in embedded perception. The proposed method is evaluated through in-field experiments, demonstrating not only performance on par with state-of-the-art Event-VIO methods but also a 38% improvement in lateral error. Collectively, these contributions aim to advance the state-of-the-art in embedded VIO by systematically exploring the algorithm-hardware design space. This work investigates foundational principles for developing efficient and accurate perception systems on edge platforms, with implications for robotics and augmented reality. It demonstrates that accurate and real-time visual-inertial perception is feasible on platforms operating below 100 mW and with memory budgets on the order of 1 MB.
Efficient and Accurate Downfacing Visual Inertial Odometry
Kühne, Jonas; Vogt, Christian; Magno, Michele; et al. (2025)
IEEE Internet of Things Journal
Visual Inertial Odometry (VIO) is a widely used computer vision method that determines an agent's movement through a camera and an IMU sensor. This paper presents an efficient and accurate VIO pipeline optimized for applications on micro- and nano-UAVs. The proposed design incorporates state-of-the-art feature detection and tracking methods (SuperPoint, PX4FLOW, ORB), all optimized and quantized for emerging RISC-V-based ultra-low-power parallel systems on chips (SoCs). Furthermore, by employing a rigid body motion model, the pipeline reduces estimation errors and achieves improved accuracy in planar motion scenarios. The pipeline's suitability for real-time VIO is assessed on an ultra-low-power SoC in terms of compute requirements and tracking accuracy after quantization. The pipeline, including the three feature tracking methods, was implemented on the SoC for real-world validation. This design bridges the gap between high-accuracy VIO pipelines that are traditionally run on computationally powerful systems and lightweight implementations suitable for microcontrollers. The optimized pipeline on the GAP9 low-power SoC demonstrates an average reduction in RMSE of up to a factor of 3.65x over the baseline pipeline when using the ORB feature tracker. The analysis of the computational complexity of the feature trackers further shows that PX4FLOW achieves on-par tracking accuracy with ORB at a lower runtime for movement speeds below 24 pixels/frame.

Publications 1 - 10 of 10

Jonas Kühne

Last Name

First Name

ORCID

Organisational unit

Filters

Search Results