Otmar Hilliges

Last Name

Hilliges

First Name

Otmar

ORCID

0000-0002-5068-3474

Show all metadata

Search Results

Publications 1 - 10 of 134

Using Synthetic Data in Supervised Learning for Robust 5-DoF Magnetic Marker Localization
Wu, Mengfan; Langerak, Thomas; Hilliges, Otmar; et al. (2024)
IEEE Transactions on Magnetics
Tracking passive magnetic markers plays a vital role in advancing healthcare and robotics, offering the potential to significantly improve the precision and efficiency of systems. This technology is key to developing smarter, more responsive tools and devices, such as enhanced surgical instruments, precise diagnostic tools, and robots with improved environmental interaction capabilities. However, traditionally, the tracking of magnetic markers is computationally expensive due to the requirement for iterative optimization procedures. Moreover, these methods depend on the magnetic dipole model for their optimization function, which can yield imprecise outcomes due to the model's significant inaccuracies when dealing with short distances between non-spherical magnet and sensor. Our article introduces a novel approach that leverages neural networks (NNs) to bypass these limitations, directly inferring the marker's position and orientation to accurately determine the magnet's five degrees of freedom (5 DoFs) in a single step without initial estimation. Although our method demands an extensive supervised training phase, we mitigate this by introducing a computationally more efficient method to generate synthetic, yet realistic data using Finite Element Methods simulations. Our novel method uses the rotational symmetry of axis-symmetric magnetic markers to transform the 3-D simulations into 2-D. The benefits of fast and accurate inference significantly outweigh the offline training preparation. In our evaluation, we use different cylindrical magnets, tracked with a square array of 16 sensors. We perform the sensors' reading and position inference on a portable, NN-oriented single-board computer, ensuring a compact setup. We benchmark our prototype against vision-based ground-truth data, achieving a mean positional error of 4 mm and an orientation error of 8 degrees within a $0.2\times 0$ . $2\times 0$ .15 m working volume. These results showcase our prototype's ability to balance accuracy and compactness effectively in tracking 5 DoFs.
PALM: Predicting Actions through Language Models
Kim, Sanghwan; Huang, Daoji; Xiang, Yongqin; et al. (2025)
Lecture Notes in Computer Science ~ Computer Vision – ECCV 2024
Understanding human activity is a crucial yet intricate task in egocentric vision, a field that focuses on capturing visual perspectives from the camera wearer's viewpoint. Traditional methods heavily rely on representation learning that is trained on a large amount of video data. However, a major challenge arises from the difficulty of obtaining effective video representation. This difficulty stems from the complex and variable nature of human activities, which contrasts with the limited availability of data. In this study, we introduce PALM, an approach that tackles the task of long-term action anticipation, which aims to forecast forthcoming sequences of actions over an extended period. Our method PALM incorporates an action recognition model to track previous action sequences and a vision-language model to articulate relevant environmental details. By leveraging the context provided by these past events, we devise a prompting strategy for action anticipation using large language models (LLMs). Moreover, we implement maximal marginal relevance for example selection to facilitate in-context learning of the LLMs. Our experimental results demonstrate that PALM surpasses the state-of-the-art methods in the task of long-term action anticipation on the Ego4D benchmark. We further validate PALM on two additional benchmarks, affirming its capacity for generalization across intricate activities with different sets of taxonomies.
AvatarPose: Avatar-guided 3D Pose Estimation of Close Human Interaction from Sparse Multi-view Videos
Lu, Feichi; Dong, Zijian; Song, Jie; et al. (2025)
Lecture Notes in Computer Science ~ Computer Vision - ECCV 2024
Despite progress in human motion capture, existing multi-view methods often face challenges in estimating the 3D pose and shape of multiple closely interacting people. This difficulty arises from reliance on accurate 2D joint estimations, which are hard to obtain due to occlusions and body contact when people are in close interaction. To address this, we propose a novel method leveraging the personalized implicit neural avatar of each individual as a prior, which significantly improves the robustness and precision of this challenging pose estimation task. Concretely, the avatars are efficiently reconstructed via layered volume rendering from sparse multi-view videos. The reconstructed avatar prior allows for the direct optimization of 3D poses based on color and silhouette rendering loss, bypassing the issues associated with noisy 2D detections. To handle interpenetration, we propose a collision loss on the overlapping shape regions of avatars to add penetration constraints. Moreover, both 3D poses and avatars are optimized in an alternating manner. Our experimental results demonstrate state-of-the-art performance on several public datasets.
Learning to find eye region landmarks for remote gaze estimation in unconstrained settings
Park, Seonwook; Zhang, Xucong; Bulling, Andreas; et al. (2018)
Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, ETRA '18
Conventional feature-based and model-based gaze estimation methods have proven to perform well in settings with controlled illumination and specialized cameras. In unconstrained real-world settings, however, such methods are surpassed by recent appearance-based methods due to difficulties in modeling factors such as illumination changes and other visual artifacts. We present a novel learning-based method for eye region landmark localization that enables conventional methods to be competitive to latest appearance-based methods. Despite having been trained exclusively on synthetic data, our method exceeds the state of the art for iris localization and eye shape registration on real-world imagery. We then use the detected landmarks as input to iterative model-fitting and lightweight learning-based gaze estimation methods. Our approach outperforms existing model-fitting and appearance-based methods in the context of person-independent and personalized gaze estimation.
Learning to assemble: Estimating 6d poses for robotic object-object manipulation
Stevšić, Stefan; Christen, Sammy; Hilliges, Otmar (2020)
IEEE Robotics and Automation Letters
Learning Human-to-Robot Handovers from Point Clouds
Christen, Sammy; Yang, Wei; Pérez-D'Arpino, Claudia; et al. (2023)
2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
We propose the first framework to learn control policies for vision-based human-to-robot handovers, a critical task for human-robot interaction. While research in Embodied AI has made significant progress in training robot agents in simulated environments, interacting with humans remains challenging due to the difficulties of simulating humans. Fortunately, recent research has developed realistic simulated environments for human-to-robot handovers. Leveraging this result, we introduce a method that is trained with a human-in-the-loop via a two-stage teacher-student framework that uses motion and grasp planning, reinforcement learning, and self-supervision. We show significant performance gains over baselines on a simulation benchmark, sim-to-sim transfer and sim-to-real transfer. Video and code are available at https://handover-sim2real.github.io.
Structured Prediction Helps 3D Human Motion Modelling
Aksan, Emre; Kaufmann, Manuel; Hilliges, Otmar (2020)
17th International Conference on Computer Vision (ICCV 2019)
EMDB: The Electromagnetic Database of Global 3D Human Pose and Shape in the Wild
Kaufmann, Manuel; Song, Jie; Guo, Chen; et al. (2023)
2023 IEEE/CVF International Conference on Computer Vision (ICCV)
We present EMDB, the Electromagnetic Database of Global 3D Human Pose and Shape in the Wild. EMDB is a novel dataset that contains high-quality 3D SMPL pose and shape parameters with global body and camera trajectories for in-the-wild videos. We use body-worn, wireless electromagnetic (EM) sensors and a hand-held iPhone to record a total of 58 minutes of motion data, distributed over 81 indoor and outdoor sequences and 10 participants. Together with accurate body poses and shapes, we also provide global camera poses and body root trajectories. To construct EMDB, we propose a multi-stage optimization procedure, which first fits SMPL to the 6-DoF EM measurements and then refines the poses via image observations. To achieve high-quality results, we leverage a neural implicit avatar model to reconstruct detailed human surface geometry and appearance, which allows for improved alignment and smoothness via a dense pixel-level objective. Our evaluations, conducted with a multi-view volumetric capture system, indicate that EMDB has an expected accuracy of 2.3 cm positional and 10.6 degrees angular error, surpassing the accuracy of previous in-the-wild datasets. We evaluate existing state-of-the-art monocular RGB methods for camera-relative and global pose estimation on EMDB. EMDB is publicly available under https://ait.ethz.ch/emdb.
TempCLR: Reconstructing Hands via Time-Coherent Contrastive Learning
Ziani, Andrea; Fan, Zicong; Kocabas, Muhammed; et al. (2022)
2022 International Conference on 3D Vision (3DV)
We introduce TempCLR, a new time-coherent contrastive learning approach for the structured regression task of 3D hand reconstruction. Unlike previous time-contrastive methods for hand pose estimation, our framework considers temporal consistency in its augmentation scheme, and accounts for the differences of hand poses along the temporal direction. Our data-driven method leverages unlabelled videos and a standard CNN, without relying on synthetic data, pseudo-labels, or specialized architectures. Our approach improves the performance of fully-supervised hand reconstruction methods by 15.9% and 7.6% in PA-V2V on the HO-3D and FreiHAND datasets respectively, thus establishing new state-of-the-art performance. Finally, we demonstrate that our approach produces smoother hand reconstructions through time, and is more robust to heavy occlusions compared to the previous state-of-the-art which we show quantitatively and qualitatively.
SPEC: Seeing People in the Wild with an Estimated Camera
Kocabas, Muhammed; Huang, Chun-Hao P.; Tesch, Joachim; et al. (2021)
2021 International Conference on Computer Vision (ICCV)
Due to the lack of camera parameter information for in-the-wild images, existing 3D human pose and shape (HPS) estimation methods make several simplifying assumptions: weak-perspective projection, large constant focal length, and zero camera rotation. These assumptions often do not hold and we show, quantitatively and qualitatively, that they cause errors in the reconstructed 3D shape and pose. To address this, we introduce SPEC, the first in-the-wild 3D HPS method that estimates the perspective camera from a single image and employs this to reconstruct 3D human bodies more accurately. First, we train a neural network to estimate the field of view, camera pitch, and roll given an input image. We employ novel losses that improve the calibration accuracy over previous work. We then train a novel network that concatenates the camera calibration to the image features and uses these together to regress 3D body shape and pose. SPEC is more accurate than the prior art on the standard benchmark (3DPW) as well as two new datasets with more challenging camera views and varying focal lengths. Specifically, we create a new photorealistic synthetic dataset (SPEC-SYN) with ground truth 3D bodies and a novel in-the-wild dataset (SPEC-MTP) with calibration and high-quality reference bodies. Code and datasets are available for research purposes at https://spec.is.tue.mpg.de/.

Publications 1 - 10 of 134

Otmar Hilliges

Last Name

First Name

ORCID

Organisational unit

Filters

Search Results