Scaling-up Soil Quality Assessments: Efficient Infrared-spectroscopic Workflows across Space and Time

Abstract

Modern soil science relies upon the measurement of quantitative and qualitative soil attributes, and is, whenever possible, based on data that we typically measure in the laboratory and also in the field. Soil is a critical component of farming systems and is a limited resource, but also tends to be more and more degraded and lost due to human activities. Thus, sustainable soil management practices are needed more than ever to sustain and improve chemical, physical and biological aspects of soil quality. These are related to key ecosystem functions such as supplying water and nutrient to crops, carbon sequestration, nutrient recycling, and the promotion of biodiversity. Therefore, soil systems require a lot of data so that we can understand, localize or generalize processes in soils, and accordingly manage them from environmental and agronomic viewpoints. For example, we need a better integration of knowledge of how soil management practices (i.e, tillage, fertilization) affects the dynamics of soil's properties, to better manage this precious resource, to protect life across the ecological landscapes, and feed the world's population. Soil properties emerge in function of their mineral and organic compositional diversity, but also processes governed by living components, across space and time. With the classical methods, we have not managed to collect enough static and dynamic parameters of soils' direct physical, chemical and biological composition and properties, for basic soil research and soil decision making (i.e., land planning, construction, environmental and agricultural policy). In Switzerland, only about 10\% to 15\% of agricultural soil landscapes have been sufficiently characterized to make informed land use decisions. This is mainly because most of the classical methods with established measurement protocols and procedures need a lot of time and financial capacities, for example for soil sample collection and processing. To complement and scale up estimates of the soils properties, diffuse reflectance infrared (IR) spectroscopy in the mid-IR and visible near infrared (vis--NIR) range are powerful methods to rapidly integrate the chemical and physical complexity of soils. Soil spectral libraries (SSLs) are harmonized collections of spectroscopic measurements and analytical reference values. The SSL can be used to estimate soil properties on new soil samples collected, primarily within but not necessarily limited to the geographical regions covered. For robust predictions, a gradient from global machine learning methods using all data, to local chemometric modeling methods are used. The research of my thesis evolved around creating SSLs from collected samples and soil archives. I developed spectroscopic data processing and modeling workflows to evaluate the trade-off between accuracy and requirement of updating with analytical measurements for agronomic soil quality assessments and local estimation for temporal soil monitoring. My thesis consists of the general introduction, three chapters, each of which represent this progressive evolution of local or regional modeling, general modeling and transfer learning. Finally, there is an overall conclusion of the work done and I give a research outlook. In the first chapter, I built cost-effective diagnostic databases of soil quality in four project regions under yam production. I focused on specific soil proxies related to different yam production landscapes that then allow to scale up for other similar landscapes in West Africa. There I sampled typical soil variation in 20 fields of smallerholder farmers in each region, and complemented the soil library with 14 samples from the Land Health Degradation Framework. The purely local calibrations across all samples that we built for total carbon (C), nitrogen (N), sulfur (S), exchangeable calcium, effective CEC, diethylenetriaminepentaacetic acid (DTPA)-extractable iron and clay content gave excellent estimates ($R^2 > 0.75$), which can be recommended for screening major soil constraints at new sites within the region. Despite the small size of the library and a gradient in inherent soil fertility (texture; organic carbon (OC)) across the soil ecoregions (humid forest to Northern Guinean savannah), the calibration with a standard multivariate linear method gave good results that are comparable to SSLs developed with higher sample number and local soil sampling densities. In the second chapter, I led the development of a mid-IR SSL for Switzerland ($n =4374 $). It was oriented towards local soil estimation and monitoring and based on time series data and soils from the Swiss Soil Monitoring Network (NABO) from 71 agricultural monitoring sites since 1985 ($n = 596$), and single-time measurements made at 1094 sites from the Swiss Biodiversity Monitoring program (BDM; $n = 3778$). Of the 16 properties we tested, ten showed good discrimination capacity ($R^2 > 0.72$) using mid-IR measurements and rule-based predictions with the {\sc cubist} algorithm. Of these, total C, OC, total N, pH and clay showed very good agreement with the analytical measurements ($R^2 > 0.8$), and were almost unbiased across all data. We also designed a strategy for site-local adaptation based on performance-driven selection ({\sc rs-local}), which we tuned with 2 reference observations for each of the 71 modeling sites and relevant observations from the SSL. Using such a transfer approach reduced the root mean squared error (RMSE) for total C projections of repeated time measurements more than four times (RMSE(C)~=~\SI{0.7}{g\,kg^{-1}}) for a monitoring site compared to the general rules. Interestingly, we also found substantial soil heterogeneity in the respective subsets from the SSL that were selected for localized modeling. The results suggest that the relatively large and diverse SSL has good potential to facilitate temporal soil monitoring at the plot level (10$\times$10\,m). Finally, in the third chapter, I determined the uncertainty and effectiveness of spectroscopic measurement and modeling in combination with the established Swiss mid-IR SSL for detecting soil OC changes at individual plots at a long-term experiment (LTE; $n = 311$; five sampling times between 2002 and 2018) for the evaluation of sustainable farming practices. The estimation of measured OC changes between all possible consecutive time points and individual combinations of plot and depth had high cross-validation accuracy with partial least squares regression (PLSR) across all data points ($n = 311$; RMSE($\Delta$C)~=~\SI{1.9}{g\,kg^{-1}}). The approach was only 1.3 times more inaccurate for cluster-based and data-driven sample transfer from the SSL using the {\sc rs-local} algorithm. The transfer was highly effective because it only needed 2\% of the LTE samples to achieve marginally lower prediction accuracy than the purely local calibration. Despite the promising results, high small-scale soil variation in mineralogy might reduce the information transfer that is related to the functional change of SOM. To mitigate this effect, there is the need for either better representation of the soil conditions at specific LTEs in SSLs, or more innovation in the learning scheme to normalize soil changes directly via the spectra. The establishment of the first version of the mid-IR SSL of Switzerland with the NABO and BDM collections was a key deliverable to test the suitability of SSLs for systematic soil monitoring over time. With chapters two and three, my PhD research has been one of its first country-to-plot level transfers of a large spectroscopic collection. With the findings of this thesis I conclude that the operationalization of SSLs needs adequately designed workflows for soil information transfer and modeling that are tailored to the soil conditions at the respective monitoring locations. For effective learning, it matters how we present data to predictive algorithms. Global modeling with all data in SSLs is useful if data is really scarce and can sometimes work out of the box. Therefore, it is important to continuously update SSLs with new soil records. Simultaneously, to make best use of them, I stand for that we should infuse local adaptation and also independent validation samples to extract and verify knowledge for specific uses, such as soil monitoring, digital soil mapping, agronomic soil testing or soil OC accounting.