FAIR DATA Fund use case

The NERVE project: a neu­ro­mor­phic radar and vision ensem­ble dataset for dis­tance sens­ing and peo­ple detec­tion

Authors: Amir­reza Youse­fzadeh (FAIR Data Fund grantee, Uni­ver­si­ty of Twente) and team made of Omar Man­sour, Pietro Mar­tinel­lo, Ethan milon, Guangzhi Tang, Mano­lis Sifalakis, Ying­Fu Xu

Edi­tor: Iulia Popes­cu

Detect­ing a per­son­’s posi­tion accu­rate­ly is cru­cial for the safe­ty of auto­mat­ed sys­tems in human-machine inter­ac­tions. For instance, in indus­tri­al set­tings, co-robots work­ing along­side human tech­ni­cians must pre­cise­ly iden­ti­fy and adapt to human move­ments to ensure seam­less col­lab­o­ra­tion. Sim­i­lar­ly, self-dri­ving cars must accu­rate­ly locate pedes­tri­ans to nav­i­gate safe­ly through urban envi­ron­ments.

Con­ven­tion­al solu­tions of 3D sens­ing require high-res­o­lu­tion dis­tance sen­sors which are usu­al­ly expen­sive and cost­ly to process. This is why there is a need for low-cost and fast solu­tions for dis­tance-sens­ing and peo­ple-detec­tion.

A new approach

Sen­sor fusion enhances per­for­mance by inte­grat­ing mul­ti­modal sen­so­ry data, sur­pass­ing what could be achieved using any indi­vid­ual sen­sor alone. Hence, it is pos­si­ble to achieve fast dis­tance-sens­ing by fus­ing infor­ma­tion from mul­ti­ple sen­sors.

In sup­port of this approach, the NERVE neu­ro­mor­phic sen­sor fusion dataset com­bines low res­o­lu­tion FMCW radars with low-laten­cy neu­ro­mor­phic cam­eras. The dataset is tai­lored for mul­ti­modal peo­ple-detec­tion and dis­tance-sens­ing with ground truth labels gen­er­at­ed from an accu­rate RGB cam­era and LiDAR sen­sor, ensur­ing reli­able data for test­ing pur­pos­es. This project explored the fea­si­bil­i­ty of accu­rate dis­tance-sens­ing by fus­ing event-based neu­ro­mor­phic cam­eras with FMCW radars using deep learn­ing approach­es.

Dataset Ver­i­fi­ca­tion

The ver­i­fi­ca­tion process of the dataset has made use of the exten­sive sen­sor diver­si­ty, allow­ing for seam­less spa­tial map­ping between sen­sors of dif­fer­ent res­o­lu­tions and allow­ing for a pre­cise tem­po­ral align­ment of all sen­sors.

FAIR Data Fund comes in handy

The FAIR Data Fund helped to work in a much more struc­tur­al and strate­gic way of orga­niz­ing data; and using meta­da­ta and think­ing more broad­ly about prob­lems such as adapt­abil­i­ty and usabil­i­ty. The team con­cludes… “in our opin­ion, FAIR prin­ci­ples are not only about cre­at­ing trans­par­ent datasets, it is about look­ing left and right on how every­one regard­less of their affil­i­a­tions not only likes to work but should work in order to guar­an­tee the most opti­mal results. Thank you to the 4TU.ResearchData team for their pro­fes­sion­al­ism and amaz­ing sup­port with our project!”

Dataset avail­able onto the 4TU.ResearchData repos­i­to­ry: https://data.4tu.nl/private_datasets/yiqrxzXEiRQ7Bjtbf9McXbwBJFYq5w9DLxID2cdI1h8