Massive Benchmarking of Vision Algorithms Using Synthetic Video Variants

Open PhD position (3 years)

We are looking for highly motivated PhD candidates in software engineering (software testing and variability) and/or video processing. The work will be conducted in close collaboration with InPixal, a company that is developing vision algorithms.

Contacts: Mathieu Acher (mathieu.acher@irisa.fr), Benoit Baudry (benoit.baudry@inria.fr)

Keywords: video analysis, software testing, variability modeling, big data, image analysis, statistics

Description

Numerous vision algorithms for video analysis have been and will be developed, each one focusing on a specific task (e.g., segmentation, registration, compression, object recognition, tracking). Even for one category of algorithm, many alternatives exist; a one-size-fits-all solution capable of handling the diversity of scenarios and signal qualities is hardly conceivable. An algorithm may excel in tracking a vehicle in a desert but completely fails when the luminance is low or some distractors are present in the video scene. Another algorithm may be less performant in a normal situation, but more robust when luminance is changed and distractors are added. The diversity of situations poses a challenge for both developers and users of video algorithms. For consumers (users), the practical problem is to determine what algorithms are likely to fail or excel in certain conditions before the actual deployment in realistic settings. For providers (developers), how to have confidence and produce evidence that the algorithms are capable of handling certain situations?

A fundamental challenge, which we aim at addressing in this work, consists in systematically testing these algorithms in different situations. We have developed ViViD [1, 2, 3], a tool for the synthesis of video variants with a wide range of characteristics. We can vary luminance, add distractors or objects, and so on. ViViD has been developed in the context of an industrial project involving consumers and providers of video processing algorithms.

We are now at the point of being able to synthesize thousands of videos and teras of data.

The goal of the PhD thesis is to exploit synthetic video variants with a wide range of characteristics to test/benchmark the algorithms. Specifically we want to perform large-scale experiments on thousands of videos and dozens of algorithms (e.g., tracking algorithms). The PhD shall address the following research questions: Can we determine and even predict some qualities of vision algorithms (e.g., performance, reliability) with synthetic video variants and dynamic code analysis? Can we compare competing algorithms? Can the video variants reveal bugs in vision algorithms? The work will be conducted in close collaboration with InPixal, a company that is developing vision algorithms and is part of ViViD. The company is already using the generator and the methodology, but numerous improvements are expected to mature the systematic exploitation of the results.

References

• José A. Galindo, Mauricio Alferez, Mathieu Acher, Benoit Baudry, and David Benavides. A Variability-based Testing Approach for Synthesizing Video Sequences (2014). In International Symposium on Software Testing and Analysis (ISSTA'14) PDF
• Mathieu Acher, Mauricio Alferez, José A. Galindo, Pierre Romenteau, and Benoit Baudry. ViViD: A Variability-Based Tool for Synthesizing Video Sequences (2014). In 18th International Software Product Line Conference (SPLC'14) PDF
• Mauricio Alférez, José A. Galindo, Mathieu Acher, and Benoit Baudry. Modeling Variability in the Video Domain: Language and Experience Report (2014). Research Report RR-8576 PDF

Working Environment

Please send your application (PDF) as soon as possible. Screening of applications starts immediately and continues until the position is filled. Please send CV, PDFs of Master thesis (if any; or draft), recommendation letters.

Other jobs at DiverSE