Objective 1 – Designing a multi-modal, modular, low power, real-time, “always-on’, user sensing speech and vision system, cutting power consumption of existing systems by factor of 10-50x
Designing a Face and body Analysis system for Natural Computer Interaction (FANCI) comprising one or more depth and image sensors, one or more microphones and a dedicated computer vision DSP for running sophisticated algorithms. The FANCI system is intended to provide an interactive user sensing system, which senses the user’s intentions and reacts accordingly, with system reactive feedback contributing to a paradigm of ambient and pervasive computing. The system is intended to be an “always-on” ubiquitous reference speech and vision platform capable of functioning both as a standalone solution or embedded into more complex systems such as those used in the creative industries. Since vision is the most demanding sensor in terms of power consumption, an optimised implementation will be proposed for the core device, leading to an “always on” system able to run below 5mW in always-on mode and below 100mW in fully active mode, this for 28nm process technology (not including the camera power). This is a huge step forward from existing systems running at few Watts. The FANCI system will include open hardware and software architecture which can run multiple user sensing technologies (such as face, head and eye monitoring, hands & finger gesture recognition), concurrently, with flexibility to add/remove modules as needed. Hence this will facilitate the adoption of the whole concept on a wider scale, benefiting the maximum European industries possible, including Automotive, Healthcare, Security, Gaming, and ICT industries.
Objective 2 – Developing the FANCI API suitable for application developers who are non-domain experts
The project aims to develop the associated software architecture at two levels. First, a high-level FANCI API will be present in the platform which abstracts low level speech and face processing and provides rich features such as voice and pitch identification, person verification, emotional state, head pose, gaze tracking and facial gesture tracking etc. Second, a middleware for an application processor will provide functionality for communication with the platform and for installing and configuring the FANCI-based applications that will run on it.
Objective 3 – Providing a real-time Cloud-based API to lower the barrier of entry to access FANCI capabilities for smaller SME’s within the creative industries
In delivery of the FANCI platform to market, a highly optimised core processing capability for embedded systems will be created. Access to these capabilities via embedded products will take time, and therefore access to FANCI capabilities via a real-time Cloud-based API of some of the user sensing methods will be released before embedded products are available. The objective is to target smaller SME’s in the creative industries, allowing low-cost highly innovative and interactive real-time mobile and web applications to be created. This action is intended to significantly lower the barrier of entry for creative industries, especially smaller SME’s, to access the advanced capabilities of FANCI by being able to create digital content and experiences that are able to sense and react to users using ordinary webcams or mobile phone cameras. Success will be measured by the number of SME’s that have enrolled in the cloud based API’s and the number of web based and mobile applications that are built.
Objective 4 – Building a reference platform which serves as an example for ultra low-cost devices, applicable to a range of mass market applications and products.
The reference platform will include various input sensing modules such as a depth sensing camera and a microphone, connectivity to feedback modules such as speakers and display, and will be able to run the sophisticated algorithms to manipulate the user sensing inputs and generate a system reaction to the feedback modules. The reference platform is intended to be an example for system companies looking to build sub 15 Euro devices composed of a SoC coupled with a depth sensing camera and microphones.
Objective 5 – Validating FANCI capabilities in automotive infotainment user experiences
Demonstrating and assessing the whole system in an automotive end-user scenario with an integrated Head-unit demonstrator, which will integrate Driver Assistance functions with Infotainment (Navigation, Connectivity, Media, Tuner, etc.). The various sensors for face, gesture, eyes and voice will be linked to the head-unit, and each scenario will involve the creation of a use-case specific FANCI Application for the device, based on the middleware API. Specific measures of success will include a) application development, b) app deployment time, c) ease of configuration, d) App/device performance, particularly with respect to the intended ‘always on’ capability. In addition, user-centric measures of success will include e) overall user satisfaction, f) learning curve of a naïve user for certain common tasks, and other metrics that measure the user experience of the whole system.