面向眼控系统交互式界面元素的视觉表征与评价机制研究

项目来源

国家自然科学基金(NSFC)

项目主持人

牛亚峰

项目受资助机构

东南大学

立项年度

2018

立项时间

未公开

项目编号

71801037

研究期限

未知 / 未知

项目级别

国家级

受资助金额

19.00万元

学科

管理科学-管理科学与工程-工业工程与质量管理

学科代码

G-G01-G0108

基金类别

青年科学基金项目

关键词

视觉表征 ; 工效学评价 ; 交互式界面元素 ; 眼控系统 ; 视觉表征 ; 工效学评价 ; 交互式界面元素 ; 眼控系统

参与者

周蕾;张晶;汤文哲;王欣悦;肖玮烨;李贤;罗子敬

参与机构

上海理工大学;南京林业大学;东南大学

项目标书摘要:随着计算机技术及人工智能的迅猛发展,符合人类自然认知习惯的眼控技术成为人机交互发展的主流技术之一。交互式界面元素成为眼控系统的第一交互入口,其视觉表征直接影响眼机交互效率。本项目以贴近人类认知本源属性的视角开展眼控系统交互式界面元素的视觉表征和交互设计,研究眼动生理特征形式和交互式界面元素的映射机理,寻求交互式界面元素的眼睛动作流设计和最佳动作组合,探索交互式界面元素的工效学评价指标,分析“Midas接触”和“低空间精确度”的设计规范及其规避策略,以期建立一套科学的面向眼控系统交互式界面元素视觉表征优化设计方法与工效学评价机制,为人工智能、先进交互环境下开展“眼机交互—界面设计”研究提供理论支撑和技术手段。研究成果可应用于军事眼控装备系统中去,具有广泛的应用前景,也可促进设计学、工效学与眼机交互领域的交叉融合,具有显著的理论意义和学术价值。

Application Abstract: With the rapid development of computer technology and artificial intelligence,eye-control technology,which conforms to the natural cognition habit of human,has become one of the mainstream technologies of human-computer interaction development.Interactive interface element is the first interactive entry of eye-control system,and its visual representation directly affects the interaction efficiency.This project carries out visual representation and interaction design of the eye control system interactive interface elements from the perspective of human cognitive origin attribute.Mapping mechanism between eye movement physiological characteristics form and interactive interface elements,and eye movement flow design and best action combinations for interactive interface elements are studied.The ergonomic evaluation index of interactive interface elements is explored.Design specification and its countermeasure of"Midas touch"and"low precision space"is analyzed.In order to establish a set of visual representation optimization design method and ergonomic evaluation mechanism for eye control system interactive interface element,providing theoretical support and technical means for the research of"eye-machine interaction-interface design"in artificial intelligence and advanced interactive environment.Research results can be applied to military eye control equipment system,have extensive application prospect,can promote the intersection of design,ergonomics and eye-machine interaction,have significant theoretical meaning and academic value.

项目受资助省

江苏省

项目结题报告(全文)

本项目主要完成了下述工作,并取得了相应研究成果:分析了交互式界面元素的图形符号、色彩、反馈等信息编码规律,探究了图形符号、色彩搭配、交互反馈等成分的最佳组织形式和表征规则。建立了交互式界面元素的布局形态设计体系,完善了眼控系统界面设计功能分区及符合视觉运动规律的界面布局。完成了对界面元素有效交互面积、间距和交互距离的工效学研究,提出了眼控系统交互式界面元素的设计标准。构建了眨眼、凝视、扫视、平滑跟踪、闭眼等眼睛动作指令与单个交互式界面元素的自然逻辑关系和映射,完成了对眼控最佳动作组合形式和凝视触发时间的实验研究。综合运用设计科学、认知心理学、行为学等学科理论和技术手段,探究了多个交互式界面元素条件下眼睛动作流最佳纵向深度与横向容量。从设计学和工效学角度提出了解决“Midas接触”和“低空间精确度”的策略和建议,构建了面向眼控系统交互式界面元素的视觉表征优化设计策略。基于用户交互反馈过程中的用户行为和生理表现,提出了面向交互式界面元素用户认知的关键性脑电成分和脑电指标,确定了交互反馈过程中的行为指标(眼动数据、行为数据)和脑生理指标(ERP成分)的阈值范围,及其两者之间的关联性,确定了交互式界面元素的工效学评价指标。项目实施以来,项目组成员以一作/通讯作者身份共发表17篇论文,其中SCI/SSCI检索和录用7篇,EI检索和录用8篇,项目负责人以第一作者出版学术专著1本,获得授权发明专利2项,获得省部级科研奖励3项。11篇硕士学位论文受到本项目资助。此外,项目组与多家研究所和企业积极开展科研合作,并把项目研究成果推广到自闭症儿童注意力康复软件的开发中,显示了较高的应用价值。

  • 排序方式:
  • 2
  • /
  • 1.The Effects of Spatially-Informed Alarm Types and Presentation Rates on Pilot Performance

    • 关键词:
    • Cockpits (aircraft);Fighter aircraft;Air combat;Alarm presentation rate;Alarm type;Cognitive workloads;Earcons;Human-machine systems;Pilot performance;Situational awareness;Spatial informations;Voice alarms
    • Zhu, Ke;Li, Yaning;Wang, Yiyan;Tian, Jingze;Huang, Jianwei;Yang, Wenjun;Niu, Yafeng
    • 《29th International Conference on Mechatronics and Machine Vision in Practice, M2VIP 2023》
    • 2023年
    • November 21, 2023 - November 24, 2023
    • Queenstown, New zealand
    • 会议

    The efficiency of human-machine systems mainly constrains situational awareness in modern air combat. This study aims to investigate the effects of alarm types with spatial information (voice alarms, auditory icon alarms, earcon alarms) on pilot performance and cognitive workload at different alarm presentation rates (low, medium, high) in aircraft operations. Research findings suggest that alarm presentation rate and alarm type have no significant impact on flight performance, while reaction time decreases as the alarm presentation rate increases. When exposed to earcon alarms and auditory icon alarms, participants reaction speed significantly faster compared to voice alarms, with no significant difference between visual and earcon alarms. Furthermore, the study revealed that visual alarms impose a higher cognitive load on participants. These research conclusions can contribute to the rational design of sound alarm in future fighter cockpit systems. © 2023 IEEE.

    ...
  • 2.Research on the Effect of Visual Warming Information Presentation on Attention in Fighter Tracking Task

    • 关键词:
    • ;
    • Zhu, Jingxin;Qu, Mengyuan;Tian, Jingze;Wang, Yiyan;Huang, Jianwei;Yang, Wenjun;Niu, Yafeng
    • 《2023 IEEE International Conference on Industrial Engineering and Engineering Management, IEEM 2023》
    • 2023年
    • December 18, 2023 - December 21, 2023
    • Singapore, Singapore
    • 会议

    The influence of several visual warning presentation techniques on attention during the fighter tracking task is examined in this research. Concentrate on analyzing the visual channel information form used in alarm systems, as well as how tracking tasks and alert reaction tasks are conveyed by this information. To investigate the ideal visual angle of separation (VAS) and flashing frequency of the visual warning information of the fighter tracking task, the tracking inaccuracy of the tracking task, the reaction time and response accuracy of the warning response task, and the subjective workload of the participants were measured. The pilot performs well in all areas when there is a visual saccade warning signal with a VAS of 10°; for minor warning content, when the VAS is at 20° or 30°,the flashing frequency of 5Hz will not result in a significant load experience, and interference with the primary task can enhance the performance of warning and attention. © 2023 IEEE.

    ...
  • 3.A Dynamic Adjustment System of Vehicle Rearview Mirror using Gaze Detection

    • 关键词:
    • Eye tracking;Vehicle transmissions;Blind spots;Driving blind spot;Dynamic adjustment;Gaze control;Gaze detection;Human-vehicle system interaction;Multi-modelling;Rear-view mirrors;System interactions;Vehicle system
    • Han, Zijian;Niu, Yafeng;Yang, Wenjun
    • 《15th IEEE International Conference on Human System Interaction, HSI 2022》
    • 2022年
    • July 28, 2022 - July 31, 2022
    • Level 3, 360 Collins St, Melbourne, VIC, Australia
    • 会议

    Multi-model interaction research in the cockpit has yielded many interactive modes. Yet, the research on visual interaction is primarily focused on vehicle-to-human transmission. By utilizing eye-tracking technology, we propose a novel approach to introduce the driver's dynamic attention to human-vehicle system interactions. In our proposed system, the angle of the rearview mirror can be dynamically adjusted according to the driver's Area of Interest (AOI), which is obtained by identifying the driver's gaze point. In the experimental verification results, our solution is faster than the manual adjustment scheme to adjust the target in the rearview mirror to the center of the area. In real-world applications, this method can solve the blind spots of rearview mirrors that may exist in dynamic driving tasks, especially in the variant task. Moreover, the method can be compatible with the Driver Monitor System (DMS) in the intelligent cockpit, which can be implemented easily and at a low cost.
    © 2022 IEEE.

    ...
  • 4.Blink recognition using flexible graphene stress sensor and RFID chipless tagging technology

    • 关键词:
    • Frequency domain analysis;Stress measurement;Electrical signal;Eye-controled;Eyeblink;Flexible sensor;Graphene stress sensor;Orbicularis;People with disabilities;Resistance change;Stress sensor;Time and frequency domains
    • Li, Yaning;Xiao, Lang;Yang, Wenjun;Niu, Yafeng
    • 《15th IEEE International Conference on Human System Interaction, HSI 2022》
    • 2022年
    • July 28, 2022 - July 31, 2022
    • Level 3, 360 Collins St, Melbourne, VIC, Australia
    • 会议

    For people with disabilities who are incapacitated, there are few ways to interact with the computer, and using blink to control can be one of the solutions. This paper first converts the contraction signal of the orbicularis oculi muscle into the electrical signal of graphene resistance change during blinking by a graphene flexible sensor, and then extracts the time and frequency domain features of the electrical signal of graphene resistance change during human blinking control, and compares them with the previously established personal blink data database to realize the control command.
    © 2022 IEEE.

    ...
  • 5.A method for distinguishing between involuntary and voluntary blinks based on graphene sensor

    • 关键词:
    • ;Blink detections;Detection range;Electrical signal;Graphene sensor;Involuntary blink;Judging methods;Orbicularis;Resistance change;Temporal features;Voluntary blink
    • Qu, Mengyuan;Zuo, Hongrui;Yang, Wenjun;Niu, Yafeng
    • 《15th IEEE International Conference on Human System Interaction, HSI 2022》
    • 2022年
    • July 28, 2022 - July 31, 2022
    • Level 3, 360 Collins St, Melbourne, VIC, Australia
    • 会议

    In this study, we develop a judging method and device to distinguish between involuntary and voluntary blinks. This method mainly uses graphene sensor, which is attached to the orbicularis oculi muscle, to collect electrical signals of graphene resistance changes, and attains the detection range of the temporal features and peak features of different types of blinks to build a database. On this basis, the temporal, and peak features of the electrical signal of obtained single blink are extracted to complete the recognition of one single blink. The introduced solution allows for stable, reliable, and accurate blink recognition, excluding the influence of the environment.
    © 2022 IEEE.

    ...
  • 7.Research on Position Advantage of Gaze Interactive Control in Eye Control System

    • 关键词:
    • Position control;Eye movements;Cognitive loads;Digital interfaces;Eye control;Interactive control;Operation efficiencies;Priority order;Scanning velocity
    • Zhang, Yating;Xue, Chengqi;Niu, Yafeng;Yang, Lixin
    • 《7th IEEE International Conference on Industrial Engineering and Applications, ICIEA 2020》
    • 2020年
    • April 16, 2020 - April 21, 2020
    • Bangkok, Thailand
    • 会议

    In the design of digital interface of eye control system, interactive control is an important element, and it's convenience directly affects the operation efficiency of the system. In order to reduce the cognitive load of users in the process of eye movement interaction and improve the user experience of eye control system. An ergonomic experiment is designed to explore the input characteristics of gaze interactive controls. The main purpose is to determine the position advantages of the interactive controls. By comparing the scanning velocity between interactive controls, the priority order between interactive controls is obtained. It provides a scientific theoretical basis for the arrangement of the position of interactive controls in the eye control system interface design. © 2020 IEEE.

    ...
  • 8.Eye control system development and research of effects of color of icons on visual search performance based on the system

    • 关键词:
    • Mammals;Control systems;Color combination;Controlled system;Icon;Interactive approach;Interface designs;Literature reviews;Physical disability;System development
    • Cui, Jiaqi;Niu, Yafeng;Xue, Chengqi;Cai, Xijiang;Xie, Yi;Shi, Bingzheng;Qiu, Lincun
    • 《3rd International Conference on Intelligent Human Systems Integration, IHSI 2020》
    • 2020年
    • February 19, 2020 - February 21, 2020
    • Modena, Italy
    • 会议

    Traditional interactive approach which has been developed very mature has evolved from mouse control to many other forms. Eye controlled, as one of new interaction form, will help people with physical disability to manipulate devices as normal users. Meanwhile, because of its natural and high-efficiency, eye controlled system could be the future trend not only for the disabled but also ordinary people. This study proposed an eye controlled system frame to help researchers and designers to explore eye control system interface design because eyes are the only real-time input medium for eye control system. This study firstly analyzes and summarize the difference of eye-control system and traditional interface through surveys and literature reviews. Then according to the uniqueness of eye-controlled interaction, an eye control interactive logic algorithm is developed through the Tobii API combined with the Unity API. Experiments investigated the effects of color combination of icon on visual search performance were done based on the proposed system. © Springer Nature Switzerland AG 2020.

    ...
  • 9.A Cognitive Study of Multicolour Coding for the Head-up Display (HUD) of Fighter Aircraft in Multiple Flight Environments

    • 关键词:
    • Codes (symbols);Ability testing;Subjective testing;Cognitive performance;Color psychology;Flight environment;General information;Information coding;Performance indicators;Recognition abilities;Scientific basis
    • Ling, Bai;Bo, Li;Bingzheng, Shi;Lingcun, Qiu;Chengqi, Xue;Yafeng, Niu
    • 《2018 9th Asia Conference on Mechanical and Aerospace Engineering, ACMAE 2018》
    • 2019年
    • December 29, 2018 - December 31, 2018
    • Singapore, Singapore
    • 会议

    By analyzing flight environments and HUD key information of fighter combat mission, based on the theory of color psychology and the regulation of the avionics system, this study simplified 10 kinds of typical flight environments and combined with 7 types of key information colors into 70 color schemes. Taking reaction accuracy and reaction time as performance indicators and testing participants' subjective feelings, to evaluate pilots' recognition ability of key information and general information with different color schemes in different flight environments, so as to provide the scientific basis for information coding design of flight HUD. The experimental results showed that the cognitive performance of the bicolor coding is significantly better than that of the monochrome coding by comparing the mean reaction time in various environments, and the cognitive performance of color schemes with the hue of 320 or 340 was better than that of other color schemes in most environments. © Published under licence by IOP Publishing Ltd.

    ...
  • 10.HUD Layout Adaptive Method for Fighter Based on Flight Mission

    • 关键词:
    • Ergonomics;Intelligent systems;Adaptive methods;Eye movement datum;Fitness functions;Fixation point;Flight mission;Interest;Interface layout;Output interface
    • Tian, Xiaoyue;Niu, Yafeng;Xue, Chengqi;Xie, Yi;Shi, Bingzheng;Li, Bo;Qiu, Lingcun
    • 《2nd International Conference on Human Systems Engineering and Design: Future Trends and Applications, IHSED 2019》
    • 2020年
    • September 16, 2019 - September 18, 2019
    • Munich, Germany
    • 会议

    In order to solve the problem of inadequate display of important information in the mission process, which leads to low cognitive of pilots, an HUD layout adaptive method based on flight tasks is proposed in this paper. Specific methods include the following steps: Dividing HUD interface according to function; classifying the pilot’s field of view; collecting the pilot’s eye movement data in the task phase and calculating sub-region’s interest value; gridding HUD interface; calculating fitness function; adjusting the layout according to rules, selecting the interface with the largest value as the final HUD output interface. This method can effectively improve pilots’ attention allocation and provide important reference for HUD interface layout. © Springer Nature Switzerland AG 2020.

    ...
  • 排序方式:
  • 2
  • /