项目来源

韩国国家研究基金(NRF)

项目主持人

김현희

项目受资助机构

명지대학교

项目编号

2017S1A5A2A01023839

立项年度

2017

立项时间

未公开

项目级别

国家级

研究期限

未知 / 未知

受资助金额

19500000.00韩元

学科

복합학

学科代码

未公开

基金类别

학술·인문사회사업-SD-Research-기초연구지원인문사회

뇌파측정기술 ; 사건관련유발전위 ; 영상물 ; 서포트 벡터 머신 ; 정보 검색 ; 시각적 주목 산출 모형 ; 내츄럴 사용자 인터페이스 ; 비디오 요약 ; 의미 격차 ; 적합성 ; 단순 탐색 ; 복합 탐색 ; electroencephalography(EEG) ; event related potentials(ERP) ; videos ; support vector machines(SVM) ; information retrieval ; visual attention model ; natural user interface(NUI) ; video summarization ; semantic gap ; relevance ; simple search ; complex search ; N300 ; N400 ; P600 ; P3b

参与者

未公开

参与机构

未公开

项目标书摘要：Research Summary:This study proposed and evaluated methods to understand how do viewers get to know the topics of multimedia(video,sound,image)while watching them using event-related potential(ERP).ERP is a method that measures an electrical activity that is specifically related to an event(e.g.,a sensory stimulus or a cognitive task).The followings are divided into three main areas of study.1)Design and Evaluation of Video Summarization Methods We proposed and evaluated a video summarization method using ERP N400/P600 components and a maximal marginal relevance(MMR)that permits to construct diverse summaries to generate a semantically meaningful video skim.To identify the brain response of a viewer for subject-related(or irrelevant)video shots,we performed the first ERP experiment using twenty-five participants and four documentary videos.In order to construct video skims based on the ERP test results,an ERP/MMR model was implemented.The proposed ERP/MMR model enables to construct a video skim,consisted of non-overlapping video shots,reflecting people's cognitive responses.We conducted implicit and explicit evaluations to evaluate our proposed ERP/MMR-based method.The implicit evaluation was conducted by comparing video skims constructed by our proposed model and shot boundary detection(SBD)method used as a competitive baseline with standard video skims generated by multimedia experts.Whereas the explicit evaluation was performed with 39 participants by measuring how efficiently video skims constructed by the proposed model are used for their information browsing comparing to video skims generated by the SBD method.The results showed that in the implicit and explicit evaluations,the average scores of the ERP/MMR methods are statistically higher than the average score of the SBD method,respectively.2)Understanding the Process of Information Relevance Judgment The second ERP experiment was conducted to determine what kinds of their ERP responses occur to relevant or irrelevant ones when participants search images or videos.Four videos(including their video shots and frames),eight queries,consisted of four complex search queries and four simple search queries,and 36 subjects were used for the second experiment.As a result of the second experiment,two media(videos,images)showed different ERP effects regardless of the type of search(simple search,complex search).That is,P600/N400 effects appeared while searching videos,whereas P3b effect appeared while searching images.3)Design of a Sound Summarization Method The following details how to design the sound summarization method.In order to investigate how do viewers(participants)get to know the topic of sound material,we constructed a cognitive information processing model based on Mayer's multimedia learning model and Baddeley's working memory model.We then proposed a framework for the design of a sound summarization method by combining N400/P600 components with the proposed cognitive information processing model.For this end,we established and verified research hypotheses using Video 3(speech+images)used in the second ERP experiment,sound material(speech+subtitles)extracted from Video 3,and 36 participants.The experiment results showed that the P600 is a key component for the construction of a sound summary.In sum,the results of this study can be applied to the generation of generic video skims(or sound summaries)and personalized video skims(or sound summaries),and to the design of interactive interfaces of information retrieval systems to develop NUI(natural user interface)-based search systems.

Application Abstract: Research Summary:This study proposed and evaluated methods to understand how do viewers get to know the topics of multimedia(video,sound,image)while watching them using event-related potential(ERP).ERP is a method that measures an electrical activity that is specifically related to an event(e.g.,a sensory stimulus or a cognitive task).The followings are divided into three main areas of study.1)Design and Evaluation of Video Summarization Methods We proposed and evaluated a video summarization method using ERP N400/P600 components and a maximal marginal relevance(MMR)that permits to construct diverse summaries to generate a semantically meaningful video skim.To identify the brain response of a viewer for subject-related(or irrelevant)video shots,we performed the first ERP experiment using twenty-five participants and four documentary videos.In order to construct video skims based on the ERP test results,an ERP/MMR model was implemented.The proposed ERP/MMR model enables to construct a video skim,consisted of non-overlapping video shots,reflecting people's cognitive responses.We conducted implicit and explicit evaluations to evaluate our proposed ERP/MMR-based method.The implicit evaluation was conducted by comparing video skims constructed by our proposed model and shot boundary detection(SBD)method used as a competitive baseline with standard video skims generated by multimedia experts.Whereas the explicit evaluation was performed with 39 participants by measuring how efficiently video skims constructed by the proposed model are used for their information browsing comparing to video skims generated by the SBD method.The results showed that in the implicit and explicit evaluations,the average scores of the ERP/MMR methods are statistically higher than the average score of the SBD method,respectively.2)Understanding the Process of Information Relevance Judgment The second ERP experiment was conducted to determine what kinds of their ERP responses occur to relevant or irrelevant ones when participants search images or videos.Four videos(including their video shots and frames),eight queries,consisted of four complex search queries and four simple search queries,and 36 subjects were used for the second experiment.As a result of the second experiment,two media(videos,images)showed different ERP effects regardless of the type of search(simple search,complex search).That is,P600/N400 effects appeared while searching videos,whereas P3b effect appeared while searching images.3)Design of a Sound Summarization Method The following details how to design the sound summarization method.In order to investigate how do viewers(participants)get to know the topic of sound material,we constructed a cognitive information processing model based on Mayer's multimedia learning model and Baddeley's working memory model.We then proposed a framework for the design of a sound summarization method by combining N400/P600 components with the proposed cognitive information processing model.For this end,we established and verified research hypotheses using Video 3(speech+images)used in the second ERP experiment,sound material(speech+subtitles)extracted from Video 3,and 36 participants.The experiment results showed that the P600 is a key component for the construction of a sound summary.In sum,the results of this study can be applied to the generation of generic video skims(or sound summaries)and personalized video skims(or sound summaries),and to the design of interactive interfaces of information retrieval systems to develop NUI(natural user interface)-based search systems.