高清3D裸眼视频内容生成与编码

项目来源

国家自然科学基金(NSFC)

项目主持人

蒋刚毅

项目受资助机构

宁波大学

立项年度

2013

立项时间

未公开

项目编号

U1301257

研究期限

未知 / 未知

项目级别

国家级

受资助金额

255.00万元

学科

联合基金领域-电子信息领域

学科代码

L-L05

基金类别

联合基金项目-重点支持项目-NSFC-广东联合基金

关键词

3D视频内容生成 ; 3D视觉舒适度 ; 裸眼3D显示 ; 3D视频的体验质量 ; 3D视频编码 ; 3D video coding ; 3D Video Content Generation ; Autostereoscopic Display ; 3D visual comfort ; 3D video QoE

参与者

安平;张永兵;邵枫;张磊;韩军;王晓东;冯妮娜;蒋志迪

参与机构

上海大学;清华大学深圳研究生院

项目标书摘要:高清3D裸眼视频系统能提供立体感、临场感等全新视觉体验,高质量3D内容生成、高效编码是其走向应用的关键。与单视点视频系统相比,高清3D裸眼视频系统存在3D节目观看时可能的视疲劳、3D内容匮乏与制作过程复杂、3D数据海量、整体系统用户3D视觉体验质量等问题。现有方法在3D内容采集与重建很少考虑裸眼3D显示的舒适性、3D视频编码失真的视觉感知,也很少考虑从3D系统体验质量来设计各个环节。.本项目从研究影响3D视觉舒适性、编码失真的感知特性、用户体验质量的因素出发,在设计主观感知实验、统计分析各因素影响的基础上,建立数学模型,并对3D舒适度、3D感知失真、3D视觉体验质量进行定量分析与客观描述;提出基于视觉舒适度模型约束的3D内容采集与重建、基于感知失真测度模型的高效3D视频编码、基于用户体验质量预测模型的3D系统设计等理论与方法,以获得最佳用户体验质量的3D内容、高效率的3D视频压缩。

Application Abstract: High definition 3D video systems with autostereoscopic display can provide new visual experiences such as stereoscopic perception,sense of immediacy,etc..High quality 3D content generation and high efficient coding are keys of applying the systems into applications.However,compared with mono-view video system,there are still very important problems to be solved,such as visual discomfort when watching 3D programs,lack of 3D contents,high complexity of 3D content generation and compression,huge amount of 3D data,user’s 3D visual quality of experience(QoE)for the whole system.So far,the exiting 3D content generation and reconstruction methods have seldom considered comfort degree of autostereoscopic display,visual perception degradation created by 3D video coding distortion,and user’s 3D visual QoE in designing each part of 3D system as well..In this project,the factors influencing 3D visual comfort,perception characteristics of coding distortion,and user’s 3D visual QoE will be investigated firstly,the corresponding mathematical models will be established by means of subjective perception experiments and statistical analysis of effectiveness of these facts so as to quantitatively describe 3D visual comfort degree,perception characteristics of coding distortion,and user's 3D visual QoE.Then,the theories and methods for 3D contents generation and reconstruction within the constraint of visual comfort model,high efficient 3D video coding based on perception distortion metric,and user’s QoE prediction model based 3D system design will be proposed to obtain 3D contents with the optimal user’s QoE(or visual comfort)and achieve high efficient performance of 3D video coding.

项目受资助省

浙江省

项目结题报告(全文)

高清3D裸眼视频系统利用人眼双目视觉感知特性形成立体感、临场感,让观众更真实直观地感受世界,是新一代视频技术的发展方向。本项目致力于求解用户视觉体验质量与3D内容生成、3D视频编码等科学问题。从研究影响3D内容失真与3D视觉舒适性的视觉感知特性要素出发,通过设计主观视觉感知实验、统计分析各因素影响,对3D视觉失真测度、视觉舒适度等3D视觉体验质量进行定量分析与描述,提出了基于人眼视觉感知特性的用户体验质量评价理论与方法,并应用于基于3D视觉舒适度评价约束的3D内容采集与重建、基于感知失真评价模型的高效3D视频编码、基于用户体验质量评价的3D视频系统集成等,为实现高质量3D内容生成、高效率3D视频编码、高性能3D视频系统设计等提供了可借鉴的理论与方法,形成了相关专利技术;构建了基于彩色+深度的实时双目3D视频原型系统、高逼真3D实时成像与显示系统等面向不同应用的3D视频原型系统。为实现高质量的3D内容生成及其高效编码压缩提供了相关理论与方法。本项目发表学术论文125篇,其中国际SCI期刊论文72篇,IEEE Transactions、Optical Express等顶尖期刊长文19篇;在本领域权威国际会议上发表论文41篇,出版学术著作1部。获授权发明专利35件(含授权美国发明专利4件)。部分成果获省部级科技奖3项(一、二、三等奖各1项)、参与获国家科技进步二等奖1项。项目组主要成员获国家自然科学基金优秀青年科学基金项目、浙江省自然科学基金杰出青年基金项目、“广东特支计划”科技创新青年拔尖人才项目等;共培养博士和硕士毕业生37名,建立了一支3D视频研究领域的优秀科研队伍。

  • 排序方式:
  • 2
  • /
  • 1.An efficient 3-D mapping algorithm for RGB-D SLAM

    • 关键词:
    • Bismuth compounds;Trees (mathematics);False matches;Feature detection;Feature matching;ICP algorithms;K-d tree;Mapping algorithms;SLAM;Smoothness constraints
    • Yu, Jiadong;You, Zhixiang;An, Ping;Xia, Jie
    • 《14th International Forum of Digital TV and Wireless Multimedia Communication, IFTC 2017》
    • 2018年
    • November 8, 2017 - November 9, 2017
    • Shanghai, China
    • 会议

    Mapping algorithm is the beginning of SLAM, having a significant influence on the design of the follow-up SLAM system and final results. However, the popular mapping algorithms are not robust enough. These algorithms most are based on feature detection but can’t effectively eliminate false matches under challenging circumstances. In addition to this, there is a need for faster algorithms to process higher pixel images with the development of RGB-D sensors. This paper presents a new mapping algorithm to solve the problems. The approach converts the smoothness constraints to offer ultra-robust feature matching. Then, it takes the image smoothness as a statistical likelihood to sore the feature points’ adjacent region. Finally, we use Bi-direction KD-Tree to improve the ICP algorithm. Experiments are carried out with nyuv2 data set. It is challenging enough to test the performance of the new algorithm.
    © 2018, Springer Nature Singapore Pte Ltd.

    ...
  • 2.Subjective evaluation of light field images for quality assessment database

    • 关键词:
    • Image compression;Image quality;Compression algorithms;Compression methods;Light fields;Objective evaluation;Post processing;Quality assessment;Subjective evaluations
    • Shan, Liang;An, Ping;Liu, Deyang;Ma, Ran
    • 《14th International Forum of Digital TV and Wireless Multimedia Communication, IFTC 2017》
    • 2018年
    • November 8, 2017 - November 9, 2017
    • Shanghai, China
    • 会议

    Light filed imaging is becoming popular for its diversity of post-processing and a wide range of applications. Various kinds of research about light field such as light field compression methods are coming out one after the other in recent years. For better evaluation of the quality of light field images and the performance of compression algorithm, the study on quality assessment of light field is in desperate need. In this paper, in order to establish a light field quality assessment database for the subsequent research, we propose a methodology of subjective evaluation for light field image and use a 2D objective evaluation method to verify the methodology. Results show that this methodology can be successfully used to assess the quality of light field content.
    © 2018, Springer Nature Singapore Pte Ltd.

    ...
  • 3.No-Reference Hdr Image Quality Assessment Method Based on Tensor Space

    • 关键词:
    • ;
    • Guan, Feifan;Jiang, Gangyi;Song, Yang;Yu, Mei;Peng, Zongju;Chen, Fen
    • 《2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018》
    • 2018年
    • April 15, 2018 - April 20, 2018
    • Calgary, AB, Canada
    • 会议

    The full-reference image quality assessment (IQA) method are limited in practical applications. Here we propose a no-reference quality assessment method for high dynamic range (HDR) images based on tensor space. First, the tensor decomposition is used to generate three feature maps of an HDR image, considering color and structure information of the HDR image. Second, for a given HDR image, the corresponding multi-scale manifold structure features are extracted from the first feature map. For the second and third feature maps of the HDR image, multi-scale contrast features are extracted. Finally, the extracted features are aggregated by support vector regression to obtain the objective quality score of the HDR image. Experimental results show that the proposed method is superior to some representative full and no-reference methods, and even superior to the full-reference HDR IQA method, HDR-VDP-2.2, on the Nantes database. The proposed method has a higher consistency with human visual perception.
    © 2018 IEEE.

    ...
  • 4.Combining visual saliency and binocular energy for stereoscopic image quality assessment

    • 关键词:
    • Image quality;Visualization;Stereo image processing;Binocular disparity;Binocular perception;Gradient magnitude;Image information;Stereoscopic image;Stereoscopic image quality assessments;Stereoscopic vision;Visual saliency
    • Yao, Yang;Shen, Liquan;Geng, Xianqiu;An, Ping
    • 《13th International Forum of Digital TV and Wireless Multimedia Communication, IFTC 2016》
    • 2017年
    • November 9, 2016 - November 10, 2016
    • Shanghai, China
    • 会议

    With the flourishment of 3D content, the loss of quality of the stereoscopic images has been a large problem while being received by human beings. We develop a new metric in this paper to automatically assess the quality of stereoscopic images with the guidance of reference images. Visual saliency (VS) has been largely explored by researchers in the past decade to find out which areas of an image attract most attention of the viewers. We use the similarity of the VS map between original and distorted images as one of the quality-aware features since the degradation of VS map of the images can depict the quality loss in a certain degree. Meanwhile, gradient magnitude (GM) is enriched with image information, and GM similarity is exploited as another feature. While the difference of binocular energy between original and distorted versions reflects the severities of distortion, it can also act as weights between stereo pairs to simulate the binocular perception properties. Therefore, we introduce the difference of binocular energy as part of the features. The depth/disparity information between stereo pairs contains much properties of stereoscopic vision, and we extract features from disparity map. Finally, in order to take advantage of all the features, we utilize support vector machine based regression module to derive the overall quality score. Experimental results show that the proposed algorithm can assess the image quality in a manner of high consistency with human judgments.
    © Springer Nature Singapore Pte Ltd. 2017.

    ...
  • 5.Stereoscopic image quality assessment using wavelet decomposition and natural scene statistics

    • 关键词:
    • Regression analysis;Extraction;Gaussian distribution;Wavelet decomposition;Image quality;Stereo image processing;Generalized Gaussian distribution;Generalized Gaussian Distributions;Natural scene statistics;Stereoscopic image quality;Stereoscopic image quality assessments;Subjective assessments;Support vector regression (SVR);Wavelet coefficients
    • Geng, Xianqiu;Shen, Liquan;Yao, Yang;An, Ping
    • 《13th International Forum of Digital TV and Wireless Multimedia Communication, IFTC 2016》
    • 2017年
    • November 9, 2016 - November 10, 2016
    • Shanghai, China
    • 会议

    Recently, stereoscopic image quality assessment (SIQA) has been attracted more attention in academia and industry nowadays. In this paper, a wavelet decomposition and natural scene statistics based no reference stereoscopic image quality assessment algorithm is proposed. Our motivation is based on the observation that the statistics of the wavelet coefficients can be effectively captured by a generalized Gaussian distribution (GGD), and the distributions of image with different distortion have different shape and spread. The fitting parameters of GGD are extracted as the features. In this paper, stereoscopic image relevant information including stereo pairs, cyclopean image and binocular disparity are regarded as the factors that affecting stereoscopic image quality, and they are involved in the process of feature extraction. Support vector regression (SVR) is utilized to learn a regression model to predict the quality of stereoscopic image. Experimental results demonstrate that the proposed algorithm achieves high consistency with subjective assessment on two public available 3D image quality assessment databases.
    © Springer Nature Singapore Pte Ltd. 2017.

    ...
  • 6.A joint spatial-temporal 3D video stabilization algorithm

    • 关键词:
    • Stabilization;Parameter estimation;Bandpass filters;Geometrical optics;Three dimensional computer graphics;3-D (three-dimensional);3D video stabilizations;Histogram statistics;Motion parameters;Speeded up robust features;Subjective assessments;SURF feature;Video stabilization
    • Zhou, Jie;You, Zhixiang;An, Ping;Wu, Xinliang;Du, Tengyue
    • 《13th International Forum of Digital TV and Wireless Multimedia Communication, IFTC 2016》
    • 2017年
    • November 9, 2016 - November 10, 2016
    • Shanghai, China
    • 会议

    This paper presents a 3D (Three dimensional) video stabilization algorithm combined with a joint spatial and temporal strategy. On the temporal axis, SURF (Speeded-Up Robust Features) are extracted from the consecutive frames and then motion parameters are estimated, with which we calibrate and compensate the video frames after smoothing the motion parameters using Kalman filtering. Then, on the spatial axis, a histogram statistics method based on the extracted features is applied to detect the vertical parallax between the two views. Adjustments are implemented only when the parallax is larger than the safety threshold, which is conducted through subjective assessment, to maintain the consistency of 3D videos. The experimental results have shown that the proposed method is effective to reduce the vertical instability and inconsistency between binocular views and improve the quality and comfortableness of 3D videos.
    © Springer Nature Singapore Pte Ltd. 2017.

    ...
  • 7.A new tone-mapped image quality assessment approach for high dynamic range imaging system

    • 关键词:
    • Regression analysis;Image quality;Mapping;High dynamic range images;High dynamic range imaging;Low dynamic range;Quality assessment;Quality features;Regression model;Tone mapping operators;Visual quality assessment
    • Song, Yang;Jiang, Gangyi;Jiang, Hao;Yu, Mei;Shao, Feng;Peng, Zongju
    • 《24th IEEE International Conference on Image Processing, ICIP 2017》
    • 2017年
    • September 17, 2017 - September 20, 2017
    • Beijing, China
    • 会议

    Tone-mapping operators are designed to apply high dynamic range (HDR) images on widely-used low dynamic range (LDR) devices. Developing well-performed tone-mapped image quality assessment (IQA) method is highly desired because traditional IQA method cannot be adopted in cross dynamic range quality measuring. To this end, we proposed a quality assessment method based on image exposure property. Specifically, an image exposure property determination model is utilized to segment HDR image into different exposure region. Then, quality features are extracted according to the distortion characteristics of each exposure region. Finally, the quality of tone-mapped image can be acquired by a trained regression model. Validation experiments on public database show that the proposed method can accurately predict the quality of tone-mapped image.
    © 2017 IEEE.

    ...
  • 8.3D holoscopic images coding scheme based on viewpoint image rendering

    • 关键词:
    • Three dimensional computer graphics;Codes (symbols);Image understanding;Image enhancement;Rendering (computer graphics);3-d acquisitions;3D holoscopic images;Coding methods;HEVC;Quality improvement;Storage and delivery;Video sequences;Viewpoint images
    • Yang, Ling;An, Ping;Liu, Deyang;Ma, Ran
    • 《13th International Forum of Digital TV and Wireless Multimedia Communication, IFTC 2016》
    • 2017年
    • November 9, 2016 - November 10, 2016
    • Shanghai, China
    • 会议

    3D holoscopic imaging can provide immersive 3D viewing experiences, which is considered to be a promising 3D acquisition and display solution. However, in order to proper storage and delivery such particular type of image, efficient coding schemes are of great importance. Therefore, in this paper, we propose a new coding scheme based on viewpoint image rendering. All the viewpoint images are rendered firstly from the 3D holoscopic contents. The total rendered viewpoint images are then arranged into a video sequence. HEVC inter coding method is utilized to remove the redundancy among the rendered viewpoint images. Experimental results show that the proposed coding scheme can achieve average 2.70 dB quality improvements for holoscopic images compared to HEVC intra standard.
    © Springer Nature Singapore Pte Ltd. 2017.

    ...
  • 9.A modified just noticeable depth difference model for 3D displays

    • 关键词:
    • Stereo image processing;Depth perception;Piecewise linear techniques;Physiological models;3-D displays;Bottleneck problem;Depth Map;Human visual systems;Just noticeable depth difference;Physiological structures;Piece-wise linear functions;Stereoscopic image
    • Li, Chunhua;An, Ping;Shen, Liquan;Li, Kai;Ma, Jian
    • 《13th International Forum of Digital TV and Wireless Multimedia Communication, IFTC 2016》
    • 2017年
    • November 9, 2016 - November 10, 2016
    • Shanghai, China
    • 会议

    With the flourishment of 3D content, more and more 3D videos need to be transmitted and stored. The contradiction between the bitrate and the quality loss of stereoscopic images becomes a bottleneck problem. To tackle the problem, the perception characteristics of human visual system (HVS) should be exploited. In this paper, we modify the just noticeable depth difference model (JNDD) and verify its effectiveness using subjective experimental results. The modified JNDD model (MJNDD) consists of a three-piecewise linear function, which is consistent with the characteristics of the physiological structure of HVS. Each segment of the three-piecewise linear function depicts the unique depth perception characteristics in the corresponding depth range. Since MJNDD obtains the support of the physiological experimental results, it fits the subjective experimental data more accurate than the state of the art JNDD models.
    © Springer Nature Singapore Pte Ltd. 2017.

    ...
  • 10.An improved 3D holoscopic image coding scheme using HEVC based on Gaussian mixture models

    • 关键词:
    • Geometrical optics;Least squares approximations;Image enhancement;Codes (symbols);Depth perception;Forecasting;3D holoscopic image;Gaussian Mixture Model;HEVC;Image coding scheme;Image prediction;Intrinsic characteristics;Least square methods;Prediction methods
    • Liu, Deyang;An, Ping;Du, Tengyue;Ma, Ran;Shen, Liquan
    • 《13th International Forum of Digital TV and Wireless Multimedia Communication, IFTC 2016》
    • 2017年
    • November 9, 2016 - November 10, 2016
    • Shanghai, China
    • 会议

    3D holoscopic system can provide continuous motion parallax throughout the viewing zone with precise convergence and depth perception, for which it is regarded as a promising technique for future 3D TV. In this paper, a 3D holoscopic image coding scheme based on Gaussian mixture models (GMM) is introduced firstly, taking full advantage of the intrinsic characteristic of such particular type of content. Due to the shortcomings of GMM based method, an improved method is thereafter put forward, in which many parameters that are insignificant in the final estimator of GMM based method are avoided, and more surrounding pixels are used to obtain the model parameters with the help of the least square method. Experimental results indicate that the improved method can obtain considerable gains over HEVC intra prediction and several other prediction methods.
    © Springer Nature Singapore Pte Ltd. 2017.

    ...
  • 排序方式:
  • 2
  • /