5th etsi speech Quality Test Event Anonymous Test Report

Quality Pie Representation

Download 0.63 Mb.

Page	3/13
Date	06.08.2017
Size	0.63 Mb.
	#27239

1 2 3 4 5 6 7 8 9 ... 13

3.2Quality Pie Representation

The huge amount of data for each device under test needs to be condensed into a concise and readable format. The ITU-T P.505 “Quality Pie” representation provides these capabilities and is therefore used during the 5^th SQTE. The focus of this conversational speech quality representation is

To provide a condensed, “quick and easy to read” overview.
To provide a variety of measurement results and compare them to the recommended values and numbers in current ITU-T or ETSI standards.
To provide a comparison to average results from previous SQTEs from other manufactures for the listening speech quality tests.
To give an indication about strength and weakness of the implementation.
To provide detailed information for engineering and development in order to decide about quality improvements.

Each pie slice represents a transmission performance parameter like the codec performance under 5% packet loss, the echo performance under single talk conditions and others. The size of each slice represents a measure for the quality of this parameter. Bigger slices indicate a better performance.


5^th ETSI SQTE IP Gateway Pie (example)	5^th ETSI SQTE IP Phone Pie (example)

Quality Pie representations were already successfully used during the previous events. The selection of parameters for the pie is adapted on the 5^th SQTE. The figures can therefore not be directly compared to the Quality Pies from previous events.

Each selected parameter is represented by a pie slice.
Interaction aspects between different parameters are not considered.
The size of each slice directly correlates to quality. The pie slice size is area equivalent
The minimum requirement or a recommended limit for a parameter is indicated by the inner red circle. If the measured parameter exceeds the requirement or indicates a better quality, the red circle is not visible and overlapped by the pie slice.
In addition the size of a pie slice is color coded (from yellow - low quality scores or low values - to green - high quality scores or high values).
The axis scale of each pie slice is parameter dependent.

he parameters, limits and axis scaling for the Gateway Quality Pie in the 5^th ETSI SQTE are selected as follows (clockwise):

G.711, 5% - MOS-LQO for G.711 speech coder, 5% packet loss:

The minimum requirement is 3.5 MOS-LQO representing the average score from the 3^rd and 4^th SQTE under similar test conditions (5% packet loss, 10 ms packet length instead of 20 ms).

G.711, 1%, jitter - MOS-LQO for G.711 speech coder, 1% packet loss, 20 ms jitter:

The minimum requirement is set to 4.0 MOS-LQO. This limit is derived from the 3^rd and 4^th SQTE as a compromise between the average and maximum values.

G.729, 5% - MOS-LQO for G.729 speech coder, 5% packet loss:

The requirement of 3.1 MOS-LQO represents the average score from the 3^rd and 4^th SQTE. Note that the expected quality range is low due to the specified PLC with G.729 speech coder.

G.729, 1%, jitter - MOS-LQO for G.729 speech coder, 1% packet loss, 20 ms jitter:

The limits is set to 3.5 MOS-LQO as a compromise between maximum scores of 3.7 MOS-LQO from 3^rd and 4^th SQTE and average scores of 3.6 MOS-LQO (3^rd SQTE) and 3.3 MOS-LQO (4^th SQTE).

VAD Reliability – Reliability of VAD during all speech quality and background noise tests

The pie slice is set to “ok” if the VAD performed reliable (as expected from the current VAD settings) during all related tests. The decision is made based on the MOS-LQO respectively TMOS results with and w/o VAD, bandwidth saving, activation thresholds and background noise performance tests.

G-MOS Cafe – G-MOS calculated for cafeteria noise based on ETSI EG 202 396-3

The minimum requirement is set to 4.4 G-MOS representing the expected score for a telephone set simulated by an IRS modified send filter. The score of 4.4 is met, if the gateway does not introduce any kind of signal processing (like AGC, VAD, …) neither on speech nor on background noise.

G-MOS Car – G-MOS calculated for car noise based on ETSI EG 202 396-3

The minimum requirement is set to 4.2 G-MOS representing the expected score for a telephone set simulated by an IRS modified send filter. The score of 4.2 is met, if the gateway does not introduce any kind of signal processing (like AGC, VAD, …) neither on speech nor on background noise.

Echo performance – percentage of echo tests meeting the tolerances

The pie slice reflects the result compliance during the whole set of echo tests and test conditions (echo loss, echo attenuation vs. time, detection of echo components based on Relative Approach, speech recording, infinite, 40 dB and 6 dB ERL for digital connections, POTS for analog implementations). 13 test results are analyzed for digital connections, 5 results contribute for analog gateways. If all tests pass the maximum score of 130 is achieved (“Ok”). The limit is set to 95.

DECT Echo – percentage of DECT echo tests meeting the tolerances

The slice reflects the percentage of compliant results and tolerances during the whole set of DECT echo tests except the worst case test scenario (phone on hard surface). In total 13 results are considered, the maximum number on the axis is 130 (“Ok”). The limit is set to 95 (inner red circle).

Double Talk Performance – Characterization acc. to ITU-T P.340

The double talk categories (1, 2a, 2b, 2c, 3) from ITU-T P.340 are applied. Analyses are based on objective double talk tests (level variation, echo test) and informal listening tests using different speech recordings.

BGNT (far, inf.) – Average level modulation introduced by EC, NLP and CN

The average level modulation introduced by NLP and CN during background noise transmission in sending direction coincident to the application of far end signals for the infinite ERL tests (echo-free connection) is calculated. The pie slice axis is scaled from 20 dB to 0 dB. The minimum requirement is set to 10 dB.

BGNT (far, 6dB) – Average level modulation introduced by EC, NLP and CN

The average level modulation introduced by NLP and CN during background transmission test for the 6 dB ERL echo path is analyzed. The axis scaling is the same as for the “BGNT (far, inf)” pie slice.

The speech quality analyses according to ETSI EG 202 396-3 lead to one-dimensional MOS scores. These speech transmission quality is represented by the S-MOS (speech MOS), the quality of transmitted background noise is given by the N-MOS, (noise MOS), the G-MOS indicated the overall quality (general MOS). In order to provide an efficient summary in a graphical format the resulting nine scores are displayed in a second quality pie. An example is given here.

The minimum requirement (inner circle) is given by the measured S-, N-, G-MOS values for the IRS filtered signal transmitted over the reference connection between the two ISDN simulators connected back-to-back. A tolerance of -0.1 MOS considers the accuracy of these tests.

ETSI EG 202 396-3 Gateway Pie (example)

The corresponding IP Phone Quality Pie is based on the following parameters (clockwise):

G.711, SND - TMOS in sending direction for G.711 speech coder (handset):

The minimum requirement is 4.1 TMOS representing the average score from the 4^th SQTE under similar test conditions (10 ms packet length instead of 20 ms).

G.711, 3% - TMOS in receiving direction for G.711 speech coder (handset), 3% packet loss:

The minimum requirement is set to 3.2 TMOS. This limit is derived from the 4^th SQTE under similar test conditions (10 ms packet length instead of 20 ms).

G.711, Jitter – TMOS in receiving for G.711 speech coder (handset), 20ms jitter:

The requirement of 2.7 TMOS represents the average score from the 4^th SQTE.

G.729, Jitter – TMOS in receiving for G.729 speech coder (handset), 20 ms jitter:

The limit is set to 2.5 TMOS which is the average score of the 4^th SQTE.

Echo Performance – percentage of echo tests meeting the tolerances (G.711 speech coder)

The pie slice reflects the result compliance during the whole set of echo tests and receiving playback volume settings (echo loss at nominal and maximum playback volume, echo attenuation vs. time, detection of echo components based on Relative Approach at nominal and maximum playback volume). Six test results are analysed. If all tests pass the maximum score of 120 is achieved (“Ok”). The limit is set to 95.

Double Talk– Characterization acc. to ITU-T P.340 (G.711 speech coder, handset)

The double talk categories (1, 2a, 2b, 2c, 3) from ITU-T P.340 [5] are applied. Analyses are based on objective double talk tests (level variation, echo test) and informal listening tests using different speech recordings. The limit is set to “type 2a”.

BGNT (far) – Average level modulation introduced by EC, NLP and CN (G.711 speech coder, handset)

The average level modulation introduced by NLP and CN during background noise transmission in sending direction coincident to the application of far end signals is calculated. The pie slice axis is scaled from 20 dB to 0 dB. The minimum requirement is set to 10 dB.

HFT Echo – percentage of echo tests meeting the tolerances under hands-free conditions (G.711 speech coder)

The pie slice reflects the result compliance during the whole set of echo tests and receiving playback volume settings under hands-free conditions (echo loss nom. + max. vol., echo attenuation vs. time at nominal and maximum playback volume, detection of echo components based on Relative Approach at nominal and maximum playback volume). Eight test results are analyzed. If all tests pass the maximum score of 120 is achieved (“Ok”). The limit is set to 95.

HFT Double Talk – Characterization acc. to ITU-T P.340 (G.711 speech coder)

The double talk categories (1, 2a, 2b, 2c, 3) from ITU-T P.340 [5] are again applied. Analyses are based on objective double talk tests (level variation, echo test) and informal listening tests using different speech recordings. The limit for hands-free devices is set to “type 2c”.

HFT TMOS RCV – TMOS in receiving direction for G.711 speech coder, hands-free mode:

The limit is set to 3.1 TMOS which represents the average score of the 4^th SQTE.

WB HS TMOS RCV – TMOS in receiving direction for G.722 speech coder, handset mode:

The limit is set to 2.4 TMOS the average score of the 4^th SQTE. Note that the wideband tests during the 4^th SQTE are carried out from “mouth to ear”. It is assumed that the influence of the sending direction can be neglected.

WB HFT TMOS RCV – TMOS in receiving for G.722 speech coder, hands-free mode:

The average score of the 4^th SQTE of 2.7 TMOS was used as limit. Note that the wideband tests during the 4^th SQTE are carried out from “mouth to ear”. It is assumed that the influence of the sending direction can be neglected.

Similar to the gateway tests the S-, N- and G-MOS from speech quality analyses according to ETSI EG 202 396-3 are summarized in an additional Quality Pie. An example for the IP phones is given here.

The scores are measured for the cafeteria noise transmitted together with speech over the IP phones. The tests are carried out in narrowband handset and hands-free mode (G.711, “HS” and “HFT”) as well as in wideband handset mode (G.722 “WB HS”).

For both narrowband test cases the minimum requirement (inner circle) is set by an ISDN reference phone. The wideband limits represent a simulate wideband phone in handset mode. The so obtained scores are reduced by 0.1 in order to consider the accuracy of this analysis method.

ETSI EG 202 396-3 Terminal Pie (overview, example)

Download 0.63 Mb.

Share with your friends:

1 2 3 4 5 6 7 8 9 ... 13