Spec sfs® 2014 User’s Guide Standard Performance Evaluation Corporation (spec)


Running the Benchmark and Interpreting Results



Download 345.04 Kb.
Page8/11
Date29.01.2017
Size345.04 Kb.
#11564
1   2   3   4   5   6   7   8   9   10   11

4Running the Benchmark and Interpreting Results

This section contains information on the SPEC SFS 2014 benchmark directory structure, running the benchmark, and interpreting the benchmark metrics output generated in the summary results file.



4.1SFS Benchmark Directory Structure

The following is a quick overview of the benchmark’s directory structure. Please note that the variable “$SPEC” used below represents the full path to the install_directory, where the benchmark is installed.




  1. $SPEC

The directory contains the SPEC SFS 2014 benchmark Makefile. The SPEC SFS 2014 benchmark uses the UNIX “Makefile” structure to build tools, compile the benchmark source into executables, and to clean directories of all executables. However, note that pre-built binaries for several operating systems, and therefore compilation should not be required in most cases.

  1. $SPEC/bin

The benchmark binaries for the specific environment being used are located in the “$SPEC/bin” directory if the user has built the binaries using the Makefile provided..

  1. $SPEC/binaries

The benchmark pre-built binaries for various operating systems are located in subdirectories under the “$SPEC/binaries” directory.

  1. $SPEC/docs

The benchmark documentation is located under the “$SPEC/docs” directory.

  1. $SPEC/results

The benchmark log and results files created during a benchmark run are located in the “$SPEC/results” directory.


4.2Pre-Compiled SPEC SFS 2014 Benchmark Binaries

Included in this benchmark release are pre-compiled versions of the benchmark for various operating systems at various levels. If it becomes necessary for the user to compile a version of the benchmark source for testing, a generic UNIX makefile is provided in the benchmark top level directory ($SPEC).

The makefile may be modified or supplemented in a performance neutral fashion to facilitate the compilation and execution of the benchmark on operating systems not included within the benchmark distribution. To build the software simply type: make
The Visual Studio workspace files are also provided should one need to rebuild the Windows executables. The workspace files are located in the $SPEC, and /libwin32 subdirectories. The SPEC SFS 2014 benchmark can be built with Visual Studio C++ 2010 Express.

The following is a list of the vendors and their respective operating system levels for which the benchmark workloads have been pre-compiled and included with the benchmark distribution.

IBM Corporation

AIX 5.3


FreeBSD

FreeBSD 8.2, 10.0

Oracle Corporation

Solaris 11.1

Redhat Inc.

RHEL6, RHEL7

Apple Inc.

Mac OS X (Tiger, Leopard, and Mavericks)

Microsoft Corporation

Windows 2008R2, Windows 2012, Windows 7, Windows 8.



4.3Using the SFS Manager

This section briefly describes the usage of the Python based SfsManager provided with the SPEC Solution File Server (SFS) SPEC SFS 2014 benchmark suite. The SfsManager is used to run the benchmark. The results obtained from multiple data points within a run are also collected in a form amenable for ease of use with other result formatting tools.


This section does not cover the complete Client-Server environment setup in detail. It touches only on the portions currently handled by the SfsManager. For information on how to set up and run the SFS suite the reader is advised to refer to the section on configuring the SPEC SFS 2014 environment above.

A new Python SfsManager program exists to handle the execution of the SPEC SFS 2014 benchmark. The SfsManager was implemented to allow the same manager to run on Windows and UNIX. The manager uses SSH (Unix) and WMI (Windows) to communicate between the clients.



Example of SUT Validation


Before each load point, the client validates that it can perform all of the POSIX level operations that will be used during the benchmark. If the validation fails, then the benchmark will terminate, with errors collected in the log files.

Example of a Benchmark Run

[root@SPECsfs2014]# python SfsManager -r sfs_rc -s vdi-1cl-1fs-run01



<<< Mon Oct 13 12:38:02 2014: Starting VDI run 1 of 1: DESKTOPS=10 >>

[INFO][Mon Oct 13 12:38:02 2014]Exec validation successful


SPEC SFS2014 Release $Revision: 991 $
This product contains benchmarks acquired from several sources who

understand and agree with SPEC's goal of creating fair and objective

benchmarks to measure computer performance.
This copyright notice is placed here only to protect SPEC in the

event the source is misused in any manner that is contrary to the

spirit, the goals and the intent of SPEC.
The source code is provided to the user or company under the license

agreement for the SPEC Benchmark Suite for this product.


Test run time = 300 seconds, Warmup = 300 seconds.

Running 20 copies of the test on 1 clients

Results directory: /work/SPECsfs2014/results

Op latency reporting activated

Files per directory set to 1

Directories per proc set to 1

Using custom file size of 512000 Kbytes

Clients have a total of 1024 MiBytes of memory

Clients have 51 MiBytes of memory size per process

Clients each have 20 processes

Adjustable aggregate data set value set to 1024 MiBytes

Each process file size = 512000 kbytes

Client data set size = 110000 MiBytes

Total starting data set size = 110000 MiBytes

Total initial file space = 110000 MiBytes

Total max file space = 120000 MiBytes


Starting tests: Mon Oct 13 12:38:02 2014
Launching 20 processes.

Starting test client: 0 Host: t1466 Workload: VDI Workdir: /mnt/fs0

Starting test client: 1 Host: t1466 Workload: VDI Workdir: /mnt/fs0

Starting test client: 2 Host: t1466 Workload: VDI Workdir: /mnt/fs0

Starting test client: 3 Host: t1466 Workload: VDI Workdir: /mnt/fs0

Starting test client: 4 Host: t1466 Workload: VDI Workdir: /mnt/fs0

Starting test client: 5 Host: t1466 Workload: VDI Workdir: /mnt/fs0

Starting test client: 6 Host: t1466 Workload: VDI Workdir: /mnt/fs0

Starting test client: 7 Host: t1466 Workload: VDI Workdir: /mnt/fs0

Starting test client: 8 Host: t1466 Workload: VDI Workdir: /mnt/fs0

Starting test client: 9 Host: t1466 Workload: VDI Workdir: /mnt/fs0

Starting test client: 10 Host: t1466 Workload: VDI Workdir: /mnt/fs0

Starting test client: 11 Host: t1466 Workload: VDI Workdir: /mnt/fs0

Starting test client: 12 Host: t1466 Workload: VDI Workdir: /mnt/fs0

Starting test client: 13 Host: t1466 Workload: VDI Workdir: /mnt/fs0

Starting test client: 14 Host: t1466 Workload: VDI Workdir: /mnt/fs0

Starting test client: 15 Host: t1466 Workload: VDI Workdir: /mnt/fs0

Starting test client: 16 Host: t1466 Workload: VDI Workdir: /mnt/fs0

Starting test client: 17 Host: t1466 Workload: VDI Workdir: /mnt/fs0

Starting test client: 18 Host: t1466 Workload: VDI Workdir: /mnt/fs0

Starting test client: 19 Host: t1466 Workload: VDI Workdir: /mnt/fs0

Waiting to finish initialization. Mon Oct 13 12:38:11 2014

Mon Oct 13 12:38:35 2014 Starting INIT phase

Mon Oct 13 12:38:35 2014 Init Heartbeat __/\_/\__

Mon Oct 13 12:39:35 2014 Init Heartbeat __/\_/\__

Mon Oct 13 12:40:01 2014 Init 20 percent complete

Mon Oct 13 12:40:35 2014 Init Heartbeat __/\_/\__

Mon Oct 13 12:41:25 2014 Init 50 percent complete

Mon Oct 13 12:41:35 2014 Init Heartbeat __/\_/\__

Mon Oct 13 12:41:46 2014 Init 70 percent complete

Mon Oct 13 12:42:28 2014 Init 100 percent complete

Initialization finished: Mon Oct 13 12:42:29 2014

Testing begins: Mon Oct 13 12:42:29 2014

Waiting for tests to finish. Mon Oct 13 12:42:30 2014

Mon Oct 13 12:42:35 2014 Starting WARM phase

Mon Oct 13 12:42:35 2014 Warm Heartbeat Client 19: 0.00 Ops/sec

Mon Oct 13 12:42:58 2014 Warm-up 10 percent complete

Mon Oct 13 12:43:28 2014 Warm-up 20 percent complete

Mon Oct 13 12:43:35 2014 Warm Heartbeat Client 19: 100.47 Ops/sec

Mon Oct 13 12:43:58 2014 Warm-up 30 percent complete

Mon Oct 13 12:44:28 2014 Warm-up 40 percent complete

Mon Oct 13 12:44:35 2014 Warm Heartbeat Client 19: 99.89 Ops/sec

Mon Oct 13 12:44:58 2014 Warm-up 50 percent complete

Mon Oct 13 12:45:28 2014 Warm-up 60 percent complete

Mon Oct 13 12:45:35 2014 Warm Heartbeat Client 19: 99.26 Ops/sec

Mon Oct 13 12:45:58 2014 Warm-up 70 percent complete

Mon Oct 13 12:46:28 2014 Warm-up 80 percent complete

Mon Oct 13 12:46:35 2014 Warm Heartbeat Client 19: 100.22 Ops/sec

Mon Oct 13 12:46:58 2014 Warm-up 90 percent complete

Mon Oct 13 12:47:28 2014 Warm-up 100 percent complete

Mon Oct 13 12:47:30 2014 Starting RUN phase

Mon Oct 13 12:47:35 2014 Run Heartbeat Client 19: 100.13 Ops/sec

Mon Oct 13 12:47:58 2014 Run 10 percent complete

Mon Oct 13 12:48:28 2014 Run 20 percent complete

Mon Oct 13 12:48:35 2014 Run Heartbeat Client 19: 100.50 Ops/sec

Mon Oct 13 12:48:58 2014 Run 30 percent complete

Mon Oct 13 12:49:28 2014 Run 40 percent complete

Mon Oct 13 12:49:35 2014 Run Heartbeat Client 19: 101.00 Ops/sec

Mon Oct 13 12:49:58 2014 Run 50 percent complete

Mon Oct 13 12:50:29 2014 Run 60 percent complete

Mon Oct 13 12:50:35 2014 Run Heartbeat Client 19: 99.77 Ops/sec

Mon Oct 13 12:50:59 2014 Run 70 percent complete

Mon Oct 13 12:51:29 2014 Run 80 percent complete

Mon Oct 13 12:51:35 2014 Run Heartbeat Client 19: 99.47 Ops/sec

Mon Oct 13 12:51:59 2014 Run 90 percent complete

Mon Oct 13 12:52:29 2014 Run 100 percent complete

Tests finished: Mon Oct 13 12:52:30 2014

Shutting down clients, and communications layer...

------------------------------------------------------------

Overall average latency 0.78 Milli-seconds

Overall SPEC SFS2014 2000.18 Ops/sec

Overall Read_throughput ~ 10666.50 Kbytes/sec

Overall Write_throughput ~ 18818.27 Kbytes/sec

Overall throughput ~ 29484.77 Kbytes/sec


Reminder: The benchmark “run” may take many hours to complete depending upon the requested load and how many data points were requested. Also, some failures may take more than an hour to manifest.




Download 345.04 Kb.

Share with your friends:
1   2   3   4   5   6   7   8   9   10   11




The database is protected by copyright ©ininet.org 2022
send message

    Main page