SAR - System Activity Reporter

System Activity Reporter is an important tool that helps system administrators to get an overview of the server machine with status of different important metrics at different points of time.

If suppose you are having an issue with the system currently, Like some of your customers are unable to list some data from the

Database. The first thing that most of the Linux system administrators do is to recall the same issue when it previously occurred,

 and If you remember the day of its previous occurrence then you can easily compare the internal system statistics with the current  statistics.

SAR is very much helpful in doing exactly that.

The first thing that we need to do is check and confirm whether you have SAR utility installed on the machine.

 Which can be checked by listing all rpm's and finding for this utility.

1 CPU Usage of ALL CPUs (sar -u)

 This gives the cumulative real-time CPU usage of all CPUs. “1 3″ reports for every 1 seconds a total of 3 times.

 Most likely you’ll focus on the last field “%idle” to see the cpu load.

 01:10:01 AM       all     12.92      0.00      0.21      1.37      0.00     85.50

01:20:01 AM       all     12.56      0.00      0.23      0.88      0.00     86.33

01:30:01 AM       all     13.80      0.00      0.21      0.78      0.00     85.21

01:40:01 AM       all      8.15      0.00      0.13      0.39      0.00     91.34

01:50:01 AM       all      4.89      0.00      0.11      0.24      0.00     94.76

02:00:01 AM       all      7.01      0.00      0.14      0.36      0.00     92.49

02:10:01 AM       all     13.55      0.00      0.27      1.85      0.00     84.33

02:20:01 AM       all      9.95      0.00      0.21      0.64      0.00     89.20

02:30:01 AM       all      7.02      0.00      0.16      0.87      0.00     91.95

2-. CPU Usage of Individual CPU or Core (sar -P)

If you have 4 Cores on the machine and would like to see what the individual cores are doing, do the following.

“-P ALL” indicates that it should displays statistics for ALL the individual Cores.

In the following example under “CPU” column 0, 1, 2, and 3 indicates the corresponding CPU core numbers.

01:30:01 AM       all     13.80      0.00      0.21      0.78      0.00     85.21

01:40:01 AM       all      8.15      0.00      0.13      0.39      0.00     91.34

01:50:01 AM       all      4.89      0.00      0.11      0.24      0.00     94.76

02:00:01 AM       all      7.01      0.00      0.14      0.36      0.00     92.49

02:10:01 AM       all     13.55      0.00      0.27      1.85      0.00     84.33

02:20:01 AM       all      9.95      0.00      0.21      0.64      0.00     89.20

02:30:01 AM       all      7.02      0.00      0.16      0.87      0.00     91.95

3- Memory Free and Used (sar -r)

This reports the memory statistics. “1 3″ reports for every 1 seconds a total of 3 times.

Most likely you’ll focus on “kbmemfree” and “kbmemused” for free and used memory.

12:00:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad

12:10:01 AM    113948  12112648     99.07     35492   9002588  16200712    571064      3.40    204928

12:20:03 AM     81892  12144704     99.33     34468   9061424  16201388    570388      3.40    204120

12:30:01 AM    137436  12089160     98.88     37432   9099808  16201640    570136      3.40    204216

12:40:02 AM    138400  12088196     98.87     38592   9128920  16201872    569904      3.40    204076

12:50:01 AM    122772  12103824     99.00     40156   9139808  16201944    569832      3.40    204632

01:00:01 AM    169380  12057216     98.61     41868   9061508  16202004    569772      3.40    205168

4-Swap Space Used (sar -S)

This reports the swap statistics. “1 3″ reports for every 1 seconds a total of 3 times.

 If the “kbswpused” and “%swpused” are at 0, then your system is not swapping.

 Linux 2.6.18-194.el5 (milli.tenongroove.com)     02/13/2014

 5-Overall I/O Activities (sar -b)

                This reports I/O statistics. “1 3″ reports for every 1 seconds a total of 3 times.

Following fields are displays in the example below.

tps – Transactions per second (this includes both read and write)

rtps – Read transactions per second

wtps – Write transactions per second

bread/s – Bytes read per second

bwrtn/s – Bytes written per second

12:00:01 AM       tps      rtps      wtps   bread/s   bwrtn/s

12:10:01 AM    159.91    108.09     51.82   3141.91   1284.71

12:20:03 AM    359.40    300.74     58.66  41838.06  22521.18

12:30:01 AM    314.82    238.17     76.65  23390.49  13952.02

12:40:02 AM    277.36    255.01     22.34  33625.52   1036.20

12:50:01 AM     99.54     73.69     25.85   2332.04   1779.16

01:00:01 AM    110.22     86.61     23.61   3601.47    946.98

6-6. Individual Block Device I/O Activities (sar -d)

01:00:01 AM    dev8-0     94.28    639.08   1167.44     19.16      0.65      6.85      4.66     43.91

01:00:01 AM   dev8-16    243.98     67.84   2272.06      9.59      1.85      7.57      3.62     88.41

01:10:01 AM    dev8-0     96.15    609.59    675.88     13.37      0.64      6.67      4.62     44.40

01:10:01 AM   dev8-16    114.92   1887.28    751.40     22.96      0.77      6.69      3.54     40.67

01:20:01 AM    dev8-0     93.70    539.33    895.29     15.31      0.58      6.21      4.41     41.28

01:20:01 AM   dev8-16     96.87    144.94   1030.79     12.14      0.72      7.39      3.62     35.05

01:20:01 AM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util

01:30:01 AM    dev8-0     88.23    501.38   1042.23     17.50      0.63      7.18      5.02     44.33

01:30:01 AM   dev8-16     80.55    126.41   1034.64     14.41      0.57      7.07      3.50     28.15

01:40:01 AM    dev8-0     72.41    613.69   1002.74     22.32      0.52      7.23      4.88     35.36

01:40:01 AM   dev8-16    260.85  34977.00  27110.38    238.02     70.49    267.99      2.52     65.80

01:50:01 AM    dev8-0     87.83   1140.87    924.01     23.51      0.59      6.69      4.30     37.76

To identify the activities by the individual block devices (i.e a specific mount point, or LUN, or partition), use “sar -d”

7-7. Display context switch per second (sar -w)

This reports the total number of processes created per second, and total number of context switches per second.

 “1 3″ reports for every 1 seconds a total of 3 times.

 12:00:01 AM   cswch/s

12:10:01 AM    839.84

12:20:03 AM   1150.19

12:30:01 AM   1160.66

12:40:02 AM   1340.34

12:50:01 AM    852.09

01:00:01 AM    826.40

01:10:01 AM    804.37

01:20:01 AM    793.43

8. Reports run queue and load average (sar -q)

This reports the run queue size and load average of last 1 minute, 5 minutes, and 15 minutes.

“1 3″ reports for every 1 seconds a total of 3 times.

12:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15

12:10:01 AM         3       476      3.98      3.92      3.07

12:20:03 AM         1       475      6.53      5.21      3.90

12:30:01 AM         3       474      3.05      3.47      3.72

12:40:02 AM         2       467      3.04      3.20      3.36

12:50:01 AM         2       472      1.76      2.27      2.79

01:00:01 AM         4       476      2.52      2.52      2.61

01:10:01 AM         3       473      2.05      2.35      2.49

01:20:01 AM         4       472      2.32      2.30      2.36

09. Report Sar Data Using Start Time (sar -s)

When you view historic sar data from the /var/log/sa/saXX file using “sar -f” option, it displays all the sar data for

 that specific day starting from 12:00 a.m for that day.

Using “-s hh:mi:ss” option, you can specify the start time. For example, if you specify “sar -s 08:00:00″,

 it will display the sar data starting from 10 a.m (instead of starting from midnight) as shown below.

You can combine -s option with other sar option.

For example, to report the load average on 26th of this month starting from 10 a.m in the morning, combine the -q and -s option as shown below.

08:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle

08:10:02 AM     all     19.72      0.00      2.46     10.92      0.00     66.90

08:20:01 AM     all     22.18      0.00      3.13     10.57      0.00     64.12

Average:        all     20.95      0.00      2.80     10.75      0.00     65.51