A standard way to get information about CPU, memory, and other activity on your system is by examining info in /proc. In addition, an array of tools is available in Linux to obtain specific pieces of information. The sysstat package contains many of them, for example, iostat to collect CPU and I/O statistics, pidstat to collect health statistics for each process, and sar and sadc to collect similar statistics as cron jobs. Just as a sample, running iostat on a VM running the entire trade network and the chaincode yields the following CPU info and I/O statistics for the two virtual hard drives:
Linux 4.4.0-127-generic (ubuntu-xenial) 05/28/2018 _x86_64_ (2 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.31 0.01 0.26 0.11 0.00 99.32
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 1.11 16.71 11.00 688675 453396
sdb 0.00 0.05 0.00 2014 0
The vmstat tool similarly presents a summary of the virtual-machine-wide information as follows:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 2811496 129856 779724 0 0 7 5 127 342 0 1 99 0 0
For continuous per-process statistics, you can also use the well-known top command, and also dstat, which also generates output in CSV format for easy consumption. If you want to connect your measurement mechanisms to an ETL analytics pipeline, the nmon tool(http://nmon.sourceforge.net/pmwiki.php), which does comprehensive performance data collection and reporting in well-known formats, may be the ideal tool.
But we must also specifically profile the containers that are running the application components. The perf tool is very handy as a Linux performance counter and profiling tool. It can collect profiles on a per thread, per process, and per CPU (or processor) basis. Data collection is done by using the perf report command with different switches, which results in data being collected and stored in a file called perf.data in the folder the command was run in. This data can them be analyzed using the perf report command. In addition, bindfs (https://bindfs.org/) can be used to map symbols in a perf report to processes running inside docker containers. Lastly, perf stat can be used to collect system-wide statistics. The perf Wiki (https://perf.wiki.kernel.org/index.php/Main_Page) gives more information about how to use this tool.