© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2023
N. TolaramSoftware Development with Gohttps://doi.org/10.1007/978-1-4842-8731-6_13

13. Vulnerability Scanner

Nanik Tolaram1  
(1)
Sydney, NSW, Australia
 

The proliferation of cloud providers enables organizations to deploy applications that are affordable at scale. Deploying applications at scale is one thing, but securing applications and resources is another thing and this has become a headache for organizations everywhere. Security is a big topic, and it covers a lot of different aspects. In this chapter, you will look at one of the tools that helped in identifying vulnerabilities in the infrastructure.

You are going to look at a tool for detecting vulnerabilities inside Linux. The primary focus of the chapter is to understand how and where to use this tool and also to take a closer look at the source code to understand better how the tool works. In this chapter, you will learn
  • How a vulnerability scanner works

  • How the tool uses a different technology to achieve its objectives

  • About port scans, command line executions, and databases using SQLite in Go

Source Code

The source code for this chapter is available from the https://github.com/Apress/Software-Development-Go repository.

Vulnerability Scanners

Vulnerability scanners are tools that are used to search and report for known vulnerabilities that exist in your IT infrastructure. Every organization has an IT infrastructure that it manages in-house or in the cloud. In this infrastructure is a variety of applications, networks, and other things running, which requires constant supervision when it comes to security. Every day we read news of new vulnerabilities uncovered or exploited that can cause damage to organizations and sometimes to an extended community.

Tools like vulnerability scanners use a lot of interesting technology stacks that are useful to learn from, and this is the intention of this chapter. You will look at an open source project named Vuls (https://github.com/future-architect/vuls), which is written in Go, and look at how it implements some of the functionality it provides in Go. The objective is to apply this knowledge in your own project or use it as a knowledge base to understand how this kind of tool works. Please remember that this chapter is by no means a go-to chapter for installing or using Vuls or for vulnerability scanners.

The reason for choosing Vuls for this chapter is the fact that the project is heavily maintained and updated by the community and it has a high star rating. The project uses a database of information from different sources rather than relying on its own source, making it up to date in terms of detecting vulnerabilities.

Some of the key features that Vuls provides are
  • Support for scanning vulnerabilities for major Linux/FreeBSD operating systems

  • Can be used to scan cloud infrastructures like Amazon, Google, and more

  • Uses multiple vulnerability databases. One of the databases it uses is the National Vulnerability Database from NIST (U.S. National Institute of Standards and Technology).

  • Ability to do quick, deep, or other kinds of scanning depending on the need

  • Notifications via email or slack channel

In the next section, you will download the source code, compile it, and use it to understand how it works.

Using Vuls

In this section, you will explore Vuls and do the following:
  • Check out the code

  • Run a scan on a local machine

Checking Out the Code

Vuls requires Go 1.8, so make sure you have it installed before proceeding further. The easiest way to check out code is to use the go get command as in the following:
GO111MODULE=off go get github.com/future-architect/vuls
Make sure you have your GOPATH directory set up to the correct folder where you want to store your Go modules (in my case, my GOPATH points to /home/nanik/Gopath). Once the command has successfully run, it downloads the source code inside the src/github.com/future-architect/vuls directory inside GOPATH, like so:
...
drwxrwxr-x  2 nanik nanik   4096 Jun 26 15:48 detector
-rw-rw-r--  1 nanik nanik    596 Jun 26 15:48 Dockerfile
-rw-rw-r--  1 nanik nanik     55 Jun 26 15:48 .dockerignore
...
drwxrwxr-x  2 nanik nanik   4096 Jun 26 15:48 saas
drwxrwxr-x  2 nanik nanik   4096 Jun 26 15:48 scanner
-rw-rw-r--  1 nanik nanik    137 Jun 26 15:48 SECURITY.md
drwxrwxr-x  2 nanik nanik   4096 Jun 26 15:48 server
drwxrwxr-x  3 nanik nanik   4096 Jun 26 15:48 setup
drwxrwxr-x  2 nanik nanik   4096 Jun 26 15:48 subcmds
drwxrwxr-x  2 nanik nanik   4096 Jun 26 15:48 tui
drwxrwxr-x  2 nanik nanik   4096 Jun 26 15:48 util
The discussion in this chapter focuses on the v0.19.7 version, so you need to switch to a different branch. Change to the GOPATH directory to the src/github.com/future-architect/vuls directory and change the branch as follows:
git checkout v0.19.7
The code is all set and ready to be built. Use the make command to build it.
make build
The compilation process starts and all the related modules are downloaded. Once compilation completes, you get an executable file called Vuls. Run the application as follows:
./vuls
You will get output like the following:
Usage: vuls <flags> <subcommand> <subcommand args>
Subcommands:
      commands         list all command names
      flags            describe all known top-level flags
      help             describe subcommands and their syntax
Subcommands for configtest:
      configtest       Test configuration
Subcommands for discover:
      discover         Host discovery in the CIDR
Subcommands for history:
      history          List history of scanning.
Subcommands for report:
      report           Reporting
Subcommands for scan:
      scan             Scan vulnerabilities
Subcommands for server:
      server           Server
Subcommands for tui:
      tui              Run Tui view to analyze vulnerabilities
Use "vuls flags" for a list of top-level flags

Running Scan

Vuls require a configuration file in the .toml format. For this section, you can use the configuration file found inside the chapter13 directory called config.toml, which is as follows:
[servers.localhost]
host = "localhost"
port = "local"

The configuration specifies the machine to be scanned. In your example, it’s the localhost on your local machine. It performs the standard local mode scanning operation, which does not include port scanning as the one excluded service.

Run Vuls as follows:
./vuls scan --config <directory>/config.toml --debug --results-dir <report_directory>
Vuls runs with the configuration specified with the –config parameter and stores the report inside the directory specified by the –results-dir parameter. You get verbose output that looks like the following:
[Jun 26 18:33:45]  INFO [localhost] vuls-v0.19.7-build-20220626_181254_91ed318
[Jun 26 18:33:45]  INFO [localhost] Start scanning
[Jun 26 18:33:45]  INFO [localhost] config: /home/nanik/Downloads/config.toml
[Jun 26 18:33:45] DEBUG [localhost] map[string]config.ServerInfo{
  "localhost": config.ServerInfo{
    ServerName:         "localhost",
    User:               "",
...
    SSHConfigPath:      "",
    KeyPath:            "",
...
}
[Jun 26 18:33:45]  INFO [localhost] Validating config...
...
[Jun 26 18:33:45] DEBUG [localhost] execResult: servername:
  cmd: ls /etc/debian_version
  exitstatus: 0
  stdout: /etc/debian_version
...
[Jun 26 18:33:45] DEBUG [localhost] Executing... cat /etc/issue
[Jun 26 18:33:45] DEBUG [localhost] execResult: servername:
  cmd: cat /etc/issue
...
[Jun 26 18:33:45] DEBUG [localhost] Executing... lsb_release -ir
...
  cmd: lsb_release -ir
  exitstatus: 0
...
[Jun 26 18:33:45] DEBUG [localhost] Executing... type curl
[Jun 26 18:33:45] DEBUG [localhost] execResult: servername:
  cmd: type curl
  exitstatus: 0
  stdout: curl is /usr/bin/curl
...
Scan Summary
================
localhost    pop22.04    2378 installed, 0 updatable
The scan reports generated by Vuls contain comprehensive information about the things that have been scanned or found. The report looks like the following:
{
    "jsonVersion": 4,
    "lang": "",
    "serverUUID": "",
    "serverName": "192-168-1-3",
    "family": "pop",
    "release": "22.04",
    ...
    "ipv4Addrs": [
        "192.168.1.3"
    ],
    "ipv6Addrs": [
        ...
    ],
    "scannedAt": "2022-06-26T18:40:16.045650086+10:00",
    "scanMode": "fast mode",
    "...
    "scannedVia": "remote",
    "scannedIpv4Addrs": [
        ...
    ],
    "scannedIpv6Addrs": [
        ...
    ],
    "reportedAt": "0001-01-01T00:00:00Z",
    "reportedVersion": "",
    "reportedRevision": "",
    "reportedBy": "",
    "errors": [],
    ...
        "release": "5.17.5-76051705-generic",
        "version": "",
        "rebootRequired": false
    },
    "packages": {
        ...
        }
    },
    "config": {
        "scan": {
            "debug": true,
            "logDir": "/var/log/vuls",
            "logJSON": false,
            "resultsDir": "/home/nanik/go/src/github.com/future-architect/vuls/result",
            "default": {},
            "servers": {
                "192-168-1-3": {
                    ...
                }
            },
            "cveDict": {
                ...
            },
            "ovalDict": {
                ...
            },
            ...
        },
        "report": {
            "logJSON": false,
            ...
        }
    }
}

In the next section, you will explore some of the features provided by Vuls.

Learning From Vuls

There are many features in Vuls that are useful to learn and can be applied when developing systems or security applications. You will look at three main features that Vuls uses: port scanning, using a SQLite database, and executing on the command line from the Go application. These features will be discussed in depth in the following sections

Port Scan

A port scan is a way to perform an operation to determine which ports are open in a network. A ports is like a number that is picked by an application to listen to. For example, HTTP servers listen to port 80 while FTP servers listen to port 21. A list of standard port numbers that are followed in different operating system can be seen at www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml.

Looking at Vuls source code (scanner/base.go), you can see the following function that performs a network scan:
package scanner
import (
  ...
  nmap "github.com/Ullaakut/nmap/v2"
)
func (l *base) execExternalPortScan(scanDestIPPorts map[string][]string) ([]string, error) {
  ...
  baseCmd := formatNmapOptionsToString(portScanConf)
  listenIPPorts := []string{}
  for ip, ports := range scanDestIPPorts {
       ...
     scanner, err := nmap.NewScanner(nmap.WithBinaryPath(portScanConf.ScannerBinPath))
       ...
  return listenIPPorts, nil
}

The code uses the open source nmap library from github.com/Ullaakut/nmap to perform the scanning operation. Before getting into the details on how nmap is performed in the library, let’s get an understanding of what nmap is first. The tool nmap is a command-line tool that is used for network exploration and security auditing. It is used for gathering real-time information about the network, detecting which ports are open in a network environment, checking which IP addresses are activated in the network, and more.

Make sure you have the nmap tool install on your local machine. If you are using a Debian-based Linux distro, use the following command to install it:
sudo apt install nmap
Run nmap to check if you can run it successfully.
nmap
You get output that looks like the following:
Nmap 7.80 ( https://nmap.org )
Usage: nmap [Scan Type(s)] [Options] {target specification}
TARGET SPECIFICATION:
  ...
HOST DISCOVERY:
  ...
SCAN TECHNIQUES:
  ...
EXAMPLES:
  nmap -v -A scanme.nmap.org
  nmap -v -sn 192.168.0.0/16 10.0.0.0/8
  nmap -v -iR 10000 -Pn -p 80
Let’s take a look at the sample code that is provided inside the chapter13/nmap directory and run it as follows:
go run main.go
The application runs and scans your local machine for an open port. In my machine, the output looks like the following:
Host "127.0.0.1":
        Port 22/tcp open ssh
        Port 631/tcp open ipp
        Port 5432/tcp open postgresql
Nmap done: 1 hosts up scanned in 0.020000 seconds

The code detects three open ports, which are related to the ssh, ipp, and postgresql applications. You will get different results depending on what ports are open on your local machine.

The code snippet that uses the nmap library is as follows:
package main
import (
  ...
  "github.com/Ullaakut/nmap/v2"
)
func main() {
  ...
  scanner, err := nmap.NewScanner(
     nmap.WithTargets("localhost"),
     nmap.WithContext(ctx),
  )
  ...
}
The sample code initializes the library by calling nmap.NewScanner(..). Inside the function, the initialization code checks to ensure that the nmap tool is installed, as shown in the following code snippet:
func NewScanner(options ...Option) (*Scanner, error) {
  ...
  if scanner.binaryPath == "" {
     var err error
     scanner.binaryPath, err = exec.LookPath("nmap")
     if err != nil {
        return nil, ErrNmapNotInstalled
     }
  }
  ...
  return scanner, nil
}
The function uses the Go os/exec package to check for the existence of the nmap tool. Once the library has been initialized successfully, it calls the Run() function to perform the scan operation.
package main
import (
  ...
  "github.com/Ullaakut/nmap/v2"
)
func main() {
    ...
  result, warnings, err := scanner.Run()
    ...
  }
  ...
}
The library Run() function performs the following tasks:
  • Executes the nmap tool with the provided parameters

  • Executes the go routine to wait for the result from the nmap tool that will be parsed and converted into a struct that will be returned to the caller

The variable result is of type Run struct and is declared as follows in the library:
type Run struct {
  XMLName xml.Name `xml:"nmaprun"`
  Args             string         `xml:"args,attr" json:"args"`
  ProfileName      string         `xml:"profile_name,attr" json:"profile_name"`
  Scanner          string         `xml:"scanner,attr" json:"scanner"`
  StartStr         string         `xml:"startstr,attr" json:"start_str"`
  Version          string         `xml:"version,attr" json:"version"`
  XMLOutputVersion string         `xml:"xmloutputversion,attr" json:"xml_output_version"`
  Debugging        Debugging      `xml:"debugging" json:"debugging"`
  Stats            Stats          `xml:"runstats" json:"run_stats"`
  ScanInfo         ScanInfo       `xml:"scaninfo" json:"scan_info"`
  Start            Timestamp      `xml:"start,attr" json:"start"`
  Verbose          Verbose        `xml:"verbose" json:"verbose"`
  Hosts            []Host         `xml:"host" json:"hosts"`
  PostScripts      []Script       `xml:"postscript>script" json:"post_scripts"`
  PreScripts       []Script       `xml:"prescript>script" json:"pre_scripts"`
  Targets          []Target       `xml:"target" json:"targets"`
  TaskBegin        []Task         `xml:"taskbegin" json:"task_begin"`
  TaskProgress     []TaskProgress `xml:"taskprogress" json:"task_progress"`
  TaskEnd          []Task         `xml:"taskend" json:"task_end"`
  NmapErrors []string
  rawXML     []byte
}
The raw output from nmap, the library in the XML format, looks like the following:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE nmaprun>
<?xml-stylesheet href="file:///usr/bin/../share/nmap/nmap.xsl" type="text/xsl"?>
<!-- Nmap 7.80 scan initiated Sun Jun 26 20:39:01 2022 as: /usr/bin/nmap -oX - localhost -->
<nmaprun scanner="nmap" args="/usr/bin/nmap -oX - localhost" start="1656239941" startstr="Sun Jun 26 20:39:01 2022" version="7.80" xmloutputversion="1.04">
  <scaninfo type="syn" protocol="tcp" numservices="1000" services="...,61532,61900,62078,63331,64623,64680,65000,65129,65389"/>
  <verbose level="0"/>
  <debugging level="0"/>
  <host starttime="1656239941" endtime="1656239941">
    <status state="up" reason="localhost-response" reason_ttl="0"/>
    <address addr="127.0.0.1" addrtype="ipv4"/>
    <hostnames>
      <hostname name="localhost" type="user"/>
      <hostname name="localhost" type="PTR"/>
    </hostnames>
    <ports>
      ...
    </ports>
    <times srtt="3" rttvar="0" to="100000"/>
  </host>
  <runstats>
    <finished time="1656239941" timestr="Sun Jun 26 20:39:01 2022" elapsed="0.12" summary="Nmap done at Sun Jun 26 20:39:01 2022; 1 IP address (1 host up) scanned in 0.12 seconds" exit="success"/>
    <hosts up="1" down="0" total="1"/>
  </runstats>
</nmaprun>

Exec

The next feature that is used quite often inside Vuls is executing an external tool to perform some operation as part of the scanning process. The following are some of the commands that Vuls uses for getting network IP information, getting kernel information, updating the index of package manager, and many others.

apk update

Downloads the updated package index from repositories

/sbin/ip -o addr

Lists network devices, routing, and other network-related information

uname -r

Prints out system information

stat /proc/1/exe

Gets information about PID 1

systemctl status

Lists information that is registered with system

The commands used are different for different operating systems, but the way it is run is the same using the os/exec package.

Take a look at the sample app that is inside the chapter13/exec folder and run the sample in your terminal as follows:
go run main.go
You get output that looks like the following:
2022/06/26 21:18:28 --------------
2022/06/26 21:18:28 Running ip link
2022/06/26 21:18:28 --------------
2022/06/26 21:18:28 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
...
2022/06/26 21:18:28
2022/06/26 21:18:28 ---------------
2022/06/26 21:18:28 Running noexist
2022/06/26 21:18:28 ---------------
2022/06/26 21:18:28 %v exit status 127
2022/06/26 21:18:28 Running uname -r
2022/06/26 21:18:28 ----------------
2022/06/26 21:18:28 5.17.5-76051705-generic
The sample app uses the os/exec package to execute commands and print the output to the console. The following code snippet shows the function that uses the os/exec package:
package main
import (
  ..
  ex "os/exec"
)
func main() {
  ...
  Run("ip link")
  ...
  Run("noexist")
  ...
  Run("uname -r")
}
func Run(arg string) {
  var cmd *ex.Cmd
  cmd = ex.Command("/bin/sh", "-c", arg)
  ...
}
The Run(..) function is called with a string parameter, which is added to the parameter when calling the Command(..) function. The sample runs the argument passed into the Run(..) function as part of the /bin/sh command tool. For example, when the Run("ip link") is called, it runs it as follows:
/bin/sh -c ip link
The app specifies that the output is stored into the variable because it will be printed out into the console:
func Run(arg string) {
  ...
  cmd.Stdout = &stdoutBuf
  cmd.Stderr = &stderrBuf
  log.Println(stdoutBuf.String())
  log.Println(stderrBuf.String())
}

SQLite

In this section, you will learn how to use SQLite databases. In particular, you will learn how to use the sqlite3 library to read and write databases.

SQLite is a lightweight and self-contained SQL database that allows applications to read and store information. Applications use normal SQL syntax to perform different kinds of data manipulation such as inserting, updating, and deleting data. The lightweight and portable nature of SQLite makes it an attractive proposition to use in a project that doesn't require a centralized database. Mobile phones such as Android use SQLite databases that applications can use for their mobile apps.

Internally, Vuls uses SQLite extensively for storing data that it downloads from different sources. You will look at sample applications using SQLite. Sample code for this section can be found inside the chapter13/sqlite directory. Let’s run the sample application as follows from your terminal:
go run main.go
You get output that looks like the following:
2022/06/27 19:40:04 Initialize database -  local.db
2022/06/27 19:40:04 Creating table in -   local.db
2022/06/27 19:40:04 Inserting data into -  local.db
Reading Table:
2022/06/27 19:40:04 Total rows read -  [{0 CAD} {1 AUD} {2 AUD} {3 GBP} {4 CAD} {5 EUR} {6 USD} {7 USD} {8 CAD} {9 USD} {10 GBP} {11 CAD} {12 AUD} {13 GBP} {14 EUR} {15 GBP} {16 CAD} {17 USD} {18 AUD} {19 CAD} {20 USD} {21 CAD} {22 EUR} {23 EUR} {24 AUD}]

The sample code creates a new database called local.db and creates a new table called currencies. It also inserts a little data into it and prints out the newly inserted data into the console.

The following snippet shows the code that initialize the database:
package main
import (
  ...
  _ "github.com/mattn/go-sqlite3"
  ...
)
...
func main() {
  ...
  dbHandle = InitDB(dbname)
  ...
}
// InitDB initialize database
func InitDB(filepath string) *sql.DB {
  db, err := sql.Open("sqlite3", filepath)
  if err != nil {
     panic(err)
  }
  return db
}

The InitDB function creates the new database using sql.Open, passing in sqlite3 as the parameter. The sqlite3 parameter is used as a reference by the database/sql module to look up the appropriate driver for this. If successful, it will return the sql.DB struct stored inside the db variable

The sql.DB struct is declared in the database/sql module as follows:
type DB struct {
  ...
  connector driver.Connector
  ...
  closed            bool
  ...
  stop func()
}
Once the database has been created successfully, the code creates a table called currencies, which is performed by the following InitTable(..) function:
  ...
func InitTable(db *sql.DB) {
  q := `
  CREATE TABLE IF NOT EXISTS currencies(
     Id TEXT NOT NULL PRIMARY KEY,
     Name TEXT,
     InsertedDatetime DATETIME
  );`
  _, err := db.Exec(q)
  if err != nil {
     log.Fatal(err)
  }
}
The function executes the CREATE TABLE.. SQL command using the db.Exec(..) function. The function db.Exec(..) is used to execute the query against a database without returning any rows. The returned value is of type Result, which is not used in the InitTable(..) function. The Result struct is declared as follows in the database/sql module:
type Result interface {
  LastInsertId() (int64, error)
  RowsAffected() (int64, error)
}
After successfully creating the table, the code then proceeds to inserting data. There are two parts to this operation. The first part is to prepare the data to be inserted, which is shown in the following code snippet:
func main() {
  ...
  records := []Record{}
  for i := 0; i < 25; i++ {
     r := (rand.Intn(len(curNames)-0) + 0)
     d := strconv.Itoa(i)
     rec := Record{Id: d, Name: curNames[r]}
     records = append(records, rec)
  }
  ...
}
The code creates an array of the Record struct and populates it, where the populated array is passed in as a parameter to the InsertData(..) function as follows:
func InsertData(db *sql.DB, records []Record) {
  q := `
  INSERT OR REPLACE INTO currencies(
     Id,
     Name,
     InsertedDatetime
  ) values(?, ?,  CURRENT_TIMESTAMP)`
  stmt, err := db.Prepare(q)
  ...
  defer stmt.Close()
  for _, r := range records {
     _, err := stmt.Exec(r.Id, r.Name)
     ...
  }
}

The function uses the INSERT INTO statement, which is used inside the Prepare(..) function. This function is used to create prepared statements that can be executed in isolation later. The SQL statement uses a parameter placeholder for the values (the placeholder is marked by the ? symbol), which are included as part of the parameter when executing using the Exec(..) function. The value is obtained from the Id and Name of the Record struct.

Now that the data has been inserted into the table, the code completes the execution by reading the data from the table and printing it out to the console as follows:
func ReadData(db *sql.DB) []Record {
  q := `
  SELECT Id, Name  FROM currencies
  ORDER BY datetime(InsertedDatetime) DESC`
  rows, err := db.Query(q)
  ...
  var records []Record
  for rows.Next() {
     item := Record{}
     err := rows.Scan(&item.Id, &item.Name)
     ...
     records = append(records, item)
  }
  return records
}

The function ReadData(..) uses the SELECT SQL statement to read the fields from the table with the result sorted by the InsertedDateTime field in ascending order. The code uses the Query(..) function, returning the Rows struct and looping through it. Inside the loop, the code uses the Scan(..) function to copy fields from each row and read into the values passed in the parameter. In the code example, the fields are read into item.Id and item.Name.

The number of parameters passed to Scan(..) must match with the number of fields read from the table. The Rows struct that is returned when using the Query(..) function is defined inside the database/sql module.
type Rows struct {
  dc          *driverConn
  releaseConn func(error)
  ...
  closemu sync.RWMutex
  closed  bool
  lasterr error
  ...
}

Summary

In this chapter, you looked at an open source security project called Vuls, which provides vulnerability scanning capability. You learned about Vuls by checking out the code and performing a scan operation on your local machine.

Vuls provides a lot of functionality. In learning how Vuls works, you learned about port scanning, executing external command-line applications from Go, and writing code that performs database operations using SQLite.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.15.143.40