Integrating Metadata from YANG Catalog

If you are building a YANG model–based application to automate network device configuration or service deployment, it helps to have a holistic view of the YANG modules your application relies upon. Remember, your automation is only as good as the data models and their related metadata. Chapter 7 discussed the tools a network operator might use within the YANG Catalog that expose those metadata fields. In addition, YANG Catalog has a full set of APIs to perform searches and access the same metadata as the frontend tools.

How might you use YANG Catalog metadata within your automation application? Your network may have devices from different vendors or be running different versions of code, even from the same vendor. This means the devices could support different revisions of the same YANG modules. If your application encounters a revision of a module it doesn’t yet recognize, it could invoke the YANG Catalog API to pull down that module’s metadata and inspect its “derived-semantic-version” to see if the major version number changed between an already-known revision and this new one. If so, that indicates a non-backward-compatible change within the module. From there, inspect the module’s tree diff to see the changes relevant to your application. That further inspection can be folded into your application’s logic, or it can trigger a further manual audit from an engineer.

YANG Catalog provides a page¹ for developers who wish to contribute to it. This page describes a number of the REST APIs that it exposes. The APIs to publish metadata to YANG Catalog were already covered in Chapter 8, “Automation Is as Good as the Data Models, Their Related Metadata, and the Tools: For the Module Author” as part of tools for module design. In terms of metadata retrieval, YANG Catalog provides two main endpoints that accept Hypertext Transfer Protocol (HTTP) GET queries. The first, /api/search/modules, provides the means to retrieve all metadata for a specific module given its name, revision, and organization. This API endpoint uses commas to separate the multiple search keys. Example 9-1 shows a query for ietf-routing, revision 2018-03-13. Of course, the organization is the Internet Engineering Task Force (IETF).

 $ http https://yangcatalog.org/api/search/modules/ietf-routing,2018-03-13,ietf
{
    "module": [
        {
            "name": "ietf-routing",
            "namespace": "urn:ietf:params:xml:ns:yang:ietf-routing",
            "organization": "ietf",
            "prefix": "rt",
            "reference": "https://tools.ietf.org/html/rfc8349",
            "revision": "2018-03-13",
            "schema": "https://raw.githubusercontent.com/YangModels/yang/master/standard/ietf/RFC/[email protected]",
            "tree-type": "nmda-compatible",
            "yang-tree": "https://yangcatalog.org/api/services/tree/[email protected]",
            "yang-version": "1.1"
...

The other metadata retrieval endpoint allows you to search YANG Catalog for modules that include a specific metadata field value. This endpoint is /api/search/{key}/{value}. For example, you can retrieve all modules that use the Network Management Datastore Architecture (NMDA) schema tree type with a query to /api/search/tree-type/nmda-compatible. With both this API endpoint and the previous one, the results are every matching module and all their respective metadata fields. A variation exists, /api/search-filter, that uses the POST method and accepts parameters for the metadata values as well as the field to return. Using the same NMDA example, the request in Example 9-2 only returns the module name for those modules that are NMDA-compatible.

$ http POST https://yangcatalog.org/api/search-filter/name input:='{"tree-type":
"nmda-compatible"}'
{
    "output": {
        "name": [
            "ietf-msdp",
...

Other examples of YANG Catalog API calls are available from the contributor web page as a Postman collection.² Since all of YANG Catalog’s APIs are RESTful, they are accessed using libraries and functions that deal with HTTP methods (for example, libcurl, Python’s requests module, and so on). With the Postman collection, you can use Postman’s code-generation capability to create an initial script for interacting with YANG Catalog.

Example 9-3 shows Python code that retrieves derived semantic version metadata from YANG Catalog for a particular revision of the ietf-routing module. This is compared to a previously known semantic version for the same module. If they are different, the YANG tree is retrieved from the catalog and stored for later comparison.

import requests

url = "https://yangcatalog.org/api/search-filter/derived-semantic-version"

KNOWN_SEMVER_MAJORS = [1, 2, 3]

payload = """
{
  "input": {
    "name": "ietf-routing",
    "revision": "2018-03-13"
  }
}
"""

headers = {
    'Content-Type': "application/json"
}

response = requests.request("POST", url, data=payload, headers=headers)

j = response.json()
semver_major = j['output']['derived-semantic-version'][0].split(".")[0]

if semver_major not in KNOWN_SEMVER_MAJORS:
    response = requests.request("GET", "https://yangcatalog.org/api/services/tree/
[email protected]")
    fd = open("/models/trees/ietf-routing@2018-03-13_tree.txt", "w")
    fd.write(response.text)
    fd.close()

In addition to retrieving all module metadata for a module or set of modules, you can also search YANG Catalog for particular keywords by sending a POST to the /api/index/search endpoint. The search API, similar to the web frontend, allows you to specify keywords, search fields and other options, as well as filter the results so that only certain fields are returned. Example 9-4 shows Python code that searches for “route-distinguisher” typedefs and retrieves only the module name, node name, and node path fields in the result set.

import requests

url = "https://yangcatalog.org/api/index/search"

payload = """
{
  "search": "route-distinguisher",
  "type": "keyword",
  "case-sensitive": false,
  "include-mibs": false,
  "latest-revisions": true,
  "search-fields": ["module", "argument", "description"],
  "yang-versions": ["1.0", "1.1"],
  "schema-types": ["typedef"],
  "filter": {
      "node": ["name", "path"],
      "module": ["name"]
  }
}
"""

headers = {
    'Content-Type': "application/json",
}

response =uests.request("POST", url, data=payload, headers=headers)

print(response.text)

This example shows all the fields currently supported within the payload. The only mandatory field is “search.” If the other fields are omitted, the values shown in the code (except for filter) are used as the defaults. There is no default filter, so all module and node metadata is returned if a filter is omitted.

Fields such as search-fields, yang-versions, schema-types, and node and module filters support a list of options. All supported values are shown for yang-versions and search-fields. The schema-types field supports values of typedef, grouping, feature, identity, extension, rpc, action, container, list, leaf-list, list, and notification. The filter fields enable you to control the result set. The allowed node filter values are name, path, description, and type. The module filter supports all of the metadata fields returned for a given module. Additionally, besides a “keyword” search, the type field accepts a value of regex to perform a regular expression search.

Embedding Pyang

In addition to executing pyang as a command, you can embedded pyang’s backend into other Python applications to provide a YANG parser and easy access to the YANG module structures and schema. Why might you want to do this versus calling pyang from a shell or command script? In some cases, it might be easier to add a formatting plug-in to pyang to control its output and then call it from another script. However, as you work with large repositories of modules, and if you need to call pyang a number of times over that set of modules, the time it takes to instantiate pyang’s internal context adds a lot of time to your application’s execution.

Embedding pyang’s backend libraries into your application means that you only need to build the internal context once and then use it (or manipulate it) from within your Python code directly. This includes accessing formatting plug-ins and validating module veracity.

To get started, create one or more pyang.FileRepository objects to hold your YANG modules and then instantiate a pyang.Context object. Once that pyang.Context object is instantiated, call the add_module() method to parse and add module objects to that context or call the search_module() method to find a module by name (and optionally revision) from the repositories. Once all your modules are added, the context object provides the entry point into the pyang internals. Example 9-5 shows Python code that creates a pyang context and searches it for an “ietf-routing” module. If found, the module’s structure is pretty-printed using the pprint() method.

#!/usr/local/bin/python2

from pyang import Context, FileRepository
import sys
import optparse

YANG_ROOT = '/models/src/git/yang/'

optparser = optparse.OptionParser()
(o, args) = optparser.parse_args()

repo = FileRepository(YANG_ROOT)
ctx = Context(repo)
ctx.opts = o

mod_obj = ctx.search_module(None, 'ietf-routing')
if mod_obj:
    mod_obj.pprint()

The mod_obj returned by search_module() is of type Statement, defined in the statement.py module included with pyang. The __init__.py file within the root of the pyang distribution includes the definition for the Context class, which shows additional methods available to contexts.

To give you an idea of the performance boost you may gain by embedding pyang into your application versus invoking it externally, consider Examples 9-6 and 9-7. Example 9-6 is a shell script that invokes pyang to print the name and revision of three YANG modules. This script points pyang to the YangModels GitHub repository discussed in Chapter 7. On a given platform, this script takes about 9 seconds to run. Most of that time is spent rebuilding the pyang internals each time it launches.

#!/bin/sh

YANG_REPO="/models/src/git/yang"
RFC_ROOT="${YANG_REPO}/standard/ietf/RFC"

modules="[email protected] ietf-l3vpn-svc.yang ietf-ipv6-unicast-
routing.yang"

for m in ${modules}; do
  pyang -p ${YANG_REPO} -f name –name-print-revision "${RFC_ROOT}/${m}">
done

Example 9-7 provides the same functionality, only with pyang embedded into a Python script. The name plug-in is then invoked using the pyang context for all three modules at once. Since the context only needs to be built once instead of for each module, this script only takes 3 seconds to run. While 3 seconds versus 9 doesn’t seem like a long time, notice how extrapolating this out for a larger number of pyang invocations makes for a compelling reason to embed it instead of execute it externally.

#!/usr/local/bin/python2

from pyang import Context, FileRepository
from pyang.plugins.name import emit_name
import sys
import optparse

YANG_ROOT = '/models/src/git/yang'
RFC_ROOT = YANG_ROOT + '/standard/ietf/RFC'

optparser = optparse.OptionParser()
(o, args) = optparser.parse_args()
repo = FileRepository(YANG_ROOT)
ctx = Context(repo)
ctx.opts = o

ctx.opts.print_revision = True

mods = ['[email protected]', 'ietf-l3vpn-svc.yang',
        'ietf-ipv6-unicast-routing.yang']
mod_objs = []

for m in mods:
    with open(RFC_ROOT + '/' + m, 'r') as fd:
        mod = ctx.add_module(m, fd.read())
        mod_objs.append(mod)

emit_name(ctx, mod_objs, sys.stdout)

Pyang Plug-Ins

When you run pyang with the -f tree argument to see a tree structure of a YANG module’s schema, that tree format is implemented as a plug-in. Pyang allows for creating plug-ins to extend how it displays output (format plug-ins) as well as to alter module parsing behavior (backend plug-ins). Typically, if you are developing your own YANG modules, you may have a need to define new extensions. In that case, creating custom pyang backend plug-ins to properly validate those extensions (for example, within CI pipeline’s test phase) is extremely useful. Likewise, formatting plug-ins can be quite powerful. As an example, the YANG Catalog suite created formatting plug-ins to generate the index data³ that drives the Catalog search, as well as the YANG module tree interface⁴ seen at https://yangcatalog.org/yang-search/yang_tree/ietf-routing.

In addition to the tree display plug-in, pyang comes bundled with a number of other plug-ins, one of which is yang-data, which instructs pyang how to parse elements that are defined using RFC 8040’s yang-data extension. All of pyang’s plug-ins are found in the location of its library’s installation under the plugins subdirectory. For example, on Linux systems, this is typically /usr/lib/python2.7/site-packages/pyang/plugins.

Both types of pyang plug-ins declare a pyang_plugin_init() function and a class definition for the plug-in itself. Format plug-ins must register the formats they will provide as well as how to display the formatted output. The backend plug-ins vary depending on the YANG parsing behavior they are controlling. For example, they may register additional statement and grammar rules (as is the case for the restconf.py plug-in that parses yang-data).

Example 9-8 show the structure of the name.py pyang plug-in. This plug-in provides a “name” format that display a module’s name and any main module to which it belongs (in the case of a submodule). This plug-in also takes an optional parameter to display the revision in addition to the name. A sample of this plug-in’s output is shown in the snippet that follows Example 9-8.

 1    """Name output plugin
 2
 3    """
 4
 5    import optparse
 6
 7    from pyang import plugin
 8
 9    def pyang_plugin_init():
10        plugin.register_plugin(NamePlugin())
11
12    class NamePlugin(plugin.PyangPlugin):
13        def add_output_format(self, fmts):
14            self.multiple_modules = True
15            fmts['name'] = self
16
17        def add_opts(self, optparser):
18            optlist = [
19                optparse.make_option("--name-print-revision",
20                                     dest="print_revision",
21                                     action="store_true",
22                                     help="Print the name and revision in" +
                                            "name@revision format"),
23                ]
24            g = optparser.add_option_group("Name output specific options")
25            g.add_options(optlist)
26
27        def setup_fmt(self, ctx):
28            ctx.implicit_errors = False
29
30        def emit(self, ctx, modules, fd):
31            emit_name(ctx, modules, fd)
32
33    def emit_name(ctx, modules, fd):
34        for module in modules:
35            bstr = ""
36            rstr = ""
37            if ctx.opts.print_revision:
38                rs = module.i_latest_revision
39                if rs is None:
40                    r = module.search_one('revision')
41                    if r is not None:
42                        rs = r.arg
43                if rs is not None:
44                    rstr = '@%s' % rs
45            b = module.search_one('belongs-to')
46            if b is not None:
47                bstr = " (belongs-to %s)" % b.arg
48            fd.write("%s%s%s
" % (module.arg, rstr, bstr))

The plug-in can be invoked on a YANG module with the following command. Note that, in this case, because the --name-print-revision argument was specified, the module name and its revision are printed.

Click here to view code image

$ pyang -f name --name-print-revision ietf-routing.yang
ietf-routing@2018-03-13

In the code shown in Example 9-7, lines 9 and 10 initialize the plug-in and register its class with the pyang framework. Because this is a format plug-in, the add_output_format() method is defined at line 13. This plug-in provides one output, called “name.”

Optional arguments (like the --name-print-revision parameter in this plug-in) are added via the add_opts() method. The value of any optional parameters can be retrieved later in the plug-in via the ctx.opts object (as shown at line 37). If your plug-in doesn’t require any optional parameters, omit declaring this method.

The critical method for displaying the results of the formatting is emit(), on line 30. In the case of the name plug-in, that is done via a static emit_name() function. The emit() method is invoked by pyang with a context object, the set of modules on which pyang was called, and an output file descriptor. The context object provides access to internal pyang structures as well as optional parameter values. Write the formatting output to the output file descriptor, as shown in line 48. Calling print() is not desirable because the output may be tied to a file (that is, when the pyang --output option is specified).

Example 9-9 shows the elided contents of the restconf.py backend plug-in. This plug-in does not provide any new output formats, but it does provide pyang the necessary knowledge to support RFC 8040–defined yang-data constructs.

 1    restconf_module_name = 'ietf-restconf'
 2
 3    class RESTCONFPlugin(plugin.PyangPlugin):
 4        def __init__(self):
 5            plugin.PyangPlugin.__init__(self, 'restconf')
 6
 7    def pyang_plugin_init():
 8        plugin.register_plugin(RESTCONFPlugin())
 9
10        grammar.register_extension_module(restconf_module_name)
11
12        yd = (restconf_module_name, 'yang-data')
13        statements.add_data_keyword(yd)
14        statements.add_keyword_with_children(yd)
15        statements.add_keywords_with_no_explicit_config(yd)
16
17        for (stmt, occurance, (arg, rules), add_to_stmts) in restconf_stmts:
18            grammar.add_stmt((restconf_module_name, stmt), (arg, rules))
19            grammar.add_to_stmts_rules(add_to_stmts,
20                                       [((restconf_module_name, stmt), occurance)])
21
22        statements.add_validation_fun('expand_2',
23                                      [yd],
24                                      v_yang_data)
25
26        error.add_error_code('RESTCONF_YANG_DATA_CHILD', 1,
27                             "the 'yang-data' extension must have exactly one " +
28                             "child that is a container")
29
30    restconf_stmts = [
31        ('yang-data', '*',
32         ('identifier', grammar.data_def_stmts),
33         ['module', 'submodule']),
34
35    ]
36
37    def v_yang_data(ctx, stmt):
38        if (len(stmt.i_children) != 1 or
39            stmt.i_children[0].keyword != 'container'):
40            err_add(ctx.errors, stmt.pos, 'RESTCONF_YANG_DATA_CHILD', ())

Lines 12 through 15 add a new YANG statement for the ietf-restconf:yang-data extension. This allows pyang to recognize yang-data as a keyword in YANG modules, as well as to do further validation of the children of that keyword. That is, in the case of RFC 8040 yang-data, there is a rule that the yang-data extension must have exactly one (and only one) child that is a container. Lines 22 through 28 and the v_yang_data() function add this knowledge to pyang. Therefore, when pyang sees a module that contains yang-data, it applies the proper syntax checking to ensure the module is valid.

YANG Parsing with Libyang

Embedding pyang into your applications works very well if you are developing in Python. But what if you need YANG parsing support with other language bindings? Yes, you can fork and execute pyang as an external program, but as you saw, that can have performance implications. It will also require you to parse the textual output that comes back from it. This may require use of regular expressions or other “hacky” screen-scraping methods, which will add additional complexity to your code. Libyang⁵ is a C library that provides YANG parsing and validation functions and is used in a number of open source projects like Free Range Routing,⁶ Netopeer2,⁷ and sysrepo.⁸ It is also used to power the yanglint validator discussed in Chapter 8. Libyang is quick to adopt new YANG features and extensions and currently includes support for YANG 1.0 and 1.1, JSON and XML encoded data, default values in instance data (RFC 6243), and YANG metadata (RFC 7952). Because of C’s embeddable nature, libyang also includes Simplified Wrapper and Interface Generator (SWIG) bindings for JavaScript, which enables libyang to be used within Node.js applications.

Like pyang, libyang supports printing module data in various formats, including a tree structure based on RFC 8340. Example 9-10 shows a simple libyang program that parses the ietf-routing module and prints its tree output.

#include <stdio.h>
#include <libyang/libyang.h>
#include <glib.h>

#define YANG_ROOT "/models/src/git/yang"
#define RFC_ROOT YANG_ROOT "/standard/ietf/RFC"

int
main(int argc, char **argv) {
  struct ly_ctx *ctx = NULL;
  char *routing_mod = NULL;
  const struct lys_module *mod;

  ctx = ly_ctx_new(NULL);
  ly_ctx_set_searchdir(ctx, YANG_ROOT);

  routing_mod = g_strdup_printf("%s/[email protected]", RFC_ROOT);

  mod = lys_parse_path(ctx, routing_mod, LYS_IN_YANG);

  lys_print_file(stdout, mod, LYS_OUT_TREE, NULL);

  g_free(routing_mod);
  ly_ctx_destroy(ctx, NULL);

}

A libyang application needs to include libyang/libyang.h and link with -lyang to build and run properly. The full set of documentation is built with the libyang package and is found online at https://netopeer.liberouter.org/doc/libyang/master/, where it is built regularly.

The principal author of libyang (and the Netopeer project), Radek Krejčí, is interviewed at the end of this chapter, where he shares his perspective on enabling developers to harness the power of model-driven management and automation.

NETCONF with Ncclient

The Python ncclient package is one of the most popular libraries available for building NETCONF clients. Ncclient is the underlying library that provides NETCONF support to the netconf-console (or ncc) utility covered in Chapter 7, and it also provides the NETCONF support for the scripts that YANG Suite generates, which was also explored in Chapter 7.

Ncclient is available in the PyPi repository and installed with the following command:

$ pip install ncclient

A primary goal of ncclient is to simplify operations between client applications and NETCONF servers by offering a straightforward, “Pythonic” interface. Ncclient supports the standard <get>, <get-config>, and <edit-config> operations, as well as supporting notifications and an extensible mechanism for handling custom remote procedure calls (RPCs). Although NETCONF should be standard across vendor platforms, at times there are subtle differences, and ncclient comes built with knowledge of some of these nuances. Of course, because it is open source, you can add other vendor handlers or capabilities you require. A sample script showing ncclient performing a <get-config> operation is shown in Example 9-11.

import lxml.etree as ET
from argparse import ArgumentParser
from ncclient import manager
from ncclient.operations import RPCError

if __name__ == '__main__':

    parser = ArgumentParser(description='Usage:')

    # script arguments
    parser.add_argument('-a', '--host', type=str, required=True,
                        help="Device IP address or Hostname")
    parser.add_argument('-u', '--username', type=str, required=True,
                        help="Device Username (netconf agent username)")
    parser.add_argument('-p', '--password', type=str, required=True,
                        help="Device Password (netconf agent password)")
    parser.add_argument('--port', type=int, default=830,
                        help="Netconf agent port")
    args = parser.parse_args()

    # connect to netconf agent
    with manager.connect(host=args.host,
                         port=args.port,
                         username=args.username,
                         password=args.password,
                         timeout=90,
                         hostkey_verify=False,
                         device_params={'name': 'csr'}) as m:

        # execute netconf operation
        try:
            response = m.get_config(source='running').xml
            data = ET.fromstring(response)
        except RPCError as e:
            data = e._raw

        # beautify output
        print(ET.tostring(data, pretty_print=True))

In general, all ncclient code is wrapped within a manager object, which handles the underlying Secure Shell (SSH) connection to the device and provides a channel for sending RPCs and receiving responses. The script in Example 9-11 gets the entire running config and prints out the resulting eXtensible Markup Language (XML) text. It also demonstrates how to specify a device profile (“csr” in this case) to properly handle differences with specific vendor platforms. You may find that when interacting with different vendors, you have to specify the device parameters for a particular vendor’s operating system in order to get the desired behavior.

Instead of retrieving the entire running config, fetch portions using either subtree or XPath-based filters. For example, if you just need to request only the OpenConfig /network-instances path, change the get_config() call to what is shown in the following snippet:

Click here to view code image

response = m.get_config(source='running', filter=('xpath', '/network-instances')).xml

Within the ncclient manager object, you can send custom RPCs to the device just as you perform standard requests. Example 9-12 demonstrates invoking an RPC called “default” that resets an interface to its default configuration.

import lxml.etree as ET
from argparse import ArgumentParser
from ncclient import manager
from ncclient.xml_ import qualify
from ncclient.operations import RPCError

if __name__ == '__main__':

    parser = ArgumentParser(description='Usage:')

    # script arguments
    parser.add_argument('-a', '--host', type=str, required=True,
                        help="Device IP address or Hostname")
    parser.add_argument('-u', '--username', type=str, required=True,
                        help="Device Username (netconf agent username)")
    parser.add_argument('-p', '--password', type=str, required=True,
                        help="Device Password (netconf agent password)")
    parser.add_argument('--port', type=int, default=830,
                        help="Netconf agent port")
    args = parser.parse_args()

    # connect to netconf agent
    with manager.connect(host=args.host,
                         port=args.port,
                         username=args.username,
                         password=args.password,
                         timeout=90,
                         hostkey_verify=False,
                         device_params={'name': 'csr'}) as m:
        # execute netconf operation
        try:
            default_rpc = ET.Element(qualify('default', 'http://cisco.com/ns/yang/Cisco-IOS-XE-rpc'))
            ET.SubElement(default_rpc, qualify('interface', 'http://cisco.com/ns/yang/Cisco-IOS-XE-rpc')).text = 'GigabitEthernet3'
            response = m.dispatch(default_rpc)
            print('RPC invoked successfully!')
        except RPCError as e:
            data = e._raw

What about telemetry support with ncclient? Although it does NETCONF notifications, it does not yet have support for telemetry technologies like the IETF YANG Push. However, there are forks of ncclient’s code available that support this. Einar Nilsen-Nygaard, who shared his thoughts on tooling in Chapter 7, created one of these forks that offers YANG Push support. This ncclient fork⁹ supports all the other operations that the stock ncclient supports, plus adds additional capabilities for streaming telemetry. Since YANG Push is, as of this writing, still in development at the IETF, Einar’s modifications have not yet made it into the upstream. Einar also tracks the upstream ncclient distribution in his fork, so his can be used as a drop-in replacement for ncclient in the meantime; then, when YANG Push is ratified, the upstream distribution will have the same support. Example 9-13 subscribes to streaming telemetry memory stats from a device and prints the results it receives. The stream refreshes every second. A sample of the results are shown in Example 9-14.

import lxml.etree as ET
from argparse import ArgumentParser
from ncclient import manager
from ncclient.operations import RPCError
import time

def yp_cb(notif):
    data = ET.fromstring(notif.xml)
    print(ET.tostring(data, pretty_print=True))

def err_cb(e):
    print(e)

if __name__ == '__main__':

    parser = ArgumentParser(description='Usage:')

    # script arguments
    parser.add_argument('-a', '--host', type=str, required=True,
                        help="Device IP address or Hostname")
    parser.add_argument('-u', '--username', type=str, required=True,
                        help="Device Username (netconf agent username)")
    parser.add_argument('-p', '--password', type=str, required=True,
                        help="Device Password (netconf agent password)")
    parser.add_argument('--port', type=int, default=830,
                        help="Netconf agent port")
    args = parser.parse_args()

    # connect to netconf agent
    with manager.connect(host=args.host,
                         port=args.port,
                         username=args.username,
                         password=args.password,
                         timeout=90,
                         hostkey_verify=False,
                         device_params={'name': 'csr'}) as m:

        # execute netconf operation
        try:
            response = m.establish_subscription(yp_cb, err_cb, '/memory-statistics',   1000).xml
            data = ET.fromstring(response)
        except RPCError as e:
            data = e._raw

        # beautify output
        print(ET.tostring(data, pretty_print=True))
        while True:
            time.sleep(1)

<notification xmlns="urn:ietf:params:xml:ns:netconf:notification:1.0">
  <eventTime>2018-12-20T14:09:12.74Z</eventTime>
  <push-update xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-push">
    <subscription-id>2147483653</subscription-id>
    <datastore-contents-xml>
      <memory-statistics xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-memory-oper">
        <memory-statistic>
          <name>Processor</name>
          <total-memory>2450272320</total-memory>
          <used-memory>337016136</used-memory>
          <free-memory>2113256184</free-memory>
          <lowest-usage>2111832096</lowest-usage>
          <highest-usage>1474310140</highest-usage>
        </memory-statistic>
        <memory-statistic>
          <name>lsmpi_io</name>
          <total-memory>3149400</total-memory>
          <used-memory>3148576</used-memory>
          <free-memory>824</free-memory>
          <lowest-usage>824</lowest-usage>
          <highest-usage>412</highest-usage>
        </memory-statistic>
      </memory-statistics>
    </datastore-contents-xml>
  </push-update>
</notification>

NETCONF Clients and Servers with Libnetconf2

Libnetconf2¹⁰ is a C library alternative to ncclient that provides embedded NETCONF capabilities for your applications. Besides the language, another major difference between ncclient and libnetconf2 is that libnetconf2 provides support for adding both NETCONF client and server capabilities. That means you can use libnetconf2 like you did ncclient to connect to devices that run NETCONF servers and be able to perform <edit-config>, <get-config>, and <get> RPCs, for example. You can also use libnetconf2 to develop your own embedded NETCONF servers. Example 9-15 is a short libnetconf2 program that prints the XML contents of the OpenConfig /network-instances path from a device via a <get> RPC.

#include <nc_client.h>
#include <libyang/libyang.h>
#include <stdio.h>

int
main(int argc, char **argv) {
  struct nc_session *session;
  uint64_t msgid;
  struct nc_rpc *rpc;
  struct nc_reply *reply;
  struct nc_reply_data *data_rpl;

  nc_client_ssh_set_username("netop");
  session = nc_connect_ssh("192.168.10.48", 830, NULL);
  rpc = nc_rpc_get("/network-instances", 0, NC_PARAMTYPE_CONST);

  nc_send_rpc(session, rpc, 1000, &msgid);
  nc_recv_reply(session, rpc, msgid, 20000, LYD_OPT_DESTRUCT | LYD_OPT_NOSIBLINGS, &reply);

  data_rpl = (struct nc_reply_data *)reply;

  lyd_print_file(stdout, data_rpl->data, LYD_XML, LYP_WITHSIBLINGS);

  return 0;

}

In fact, the Netopeer2 suite, discussed previously, uses libnetconf2 to provide its NETCONF server capabilities. The Netopeer2 suite is designed to act as both a framework for building NETCONF tools and as a management system for local Linux devices. Through libnetconf2, Netopeer2 uses libyang for YANG loading, parsing, and validation. For its data store implementation, it relies on sysrepo, a YANG-native data store package.

In addition to base NETCONF over SSH, libnetconf2 supports NETCONF over Transport Layer Security (TLS) with Domain Name Security (DNS SEC) SSH key fingerprints. It also supports the fairly recent NETCONF Call Home spec, as well as does NETCONF event notifications. Its client API is designed to be very simple for performing basic RPCs, and it also provides an extensible framework to add in additional RPCs. Because it is a C library, it can be combined with frameworks such as SWIG¹¹ or various foreign function interfaces (FFIs) to provide bindings for other languages. Currently, a native Python binding is in the works as part of the libnetconf2 distribution.

Interacting with RESTCONF Servers

A compelling attribute of RESTCONF is that it behaves similarly to other HTTP-based RESTful APIs. Therefore, the libraries you use to interact with those APIs or those HTTP-based services also work for RESTCONF. Just about all modern languages have some mechanism to interact with HTTP-based services, and those same mechanisms work with REST-based APIs and RESTCONF. Some of these frameworks and libraries are shown in Table 9-1.

Table 9-1 REST Packages for Various Languages

YDK

YDK, or the YANG Development Kit,¹² enables your applications to interact with YANG data models in a more application-native way. YDK is an open source project that allows you to take YANG modules and convert them into language-native packages so you can load them into your programs and applications just like you would any other library or module.

Currently, YDK supports Python, C++, and Golang package generation. YDK works by taking the YANG modules, parsing them, and converting them into per-language code packages. Those code packages become loadable modules or libraries within your applications (for example, for Python, you import the resulting modules; for C++, the code is compiled to libraries to which you link, and so on). So, whereas before you were creating an XML-encoded blob in order to perform an <edit-config> operation on the device, now you load, for example, a Python module that has an object-oriented representation of that configuration that you want to change. Example 9-16 shows the XML needed to create a new interface using the openconfig-interfaces.yang module. Example 9-17 shows the same thing, but using the YDK-generated openconfig_interfaces module.

<interfaces xmlns="http://openconfig.net/yang/interfaces">
  <interface>
    <name>Loopback0</name>
    <config>
      <name>Loopback0</name>
      <type xmlns:ianaift="urn:ietf:params:xml:ns:yang:iana-if-type">ianaift:softwareLoopback</type>
    </config>
  </interface>
</interfaces>

intf = ydk.models.openconfig.openconfig_interfaces.Interfaces.Interface()
intf.name = 'Loopback0'
intf.config.name = intf.name
intf.config.type = ydk.models.ietf.iana_if_type.SoftwareLoopback()

Shorter, yes—and more Pythonic!

As shown in Example 9-17, once you instantiate an object representing the class of entity on which you want to operate, you then set properties on that object. When you’re ready to interact with the device, YDK provides an abstraction layer for transmitting the object and receiving the results. This abstraction layer is flexible so that it can abstract multiple transport protocols and encodings. This includes NETCONF, gRPC Network Management Interface (gNMI), and RESTCONF protocols as well as XML, JavaScript Object Notation (JSON), and protobuf encodings. Replies from the device are also handled by this abstraction layer and converted from the raw encoding back into object instantiations. Therefore, in the case of a <get>, <get-config>, or telemetry publication where the device responds with data to the application, YDK presents this data as an object following the YANG-modeled API. Example 9-18 shows a complete YDK script that configures an interface with an IP address and then reads and prints statistics from this interface.

 1    #!/usr/bin/env python
 2
 3    from ydk.services import CRUDService
 4    from ydk.providers import NetconfServiceProvider
 5
 6    from ydk.models.openconfig.openconfig_interfaces import Interfaces
 7    from ydk.errors import YError
 8
 9
10    def print_stats(**kwargs):
11        if_filter = Interfaces()
12        interfaces = kwargs['service'].read(kwargs['provider'], if_filter)
13        for interface in interfaces.interface:
14            if interface.name == kwargs['intf']:
15                if interface.state.counters is not None:
16                    print('Stats for interface {}:'.format(kwargs['intf']))
17                    print('        in_unicast_pkts : {}'.format(
18                        interface.state.counters.in_unicast_pkts))
19                    print('        in_octets : {}'.format(
20                        interface.state.counters.in_octets))
21                    print('        out_unicast_pkts : {}'.format(
22                        interface.state.counters.out_unicast_pkts))
23                    print('        out_octets : {}'.format(
24                        interface.state.counters.out_octets))
25                    print('        in_multicast_pkts : {}'.format(
26                        interface.state.counters.in_multicast_pkts))
27                    print('        in_broadcast_pkts : {}'.format(
28                        interface.state.counters.in_broadcast_pkts))
29                    print('        out_multicast_pkts : {}'.format(
30                        interface.state.counters.out_multicast_pkts))
31                    print('        out_broadcast_pkts : {}'.format(
32                        interface.state.counters.out_broadcast_pkts))
33                    print('        out_discards : {}'.format(
34                        interface.state.counters.out_discards))
35                    print('        in_discards : {}'.format(
36                        interface.state.counters.in_discards))
37                    print('        in_unknown_protos : {}'.format(
38                        interface.state.counters.in_unknown_protos))
39                    print('        in_errors : {}'.format(
40                        interface.state.counters.in_errors))
41                    print('        out_errors : {}'.format(
42                        interface.state.counters.out_errors))
43
44
45    def add_ip_to_intf(**kwargs):
46
47        interface = Interfaces.Interface()
48        interface.name = kwargs['intf']
49        subinterface = Interfaces.Interface.Subinterfaces.Subinterface()
50        subinterface.index = 0
51        addr = 
            Interfaces.Interface.Subinterfaces.Subinterface.Ipv4.Addresses.Address()
52        addr.ip = kwargs['ip']
53        addr.config.ip = kwargs['ip']
54        addr.config.prefix_length = kwargs['prefixlen']
55        subinterface.ipv4.addresses.address.append(addr)
56        interface.subinterfaces.subinterface.append(subinterface)
57
58        try:
59            kwargs['service'].update(kwargs['provider'], interface)
60        except YError as ye:
61            print('An error occurred adding the IP to the interface: {}'. format(ye))
62
63
64    if __name__ == '__main__':
65
66        provider = NetconfServiceProvider(
67            address='192.168.10.48', port=830, protocol='ssh',
              username='netops', password='netops')
68        cruds = CRUDService()
69        add_ip_to_intf(ip='192.168.20.48', prefixlen=24,
70                       intf='GigabitEthernet2', service=cruds, provider=provider)
71        print_stats(intf='GigabitEthernet2', service=cruds, provider=provider)

This example demonstrates the object-oriented approach to programming with YANG as well as the abstraction YDK provides. The abstraction is seen in lines 12 and 59. Here, the CRUDService (Create, Read, Update, Delete) handles sending the necessary RPC details to the device via the NetconfServiceProvider. As part of the initial session establishment process, YDK learns the network element’s capabilities and determines how to perform the CRUD operations. That is, if the device supports a writeable running configuration, YDK build the <edit-config> RPC at line 59 with the correct “running” target.

Lines 47 through 56 show how the openconfig-interafce.yang and openconfig-if-ip.yang modules translate to their API equivalents. The values set in these properties are checked by YDK before being sent to the network element since YDK understands the models’ structures and constraints. YDK reports any errors caused by illegal values via Python exceptions. The following snippet shows the YModelError that is thrown if you accidentally use a string instead of an integer for prefix_length in line 54:

Click here to view code image

ydk.errors.YModelError: Invalid value ass for 'prefix_length'. Got type: 'str'.
Expected types: 'int'

YDK includes a number of prebuilt packages or bundles of models—that is, modules that are already converted from their YANG counterparts into Python, C++, or Golang libraries. These include an IETF bundle, an OpenConfig bundle, and a Cisco IOS XE and IOS XR bundle. You can choose to install only the bundles you need. The IETF bundle is required at a minimum.

These bundles may not be sufficient for the application you are building, however. If there are native modules for a particular vendor’s platform that you are using—or if you’ve created your own service models—and you want to compile them into language-native packages, YDK offers a tool called the ydk-gen¹³ (or YDK Generator) that allows you to take any module or any set of modules and convert them into the same native language bindings that are included with the YDK distribution. Ydk-gen is a Python application and comes either standalone or as a Docker container, so it is easy to use, even if you don’t need it too often. API bundles are formally defined by creating JSON files that specify the bundle name (this is the model package or namespace to load), metadata about the bundle, and the module or modules to convert. Modules come from the local file system or from specific git repositories. Example 9-19 shows a sample bundle definition file that converts a single local YANG module. The definition for the IETF bundle is found at https://github.com/CiscoDevNet/ydk-gen/blob/master/profiles/bundles/ietf_0_1_1.json.

{
    "name":"yangcatalog",
    "version": "0.1.0",
    "ydk_version": "0.8.0",
    "author": "YangCatalog.org",
    "copyright": "YangCatalog.org",
    "description": "YANG Catalog API model",
    "models": {
        "file": [
            "/models/yc.o/yangcatalog.yang"
        ]
    }
}

Once the bundle definitions are created, the generate.py command turns them into a package for the desired language. Taking the yangcatalog.json file shown in Example 9-18, Python bindings are generated and installed with the commands shown in the following snippet:

Click here to view code image

$ generate.py --python --bundle /bundles/yangcatalog.json
$ pip install gen-api/python/yangcatalog-bundle/dist/ydk*.tar.gz

Now you can begin using your YANG-modeled APIs in your application. In order to build more familiarity with the flow of developing with generated APIs, it helps to have good examples. The YDK distribution includes a number of samples that span different model bundles. You can typically find an example to use to see how to work with various data types and abstraction services. If you need more help, there is an active YDK community¹⁴ where other network developers will help you out.

Pyangbind

Pyangbind¹⁵ is another framework that generates native language bindings directly from YANG modules. Pyangbind is a pyang display plug-in that takes a YANG module or set of YANG modules and outputs a Python library file that represents those YANG module definitions. Functionally, pyangbind is like the “tree” display plug-in for pyang, except instead of generating an ASCII tree structure from the YANG module, it generates Python code.

Whereas YDK includes support for the Python, C++, and Golang languages, plus the abstraction layer for interacting with devices, pyangbind only produces Python bindings for the YANG modules, and it does not include the encoding/decoding and transport abstraction layers. While you still don’t need to interact with the XML or JSON-encoded data, you still require a package (such as ncclient) in order to provide the device interaction piece. This means that pyangbind is a lighter-weight framework for working with object-oriented YANG within your applications. If you’re only writing Python applications, pyangbind might be the right fit for your object-oriented YANG API solution.

Before using pyangbind in your applications, you must first run it against the YANG module or modules you want to convert to their Python equivalents. This is similar to how you use any other pyang display plug-in, except that pyangbind does not, by default, install itself into the pyang plugins subdirectory. The following snippet shows how to execute pyangbind against the openconfig-interfaces.yang and openconfig-if-ip.yang modules:

Click here to view code image

$ pyang --plugindir /local/lib/pyangbind -f pyangbind -o oc_if.py
/modules/oc/openconfig-interfaces.yang /modules/oc/openconfig-if-ip.yang

This creates an oc_if.py file that contains all the binding definitions for openconfig-interfaces.yang and openconfig-if-ip.yang. This module can then be used to build the payload for various NETCONF or RESTCONF operations. Those payloads can be used by ncclient or the requests module for sending the data directly to the device. Example 9-20 shows how to build an RPC payload using pyangbind and then using ncclient to add an IP address to an interface.

 1    #!/usr/bin/env python
 2
 3    from oc_if import openconfig_interfaces
 4    from pyangbind.lib.serialise import pybindIETFXMLEncoder
 5    from ncclient import manager
 6
 7    def send_to_device(**kwargs):
 8        rpc_body = '<config>' +
                     pybindIETFXMLEncoder.serialise(kwargs['py_obj']) + '</config>'
 9        with manager.connect_ssh(host=kwargs['dev'], port=830,
               username=kwargs['user'],
               password=kwargs['password'], hostkey_verify=False) as m:
10            try:
11                m.edit_config(target='running', config=rpc_body)
12                print('Successfully configured IP on {}'.format(kwargs['dev']))
13            except Exception as e:
14                print('Failed to configure interface: {}'.format(e))
15
16    if __name__ == '__main__':
17
18        ocif = openconfig_interfaces()
19        intfs = ocif.interfaces
20
21        intfs.interface.add('GigabitEthernet2')
22        intf = intfs.interface['GigabitEthernet2']
23        intf.subinterfaces.subinterface.add(0)
24        sintf = intf.subinterfaces.subinterface[0]
25        sintf.ipv4.addresses.address.add('192.168.20.48')
26        ip = sintf.ipv4.addresses.address['192.168.20.48']
27        ip.config.ip = '192.168.20.48'
28        ip.config.prefix_length = 24
29
30        send_to_device(dev='192.168.10.48', user='netops',
                         password='netops', py_obj=intfs)

Lines 18 through 28 create an object representing the target interface to which the IPv4 address is added. Pyangbind includes serialization methods to turn those objects into XML or JSON-encoded data. Line 8 turns the interface object into XML using IETF rules and wraps it in a <config> element to use with ncclient.

In addition to serialization, pyangbind takes JSON or XML data from the device and deserializes that back into a Python object. Example 9-21 uses requests to gather interface stats as a JSON object, passes that object to pyangbind’s JSON deserializer, and then prints various fields. All this happens without needing to interact with any JSON data directly.

 1    import requests
 2    from pyangbind.lib import pybindJSON
 3    import oc_if
 4
 5
 6    def print_stats(response):
 7        py_obj = pybindJSON.load_ietf(
 8            response.text, oc_if, 'openconfig_interfaces')
 9        for index, intf in py_obj.interfaces.interface.iteritems():
10            if intf.name == 'GigabitEthernet2':
11                print('Bytes out : {}'.format(intf.state.counters.out_octets))
12                print('Bytes in  : {}'.format(intf.state.counters.in_octets))
13
14
15    if __name__ == '__main__':
16
17        url = 'https://192.168.10.48/restconf/data/openconfig-interfaces:interfaces'
18
19        headers = {
20            'Accept': 'application/yang-data+json',
21            'Authorization': 'Basic bmV0b3BzOm5ldG9wcw=='
22        }
23
24        response = requests.request('GET', url, headers=headers, verify=False)
25
26        print_stats(response)

Line 7 takes the raw response text as JSON and converts it into an instance of the openconfig_interfaces class. This object can be printed, manipulated, and sent back to the device, all without you needing to deal with the raw encoded data.

Q&A with Radek Krejˇcí

Radek Krejčí is a researcher at CESNET, operator of the Czech national e-infrastructure for science, research, and education. He is an experienced developer and architect of various tools for network security monitoring, but mainly for network configuration. He is the author of the Netopeer project’s open source NETCONF protocol implementation, which started as his bachelor’s thesis and evolved into a toolset providing generic YANG and NETCONF libraries, a (YANG-based) validated data store, as well as complete NETCONF server and client applications.

Question:

What do you see is most needed in terms of YANG libraries and SDKs when it comes to making it easy to realize the full value of the abstraction data model–driven management that YANG promises?

Answer:

As for any library or SDK, it is the combination of a fine API and documentation that makes it successful. It must provide developers an easy and straightforward way to do exactly what they want or need. And because of YANG’s complexity, there are many use cases, so it is quite a challenging task. The API must hide the complexity of YANG while still enabling even rare use cases.

Question:

What advice would you give to application developers who want to integrate YANG-modeled data into their Operations Support Systems (OSS) applications?

Answer:

Just do not reinvent the wheel, and try to use libraries and frameworks such as libyang and libnetconf2. YANG is very friendly to module authors, but less friendly to the tool developers. You already did the most important step, but just the first step, when you decided to use YANG to describe your data. Now, you are supposed to decide how much work it will be for you.