Setting up our templating system

We are going to use Markdown for the various content we want to host, but there are going to be certain sections of our files that we will want to use across all of the articles. These will be things such as the header, footer, and the sidebar of our pages. Instead of having to have these inserted into all of the Markdown files that we will create for our articles, we can template these in.

We will put these sections in a folder that will be known at runtime, by using the templateDirectory variable that we declared. This will also allow users of our package to change out the look and feel of our static site server without having to do anything too crazy. Let's go ahead and create the directory structure for the template section. This should look like the following:

  • Template: Where we should look for the static content across all pages
  • HTML: Where all of our static HTML code will go
  • CSS: Where our stylesheets will live

With this directory structure, we can now create some basic header, footer, and sidebar HTML files, and also some basic Cascading Style Sheets (CSS), to get a page structure that should be familiar to everyone. So, let's begin, as follows:

  1. We will write the header HTML, like so:
<header>
    <h1>Our Website</h1>
    <nav>
        <a href="/all">All Articles</a>
        <a href="/contact">Contact Us</a>
        <a href="/about">About Us</a>
    </nav>
</header>

With this basic structure, we have the name of our website, and then a couple of links that most blog sites will have.

  1. Next, let's create the footer section, like this:
<footer>
    <p>Created by: Me</p>
    <p>Contact: <a href="mailto:[email protected]">Me</a></p>
</footer>
  1. Again, fairly self-explanatory. Finally, we will create the sidebar, as follows:
<nav>
    <% loop 5
    <a href="article/${location}">${name}</a>
    %>
</nav>

This is where our templating engine comes into play. First, we are going to use <% %> character pattern to denote that we want to replace this with some static content. Next, loop <number> will let our templating engine know that we plan on looping through the next piece of content a certain amount of times before stopping the engine. Finally, the <a href="article/${location}">${name}</a> pattern will tell our templating engine that this is the content we want to put in, but we will want to replace the ${} tags with variables in an object that we pass in our code.

Next, let's go ahead and create the basic CSS for our page, as follows:

*, html {
    margin : 0;
    padding : 0;
}
:root {
   --main-color : "#003A21"; 
   --text-color : "#efefef";
}
/* header styles */
header {
    background : var(--main-color);
    color      : var(--text-color);
}
/* Footer styles */
footer {
    background : var(--main-color);
    color  : var(--text-color);
}

Most of the CSS file has been cut since a majority of it is boilerplate code. The only piece that is worth mentioning is the custom variables. With CSS, we can declare our own custom variables by using the pattern --<name> : <content>, and then we can use it later in the CSS file by using the var() declaration. This allows us to reuse variables such as colors and heights without having to use a preprocessor such as Syntactically Awesome Style Sheets (SASS).

CSS variables are scoped. This means if you define the variable for the header section, it will only be available in the header section. This is why we decided to put our colors at the :root pseudo element level since it will be available across our entire page. Just remember that CSS variables have scope similar to the let and const variables we declare in JavaScript.

With our CSS laid out, we can now write our main HTML file in our template file. We will move this outside of the HTML folder since this is the main file that we will want in order to put everything together. It will also let users of our package know that this is the main file that we will use to put together all of the pieces and that if they want to change it up, they should do it here. For now, let's create a main.html file that looks like the following:

<!DOCTYPE html>
<html>
    <head>
        <link rel="stylesheet"  type="text/css" href="css/main.css" />
    </head>
    <body>
        <% from html header %>
        <% from html sidebar %>
        <% from html footer %>
    </body>
</html>

The top section should look familiar, but we now have a new template type. The from directive lets us know that we are going to source this file from somewhere else. The next statement says it is an HTML file, so we will look inside the HTML folder. Finally, we see the name of the file, so we know that we want to bring in the header.html file.

With all of this, we can now write the templating system that we are going to use to build our pages. We will be implementing our templating system utilizing a Transform stream. While we could utilize something like a Writable stream, it makes more sense to utilize a Transform stream since we are changing the output based on some input criteria.

To implement the Transform stream, we will need to keep track of a few things so, that way, we can process our keys correctly. First, let's get us reading and sending off the proper chunks to be processed. We can do this by implementing the transform method and spitting out the chunks that we are going to replace. To do this, we will do the following:

  1. We will extend a Transform stream and set up the basic structure, as we did in Chapter 7, Streams – Understanding Streams and Non-Blocking I/O. We will also create a custom class to hold the start and end locations of a buffer. This will allow us to know if we got the start of the pattern matcher in the same loop. We will need this later. We will also set up some private variables for our class, such as the begin and end template buffers, on top of state variables such as the #pattern variable, as follows:
import { Transform } from 'stream'
class Pair {
    start = -1
    end = -1
}
export default class TemplateBuilder extends Transform {
    #pattern = []
    #pair = new Pair()
    #beforePattern = Buffer.from("<%")
    #afterPattern = Buffer.from("%>")
    constructor(opts={}) {
        super(opts);
    }
    _transform(chunk, encoding, cb) {
        // process data
    }
}
  1. Next, we will have to check if we have data held in our #pattern state variable. If we do not, then we know to look for the beginning of a template. Once we do the check for the beginning of a template statement, we can check to see if it is actually in this chunk of data. If it is, we set the start property of #pair to this location so our loop can keep going; otherwise, we have no template in this chunk and we can start to process the next chunk, as shown here:
// inside the _transform function
if(!this.#pattern.length && !this.#pair.start) {
    location = chunk.indexOf(this.#beforePattern, location);
    if( location !== -1 ) {
        this.#pair.start = location;
        location += 2;
    } else {
        return cb();
   }
}
  1. To handle the other condition (we are looking for the end of a template), we have quite a bit more state to deal with. First, if our #pair variable's start is not -1 (we set it), we know we are still processing a current chunk. This means we need to check if we can find the end template buffer in the current chunk. If we do, then we can process the pattern and reset our #pair variable. Otherwise, we just push the current chunk from the start member location of #pair to our #pattern holder at the end of the chunk, as follows:
if( this.#pair.start !== -1 ) {
    location = chunk.indexOf(this.#afterPattern, location);
    if( location !== -1 ) {
        this.#pair.end = location;
        this.push(processPattern(chunk.slice(this.#pair.start,this.#pair.end)));
        this.#pair = new Pair();
    } else {
        this.#pattern.push(chunk.slice(this.#pair.start));
    }
}
  1. Finally, if start member of #pair is set, we check for the end template pattern. If we do not find it, we just push the entire chunk to the #pattern array. If we do find it, we slice the chunk from the beginning of it to where we found our end template string. We then concatenate all this together and process it. We will then also reset our #pattern variable back to holding nothing, like this:
location = chunk.indexOf(this.#afterPattern, location);
if( location !== -1 ) {
    this.#pattern.push(chunk.slice(0, location));
    this.push(processPattern(Buffer.concat(this.#pattern)));
    this.#pattern = [];
} else {
    this.#pattern.push(chunk);
}
  1. All of this will be wrapped in a do/while loop since we want to run this piece of code at least once, and we will know that we are finished when our location variable is -1 (this is what is returned from an indexOf check when it does not find what we want). After the do/while loop, we run the callback to tell our stream that we are ready to process more data, as follows:
do {
  // transformation code
} while( location !== -1 );
cb();

With all of this put together, we now have a transform loop that should handle almost all of the conditions to grab our templating system. We can test this by passing in our main.html file and putting the following code inside of our processPattern method, like this:

console.log(pattern.toString('utf8'));

  1. We can create a test script to run our main.html file through. Go ahead and create a test.js file and put the following code in it:
import TemplateStream from './template.js';
const file = fs.createReadStream('./template/main.html');
const tStream = new TemplateStream();
file.pipe(tStream);

With this, we should get a nice printout with the template syntax that we were looking for, such as from html header. If we ran our sidebar.html file through it, it should look something like the following:

loop 5
    <a href="article"/${location}">${name}</a>

Now that we know our Transform stream's template-finding code works, we just need to write our process chunk system to handle the preceding cases we have.

To now process the chunks, we will need to know where to look for the files. Remember from before, when we declared various variables inside of our package.json file? Now, we will utilize the templateDirectory one. Let's go ahead and pass that in as an argument for our stream, like so:

#template = null
constructor(opts={}) {
    if( opts.templateDirectory ) {
        this.#template = opts.templateDirectory;
    }
    super(opts);
}

Now, when we call processPattern, we can pass in the chunk and the template directory as arguments. From here, we can now implement the processPattern method. We will handle two cases: when we find a for loop and when we find a find statement.

To process a for loop and a find statement, we will proceed as follows:

  1. We will build out an array of buffers that will be what the template held, other than the for loop. We can do this with the following code:
const _process = pattern.toString('utf8').trim();
const LOOP = "loop";
const FIND = "from";
const breakdown = _process.split(' ');
switch(breakdown[0]) {
    case LOOP:
        const num = parseInt(breakdown[1]);
        const bufs = new Array(num);
        for(let i = 0; i < num; i++) {             
           bufs[i] = Buffer.from(breakdown.slice(2).join(''));
        }
        break;
   case FIND:
        console.log('we have a find loop', breakdown);
        break;
   default:
        return new Error("No keyword found for processing! " + 
         breakdown[0]);
}
  1. We will look for the loop directive and then take the second argument, which should be a number. If we print this out, we will see that we have buffers that are all filled with the same exact data.
  2. Next, we will need to make sure that we are filling in all of the templated string locations. These look like the pattern ${<name>}. To do this, we will add another argument to this loop that will give the name of the variable we want to use. Let's go ahead and add this to the sidebar.html file, as follows:
<% loop 5 articles
    <a href="article/${location}">${name}</a>
%>
  1. With this, we should now pass in a list of variables that we are going to want to use for our templating system—in this case, one named articles that is an array of objects that have a location and name key. This could look like the following:
const tStream = new TemplateStream({
    templateDirectory,
    templateVariables : {
        sidebar : [
            {
                location : temp1,
                name     : 'article 1'
            }
        ]
    }
}

With enough to satisfy our for loop condition, we can now head back to the Transform stream and add this as an item we will process in our constructor, and send it to our processPattern method. Once we have added these items here, we will update our loop case with the following code inside of the for loop:

const num = parseInt(breakdown[1]);
const bufs = new Array(num);
const varName = breakdown[2].trim();
for(let i = 0; i < num; i++) {
    let temp = breakdown.slice(3).join(' ');
    const replace = /${([0-9a-zA-Z]+)}/
    let results = replace.exec(temp);           
    while( results ) {
        if( vars[varName][i][results[1]] ) {
            temp = temp.replace(results[0], vars[varName][i][results[1]]);
        }
       results = replace.exec(temp);                
    }
    bufs[i] = Buffer.from(temp);
}
return Buffer.concat(bufs);

Our temporary string holds all of the data that we consider a template, and the varName variable tells us where to look in our object that we pass into processPattern to do our replacement strategy. Next, we will use a regular expression to pull out the name of the variable. This specific regular expression says to look for the ${<name>} pattern while also saying to capture whatever is in the <name> section. This allows us to easily get to the name of the variable. We will also keep looping through the template to see if there are more regular expressions that pass these criteria. Finally, we will replace that templated code with the variable we have stored.

Once all of this is done, we will concatenate all of these buffers together and return all of them. That is a lot for that piece of code; luckily, the from section of our template is quite a bit easier to handle. The from section of our templating code will just look for a file with that name from our templateDirectory variable and will return the buffer form of it.

It should look something like the following:

case FIND: {
    const type = breakdown[1];
    const HTML = 'html';
    const CSS  = 'css';
    if(!(type === HTML || type === CSS)) return new Error("This is not a
     valid template type! " + breakdown[1]);
    return fs.readFileSync(path.join(templateDirectory, type, `${breakdown[2]}.${type}`));
}

We first grab the type of file from the second argument. If it is not an HTML or CSS file, we will reject it. Otherwise, we will try reading the file in and sending it to our stream.

Some of you may be wondering how we will handle the templating in the other files. Right now, if we run our system on the main.html file, we will get all the separate chunks, but our sidebar.html file is not filled out. This is one weakness of our templating system. One way around this is to create another function that will call our Transform stream a certain amount of times. This will make sure we get the templating done for these separate pieces. Let's go ahead and create that function right now.

This is not the only way to handle this. Instead, we could utilize another system: when we see template directives in a file, we add that buffer to the list of items needed for processing. This would allow our stream to process the directives instead of looping through the buffers again and again. This leads to its own problems since someone could write an infinitely recursing template and would cause our stream to break. Everything is a trade-off, and right now, we are going for ease of coding over ease of use.

First, we will need to import the once function from the events module and the PassThrough stream from the stream module. Let's update those dependencies now, like this:

import { Transform, PassThrough } from 'stream'
import { once } from 'events'

Next, we will create a new Transform stream that will bring in the same information as before, but now, we will also add in a loop counter. We will also respond to the transform event and push it onto a private variable until we have read in the entire starting template, as follows:

export class LoopingStream extends Transform {
    #numberOfRolls = 1
    #data = []
    #dir = null
    #vars = null
    constructor(opts={}) {
        super(opts);
        if( 'loopAmount' in opts ) {
            this.#numberOfRolls = opts.loopAmount
        }
        if( opts.vars ) {
            this.#vars = opts.vars;
        }
        if( opts.dir) {
            this.#dir = opts.dir;
        }
    }
    _transform(chunk, encoding, cb) {
        this.#data.push(chunk);
        cb();
    }
    _flush(cb) {
    }
}

Next, we will make our flush event async since we will utilize an async for loop, like so:

async _flush(cb) {
    let tData = Buffer.concat(this.#data);
    let tempBuf = [];
    for(let i = 0; i < this.#numberOfRolls; i++) {
        const passThrough = new PassThrough();
        const templateBuilder = new TemplateBuilder({ templateDirectory :
        this.#dir, templateVariables : this.#vars });
        passThrough.pipe(templateBuilder);
        templateBuilder.on('data', (data) => {
            tempBuf.push(data);
        });
        passThrough.end(tData);
        await once(templateBuilder, 'end');
        tData = Buffer.concat(tempBuf);
        tempBuf = [];
    }
    this.push(tData);
    cb();
}

Essentially, we will put all of the initial template data together. Then, we will run this data through our TemplateBuilder, building a new template for it to run over. We utilize the await once(templateBuilder, ‘end') system to let us treat this code synchronously. Once we have gone through the counter, we will spit out the data.

We can test this with our old test harness. Let's go ahead and set it up to utilize our new Transform stream, along with spitting the data out to a file, as follows:

const file = fs.createReadStream('./template/main.html');
const testOut = fs.createWriteStream('test.html');
const tStream = new LoopingStream({
    dir : templateDirectory,
    vars : { //removed for simplicity sake },
    loopAmount : 2
});
file.pipe(tStream).pipe(testOut);

If we now run this, we will notice that the test.html file holds our fully built-out template file! We now have a functioning template system we can use. Let's hook this up to our server.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.17.174.156