11.1.1. Quick presentation of the Quandl API

The directory https://www.quandl.com/api/v3/datasets/ gives direct access to JSON formatted data with the following command:

GET
https://www.quandl.com/api/v3/datasets/{db_code}/{data_code}.json

The JSON data structure is normalized as in this example (excerpts):

{
  "dataset": {
    "name":"World Top Incomes - Top 0.5-0.1% income share - United States",
    "description":"Units=%. Fractiles: total income excl. capital gains",
    "column_names": [ "Date", "Value" ],
    "frequency": "annual",
    "start_date": "1913-12-31",
    "end_date": "2015-12-31",
    "data": [ [ "2015-12-31", 6.29 ], … ]
  }

11.1.2. Processing an example with promises

Knowing that structure and the components of the URL, we are able to write a callback function and call the helperGetJson function (presented in Chapter 10) with the correct arguments (url and no data).

  const db = "PSE/", ds = "USA_1010120401";
  const key = "a_key_provided_by_quandl"; //no key: limit 50 req.
  const url = "https://www.quandl.com/api/v3/datasets/"+db+ds+
  ".json?api_key="+key;
  fetch(url)                            // returns 1st promise
    .then(res => res.json())     // returns 2nd promise
    .then(processQuandl)       // achieves final processing
    .catch(function(err){console.log('Fetch Error:-S', err);});

First, we build the url that refers the particular database (PSE) and dataset that we want to process. The api_key can be omitted but there is a 50 request limit in that case.

Then we use fetch to send the query and return a first promise, which we convert into JSON, and it returns a second promise, which we finally use in this function:

  function processQuandl(response){
    const set = response.dataset;
    const getShare = (t)=> t[1];       // the second array gives %
    const getYear = (t)=> parseInt(t[0].substring(0,4)); // year
    const initCurve = function(title,color){/* code presented */}
    const drawCurve = function(time,val){/* in section graphics */}
    let title = set.name +' [${set.start_date} - ${set.end_date}]';
    let rev = set.data.reverse();    // for ascending time
    initCurve(title, "green");
    drawCurve(rev.map(getYear), rev.map(getShare));
      // then we can overlay some events such as presidency terms
      // putEvents(president, data); // see: 'additional data'
    return res; // we can return a promise, if we want a next 'then'
}

Chronological data that are delivered as JSON can be directly received as JavaScript objects presenting similar properties:

• – metadata: such as a name, a date or a period (start-end dates);
• – data: a 2D array i.e. an array of arrays (all inner arrays containing two elements), t = [[date, value], …].

The array can be used for plotting the curve y = value(date), because the relation between date and value is functional (one value per date). The functions getYear and getShare provide the abscissa and ordinate respectively, by mapping separately the date and the value into two new arrays (method map). Then these arrays are passed in arguments to the function drawCurve, already detailed in chapter 9, and the function initCurve prepares the canvas and initializes some geometric values. The functions initCurve and drawCurve can be replaced by equivalent functions taken from some graphic library but the overall logic of the application is the same.

11.1.2.1. Additional data

The historical data from that example spans over a century, from 1913 to 2015, and concerns the share of GDP owned by the most wealthy people (0.5% uppermost revenues). In order to relate these data to some historical events, we can search for the presidential terms of the President of the United States during that century.

Here are the data:

  const president = [
    ["1913-1921",8,"T. Woodrow Wilson","D"], ["1921-1923",2,"Warren G. Harding","R"],
    ["1923-1929",6,"J. Calvin Coolidge","R"], ["1929-1933",4,"Herbet C. Hoover","R"],
    ["1933-1945",12,"Franklin D. Roosevelt","D"], ["1945-1953",8,"Harry S. Truman","D"],
    ["1953-1961",8,"Dwight D. Eisenhower","R"], ["1961-1963",2,"John F. Kennedy","D"],
    ["1963-1969",6,"Lyndon B. Johnson","D"], ["1969-1974",5,"Richard M. Nixon","R"],
    ["1974-1977",3,"Gerald R. Ford","R"], ["1977-1981",4,"Jimmy Carter","D"],
    ["1981-1989",8,"Ronald Reagan","R"], ["1989-1993",4,"George H. W. Bush","R"],
    ["1993-2001",8,"Bill Clinton","D"], ["2001-2009",8,"George W. Bush","R"],
    ["2009-2017",8,"Barack Obama","D"]
];

This function uses some <div> elements, whose creation and setting are not described here:

  function putEvents(table, datab) {
            // Color according to President's party
    let Dgd = "linear-gradient(to bottom,#99ccff 64%,#0000ff 100%)";
    let Rgd = "linear-gradient(to bottom,#ff99cc 66%,#ff0000 100%)";
    let date0 = parseInt(table[0][0].substring(0,4));
    table.forEach(function(tp, p, tab){
      // display a coloured <div> spanning over the President's term
      div.addEventListener("mouseover",function(e)    {
        // coding data to popup
      });
}

With the help of some CSS rules for coloring the elements, we present here the graphical output.

This kind of application does not require a deep understanding of all the facets of JavaScript. It is within the reach of a secondary school teacher, with the contribution of students and a minimum equipment.

EXERCISE.–

Use the same code with the second example (INSEE demographic data of people 60+ years in France, over the same century). Instead of raw data, display the annual %-change: rev[i] → (rev[i] - rev[i-1])/rev[i-1] (the first value rev[0] cannot be computed).

Then, for each important variation, find which event occurred 60 years earlier; that is easy and impressive.

11.2.1. Plot multiple data series against the same time axis

This is a typical example of the different steps to achieve:

• – extract the data from the source;
• – convert CSV into JSON. In this example, the ECB SDMX 2.1 RESTful web service, operational since 2017, provides an API for accessing the ECB Statistical Data Warehouse;
• – reverse the array (ascending time);
• – convert dates into JavaScript Date objects (goodDate);
• – set labels and format axes;

Here is the code (Ajax request not repeated):

const ecbTGB = {
    "name":"Euro TARGET balance (selected countries)",
    "dataSource": "http://sdw.ecb.europa.eu/browse.do?node=bbn4859",
    "Unit": "[Millions of Euro]",
    "Participants": ["ECB","BE","DE","ES","FR","IT","NL"],
    "data": [
    ["2017Aug",-212940,-22095,852511,-384426,9278,-414164,107511],
    […], … ]
}
const dygrafTable = ecbTGB.data.reverse(); // latest right
function goodDate(x){
      let xx = x[0], y = parseInt(xx);
      x[0] = new Date( xx.split(y)[1]+" 1 "+y );
      return x;
}
dygrafTable = dataTable.map(goodDate);
const dygrafDiv = document.querySelector(".plot");
let title = ecbTGB.name, width = 480, height = 200;
let labels = ['Date'].concat(source.Participants);
let axes = {x:{pixelsPerLabel: 30,
              axisLabelFormatter(x){return x.getFullYear();}
             };
const g = new Dygraph(
      dygrafDiv ,      // where to plot
      dygrafTable,     // data (or url to a CSV  file
      {title, width, height, labels, axes}
  );

11.2.2. Dynamic plot: simulating time evolution

The plot is displayed on the screen in just one frame. We may want to control that display by using as many frames as needed, to simulate an evolution, for example, at the pace of 1 s per year:

• – first, we draw only the first date (2008), as in the previous example;
• – we create a function that adds the next date (push) and we call the updateOptions method of the object Dygraph (it forces frame refresh);
• – we invoke window.setInterval, which repeats that every ms milliseconds;
• – we stop (clearInterval) when the maximal size is reached.

Except updateOptions, which is Dygraph specific, the logic is general to any similar animation (to control the speed of a plot). Here is the code:

const dynamicTable = Array.from([dygrafTable[0]]); // = 2008
const maxlen = dygrafTable.length; // stop test
function dynamicDraw(){
     dynamicTable.push(dygrafTable[dynamicTable.length]);
     if(dynamicTable.length === maxlen)
         clearInterval(window.intervalId);
     g.updateOptions( {'file':dynamicTable} );
}
let ms = 500; // every 500 ms
window.intervalId = setInterval(dynamicDraw, ms);

In the example, the interval of the vertical axis adapts itself to the range of the values read so far. We can notice an important scale change around the year 2010 (after 2.5 s): interval [-150M,200M] becomes [-500M,500M].