Created by István Satmári, last modified on Mai 15, 2017

Introduction

We decided to go with the approach of developing a single page application (SPA), which would communicate with the server via REST application interface. This way the server is lighter and does not have to deal with issues on the client's side. It allows for the server to be (almost) completely isolated, more secure, and puts weight of computing page look solely on the client's computer, which in turn increases the overall throughput of the server. Modern internet browsers make this approach possible on almost every popular platform.

We decided to stick with the basic personal computers as our platform of choice with installed browser(s) that support(s) HTML5, such as Microsoft® Edge 14 (or newer), Mozilla Firefox 50 (or newer) and Google® Chrome 49 (or newer). We used JavaScript language and Angular (https://angularjs.org/) framework for the overall UI development, which would also determine the application architecture. Angular was chosen based on its popularity, good documentation and robustness, providing the desired functionality for almost everything we needed.

Architecture

Angular separates the entire application into several 'views'. It basically maps a route to a template which is then used for display. We usually call these 'views' as 'screens'. Each screen consists of a template, which is a portion of HTML code, and controller, being a piece of JavaScript code that 'controls' the template, providing functionalities for user interaction. During development, our goal was to put most of the code that determines visuals and contents of each screen into templates, while leaving controllers 'clean', providing only the necessary functions requested by templates. This also explains our certain design decisions, some of them will be described later.

Certain sub-parts of views naturally occur more than once in an application. For that reason Angular comes with a notion of directives: a pieces of code (sometimes) with their own templates attached. These may be later reused in views, or other directives. Last, but not least, for implementation of business logic, Angular provides services, which can be used by any controller, or another services.

Folder structure

The project is structured in the following way:

  • font: external font files used by the application
  • graphics: images used by the application
  • less: stylesheets in the CSS/LESS format
  • css: pre-compiled stylesheets from less folder
  • lib: external libraries
  • test: exemplar input files and responses from the server
  • src: application source files
    • common: directives in the role of subcomponents, such as navigation bar, page footer, etc.
    • directives: generic reusable directives, such as modal window, pagination, tooltips, etc.
    • filters: generic custom angular filters
    • services: implementation of business logic, such as REST services, authentication, etc.
    • templates: application screens / views
      • may be structured to subcomponents
    • util: generic utilities

index.html, contained within the root directory, is the application's entry point.

External libraries

As mentioned, for the overall development, Angular framework was used. Additional used libraries include:

  • bootstrap: a popular, comprehensive CSS framework (http://getbootstrap.com/), speeding up overall development, providing basic styling and allowing for easy implementation of responsive pages
  • d3: a library selected for implementation of graphvis component (see Graphvis below)
  • date: provides basic functions for working with dates
  • fontawesome: a popular, icon-based, font for improving overall appearance of the application
  • jquery: better integration with bootstrap as well as speeding up development of directives
  • satellizer: JWT based authentication / authorization library
  • uiboostrap: a collection of several bootstrap modules, packed as directives, for better integration with Angular framework
  • less: interpreter of CSS/LESS stylesheets, used only during development (to optimize the application, the CSS/LESS stylesheets are pre-compiled)
  • papaparse: parser of CSV files; for testing purposes only, when running application without the server

Goals

As mentioned, during development our goal was to keep the application extensible (where possible) while also allowing for easy changes for the common cases. That is why we strived to keep logic and visuals in screens as much as possible separated. Most of the time controllers of screens do actually provide only the functionalities necessary, while leaving visuals solely up to templates of the screens. This may make certain development decisions look illogical to some as demonstrated by the following example, but helping us achieve our goal in the end:

Template:

...
<injector for="msgtxt.configdFailure">An error occurred while trying to download the task's configuration.</injector>
...

Corresponding controller:

...
$scope.messages.push('error', $scope['msgtxt.configdFailure']); 
...

The example relates to a situation when there is an uncatched error that needs to be displayed to a user. While it was possible to declare the message in the controller, by injecting it via template we can keep controllers 'clean' and allow for easy changes in the future.

Application loading

index.html being the entry point of the application references scripts used via <script src="..."> HTML element. All external libraries are loaded this way and the main source file, src/global.js, defines the way of loading all other resources. It looks up src/require.json file for determining what components need to be loaded with an exception of screens, which are then loaded via src/templates/templates.js. Usage of JSON files, such as src/require.json, allows for easier extensibility of the application, requiring programmer only to write what additional components (s)he would like to be loaded, instead of editing the actual code. This adds to clarity and speeds up development. The components (in this case we mean views, directives, services, ...) then may be separated into several files and may consist of other subcomponents. However, loading of those has to be handled by the corresponding component itself.

Loading of screens is a little less straightforward. Screens have to be mapped to a certain route. Additionally it may be useful not to have a custom controller specified for certain screens, only a template, where there is no functionality necessary (take for example home screen displaying basic information about the project, while providing no user interaction at all). For clarity we decided to represent this in src/templates/mappings.json. Each screen is represented with an object:

{
    "route": "/signup/:token",
    "folder": "signupcnf",
    "controller": "odalic-signupcnf-ctrl"
}

while "route" being the route the screen is mapped to, "folder" being the name of a folder the screen is located in (omitting src/templates/) and "controller" being a name of the corresponding controller. The name has to match the one specified in the actual controller definition. Additionally it may be equal to "generic" in which case it is assumed no custom controller is needed and only the template is loaded for the route. (Note that in this case simply an empty controller is created automatically.)

Please note the AngularJS supports a way to map a screen not only to a specific single route, but also to a pattern. In the case of /signup/:token, upon visiting /signup/anything, the mapping still holds. Not only that, we can also retrieve what :token part of the route is equal to (in this case "anything") and specify further action based on this information. Such mechanism is used, for instance, by a sign up confirmation screen, which retrieves token directly from the route (route being visited by a user upon receiving an e-mail to 'confirm the sign up by visiting the following link: http://.../signup/GcOiJUz1...'), sends it to server for evaluation and displays information about the state of sign up process to the user.

For redirecting purposes there is another type of object that may be specified:

{
    "route": "/home",
    "target": "/",
    "controller": "reroute"
}

while "route" being the route the "redirect" is mapped to, "target" being the route to redirect to, and "controller" set to "reroute" (for clarity).

Name of the template has to always be template.html while controller has to be named controller.js. Loading of subcomponents / other files is handled individually.

Screens

The whole application is divided into screens. While some of their elements are the same (e.g. navigation bar, footer, ...), for the application to be as flexible as possible, none of the elements (except header) are hard-coded in the entry point. This allows for easier implementation of special cases, e.g. when the footer on a certain screen is to be different, or no present at all, etc.

In order to not repeat large amount of code throughout the screens, we separated common screen elements into 3 subcomponents:

  • main-cnt: represents a wrapper around 'main content', i.e. the whole screen content needs to be contained within this element.
  • navbar: a configurable navigation bar, has to be put inside main-cnt element.
  • footer: a generic footer same for most of the screens.

That means an ordinary screen template will look like this:

<!-- Main Content -->
<main-cnt>
    <!-- Navigation Bar -->
    <navbar selected="home" lmenu="default-lm.json" rmenu="default-rm.json"></navbar>

    <!-- Content + Sidebar -->
    <div class="container-fluid">
        <!-- Sidebar -->
        <div class="col-sm-3">
			Sidebar content
        </div>

        <!-- Content -->
        <div class="col-sm-9">
            Main content
        </div>
    </div>
</main-cnt>

<!-- Footer -->
<footer/>

Navbar

navbar is a configurable navigation bar, which means on each screen it can be set what items should be available and what item should be highlighted as the selected one. Available items have to be specified as a relative path to a file in JSON format. Specifically, there has to be a file describing menu on the left and a file describing menu on the right on the navigation bar. These two differ in how they work.

Menu on the left is an array of objects of the following format:

  {
    "id": "home",
    "title": "Home",
    "link": "#/",
    "menu": []
  }

The exemplary object describes a single item on the navigation bar. "id" stands for identifier of the item. This can be referenced when describing what item is selected on a current screen. "title" stands for text displayed, "link" for where to redirect upon click and finally "menu" is an array of subitems. If the "menu" is not an empty array, "link" property should be omitted. The "menu" items look like this:

{
    "title": "File list",
    "link": "#/filelist"
}

where "title" stands for text displayed, "link" for where to redirect upon click. If no properties are provided, the item is identified as a separator (visual purposes).

Menu on the right consists of following type of objects:

{
    "id": "signup",
    "title": "Sign up",
    "link": "#/signup",
    "icon": "glyphicon-user",
    "condition": "!$auth.isAuthenticated()"
}

While being for the most part the same as objects in the left menu, there are some differences:

  • Items on the right have an icon attached. See http://getbootstrap.com/components/ reference for allowed icons.
  • Items on the right may not represent a menu of subitems.
  • Items on the right may specify a "condition" property, which determines whether should the item be displayed or not. Note that conditions are evaluated during each Angular digest cycle, therefore complicated conditions may cause performance issues.

Services

ODALIC being a single page application, most of the logic is handled on the server. Data exchanged between client and the server is realized via asynchronous requests (AJAX) and are (mostly) in JSON format. For data exchange a "rest" service (actual name of the service) is implemented, which transforms data sent/received, automatically injects headers required by the API and generally eases the overall work with the server's interface. The service is divided into parts corresponding to the ones described in REST API specification.

To handle generic AJAX requests a "requests" service is implemented. Most of the responses from the server have a standardized format and therefore can be automatically parsed and transformed. The service handling this is injected via "ioc['requests']", where "ioc" is a service being a very simple implementation of IoC (inversion of control) pattern (src/services/ioc/modules.json is the configuration file). The "ioc['requests']" additionally handles the case of unauthorized access to resources (redirecting to log in screen by default).

Authentication and authorization

As described in Authentication and authorization, to ensure security, JWT (JSON Web Tokens) standard is used. For this we used Satellizer library (https://github.com/sahat/satellizer), which automatically signs each AJAX request with an appropriate authorization header and allows for easy token storage on a user's computer. Additionally, the library conveniently handles requests for logging and signing up.

Several screens are associated with the authentication / authorization process:

  • signup: allows users to sign up
  • signupcnf: maps to a route received by a user in an e-mail (the e-mail requesting user's confirmation for signing up); automatically handles additional requests associated with the process and displays notifications about the current state
  • login: allows users to log into or out of the application, re-checks token (to ensure its validity) and displays the current state
  • chngpasswd: allows users to change their password
  • chngpasswdcnf: maps to a route received by a user in an e-mail (the e-mail requesting user's confirmation for changing his/her password); automatically handles additional requests associated with the process and displays notifications about the current state

Directives

While directives were designed to solve many different kinds of issues, there are some patterns we followed during development. We will demonstrate our approach, when designing directives, on an example of a confirmation modal window.

A confirmation modal window is a piece of HTML code consisting of several divisional elements (<div>) with correctly attached pre-defined classes. The classes as well as functionality is provided by the bootstrap framework. Therefore an example of how a modal window may look like, may be found here: http://www.w3schools.com/bootstrap/bootstrap_modal.asp. What remains is a way to open the modal window and a way to close it from outside the directive. (Note bootstrap already supports a way to open / close the modal window; however, a concrete element has to be selected first, which is rather impractical to do inside of a screen's controller.) For that purpose we reserved a single attribute, "bind", acting as a 'gate' to our directive's interface. An example can be seen below:

Inside of a template we can use the directive the following way:

<!-- Confirm modal window -->
<confirm bind="myobj" title="Title">
    confirm modal window content, 'yes or no' question
</confirm>

<!-- Button to open the modal window with -->
<button ng-click="open()">Open</button>

Inside of a corresponding controller we may specify the following code:

// Initialization; the object will be filled by the corresponding functions automatically
scope.myobj = {};

// On button click
scope.open = function() {
    // Open the modal
    scope.myobj.open(function (response) {
        // Upon closing the modal, this will be automatically called
        if (!response) {
            console.log('A user answered "no".');
        }
    });
};

It is worth mentioning that we strived to put visually related interface into separate attributes (e.g. modal headline is determined by the "title" attribute inside of a template). "bind" attribute serves mostly as a 'gate' to call a directive's functions. For data or functions to be consumed by a directive itself we usually created individual attributes, as demonstrated by the following example:

<button-load button-class="btn" action="f" disabled="option1.chosen">Execute</button-load>

Where action is a function to be used by the button-load directive and disabled is an expression to be evaluated by the directive (on certain events). Exceptions from the rule may happen, however.

File handling

File handling is associated with the following screens:

  • addfile: serves for uploading/attaching new files
  • filelist: displays all user's files; allows downloading, configuration and removal of the files
  • createnewtask: during a task creation/configuration, a new file may be uploaded/attached and configured

To avoid repeating ourselves, we put a portion of a code serving for uploading/attaching files into a separate subcomponent, common/fileinput. It has a form of a directive, i.e. it consists both of an HTML template and a controller handling the logic behind.

File configuration is handled on several places (all of the screens mentioned at this section). We approached the problem by creating a subcomponent common/filesettings, consisting of a modal window with corresponding controls. On opening the modal, data associated with the existing file configuration is loaded from the server and sent upon close.

Last, but not least, filelist screen has a similar implementation to taskconfigs screen (described in the following section), consisting of a simple table, displaying basic information about each file, while providing actions to further manipulate the files (configuring, downloading and removing the files).

Task handling

Task handling is associated with the following screens:

  • createnewtask: serves for creating new tasks while also allowing editing existing tasks
  • importtask: allows creating new tasks by configuration import
  • taskconfigs: displays all user's tasks, shows their basic information and allows for their basic manipulation
  • taskresult: a comprehensive display of a task's result, provided by the server; allows sending feedback and reexecuting the task

createnewtask consists, among others, of a common/fileinput component, to allow for a simple file upload / attach right during a task's creation / editing. Several form controls are available to allow a detailed task configuration.

importtask is a relatively simple screen consisting only of a text field and file input field. The selected task configuration file is processed via HTML5 file API and its data sent to the server (upon clicking the corresponding submit buttons).

taskconfig, similarly to filelist screen, is basically a table of tasks, displaying basic information, such as identifier, last modification date and description, while also providing several buttons for manipulating the tasks. The list of tasks, when obtained from the server, is handed to pagination directive. The directive processes the list, providing only a sublist to surrounding components, based on a currently selected page.

An important aspect of taskconfig is its ability to display each task's current state. Based on the state, different actions may be taken for each task state (e.g. a running task may not be removed, while only a finished task has a 'go to result' button available). The states are processed in the following manner:

  • At the beginning, a server returns a list of all tasks, while also providing information about each task's current state.
  • A list of tasks that are running, is created.
  • Each of the task's state from the list is requested each 3 seconds via a time-out function. (It is taken into a consideration that the server may take a longer time to respond, in which case the interval may be prolonged.)
  • A task, which is no longer running, is removed from the list.
  • An action, such as re-running a task, may again change a task's state. If a task is this way put into the running state, it is added to the list of running tasks.
  • Upon visiting a different screen, the time-out function is cleared.

Taskresult screen

The taskresult screen provides a comprehensive display of a task's result, which was previously computed by the server. The screen allows examining the result, provides means to store and send feedback to the server, and allows for downloading of exported result in various formats.


JSON data binding

In order to work properly, the application needs the following JSON objects from the server:

  • $scope.result - represents an actual result of the algorithm. Additionaly, the object is used as a binding variable for user changes.

  • $scope.feedback - contains saved user changes from a previous iteration of the algorithm.

  • $scope.inputFile - a user input file.

  • $scope.configuration - contains information about chosen knowledge bases, primary knowledge base, etc.

Additional important objects include:

  • $scope.ignoredColumn - represents ignored columns for feedback.
  • $scope.noDisambiguationColumn - represents ignored cells in all columns.
  • $scope.noDisambiguationCell - represents ignored cells.

An important utility object is "$scope.locked" with flags for locked/unlocked state of entities. Basically, every data change causes setting of a corresponding lock to "true" (representing locked state). It allows for sending only the actually modified data to the server.  

In the beginning, all entities are unlocked. After a user makes some data modification and reruns the algorithm with his/her feedback, the modified entities will be locked. They stay that way until a user modifies his/her feedback again.

The structure of the object is as follows:

$scope.locked = {
    tableCells: ...,
    subjectColumns: ...,
    graphEdges: ...,
    statisticalData: ...
}
  • tableCells - two-dimensional array, first dimension ranging from -1 to number of rows (-1 representing the header row), the second dimension from 0 to number of columns.
  • subjectColumns - one for each knowledge base (at most one column can be chosen/locked).
  • graphEdges - for relation changes between two columns.
  • statisticalData - for changes of data cube.

"$scope.selectedPosition" determines selected position in the table of classifications/disambiguations, e.g.:

$scope.selectedPosition = {
    column: 3,
    row: -1
}

"$scope.selectedRelation" determines selected relation between two columns in the graphvis component, e.g.:

$scope.selectedRelation = {
    column1: 2,
    column2: 0
}

The code of taskresult screen, being relatively complex, is divided into several sections:

  • classdisambiguation: generates a table showing classifications and disambiguations suggested by the algorithm. The user can edit this suggested values.
    • cdlock: is a directive which shows lock/unlock icons. Also detects user changes.
    • cdtable: generates table using cdrow directive. The first row contains headers of the input file.
      •  cdrow: represents a single row of the table. Shows a winner classification or disambiguation and provides means for editing the feedback.
    • cdmodalproposal: a modal window allowing a user to create and save his/her own classification/disambiguation for each cell in the table. A user can add his own entities (but only to the primary knowledge base).
    • cdmodalselection: a modal window showing detailed information about a picked cell from the table.
      • cdselecting: generates cdselectbox and cdsuggestion for all knowledge bases (see below).
        • cdselectbox: represents a select box using a component ui-selectbox, which is a smart version of select box that allows for displaying HTML elements inside its options (more on https://angular-ui.github.io/ui-select/).
        • cdsugestion: allows to search for an appropriate classification or disambiguation via a label.
      • cdcheckboxes: provides settings of ignored values
  • relations: shows suggested relations using the graphvis component. 
    • rlock: shows lock/unlock icons and detects user changes in the graphvis component.
    • rmodalselection: is a modal window providing details of a concrete relation.
      • rmodalproposal: allows to create and save a custom relation.
      • rselectbox: a select box similar to cdselectbox, but for relations .
    • rsugestion: allows to search for an appropriate relation via a label.
    • graphvis: represents the graph of relations (see "Grapvis" below for further details).
  • subjectcolumns: for setting a subject column for all knowledge bases.
  • statisticaldata: setting dimensions and measures and its predicates using the table directive (see below) for data cube. 
    • table: generates a table for given dimensions or measures. Allows to choose predicate in a similar manner as relations.
  • controlsbuttons: contains control buttons, which allow to browse the result, save feedback and re-run the algorithm.
  • export: buttons for exporting the data in a desired format.


LodLive

Odalic experimentally cooperates with project LodLive (https://github.com/dvcama/LodLive). The project provides means to browse resources and their related entities.

A copy of the project is lightly adapted to allow for communication with Odalic. An exit button was added as well as a button for returning a selected resource URL.

The communication between Odalic and LodLive is solved by HTML5 Message API (Window.postMessage()). For more information, please visit https://developer.mozilla.org/en-US/docs/Web/API/Window/postMessage.


Graphvis

A graphvis is a component that is a part of a relationship discovery step of the taskresult screen. Graphvis is basically just an SVG (HTML5) element with several controlling buttons.

At the beginning, a graph is created, based on the data in the task result obtained from the server. Graph is created by the principles of OOP, i.e. vertices, edges and labels are all objects that may be further manipulated. Then, since the graphvis is implemented using D3 library (https://d3js.org/), tick function is called continuously by the D3 (which represents a smallest step in the simulation, what graphvis in its principle is - a simulation).

Several actions are mapped to events, such as clicking on an edge label, or dragging a node with a mouse. The main idea behind the graphvis are states: the only moving parts of the graph are nodes. These are moving only if attractive / repulsive forces are active and if the graph is not stabilized. We can turn the forces off, thus putting the graph into a static state. This serves us for allowing a user to create his/her own links between the nodes. (The states are changed by clicking on the corresponding buttons.) Additionally, the states are applied also when a node is fixed due to a user moving it around (which may be further released by double-clicking on it). This state is preserved even when changin between 'link creation' and 'node dragging' mode.

Knowledge base configuration handling

Task handling is associated with the following screens:

  • kbconfig: serves either for creating new knowledge base configurations or editing the exsting ones
  • kbimport: allows creating new knowledge base configurations by configuration import
  • kblist: displays all user's knowledge base configurations, shows their basic information and allows for their basic manipulation
  • setproperties: serves for defining new, or editing existing ones, predicates and classes groups

kblist is in many ways similar to other 'listing' screens, such as filelist or taskconfig. It consists of a table, pagination directive (not to overwhelm the user with all of the configurations on 1 page in case of many configurations defined), and several buttons to allow the manipulation of the configurations.

kbconfig is a rather more complicated screen consisting of several controls to allow detailed specification of a knowledge base configuration. tabset (provided by the library uibootstrap) groups relevant controls under common tabs and allows switching among them. A new directive, for specifying string arrays by a user, has been added, cilistbox (e.g. for specifying skipped attributes in the 'search' tab). To allow defining and editing predicates and classes groups, a new screen had to be added - setproperties. This led to a problem with data persistence: normally, when switching between screens, user-entered data is lost. To avoid that, we used persist service to keep the user-entered data while browsing setproperties screen and reload the data upon entering kbconfig again.

kbimport shares a very similar implementation with importtask screen.