ReactorWare - Data graphs with rules

Graph Database

After many years of projects using conventional RDBMS architecture I formed ideas about a more flexible approach for very complex data models. Instead of encapsulating types in tables and foreign keys between them, I created a structure of tables which contains a single table of all relationships between keys and a set of tables for primitive data types, e.g. strings, dates etc., referred to as atom tables. Each key is either a node or an edge between two (or more - hyper edge) nodes. Each node and edge have a type id which corresponds to some type in the data model. Any attributes (atoms) for a key (nodes and edges) are stored in the atom tables using the same key and an attribute id (analogous to the type id). This atomic data can be very efficiently indexed. Complex data can be stored using JSON. Because all values of the same primitive type are stored in the same table it is very easy to build search queries.

This design allows for very complex data models which are easily modified without altering the database structure. Model information is stored outside of the database. I currently use XML and JSON structures to do so.

I am currently using this database design in various production systems.

At the time I developed early versions of this there were no graph databases available, nor were they publicly under discussion. Currently there are a number of very good graph databases available. All variations on a theme.

Reactor

To utilise the graph database I have developed a Java based middle ware, dubbed Reactor.

It provides a means to process queries of the graph database and persist data to the graph database. It also provides units of functionality (reactors) which can perform autonomous operations on data. These units are not structured in parallel with the data model (as they probably would be in a traditional OO approach), but rather as per task that needs to be performed. Such a task could manipulate multiple related objects. The functionality in these units can be combined into more complex operations, as might be performed by a rule engine.

These units can also provide calculated derived data for processing elsewhere, such as document generation.

The middle ware is currently hosted in a Tomcat application server and published using servlets. It could easily be hosted in a node.js environment. It has also been deployed in containers using Docker.

Rule Engine

Systems grow. Data models change. Code not so fast. To solve the problem of how to structure the business logic code in response to fast changing models, I started looking at rule engines, specifically Drools. I wanted the ability to be able to add ad hoc pieces of business logic without having to modify existing code and more importantly without having to consider a rule’s impact on other rules. I have integrated such a rule engine in the software I write in conjunction with the graph database.

When rules fire, they have access to the data domain via the graph database and units of code (reactors) with which to perform data manipulations or complex tasks. Adding a new rule will make use of the same reactors but can add its own logic based on conditions applicable to it. These rules can quickly be added without any change to the basic codebase. At times new types of data may require code to be modified, or new code to be added with applicable functionality. But this is usually limited to fundamental types, and rarely when adding new relationships between data.

The rule-based approach is also very well suited to parameterise business logic. I use it to configure timelines for processes related to intellectual properties in different jurisdictions.

REST

This is my preferred way to communicate with a server-based application. Initially the stateless approach seemed odd from a Desktop application perspective, but it is a very good fit for a graph database / rule engine system.

The graph database is queried and searched using a path analogy as opposed to SQL. It is thus ideally suited to expressing queries as URLs allowing great flexibility on the front-end without having to provide views for everything server-side.

There is a single GET method for querying the entire data domain using a very expressive syntax.

Example: Retrieve all Agents for a specific Renewal.
./select(@ip.admin.Renewal,%20160232)/edgesOut/end(@ip.admin.Agent)

or more concisely:
./select(@ip.admin.Renewal,%20160232)/adjacentOut(@ip.admin.Agent)

There is a generic POST method which takes JSON data representing changes in data nodes in a graph. These are inserted into a session in the rule engine which will fire rules as defined and process the data. This is then persisted to the graph database for storage if necessary.

Data exchange is either JSON or XML.

HTML 5

Currently I develop HTML5, JavaScript based front ends communicating with a REST based API which serves and processes graph data structures. I avoid using frameworks such as Angular and React, but instead opt for libraries with specific functionality integrated into my own application structure.

I use knockout.js and Vue.js for data-binding and some jQuery here and there.

I structure my JavaScript code using an OO approach for the basic units, with a functional approach for processing UI events and REST API interaction. The graph approach to data modeling is replicated on the client side to a large extent, i.e. there is an API for processing the graph data structures provided by a suitable REST service.

Regarding frameworks I have found that I can make code evolve easier and last longer without them. I tend to keep my code as independent of third party code and as minimal as possible to allow me to adapt it easily to changing environments.

XSLT

As part of a major project over the last 20 years I have had to generate documents based on templates merged with data from the software system.

Early on I developed an engine which processed XHTML documents with directives embedded which would merge the requested data. XHTML as a document format has its limitations. Most notably page formatting during printing and WYSIWYG editing. When the ODF format was implemented by OpenOffice and .docx, .xlsx etc. by Microsoft, it became possible to use XSLT to merge XML data into modified .docx documents. Essentially the .docx documents were wrapped as XSL stylesheets and then transformed back into .docx. These can then be further edited using Word etc.

The XML data can be generated in any number of ways. Currently I use JavaScript in a node.js like environment to interact with the middle ware that hosts the graph database and rule engine to access units of code that provide calculated or in some way derived data. The JavaScript allows for very easy compositing of custom data views, either as XML or JSON, which can then be transformed using the templates.

The entire process is stream based. It has proven very flexible and provides quality documents.

Open Source

In the late nineties I was made aware of RedHat. I then started using Linux exclusively and have since then actively pursued development utilising open source components wherever possible. At this stage the entire software stack I develop for is open source. I have found it to be an exceptionally reliable and versatile environment.