Common Browser Server Services

Devapt project is deprecated, no security fix and no updates.

Welcome on Devapt project documentation (v2 or more)

With Devapt you can easily develop powerfull application.

Devapt libraries are usefull to quickly develop applications with more JSON configuration and less code.

What is Devapt ?

The Developpers Application Toolkit (Devapt) is a usefull package for developers: create quickly and securely a complex distributed set of applications or a simple alone application into one or more nodejs.

This project is based on Javascript (ECMAscript 6 transpiled in ES5) and nodejs. Devapt contains a server runtime and a set of client features. Devapt help you to write less code and to use easily more features.

The principle is simple: you write some configuration files for distributed nodes, models, views and menus and the you start the application. The framework will automatically generates the server and manages common features (logges, traces, metrics, restful, security...). Simply launch index.html and your application is up a rich user interface and many features.

Built-in features cover many important subjects as: authentication, restfull, http server, distributed services, logs, metrics... Main characteristics of DEVAPT architecture is dynamic behaviors, reactive programming, object oriented ES6 javascript, gulp build chain All of this on NodeJS.

My main initial purpose was to provide me a simple way to play with new scientifical or computer concepts by writing some labs applications.

Where to find Devapt ?

Devapt is a set of core libraries to help producing applications:

Project GitHub API Manual NPM Demo Issues
devapt-core-common Repo Manual NPM Demo Issues
devapt-core-browser Repo Manual NPM Demo Issues
devapt-core-server Repo Manual NPM Demo Issues
devapt-core-services Repo Manual NPM Demo Issues

Others libraries provide usefull features.

Project GitHub API Manual NPM Demo Issues
devapt-features-foundation6 Repo Todo NPM Demo Issues

Applications are good examples:

Project GitHub API Manual NPM Demo Issues
devapt-app-testbus Repo Todo NPM Demo Issues

How does Devapt work ?

With Devapt you define:

  • A topology, simple (one application, one server, one service) or complex (many services distributed for many applications on many distributed servers).
  • Some resources (models, views, menubars, menus)
  • Security rules

A topology contains:

  • nodes: A node is a nodejs process with an unique name. You can have many nodes on the same machine. Each node communicates with other node through messaging buses (messages, metrics, logs).

A node can have one or many servers.

  • servers: A server provides one or more services and listen client connections through a couple (host, port).

A server has a unique name and can be one of types: restify, express, socket.io (coming soon)...

  • services: A service offers one feature to clients. A service can be one of: static assets providers, generic middleware provider, RESTfull provider, resources provider...
  • applications: An application contains some services and is provided through nodes and servers.

An application can have one service on one server on one node. Or many services on many servers on many nodes.

An application contains:

  • modules: functional features for applications (a set of preconfigured UI for example).
  • plugins: technical fearures for applications (a rendering provider for example).
  • security: defines authentication and authorization rules.

The rendering engines are plugins and rendering classes are stateless: state is stored in a Redux store. See Getting Started with Redux for flow concepts.

Devapt is a smart glue between many powerfull projects

Thanks for all projects leaders and contributers.

The given list is an extract of all used or inspired projects.

Main dependancies:

NodeJS: https://nodejs.org

Servers and datas access:

Security:

Foundations:

Rendering:

Dev and build dependancies: (coming soon.)

Version

Current active version is 2.x.x

Versionning use standard method: M.m.p with M is a major change, m a minor change, p a patch.

License

Copyright Luc BORIES 2015-2016

Apache Version 2 license.

See LICENSE on GitHub

Please enjoy using devapt projects and let me know if they are usefull for you.

Next

Overview

With DEVAPT you can easily develop powerfull application. Built-in features cover many important subjects as: authentication, restfull, http server, distributed services, logs, metrics... Main characteristics of DEVAPT architecture is dynamic behaviors, reactive programming, object oriented ES6 javascript, gulp build chain All of this on NodeJS.

Projects structure

All devapt-* github repositories have the same structure: a master branch with the last tagged tree, a develop branch with latest comited updates and tags.

All devapt-* projects have a common folders tree: project root\

  • bower_components (only for browser features)
  • build: building scripts
  • dist: build codes and resources
    • dist/js: build javascript codes
    • dist/resources: build resources configuration
  • node_modules: nodes packages
  • public: shared assets for browsers
    • public/app_name/css: CSS assets
    • public/app_name/html: HTML assets
    • public/app_name/img: images assets
    • public/app_name/js: javascript assets
  • public/favico.png: browser project icon
  • src: project sources
    • src/js: javascript sources
    • src/resources: resources configuration sources
  • tmp: temporary application files (logs)
  • bower.json: browser packages configuration
  • gulpfile.js: main build script
  • LICENSE: project license
  • package.json: nodes packages configuration
  • README.md: project description

Some differences appears between library and application folder.

Versionning use standard method: M.m.p with M is a major change, m a minor change, p a patch.

Development

git clone git://github.com/lucbories/devapt-core-common.git
cd devapt-core-common
npm install
npm test

Same commands for devapt-core-browser, devapt-core-server, devapt-core-services

Contribution

Feel free to contribute, you're welcome.

Check issues and solve it:

Submit your work:

If you submit a pull request, please be sure to add or update corresponding test cases, and ensure that npm test continues to pass.

Installation

Via npm on Node:

cd my_project/
npm init
npm install devapt-core-common --save
npm install devapt-core-browser --save
npm install devapt-core-serices --save
npm install devapt-core-server --save

To install and launch TestBus applicaton :

cd apps/
git clone git://github.com/lucbories/devapt-app-testbus.git
cd devapt-app-testbus

npm install --save-prod
or yarn install -- production

npm run nodeA
or yarn run nodeA

or
npm run nodeA
npm run nodeB

or
npm run nodeA
npm run nodeB
npm run nodeC

open a browser on http://my_domain:8080/assets/html/index.html

Usage

To install a devapt application, follow:

cd apps/
git clone git://github.com/lucbories/devapt-app-testbus.git
cd devapt-app-testbus

npm install --save-prod

To create an empty application from scratch (not recommended for devapt newbies):

cd my_app
npm init
npm install devapt-core-common --save
npm install devapt-core-browser --save
npm install devapt-core-services --save
npm install devapt-core-server --save

Create folders tree:

-build
-dist
-public
-public/assets
-public/assets/css
-public/assets/html
-public/assets/img
-public/assets/js
-src/js
-src/resources
-tmp

Create your first Devapt based HTML page in public/assets/html/mypage.html:

<html>

<head>
    <meta charSet="utf8"/>
    <title>Devapt TestBus Home</title>

    <link rel="icon" type="image/png" href="/assets/img/favico.png"/>

    <link type='text/css'  rel="stylesheet" href="/assets/css/normalize.css" media="all" />

    <style type='text/css'>
        ... your styles
    </style>
</head>


<body>

<header>
    <center>
        <h1>MY FIRST APP</h1>
        <h8><a href="/assets/html/mypage.html">Home</a></h8>
    </center>
</header>

<hr>

<script>window.__INITIAL_STATE__ = {
    "credentials":{
        "tenant":"tenantA",
        "env":"dev",
        "application":"testbus",

        "token":"",
        "user_name":"",
        "user_pass_digest":"",

        "ts_login":0,
        "ts_expiration":0,

        "errors_count":0,
        "renew_count":0,

        "hash":undefined
    },
    "commands":{}
}
</script>

<div id="content">
... your HTML code
</div>

<script type="text/javascript" src="/socket.io/socket.io.js"></script>
<script type="text/javascript" src="/assets/js/vendor/browser.min.js"></script>
<script type="text/javascript" src="/assets/js/devapt-core-browser.js"></script>
<script type="text/javascript" src="/assets/js/devapt-bootstrap.js"></script>
<script type="text/javascript" src="/assets/js/app.js"></script>

</body>

</html>

Now you can write your Javascript code in public/assets/js/app.js:

window.devapt().on_runtime_created(
    function()
    {
        console.log('runtime ready for application script')

        write your code here: devapt init is finished and you can use Devapt function:

        window.devapt(): Devapt bootstrap root with some helpers
        window.devapt().runtime():main Devapt client runtime
        window.devapt().ui(): main Devapt user interface manager

        Example: get client session unique id
        var session_uid = window.devapt().runtime().get_session_uid()

        Example: ping master node
        var target_name = 'master' (target is a node or a node part, node is server host or remote host or remote browser session)
        var ping_svc_cfg = undefined (reserved for later)
        var ping_svc_promise = window.devapt().get_service('pingpong', ping_svc_cfg)
        var on_response_callback = function() {
            console.log('pong received')
        }
        window.devapt().request_service(ping_svc_promise, 'devapt-ping', target_name, on_response_callback)
    }
)

Devapt use configuration files to discover and build a runtime of dynamic resources.

First we declare our unique node called "nodeA", this node is the topology master and the application provider: File: dist/resources/nodes/nodeA.json

{
    "is_master":true,
    "name":"NodeA",

    "master":{
        "name":"NodeA",

        "msg_bus":{
            "package":"socketio",
            "protocole":"ws",
            "type":"Server",
            "host":"localhost",
            "port":5000
        },
        "logs_bus":{
            "package":"socketio",
            "protocole":"ws",
            "type":"Server",
            "host":"localhost",
            "port":5001
        },
        "metrics_bus":{
            "package":"socketio",
            "protocole":"ws",
            "type":"Server",
            "host":"localhost",
            "port":5002
        }
    },

    "settings_provider": {
        "source":"local_file",
        "relative_path":"world.json"
    }
}

Secondly we describe our application features topology: File: dist/resources/world/world.json

{
    "nodes":"nodes":{
        "NodeA":{
            "host":"localhost",
            "is_master":true,
            "servers":{
                "NodeALocal8080":{
                    "type":"express",
                    "port":8080,
                    "protocole":"http",
                    "middlewares":[],
                    "use_socketio":true,

                    "security": {
                        "authentication": {
                            "enabled":false,
                            "plugins":[]
                        }
                    }
                },
                "NodeALocalLogs":{
                    "type":"logs",
                    "port":"N/A",
                    "protocole":"bus",
                    "middlewares":[],
                    "use_socketio":false,

                    "security": {
                        "authentication": {
                            "enabled":false,
                            "plugins":[]
                        }
                    }
                }
            }
        }
    },

    "tenants":{
        "tenantA":{
            "applications":{
                "assets":{
                    "title":"Assets provider",
                    "url":"assets",
                    "license":"APACHE-LICENSE-2.0",

                    "provided_services":[],
                    "used_services":[],

                    "used_packages": ["my_app_pkg"],
                    "used_plugins":["DefaultRendering"],

                    "assets":{
                        "css":[],
                        "js":[],
                        "img":[],
                        "index":""
                    }
                }
            },
            "packages":{
                "my_app_pkg":{
                    "base_dir":null,
                    "resources":[],
                    "templates":[],
                    "includes":[],
                    "commands":[],
                    "services":{
                        "pingpong":{
                            "type":"ping_svc"
                        },
                        "html_assets":{
                            "type":"html_assets_svc",
                            "routes":[
                                {
                                    "route":"/",
                                    "directory":"../../public/assets/",
                                    "required_prefixes":[],
                                    "required_suffixes":[".js", ".js.map", ".css", ".html", ".png", ".jpg"]
                                }
                            ],
                            "security":{
                                "enabled":false,
                                "authentication":{
                                    "enabled":false,
                                    "engines":[]
                                },
                                "authorization":{
                                    "enabled":false,
                                    "engines":[],
                                    "permissions":{
                                        "*":["*"]
                                    }
                                }
                            }
                        }
                    }
                }
            },
            "security":{}
        }
    },

    "deployments":{
        "tenantA":{
            "assets":{
                "assets":{},
                "services":{
                    "topology":{ "servers":["NodeALocal8080"], "filters":["*"] },
                    "html_assets":{ "servers":["NodeALocal8080"], "filters":["*"] },
                    "pingpong":{ "servers":["NodeALocal8080"], "filters":["*"] }
                }
            }
        }
    },

    "security":{
        "is_readonly":false,

        "datasources":[],

        "authentication": {
            "enabled":false,

            "plugins":{},

            "default_plugins": []
        },

        "authorization": {
            "enabled":false
        }
    },

    "plugins":{
        "DefaultRendering":{
            "type":"rendering",
            "file":"../../../node_modules/devapt-core-common/dist/js/default_plugins/rendering_default_plugin"
        },
        "DefaultServices":{
            "type":"services",
            "file":"../../../node_modules/devapt-core-services/dist/js/default_plugins/services_default_plugin"
        }
    },

    "loggers":{},

    "traces":{}
}

Launch your runtime nodes and servers:

cd my_app
node node_modules/devapt-core-server/dist/js/start_node.js -n NodeA -t dist/resources/world -b node_modules/devapt-core-server/dist/js -s NodeA:NodeALocal8081

Example with settings in many files:

"resources//worldworld.json":

{
    "nodes":"nodes.json",

    "tenants":{
        "tenantA":{
            "applications":"applications.json",
            "packages":"tenantA_packages.json",
            "security":{}
        }
    },

    "deployments":"deployments.json",

    "security":"security.json",

    "plugins":"plugins.json",

    "loggers":"loggers.json",

    "traces":"traces.json"
}

"nodes.json":
{
    "nodes":{ same settings than above }
}

"services.json":
{
    "services":{ same settings than above }
}

Repeat on each files on the same manner.

Design

Devapt is full stack solution. So it's usefull to provide an integrated framework to build your application with configuration files. On the other hands it's difficult to have an overview of all parts and undestand how all this parts work together.

First, devapt is provided throught specialized core projects and addon features. If you write an application you don't need to know all details. But if you want to add features, you should understand on what starting to build your feature. Details of projects

Main devapt concepts are:

  • Node: each application can be distributed on one or many nodes. Each node run on a NodeJS instance with some inter node communication channels. One or many nodes can run on one NodeJS instance. One or many NodeJS instance can run on one hostnames.
  • Node settings: JSON file which describe one or many nodes with their names, security, debug trace and communication channels.
  • Application settings or configuration or description: main JSON file which define application features, packages, resources (UI components...), debug traces and security.
  • Bus: all nodes communicate together with messages throught a communication bus. An application can have one or more buses, one for metrics, one for logs and one between components.
  • Tenant: a tenant is a set of applications and packages with its own security access. for example, we can have one tenant per company on a shared server (SaaS).
  • Service: a service provide a feature which responds some datas on a request. For example, a ping pong service responds a 'pong' on each 'ping' request.
  • Server: a server (not a hostname) is an engine which provides services and others features. For example, a logs server exists to process received logs.
  • Metric: metrics are collected datas from a node, an application, a server, a service... which help to monitors system health.
  • Rendering: it's the process of producing DOM content for the browser page.
  • Virtual DOM: it's a technology which can produce DOM content outside of browser and to apply differential changes on a page. It enhance performance and decrease browser load.
  • Security: security is an important need for all usages. Security is everywhere in devapt apps but all parts aren't finished yet. (TODO)

How an application work

An application has one directory but two targets: server and browser. On server side, an application launches nodes and services. On browser side, the page receive its content from the server: it's a lightweight HTML page with two scripts: a credentials record and an initial application state. Each user action on the interface call some scripts and request server if needed.

Features

Distributed

Messaging

Metrics

Rendering

Security

Servers

Services

Starting process

Examples

Define two routes to serve static files:

    "html_assets_1":{
        "type":"html_assets",
        "routes":[
            {
                "is_global":true,
                "route":"/",
                "directory":"/.../devapt-app-devtools/dist/assets",
                "default":"index.html"
            },

            {
                "route":"/css/.*.css",
                "directory":"./public"
            }
        ]
    }

Tutorials

Some usefull applications are:

  • DEVTOOLS which actualy display settings, logs, buses messages, metrics and topology. DEVTOOLS app project
  • TESTBUS wich helps to test distributed features: basic ping/pong communication, messaging, logs, topology. TESTBUS app project
  • TESTUI wich helps to test user interface features. (coming soon) TESTUI app project

Projects map

Devapt projects refcard.

Development Map

Projects core dependencies (package.json)

Project devapt-core-common devapt-core-browser devapt-core-services devapt-core-server
devapt-app-testbus x x x x
devapt-app-testui x x x x
devapt-app-devtools x x x x
devapt-core-doc x x x x
devapt-core-common
devapt-core-browser x
devapt-core-services x
devapt-core-server x x
devapt-features-foundation6 x
devapt-features-cytoscape x
devapt-features-sparklines x
devapt-features-labs x x
devtools x x

Projects features dependencies (package.json)

Project devapt-features-foundation6 devapt-features-cytoscape devapt-features-sparklines devapt-features-labs devtools
devapt-app-testbus
devapt-app-testui
devapt-app-devtools x
devapt-core-common
devapt-core-browser
devapt-core-services
devapt-core-server
devapt-features-labs
devapt-app-labs x x
devtools x x x

devapt-core-doc

DEVAPT CORE DOC project

Purpose: generate core projects documentation (API, manuals, tutorials)

Devapt projects dependencies:

  • devapt-core-browser
  • devapt-core-common
  • devapt-core-server
  • devapt-core-services

devapt-core-common

DEVAPT CORE COMMON project

Purpose: common features for all others projects.

Devapt projects dependencies:

  • none

devapt-core-services

DEVAPT CORE SERVICES project

Purpose: services implementation for browser/server projects.

Devapt projects dependencies:

  • devapt-core-common

Provided services:

  • assets routes
  • logs stream provider
  • messages/logs/metrics stream provider and dispatching
  • filtered metrics stream provider
  • middleware based routes provider
  • ping/pong provider
  • resources configuration provider
  • security login/signup/logout/... provider (coming soon)
  • topology description provider

devapt-core-server

DEVAPT CORE SERVER project

Purpose: server features. Provides assets, resources, services, security, plugins, server runtime and metrics.

Devapt projects dependencies:

  • devapt-core-common
  • devapt-core-services

devapt-core-browser

DEVAPT CORE BROWSER project

Purpose: browser features. Provides commands, client runtime and components.

Devapt projects dependencies:

  • devapt-core-common

devapt-app-testbus

TESTBUS app project

Purpose: messages bus testing application.

Devapt projects dependencies:

  • devapt-core-common
  • devapt-core-browser
  • devapt-core-server
  • devapt-core-services

devapt-app-testui

TESTUI app project

Purpose: user interface testing application.

Devapt projects dependencies:

  • devapt-core-common
  • devapt-core-browser
  • devapt-core-server
  • devapt-core-services

devapt-features-foundation6

DEVAPT FOUNDATION 6 features project

Purpose: Foundation 6 rendering features.

Devapt projects dependencies:

  • devapt-core-common

devapt-features-cytoscape

DEVAPT CYTOSCAPE features project

Purpose: Cytoscape rendering features.

Devapt projects dependencies:

  • devapt-core-common
  • devapt-core-browser

devapt-features-sparklines

DEVAPT SPARKLINES features project

Purpose: Sparklines rendering features.

Devapt projects dependencies:

  • devapt-core-common
  • devapt-core-browser

devapt-features-labs

DEVAPT LABS features project

Purpose: labs features. Provides calculus and plots.

Devapt projects dependencies:

  • devapt-core-common
  • devapt-core-browser

devapt-app-labs

DEVAPT LABS app project

Purpose: Amazing labs application.

Devapt projects dependencies:

  • devapt-core-common
  • devapt-core-server
  • devapt-core-services
  • devapt-features-foundation6
  • devapt-features-labs

devapt-devtools

DEVTOOLS project

Purpose: development tools features.

Devapt projects dependencies:

  • devapt-core-common
  • devapt-features-cytoscape
  • devapt-features-foundation6
  • devapt-features-sparklines

devapt-app-devtools

DEVTOOLS app project

Purpose: development tools application.

Devapt projects dependencies:

  • devapt-core-common
  • devapt-core-server
  • devapt-core-services
  • devapt-devtools
  • devapt-features-cytoscape
  • devapt-features-foundation6
  • devapt-features-sparklines

Roadmap

2.0.x breaking changes and new features (CURRENT)

MAIN GOALS

When I was working on devtools features with devapt 1.0.x, I meet some trouble with missing features. An other problem was to deal with topology definition and deployment information previously stored in many pieces of code. It was difficult to obtain a global view of defined or deployed topology. So I pull out deployment information from topology definition.

One missing feature for fine deployment was the lake of customer and version of defined items.

Main added features:

  • isomorphic Credentials class to deal with session information on browser or on server (tenant, application, username, token...). (DONE)
  • isomorphic data management classes: DataStore, DataRecord, DataCollection...
  • topology management classes: TopologyDefineItem...(DONE)
  • topology deployment classes: TopologyDeployLocalNode... (DONE)
  • browser router (for single page application and more) (DONE)
  • isomorphic cache management (DONE)
  • versionned Collection to register versionned items (CURRENT)

Main breaking changes:

  • settings json format has evolved
  • module was renamed to package
  • change Collection.$items from [] to {}, use lodash as collection backend

New topology tree is now:

world: the root of the topology

-nodes
--nodeA
---serverA1
---serverA2

-tenants
--tenantA
---applications
----applicationA
-----url
-----license
-----provided_services
-----used_services
-----used_packages
-----used_plugins
-----assets
---packages
----packageA
-----base_dir
-----resources
-----services
---security

-plugins
--pluginA
---type
---file or package

-deployments
--tenantA
---applicationA
----assets
----services

-security
-loggers
-traces

That's why previous roadmap is outdated, sorry for the changes.

enhance core features, architecture and stability (previous 1.1.x)

PROJECT

  • refactor directories into common/, browser/, server/ (DONE)
  • implement Unit tests for base classes (CURRENT)
  • use babel-register to debug without build to dist/ directory (DONE)
  • test on nodejs 4.4, 5.1, 6.x (DONE)

FEATURES

  • implement browser router for single page application (DONE)
  • enhance devapt-devtools: runtime registry viewing (DONE)
  • enhance devapt-devtools: runtime topology viewing (DONE)
  • enhance devapt-devtools: metrics on topology (TODO)
  • enhance devapt-devtools: application Redux state viewing (TODO)
  • enhance devapt-devtools: application Redux state history player (TODO)
  • implement unit test for instances, traces, loggers... (CURRENT)
  • implement topology registry (DONE)
  • implement topology runtime (DONE)
  • implement generic api for state storing (Redux, Map) (DONE)
  • implement cache management classes for browser and server (DONE)

enhance datas access features (previous 1.3.x)

PROJECT

  • document datas access layers (TODO)
  • write Unit tests for datas access classes (TODO)

FEATURES

  • enhance devapt-devtools: create datas access dashboard (TODO)
  • enhance devapt-devtools: create datas admin backend (TODO)
  • implement homogeneous datas access for all datasources (DONE)
  • enhance sequelize datasource for SQL databases:MySQL, MariaDb, PostgreSQL, SQLite, MSSQL (DONE)
  • implement waterline ORM layer (TODO)
  • implement Redis datasource for caching, messaging, NoSQL database (TODO)
  • implement datas management classes for browser and server (DONE)
  • implement datas cache on distributed nodes (TODO)

2.1.x enhance distributed features and documentatioon (LATER)

PROJECT

  • update web site on github with project/, technical/... (TODO)
  • document build process (TODO)

FEATURES

  • implement distributed services consumers (TODO)
  • implement master configuration changes: receive and propagate complete/add/remove/update (TODO)
  • implement master node revocation and promotion (TODO)
  • implement node settings filter (TODO)
  • implement swagger API player (TODO)
  • implement circuit breaker on browser (TODO)

2.2.x enhance security features (LATER)

PROJECT

  • document security process (TODO)
  • write Unit tests for security classes (TODO)

FEATURES

  • enhance devapt-devtools: create security dashboard (TODO)
  • enhance devapt-devtools: create security admin backend (TODO)
  • implement security routes /login, /signup, /logout (TODO)
  • implement Passport security plugin (TODO)
  • implement authorization with ACL (TODO)
  • implement security token (TODO)
  • implement registry setting validation with a json schema (TODO)

2.3.x implement Cloud features (LATER)

PROJECT

  • document Cloud integration process (TODO)
  • write Unit tests for Cloud integration classes (TODO)

FEATURES

  • implement AWS DynamoDB datasource (TODO)
  • implement AWS Lambda integration (TODO)
  • implement AWS Security integration (TODO)