NODIX FRAMEWORK

TPO BINARY MODULES

1. Why tpo modules ?

The motivation for developing binary modules, is that existing binary format ( DLL or ELF ) are strongly tied to their own operating system, and can't be executed easily on other platform, even if the machine code only involve cpu instruction, the convention for exported function name and memory model prevent the use of one binary from one operating system to another, even if the code doesn't use operating system specific code.

Tpo module are inter OS dynamically linked position independent binary module. In the process of developing a micro kernel project, i wanted to avoid dependency on a particular compiler or operating system for the binaries produced from the host system, but still being able to compile and link binary executables and dynamically linked libraries from them using their own tool chain, but without having any dependency on a particular tool chain for the module to be loaded and executed on micro kernel architecture with minimalist C Library and runtime.

The nodix framework allow to abstract dependencies on compiler runtime and operating system in order to have safe portable binary linked with each other regardless of the compiler and operating system used to produce them.

2. Internal structure.

START	SIZE
0	128	Module name
128	4	Length of the string buffer
132	4	Type of function decoration (GCC or MSVC)
136	4	Number of sections
140	28 byte header	Length and other information about the section
168	Section size	Section data aligned on 16 bytes boundaries.
168 + section size	12	Import definition 4 bytes offset of the name of the lib in the string buffer 4 bytes offset of the function name in the string buffer 4 bytes offset of the import location
168 + section size + imps[]	8	Export definition 4 bytes offset of the function name in the string buffer 4 bytes offset of the export location
168 + section size + imps[] + exps[]	8	Internal relocation for position independent code. 4 bytes base address for the section relocation 4 bytes offset from the base address.

3. Creation of tpo modules from native binaries.

The only tool required to produce tpo module is "modmaker", which is a command line tools that convert a dll or a .so into a binary module.

The source code and build instruction for linux and windows can be found on the github at this address mod_maker

The command line tool reads the sections, imported and exported symbols, as well as relocation table, and convert it to a binary format that can be loaded by the libcon runtime on any intel x86 32 bit system (or x64 with x32 emulation, no support for native x64 yet).

To create a tpo from a .so or dll, just run the command 'modmaker myfile.[so|dll] [output directory]'.

The only restriction is the dynamic loader doesn't handle non initialized data section, because code reference to non initialized data point to location that are not in the binary, and it has to be loaded 'in place' as position independent code in the micro kernel architecture with an allocation-free mode. (the allocator is in a tpo module). Memory allocation routines or stack can be used to create uninitialized memory at runtime.

The output should look like this for example for therpc module that handle the request for the javascript wallet :

using file : rpc_wallet.dll ->export/modz/

PE HEADER Symbols : 0 , 0 flag
: 2102 
import table : 68b8 - 80
function name : _tree_manager_get_child_value_i64@12 
reloc addr : 00000080@00006000 => 6ec2 
function name : _tree_manager_get_node_hash@12 
reloc addr : 00000084@00006000 => 6ea0 
function name : _tree_manager_write_node_btcaddr@12 
reloc addr : 00000088@00006000 => 6e7a 
function name : _tree_manager_set_child_value_float@16 
reloc addr : 0000008c@00006000 => 6e50 
function name : _tree_manager_node_add_child@8 
reloc addr : 00000090@00006000 => 6e2e 
function name : _tree_manager_create_node@12 
reloc addr : 00000094@00006000 => 6e0e 
function name : _tree_manager_set_child_value_hash@12 
reloc addr : 00000098@00006000 => 6de6 
function name : _tree_manager_get_node_istr@16 
reloc addr : 0000009c@00006000 => 6dc4 
function name : _tree_manager_get_child_value_i16@12 
reloc addr : 000000a0@00006000 => 6d9c 
function name : _tree_manager_add_child_node@16
 
 
    
export table : 6630 - 647
export table name : rpc_wallet
- functions : 22 - names : 22 
name: _getaddressscanstatus@12@rpc_wallet - func addr: 0x00001e80 0x00001000 - ordinal : 1 
name : _getblockcount@12@rpc_wallet -func addr: 0x000026a0 0x00001000 - ordinal : 2 
name : _getinfo@12@rpc_wallet - func addr: 0x00002380 0x00001000 - ordinal : 3 
name : _getlastblock@12@rpc_wallet - func addr: 0x00002070 0x00001000 - ordinal : 4 
name : _getprivaddr@12@rpc_wallet - func addr: 0x00005580 0x00001000 - ordinal : 5 
name : _getpubaddrs@12@rpc_wallet - func addr: 0x00005710 0x00001000 - ordinal : 6 
name : _getstaketx@12@rpc_wallet - func addr: 0x00004ac0 0x00001000 - ordinal : 7 
name : _getstaking@12@rpc_wallet - func addr: 0x00004f60 0x00001000 - ordinal : 8 
name : _importaddress@12@rpc_wallet - func addr: 0x000026d0 0x00001000 - ordinal : 9 
name : _importkeypair@12@rpc_wallet - func addr: 0x00005150 0x00001000 - ordinal : 10 
name : _listreceived@12@rpc_wallet - func addr: 0x00002ee0 0x00001000 - ordinal : 11 
name : _listreceivedbyaddress@12@rpc_wallet - func addr: 0x00003650 0x00001000 - ordinal : 12 
name : _listspent@12@rpc_wallet - func addr: 0x00003000 0x00001000 - ordinal : 13 
name : _liststaking@12@rpc_wallet - func addr: 0x000042d0 0x00001000 - ordinal : 14 
name : _listtransactions@12@rpc_wallet - func addr: 0x00002840 0x00001000 - ordinal : 15 
name : _listunspent@12@rpc_wallet - func addr: 0x00003520 0x00001000 - ordinal : 16 
name : _pubkeytoaddr@12@rpc_wallet - func addr: 0x000037a0 0x00001000 - ordinal : 18 
name : _set_node@8@rpc_wallet - func addr: 0x00001010 0x00001000 - ordinal : 19 
name : _signstakeblock@12@rpc_wallet - func addr: 0x000044d0 0x00001000 - ordinal : 20 
name : _signstaketx@12@rpc_wallet - func addr: 0x000047c0 0x00001000 - ordinal : 21 
name : mod_name_deco_type@rpc_wallet - func addr: 0x00008000 0x00008000 - ordinal : 17 
text section : name : .text 
section header flags : 60000020 
relocations : 0 
relocations ptr: 0 
vaddr : 00001000 
paddr : 000048cd 
size : 18944 
externs : 0 
syms : 0 imports : 0 
rdata section :
name : .rdata 
section header flags : 40000040 
relocations : 0 
relocations ptr: 0 
vaddr : 00006000 
paddr : 00001368 
size : 5120 
externs : 0 
syms : 0 
symboles :
imports : 88 
module : libbase - function : _tree_manager_get_child_value_i64@12 - reloc addr : 0x00000080 
module : libbase - function : _tree_manager_get_node_hash@12 - reloc addr : 0x00000084  
module : libbase - function : _tree_manager_write_node_btcaddr@12 - reloc addr : 0x00000088  
module : libbase - function : _tree_manager_set_child_value_float@16 - reloc addr : 0x0000008c  
module : libbase - function : _tree_manager_node_add_child@8 - reloc addr : 0x00000090  
module : libbase - function : _tree_manager_create_node@12 - reloc addr : 0x00000094

4. High level encapsulation

Modules can be encapsulated either as 'local' application modules, or as distributed modules which will be used behind a network protocol such as http rpc or cgi.

4.1 Application modules

Application modules export function can be used by other binary modules or scripts.

If they are called locally from another binary modules, the is not special requirement other than using OS_API_C_FUNC macro to declare the function prototype.

Any module compiled using the same std_def.h file will be able to import and call any function exported by any other module regardless of the host operating system and/or compiler used to produce it.

For exported interfaces called from the script language, they are required to have between 0 and 2 arguments which must be nodes references.

4.2 Common Gateway Interface modules.

CGI modules are made to be executed behind http request.

Using the node's service configuration entry

{"base" : "/my_mod/" , "type" : "cgi" ,(NODE_MODULE_DEF) "mod_name" :{ "file":"modz/my_cgi_mod.tpo"}}

The modules exported function will be made available with an url like

http://node-ip/my_mod/my_function

Additional parameters can be passed as additional path segment in the url, or via query strings, post data in json, or cookies.

The function prototype for CGI module is :

OS_API_C_FUNC(int ) my_function( const char * params , const struct http_req * req , mem_zone_ref_ptr result)

Params is a pointer the url path segments after the method name.

Req is a pointer to a structure containing informations about the http request such as query string variables and http request header.

Result is a dynamic object which will be converted to json and returned in the http response .

4.3 RPC modules.

RPC modules are made to handle json/rpc request over the http protocol.

Using the node's service configuration entry

{"base" : "/jsonrpc", "type" : "rpc" , (NODE_MODULE_DEF) "mod_name" : { "file":"modz/my_rpc_mod.tpo"}}

RPC methods exported from the modules will be available at

http://node-ip/jsonrpc

The HTTP POST contain the rpc request formated as json containing themethod name, and the parameters.

The function prototype for RPC modules is :

OS_API_C_FUNC(int )rpc_method( mem_zone_ref_const_ptr params , unsigned int rpc_mode , mem_zone_ref_ptr result)

Params can be either an array containing the ordered list of parameters, or an object containing parameters as named key, depending on the value of rpc_mode.

rpc_mode indicate if parameters are passed as ordered array or key/value pair objects.

Result is the dynamic object that will be converted to json and sent as the http response.

5. How to use tpo modules ?

The full API to use the modules is described in the Libcon section .

From C/C++ application using the runtime.

Binary modules can be loaded and executed from a C or C++ program using the module API by linking with the libcon library (statically or dynamically) and including the header definitions.

#include <base/std_def.h>
#include <base/std_mem.h>
#include <base/mem_base.h>
#include <base/std_str.h>
#include <fsio.h>
#include <mem_stream.h>
#include <tpo_mod.h>

int main(int argc, char **argv)

{

int done = 0;

//base initialization
init_mem_system ();

//init 8Mo of memory
init_default_mem_area (8 * 1024 * 1024);

//init network
network_init ();

//set home dir
set_exe_path ();
set_home_path ( "purenode" );

//load node modules

load_module("modz/libbase.tpo", "libbase", &libbase_mod);
load_module("modz/protocol_adx.tpo", "protocol_adx", &protocol_mod);
load_module("modz/block_adx.tpo", "block_adx", &block_mod);
load_module("modz/iadixcoin.tpo", "iadixcoin", &iadix_mod);

//load application entry points

app_init = get_tpo_mod_exp_addr_name ( &iadix_mod, "app_init", 0);

app_start = get_tpo_mod_exp_addr_name ( &iadix_mod, "app_start", 0);
app_loop = get_tpo_mod_exp_addr_name ( &iadix_mod, "app_loop", 0);
app_stop = get_tpo_mod_exp_addr_name ( &iadix_mod, "app_stop", 0);

if (!app_init(PTR_NULL))
{

console_print ( "could not initialize app " );
console_print ( iadix_mod.name );
console_print ( "\n" );
return 0;

}

//initialize application context

daemonize ("purenode");

//start application
app_start (PTR_NULL);

while (isRunning())
{

app_loop(PTR_NULL);

}

app_stop(PTR_NULL);

}

The dynamic loader take care of registering module exported interface, as well as symbol decoration differences between compilers, as long as the calling convention is identical (compiled with same std_def.h)

The same binary module can be loaded and executed from any operating system host, provided it can execute x86 machine code from memory. Parameters types are handled automatically by the runtime based on runtime type information from the libbase tree.

Using node script definition

let NODE_MODULE_DEF protocol_adx = `{"order":0, "file" : "modz/protocol_adx.tpo"}`
let NODE_MODULE_DEF block_adx = `{"order":1, "file" : "modz/block_adx.tpo"}`
let NODE_MODULE_DEF wallet = `{"order":2, "file" : "modz/wallet.tpo"}`
let NODE_MODULE_DEF node_adx = `{"order":3, "file" : "modz/node_adx.tpo"}`
let NODE_MODULE_DEF nodix = `{"order":4, "file" : "modz/nodix.tpo"}`

The runtime will automatically load the modules from the file at specified path, and store the module internal representation in a child node of the module container object. If a module with the same path is already loaded, it will copy a reference to it in the target variable.

let NODE_GFX_OBJECT configuration = ` { "name" : "nodix", "seed_node" : { "host" : "iadix.com", "port" : 16714 } } `

proc my_proc = ` node_adx.connect_peer_node ( configuration.seed_node ) `

Functions exported in the binary modules can then be invoked as an abstract interface to manipulate objects with the corresponding abstract type.

Informations about the interfaces implemented in the modules can be accessed dynamically through the NODE_MODULE_DEF container object, which provide an abstraction layer over binary module export definition.

Modules interfaces exported over rpc or cgi services can be invoked via the corresponding HTTP API eg. from javascript