Distributed dynamic object tree memory model.
The libbase framework is based on the memory reference mechanism implemented in libcon to provide lockless atomic reference counted acess to dynamic distributed object with support for multi typed arrays, memory aligned for SIMD operations, and built in primitives for blockchain technologies.
All access to distributed objects is made lockless and thread safe with the built-in double buffered write on object childs list, all read only access are lockless, and only blocking when more than one thread modify the list .
Nodix is very similar in design to Emerald, a language made in the early 90's as an attempt to bridge the gap between programing local and distributed Object Oriented applications.
Object Structure in the Emerald System, Oopsla 1986.
2. Emerald Objects
All entities in the Emerald system are objects. This includes small entities, such as Booleans and integers, and large entities, such as directories and compilers.
While different objects may be implemented with different techniques, all objects exhibit uniform semantics. An object can be manipulated only through invocation; no external access to an object's data is permitted.
Objects can be invoked remotely and can move from node to node. Each Emerald object has four components:
1. A name, which uniquely identifies the object within the network.
2. A representation, which consists of the data stored in the object. The representation of a programmerdefined object is composed of a collection of references to other objects.
Emerald supports concurrency both between objects and within an object. Within the network many objects can execute concurrently. Within a single object, several operation invocations can be in progress simultaneously, and these can execute in parallel with the object's internal process. To control access to variables shared by different operations, the shared variables and the operations manipulating them can be defined within a monitor [10, 16]. Processes synchronize through builtin condition objects. An object's process executes outside of the monitor, but can invoke monitored operations should it need access to shared state.
3. Abstract Types
Central to Emerald is the concept of abstract type. An abstract type defines a collection of operation signatures, that is, operation names and the types of their arguments and results. All identifiers in Emerald are typed: the programmer must declare the abstract type of the objects that an identifier may name. An abstract type is represented by an Emerald object that specifies such a list of signatures.
The relationship between abstract types and object implementations is many-to.one in both directions. A single object may conform to many abstract types, and an abstract type may be implemented by many different objects. Although Emerald requires that the abstract type of each identifier be manifest, the type of the object that is to be assigned to an identifier may not be known until run time. In such a case, the conformity check will be performed at run time. However, very often enough information will be available at compile time for conformity to be checked statically.
4. Object Creation
As described above, an identifier in Emerald programs has an abstract type, and an object must conform to that abstract type to be named by the identifier. However, Emerald objects do not require a Class object for their creation. In most object-based systems, the programmer first specifies a class object that defines the structure and behavior of all its instances. The class object also responds to new invocations to make new instances. In contrast, an Emerald object is created by executing an object constructor. An object constructor is an Emerald expression that defines the representation, the operations, and the process of an object.
6. Supporting Multiple implementations
The most important goal of the Emerald design is the support of a uniform object model. The semantics of all objects, whether large or small, local or distributed, should be independent of the implementation technique. This uniformity should hold both for the progranuner who builds objects and types, and for the application that invokes them.
7. Distribution Support
Emerald is designed for the construction of distributed applications. As previously stated, we believe that objects are an excellent way of structuring such programs because they provide the units of processing and distribution. This belief has been confirmed by our experience with distributed applications in Eden [1-3, 6]. The tendency of many distributed systems is to hide distribution from the programmer. For example, in Xerox RPC [5], remote procedure calls were added to Cedar Mesa. In so far as it was possible, remote procedure calls were designed to be semantically identical to local procedure calls. This is obviously a desirable property and is what makes RPC so attractive; programs can be written and debugged on a single node using local procedures and then easily distributed.
Emerald supports the same notion with object invocation. All objects are manipulated through invocation, and all invocations are location independent; it is the responsibility of the run-time system to locate and transfer control to the target object. Remote invocation achieves the same benefits as remote procedure call.
While it is crucial that invocation be location independent, it is not necessary that an object's location be invisible. Many applications may choose to ignore distribution, but others may wish to benefit from location dependence. For example, a replication manager may wish to distribute object replicas on different nodes, or two applications may wish to be co-located during periods of high activity. Applications that are concerned with distribution may wish to discover and modify objects' locations, but they still benefit from location-independent invocation.
9. Conclusions
Languages like Smalltalk rely heavily on the concept of Class. However, Classes have at least three functions:they generate instances, they act as a repository for the code of those instances, and (through the inheritance hierarchy) they provide a classification scheme for instances. Emerald allocates these functions to separate mechanisms: objects are created by explicit constructors, code sharing is managed by the kernel, and abstract types provide a classification scheme that is independent of an object's implementation.
Nodix's nodes can conceptually be viewed as Emerald mobile objects defined in json with optional abstract type.
Applications create and manipulate distributed objects as a reference to a generic node, and operations on those objects as well specialized constructor are invoked either via interfaces defined in modules (binary or script), or using asynchronous message handling mechanism similar to SmallTalk.
Each node is associated with an abstract type, a name, and its data representation, and can also contain a dynamic list of reference to other nodes. This dynamic list can be used both to represent typed element for arrays and list or key/value pairs for objects.
Abstract types are represented as an integer, and built-in types have a static mapping between type name and type identifiers.
Nodes can be considered as simple type (integer, float, string, hash,vec3, mat3x3, der signature), or typed list of heteroclytes nodes (multi type arrays), or objects.
Nodes data is only manipulated using runtime function passing a reference to the node, as well as the type of the input/output value.
RPC is achieved using a specific module interface bound to json/rpc protocol over http through the node's service configuration, for easy communication with javascript. All RPCs are executed in parallel if the client language support asynchronous http request.
The API to create new nodes, access the tree, as well as built in abstract type identifiers are defined in libbase/include/tree.h
Memory model of distributed objects.
API definition in C
The C api provide the high level semantic to manipulate distributed object.
They are divided in two type of operations
- operations affecting leaf node data (simple type mode).They will always try to convert the node's data to the requested type, or fail if they can't access the value as the requested type.
- operations affecting the node's child list (object / array mode).They will always affect a temporary write buffer with exclusive access, so all operation on dynamic arrays and object keys are 'thread atomic', they only appear as a whole when they are fully completed, or fail as a whole without affecting the original node.
A generic constructor function is used to create new 'root nodes' associated with a type, and references to it are released manually using the generic libcon api.
A specialized destructor is automatically added on nodes to release references on their child lists recursively when their reference count reach zero.
Simple type node manipulation.
General node manipulation
int tree_manager_create_node (const char *name,unsigned int type,mem_zone_ref_ptr ref_ptr);
const char * tree_mamanger_get_node_name (mem_zone_ref_const_ptr node_ref);
unsigned int tree_mamanger_get_node_type (mem_zone_ref_const_ptr node_ref);
unsigned int tree_mamanger_get_node_type_str (const char *name);
int tree_manager_node_dup (mem_zone_ref_ptr new_parent, mem_zone_ref_const_ptr src_ref_ptr, mem_zone_ref_ptr new_ref_ptr);
int tree_manager_node_dup_one (mem_zone_ref_ptr src_ref_ptr, mem_zone_ref_ptr new_ref_ptr);
node write functions
tree_manager_write_node_[type] (mem_zone_ref_ptr node_ref,mem_size ofset,type value); | |
|---|---|
| uint64_t value | write the specified 64 bits integer value in the node's data at the specified ofset, and automatically allocate memory if necessary. |
| unsigned int value | write the specified 32 bits integer value in the node's data at the specified ofset, and automatically allocate memory if necessary. |
| unsigned short value | write 16 bits integer in the node's data at the specified ofset, and automatically allocate memory if necessary. |
| unsigned char value | write 8 bits integer in the node's data at the specified ofset, and automatically allocate memory if necessary. |
| int value | write the specified 32 bits signed integer value in the node's data at the specified ofset, and automatically allocate memory if necessary. |
| int64_t value | write the specified 64 bits signed integer value in the node's data at the specified ofset, and automatically allocate memory if necessary. |
| short value | write the specified 16 bits signed integer value in the node's data at the specified ofset, and automatically allocate memory if necessary. |
| mem_ptr value | write the specified pointer in the node's data at the specified ofset, and automatically allocate memory if necessary. |
| float value | write the specified 32 bits float value in the node's data at the specified ofset, and automatically allocate memory if necessary. |
| double value | write the specified 80 bits double value in the node's data at the specified ofset, and automatically allocate memory if necessary. |
| const hash_t hash | write the specified 32 bytes (256bits) hash in the node's data at the specified ofset, and automatically allocate memory if necessary. |
| const btc_addr_t addr | write the specified 34 characters base58 bitcoin public address in the node's data at the specified ofset, and automatically allocate memory if necessary. |
| const struct string *str | copy the specified string in the node's data at the specified ofset, and automatically allocate memory if necessary. |
| const vec_4uc_t val | write the 4 specified unsigned char (eg RGBA color) in the node's data at the specified ofset, and automatically allocate memory if necessary. |
| const char *str | copy the specified C string in the node's data at the specified ofset, and automatically allocate memory if necessary. |
| const_mem_ptr vint | copy the specified bitcore variable integer binary representation in the node's data at the specified ofset, and automatically allocate memory if necessary. |
| unsigned char *sign, size_t sign_len | write the DER signature in the node's data from the binary representation at the specified ofset, and automatically allocate memory if necessary. |
| unsigned int value | Atomically swap the specified node's data with the specified 32 bits value if the current value is zero and return 1, or return 0 otherwise. |
node read functions
tree_manager_read_node_[type] (mem_zone_ref_ptr node_ref,mem_size ofset,type *value); | |
|---|---|
| uint64_t *val | read the node's data and convert it to a 64 bit unsigned integer. |
| unsigned int *val | read the node's data and convert it to a 32 bit unsigned integer. |
| unsigned short *val | read the node's data and convert it to a 16 bit unsigned integer. |
| unsigned char *val | read the node's data and convert it to a 8 bit unsigned integer. |
| int64_t *val | read the node's data and convert it to a 64 bit signed integer. |
| int *val | read the node's data and convert it to a 32 bit signed integer. |
| short *val | read the node's data and convert it to a 16 bit signed integer. |
| mem_ptr *value | read the node's data and convert it to a memory pointer. |
| float *val | read the node's data and convert it to a 32 bit float. |
| double *val | read the node's data and convert it to a 80 bit double. |
| hash_t hash | read the node's data and convert it to a 256 bit hash. |
| btc_addr_t addr | read the node's data and convert it to a 34 bytes bitcoin public address. |
| mem_ptr vstr | read the node's data and convert it to a bitcoin variable string into the destination memory. |
| vec_4uc_t val | read the node's data and convert it to a 4 bytes vector. |
| char *str,unsigned int str_len,unsigned int base | read the node's data and copy it to C string, of specified size, using the specific radix for conversion from integers data. |
| struct string *str,unsigned int base | read the node's data and copy it to a string, using the specific radix for conversion from integers data. |
| unsigned int val | Evaluate the node's data as 32 unsigned integer and compare it with the given 32 bits integer value. |
| unsigned int crc | compare a given hash to the node's name. |
Raw write on nodes data
int tree_manager_allocate_node_data (mem_zone_ref_ptr node_ref,mem_size data_size);
Allocate node's data to the specified size.
mem_ptr tree_manager_expand_node_data_ptr(mem_zone_ref_ptr node_ref, mem_size ofset, mem_size size);
int tree_manager_write_node_data (mem_zone_ref_ptr node_ref,const_mem_ptr data,mem_size ofset,mem_size size);
Write raw binary data in the node's data at speficied ofset
int tree_manager_copy_node_data (mem_zone_ref_ptr dst_node,mem_zone_ref_const_ptr src_node);
copy the raw binary data from source node to destination node.
raw read on node data
mem_ptr tree_mamanger_get_node_data_ptr (mem_zone_ref_const_ptr node_ref,mem_size ofset);
size_t tree_mamanger_get_node_data_size (mem_zone_ref_const_ptr node_ref);
List and object node manipulation.
General children manipulation
int tree_mamanger_get_parent (mem_zone_ref_const_ptr node_ref,mem_zone_ref_ptr p_ref);
int tree_manager_get_ancestor_by_type (mem_zone_ref_const_ptr node_ref,unsigned int node_type,mem_zone_ref_ptr p_ref);
size_t tree_manager_get_node_num_children (mem_zone_ref_const_ptr p_node_ref);
int tree_manager_get_child_at (mem_zone_ref_const_ptr parent_ref_ptr ,unsigned int index,mem_zone_ref_ptr ref_ptr);
int tree_manager_find_child_node (mem_zone_ref_const_ptr parent_ref_ptr ,unsigned int crc_name,unsigned int type,mem_zone_ref_ptr ref_ptr);
int tree_manager_add_child_node (mem_zone_ref_ptr parent_ref_ptr,const char *name,unsigned int type,mem_zone_ref *ref_ptr);
unsigned int tree_manager_node_add_child (mem_zone_ref_ptr parent_ref_ptr,mem_zone_ref_const_ptr child_ref_ptr);
int tree_manager_copy_children (mem_zone_ref_ptr dest_ref_ptr,mem_zone_ref_const_ptr src_ref_ptr);
int tree_manager_copy_children_ref (mem_zone_ref_ptr dest_ref_ptr, mem_zone_ref_const_ptr src_ref_ptr);
child node write functions
tree_manager_set_child_value_[type] (mem_zone_ref_ptr p_node_ref, const char *name, type value); | |
|---|---|
| uint64_t value | write the specified 64 bits integer value in the node's child with the specified name, and automatically allocate memory in the node's child if necessary. |
| unsigned int value | write the specified 32 bits integer value in the node's child with the specified name, and automatically allocate memory in the node's child if necessary. |
| unsigned short value | write the specified 16 bits integer value in the node's child with the specified name, and automatically allocate memory in the node's child if necessary. |
| int64_t value | write the specified signed 64 bits integer value in the node's child with the specified name, and automatically allocate memory in the node's child if necessary. |
| int value | write the specified signed 32 bits integer value in the node's child with the specified name, and automatically allocate memory in the node's child if necessary. |
| mem_ptr value | write the specified pointer value in the node's child with the specified name, and automatically allocate memory in the node's child if necessary. |
| float value | write the specified float value in the node's child with the specified name, and automatically allocate memory in the node's child if necessary. |
| double value | write the specified double value in the node's child with the specified name, and automatically allocate memory in the node's child if necessary. |
| const hash_t str | write the specified 256bit hash value in the node's child with the specified name, and automatically allocate memory in the node's child if necessary. |
| const btc_addr_t str | write the specified 34 bytes btc public address value in the node's child with the specified name, and automatically allocate memory in the node's child if necessary. |
| const struct string *str | write the specified string value in the node's child with the specified name, and automatically allocate memory in the node's child if necessary. |
| const_mem_ptr vint | write the specified bitcore variable integer value in the node's child with the specified name, and automatically allocate memory in the node's child if necessary. |
| const vec_4uc_t value | write the specified 4 bytes vector value in the node's child with the specified name, and automatically allocate memory in the node's child if necessary. |
| const char *str | write the specified C string value in the node's child with the specified name, and automatically allocate memory in the node's child if necessary. |
| const struct gfx_rect *rect | write the specified rect structure in the node's child with the specified name, and automatically allocate memory in the node's child if necessary. |
| float x, float y, float z | write the specified 3 float vector value in the node's child with the specified name, and automatically allocate memory in the node's child if necessary. |
| size_t idx,float x, float y, float z | write the specified 3 float vector value in the node's child with the specified name at the specified index, and automatically allocate memory in the node's child if necessary. |
| float *mat3x3 | write the specified 3x3 float matrix in the node's child with the specified name, and automatically allocate memory in the node's child if necessary. |
| ipv4_t value | write the specified ip value in the node's child with the specified name, and automatically allocate memory in the node's child if necessary. |
child node read functions
tree_manager_get_child_value_[type] (mem_zone_ref_const_ptr p_node_ref, unsigned int crc_name, type *value); | |
|---|---|
| uint64_t value | read the a bits integer value from the node's child with the specified name. |
| unsigned int value | read a 32 bits integer value from the node's child with the specified name. |
| unsigned short value | read a 16 bits integer value from the node's child with the specified name. |
| int64_t value | read a signed 64 bits integer value from the node's child with the specified name. |
| int value | read a signed 32 bits integer value from the node's child with the specified name. |
| mem_ptr value | read a pointer value from the node's child with the specified name. |
| float value | read a float value from the node's child with the specified name. |
| const hash_t str | read a 256bit hash value from the node's child with the specified name. |
| const btc_addr_t str | read a 34 bytes btc public address value from the node's child with the specified name. |
| const struct string *str | read a string value from the node's child with the specified name. |
| const vec_4uc_t value | read a 4 bytes vector value in the node's child with the specified name. |
| const char *str | read a C string value in the node's child with the specified name. |
| const struct gfx_rect *rect | read a rect structure in the node's child with the specified name. |
| ipv4_t value | read a ip value in the node's child with the specified name. |
Childs node iteration
int tree_manager_get_first_child (mem_zone_ref_const_ptr p_node_ref, mem_zone_ref_ptr child_list, mem_zone_ref_ptr *p_node_out_ref);
int tree_manager_get_next_child ( mem_zone_ref_ptr child_list, mem_zone_ref_ptr *p_node_out_ref);
int tree_manager_get_last_child (mem_zone_ref_const_ptr p_node_ref, mem_zone_ref_ptr child_list, mem_zone_ref_ptr *p_node_out_ref);
int tree_manager_get_prev_child (mem_zone_ref_const_ptr child_list,mem_zone_ref_ptr *p_node_out_ref);
int tree_node_find_child_by_name (mem_zone_ref_const_ptr p_node_ref, const char *name, mem_zone_ref_ptr p_node_out_ref);
unsigned int tree_node_list_child_by_type (mem_zone_ref_const_ptr p_node_ref, unsigned int type, mem_zone_ref_ptr p_node_out_ref, unsigned int index);
int tree_node_find_child_by_type_value (mem_zone_ref_const_ptr p_node_ref, unsigned int type,unsigned int value ,mem_zone_ref_ptr p_node_out_ref);
int tree_node_find_child_by_type (mem_zone_ref_const_ptr p_node_ref, unsigned int node_type ,mem_zone_ref_ptr p_node_out_ref,unsigned int index);
int tree_node_find_child_by_id (mem_zone_ref_const_ptr p_node_ref, unsigned int node_id ,mem_zone_ref_ptr p_node_out_ref);
int tree_find_child_node_by_id_name (mem_zone_ref_const_ptr p_node_ref, unsigned int child_type, const char *id_name, unsigned int id_val, mem_zone_ref_ptr out_node);
int tree_find_child_node_idx_by_id (mem_zone_ref_const_ptr p_node_ref, unsigned int child_type, unsigned int child_id, unsigned int *out_idx);
int tree_find_child_node_by_member_name (mem_zone_ref_const_ptr p_node_ref, unsigned int child_type, unsigned int child_member_type,const char *child_member_name,mem_zone_ref_ptr out_node);
int tree_find_child_node_by_member_name_hash(mem_zone_ref_const_ptr p_node_ref, unsigned int child_type, const char *child_member_name,hash_t hash, mem_zone_ref_ptr out_node);
Advanced children manipulation
int tree_swap_child_node_by_id (mem_zone_ref_ptr p_node_ref,unsigned int id_val,mem_zone_ref_ptr node);
int tree_manager_swap_child_ref (mem_zone_ref_const_ptr parent_ref_ptr, unsigned int crc_name, unsigned int type, mem_zone_ref_ptr ref_ptr);
void tree_manager_sort_childs (mem_zone_ref_ptr parent_ref_ptr,const char *name,unsigned int dir);
int tree_manager_list_child_type (mem_zone_ref_ptr child_list, unsigned int type, unsigned int *index, mem_zone_ref_ptr *p_node_out_ref);
int tree_node_eval_i64 (mem_zone_ref_const_ptr p_node_ref, const char *key, enum op_type op,int64_t ivalue);
int tree_node_keval_i64 (mem_zone_ref_const_ptr p_node_ref, const struct key_val *key);
int tree_node_keval_hash (mem_zone_ref_const_ptr p_node_ref, const struct key_val *key);
int tree_node_keval_str (mem_zone_ref_const_ptr p_node_ref, const struct key_val *key);
int tree_node_keval (mem_zone_ref_const_ptr p_node_ref, struct key_val *key);