1 Introduction

This user guide provides a comprehensive overview of IvoryScript, a functional programming language designed for dynamic data types, uniform data persistence and explicit control over evaluation.

It explains key concepts and practical usage, highlighting distinctive features compared to more conventional languages. Whilst this guide focuses on practical aspects of the language, a formal specification of the syntax and semantics is provided by the reference manual.

Many of the syntactic constructs ultimately transform into a much simpler core language. One particular feature of which is that all evaluation is explicit, so the meaning of reduction and evaluation operators and coercion for different syntactic terms is explained fully to enable a comprehensive understanding of the meaning of expression values.

Although the language is generally purely functional, mutable heap variables are supported for changes of application state.

Programming in the language involves scripts written in its grammar, which a compiler translates into either bytecode for an interpreter or C++ source code for further compilation by external tools.

1.1 Core language and modules

IvoryScript has a deliberately minimal core language. The core provides the fundamental building blocks: types, expressions, function application, lazy and strict evaluation, name literals, and the class and instance mechanism. It does not include concrete sequence operations, arithmetic operators, selection, or any built-in collection type. These are all supplied by modules.

The most important module is the Prelude, which is loaded automatically for every script. The Prelude defines the foundational classes that give IvoryScript its characteristic expressive power:

• Num, Eq, Ord — arithmetic and comparison;

• Select — dot-notation property selection;

• Seq, IndexableSeq, ReverseableSeq — sequence operations;

• Textual, Show — text formatting and output.

Because selection and arithmetic are class-based rather than built-in, a dot expression such as store.value and an addition such as x + y are both ordinary function applications resolved through the class system. This uniformity means the same operators work seamlessly for any type that provides the appropriate instance — user-defined types included.

Beyond the Prelude, a family of modules covers the requirements of typical application development. These fall into several natural groups:

• Sequences. List and ListFunction provide the lazy list type [a] and its processing functions; StringList adds string-as-character-sequence operations. Vector and Array cover mutable and fixed-size indexed collections; Tree supports hierarchical data. All share the Seq and IndexableSeq interfaces defined in the Prelude, so the same operations apply across collection types. The bracket notation [e1, e2, ...] used in list literals also appears in collection constructors — for example PropertySet [alice: 42, bob: "Hello, world"] first builds a [Property] list before the constructor processes it.

• Name/property collections. Property, PropertySet, Binding, NameAnyMap, and their companion function modules form the core of IvoryScript's application modelling approach: arbitrary sets of named values that can be assembled, inspected, and modified at runtime without a fixed schema. HashTable and HashTableFunction provide efficient key-indexed storage when the key set is not known in advance.

• I/O and streams. IO covers basic input/output; Streams provides typed byte-stream reading and writing with StreamExtract and StreamInsert instances across all transferable types; Serialisable adds higher-level object serialisation.

• Persistence. TransientStore and TransientDataStore give scripts named, persistent key-value roots backed by the environment model; SimpleRoot provides in-process global state without serialisation overhead.

• Numeric and mathematical. Arith extends the Prelude's Num class with rounding, absolute value, and conversion functions; MathConst supplies standard constants; NumAggregate covers sum, product and statistical aggregates over sequences; IntSeq provides number-theoretic sequences such as primes.

• Dynamic and polymorphic operations. The Any module defines the Any type used in property collections; AnyNum, AnySeq and AnyBindingSet extend numeric, sequence and binding-set operations to dynamically-typed values, enabling generic code that works across heterogeneous collections.

The reference manual covers the core language exclusively. This guide is structured in two parts. Part 1 describes core language features: types, expressions, names, evaluation, and the language-level optimisations that matter for practical use. Part 2 introduces the Prelude and the essential modules required for most practical scripts, including list processing, persistence, and name/property collections.

Readers new to IvoryScript are recommended to begin with Chapters 1–7 (lexical structure, script forms, namespaces, types, evaluation, and expression forms), then move directly to Chapter 14 (the Prelude) and Chapter 15 (Sequences). Those seven chapters provide sufficient grounding to write useful, idiomatic scripts. The remaining Part 1 chapters — name optimisation, coercion, tail recursion, garbage collection, and embedded deployment — and the later Part 2 modules can be consulted as specific needs arise.

As IvoryScript continues to evolve, this guide will be updated to reflect new features - such as derived classes etc.

2 Lexical overview

Scripts consist of ASCII characters that are are subdivided into tokens based on the following rules.

2.1 Comments

Two types of comments are supported:

• Single-line comments start with -- and continue to the end of the line.

• Nested comments start with {- and end with -}. Comment symbols within string codes do not introduce comments.

2.2 Whitespace and tokens

Whitespace characters (spaces, tabs, newlines) serve as separators between tokens like identifiers, keywords, literals and operators. Consecutive whitespace characters are treated as a single separator.

2.3 Names

A name consists of letters, digits and underscores. It must start with a letter, and names are used in different contexts dependent on the case of the first letter. e.g. x, Ptr and temp_123 etc.

2.4 Keywords

Lowercase words like

class, inline, let, module and type etc.

are reserved keywords with special meaning and so unavailable for use as general identifiers. For a full list see.

2.5 Special symbols

Special symbols, such as ;, (), [], :=, ->, ::, !, ^ and #, have specific meaning as described in the language syntax. For a full list see.

2.6 Numeric literals

Numeric literals can be either integer or floating-point:

• Integer literals consist of digits (e.g. 6144). Hexadecimal form if there is a 0x or 0X prefix (e.g. 0x23FF).

• Floating-point literals consist of digits, a decimal point and may include an exponent (e.g. 3.14e2).

2.7 Character and string literals

Character literals are single characters enclosed in single quotes, and they support 'C' style escape sequences (e.g. '\n' for newline). For a full list see.

String literals are sequences of characters enclosed in double quotes and also support the same escape sequences (e.g. 'Hello, world\n').

3 Script forms: Order and Module

IvoryScript supports two primary script forms: Order and Module.

3.1 Order script

An Order script is typically associated with an application, such as a session in the console. It comprises a sequence of expressions that are executed in order. Each step in a sequence is typically either an action or a value to be shown.

An example of an Order script:

tan (radians 45);
sqrt 2;
let {
   x :: Int = 10;
   y :: Int = x * 2;
   z :: Int = y + 5
} in
   z
      

For the last, the variables x, y and z are assigned sequentially, and then the value of z is shown.

3.2 Module script

Module scripts are designed to define reusable names associated with types, classes, functions etc. Modules serve as libraries or packages that can be imported and used elsewhere in an application. Although the primary use is for declarations and definitions that are initialised in definition order, initialisation can cause side effects - although this is not something to be generally recommended.

An example of a Module script:

module MathOperations where {

square :: Int -> Int;
square x = x * x;

cube :: Int -> Int;
cube x = x * x * x;

sideEffect :: Int = {show "Module initialised"; 42}

}
      

which defines reusable functions, but also includes a side effect that is executed when the module is initialised.

4 Namespaces

Names may occur in four distinct namespaces.

• Type namespace

• Class namespace

• Module namespace

• Value namespace

4.1 Naming convention

In IvoryScript, Type, Class, and Module names must begin with a capital letter. In contrast, names in the Value namespace can begin with either an uppercase or lowercase letter.

A name can appear in multiple namespaces. For example, Point can be a type constructor in the Type namespace and a data constructor in the Value namespace:

type Point a = Point a a               -- 'Point' as both type constructor and data constructor
let p :: Point Double = Point 1.0 1.0  -- 'Point' constructs a value of type Point
      

In this case, Point is a type constructor when defining the type and a data constructor when creating a value.

4.2 Type namespace

The Type namespace holds type constructor names, including built-in types (Int, Double) and custom types.

type Point a = ...
      

Here, Point is introduced as a type constructor in the Type namespace.

4.3 Class namespace

The Class namespace contains type classes, which define sets of functions or operations that can be implemented for various types.

class Point a where
  move :: a -> a
      

In this case, Point is a type class in the Class namespace that defines a move function.

4.4 Module namespace

The Module namespace holds module names, which group related functions, types, and values. Modules allow for code organization and reusability.

module Point where
  origin = Point 0 0
      

Here, Point is the name of a module in the Module namespace.

4.5 Value namespace

The Value namespace holds variables, functions, constants, and data constructors—names representing values or operations on them.

let p = Point 1.0 1.0
      

In this example, Point is a data constructor in the Value namespace, used to create a value of type Point.

4.6 Summary

5 Data types and signatures

5.1 Introduction

IvoryScript’s type system is designed to maintain clarity and safety, ensuring that values are used consistently with their declared types. This chapter introduces the core types in IvoryScript, including primitive types, type signatures, function types, and the essential Ptr a. Types such as List and Bool are not part of the core language but are expected to be present in the prelude or list modules.

5.2 Primitive types

IvoryScript supports several primitive types that serve as the foundation for more complex structures. These include:

• Int: Represents integer numbers.

                 
  let x = 42  -- x is inferred as Exp Int
  let y = !x  -- y is of type Int
          
              

• Double: Represents floating-point numbers.

                 
  let z = 3.14  -- z is inferred as Exp Double
  let w = !z    -- w is of type Double
          
              

• Char: Represents a single character.

                 
  let letter = 'a'  -- letter is inferred as Exp Char
  let strictLetter = !letter  -- strictLetter is of type Char
          
              

• String: Represents a sequence of characters (a list of Char).

                 
  let greeting = "Hello, World!"  -- greeting is inferred as Exp String
  let strictGreeting = !greeting  -- strictGreeting is of type String
          
              

Primitive types are inferred as lazy expressions (Exp a) by default. When a value is forced with ! or #, it changes to its strict type a.

5.3 Name and Type

IvoryScript introduces two critical built-in types:

• Name: Typically, name values result from a literal, such as #Mary, or from the special binding syntax, e.g., Mary:"Had a little lamb". For the latter, a pattern Bind name _ matches the name.

• Type: Type values can be created from a literal type signature, such as let typeOfVar = #::Int, and are also the return value of the typeOf function, which provides reflection capabilities.

These types are essential for working with more dynamic aspects of IvoryScript, allowing you to reference and manipulate names and types directly.

5.4 Type signatures

Type signatures in IvoryScript explicitly declare the types of values and functions. These signatures ensure correctness and make the function's input and output types clear.

The general form of a type signature is:

         
name :: Type
    
      

For example, a function that adds two integers might have the following type signature:

         
add :: Int -> Int -> Int
    
      

This indicates that add takes two arguments of type Int and returns an Int. For lazy values, the signature might be:

         
add :: Exp Int -> Exp Int -> Exp Int
    
      

5.5 Function type signatures

Function types are a core part of IvoryScript’s type system. They define how functions take arguments and return results.

The arrow (->) separates argument types from the return type:

         
multiply :: Int -> Int -> Int
    
      

This shows that multiply takes two Int values and returns an Int. For functions that operate on lazy values:

         
multiply :: Exp Int -> Exp Int -> Exp Int
    
      

Function types can also be higher-order, where a function returns another function:

         
compose :: (b -> c) -> (a -> b) -> (a -> c)
    
      

Here, compose is a function that takes two functions as arguments and returns a new function.

5.6 Lazy and strict types

IvoryScript distinguishes between lazy and strict types:

• Exp a: Represents a lazy expression of type a. Lazy expressions are not immediately evaluated and only reduced when explicitly forced.

                 
  let x = 42  -- x is of type Exp Int
  let y = #42 -- y is of type Int
          
              

• a: Represents a strict, fully evaluated type.

Values are typically inferred as Exp a unless explicitly forced to be strict using the ! or # operators. This allows IvoryScript to manage evaluation efficiently.

5.7 Tuple types

Tuples group multiple values of potentially different types into a single structure. They are enclosed in parentheses, with values separated by commas.

         
let point = (3, 4)  -- point is of type Exp (Int, Int)
    
      

Strict tuples can be defined as well:

         
let strictPoint = !(point)  -- strictPoint is of type (Int, Int)
    
      

Type signatures for functions that use tuples follow this pattern:

         
first :: Exp (a, b) -> Exp a
    
      

This signature shows that the first function takes a lazy tuple and returns the first element lazily.

5.8 Ptr types

The Ptr a type is an essential polymorphic primitive type in IvoryScript. It represents a pointer to a value of type a and is primarily used for referencing memory or interacting with lower-level aspects of the runtime system. The use of Ptr a is primarily for operations that involve state changes or for efficient memory access in embedded or performance-critical applications.

• Ptr a: Represents a pointer to a value of type a.

                 
  let ptr = Ptr someValue  -- ptr is of type Exp (Ptr a)
          
              

The primary intention of Ptr a is for managing state changes or mutable references within an application. Although Ptr a can be used imperatively, doing so is generally less efficient than a pure, tail-recursive functional approach.

Pointers allow interaction with application state in a controlled manner:

         
let p = Ptr 10  -- p is of type Ptr Int
    
      

While Ptr a provides a way to handle mutable state, its use should be limited to genuine scenarios requiring changes to application state. Pure functional recursion often achieves similar results in a more efficient and declarative manner.

5.9 Conclusion

IvoryScript’s type system introduces essential types such as Name, Type, and Ptr a, alongside primitive types like Int and Char. The distinction between lazy (Exp a) and strict (a) types ensures that evaluations are performed only when necessary, allowing for efficient management of resources. Function types further provide a foundation for defining behavior, whether working with lazy or strict values.

6 Values, reduction and evaluation

6.1 Strict and lazy values

Everything in IvoryScript is treated as a value, including expressions that require further reduction or evaluation. Values can be strict or lazy, depending on their declaration.

A strict value is fully evaluated when used, while a lazy value represents an expression that can undergo further reduction when needed.

There is nothing special about functions returning lazy values. Coercion often transforms lazy values into their appropriate forms without requiring explicit reduction or evaluation.

6.2 Coercion and function types

The coercion mechanism automatically adjusts values to fit the context in which they are used, reducing the need for explicit use of (!) or !. However, because of this flexibility, it is essential to declare a function’s type, particularly for recursive functions.

The example below demonstrates how IvoryScript represents lazy and nested expressions:

         
typeOf (take 100 (!primes))
-- Output: Exp (Exp [Int])
    
      

In this case, Exp (Exp [Int]) indicates that the result of take 100 operates on a value that itself still requires further evaluation, even after a single reduction step. The outer Exp refers to the need for additional reductions to retrieve the result, while the inner Exp [Int] signifies that the list of integers has yet to be fully evaluated. Thus, Exp represents the delayed or pending computation that will eventually yield the final value.

This is quite different from many other languages in common use, where lazy values or deferred evaluations are less explicit or hidden from the developer. In IvoryScript, this flexibility requires managing types carefully, especially with recursive functions or lazy expressions that may need multiple steps of reduction before becoming usable values. Declaring types such as Exp explicitly allows control over how and when evaluation occurs, providing the benefit of lazy computation while maintaining clarity about the process.

7 Expression forms

7.1 Introduction

This chapter provides an overview of the primary expression forms in IvoryScript. The goal is to offer a practical understanding of how different expressions are used, balancing between detailed reference and introductory examples.

7.2 Literal expressions

Literal expressions represent direct values in IvoryScript. They include numbers, strings, booleans, and more.

Example:

         
42         -- Integer literal
"hello"    -- String literal
True       -- Boolean literal
    
      

Literal expressions are straightforward and are used to represent fixed values that do not require further reduction.

7.3 Variable expressions

Variable expressions refer to names that have been previously defined in a scope. They are used to access values stored in variables.

Example:

         
let {
    x = 10;
    y = x * 2
} in
    y   -- Variable 'y' refers to the result of 'x * 2'
    
      

Variables in IvoryScript are immutable, meaning once a value is assigned, it cannot be changed. Variable expressions rely on earlier bindings within their scope.

7.4 Function application

Function application is the most common way of invoking functions in IvoryScript. It consists of calling a function with one or more arguments.

Example:

         
let {
    square x = x * x;
    result = square 5
} in
    result  -- Applies the function 'square' to the argument 5
    
      

Function application is reduced by passing the argument(s) to the function and reducing the function body with these arguments.

7.5 Lambda expressions

Lambda expressions represent anonymous functions in IvoryScript. They are defined using the syntax \x -> expression where x is the argument.

Example:

         
let {
    add = \x y -> x + y;
    result = add 3 7
} in
    result  -- Applies the lambda function to arguments 3 and 7
    
      

Lambda expressions are useful for defining functions inline, especially when the function is only needed in a limited scope.

7.6 Let expressions

The let expression is used to introduce local bindings. Variables defined in a let block are only visible within that block.

Example:

         
let {
    x = 5;
    y = x + 3
} in
    y   -- Returns 8, as 'x' and 'y' are local to the let block
    
      

7.6.1 Introduction to 'let'

The let construct in IvoryScript is a fundamental building block for defining local variables and functions. It provides a way to bind expressions to names within a particular scope, allowing for modularity and reusability in code. The let construct is versatile, supporting both sequential and mutually recursive definitions.

7.6.2 Basic 'let' Usage

The most common usage of let involves binding expressions to names. These bindings are local to the block in which they are defined, and they are evaluated in sequence. Here is a basic example:

               
let {
    x = 10;
    y = x * 2;
    z = y + 5
} in
   z
    
            

In this example, the variables x, y, and z are defined locally within the let block, and the result of the expression is the final value of z. The variables are evaluated in the order they are written, making let a simple and powerful tool for local definitions.

7.6.3 Let and Function Definitions

The let construct also supports function definitions within its scope. This allows for defining local functions that can operate on the values bound in the same let block. Here's an example of a let block defining a function:

               
let {
    add x y = x + y;
    result = add 5 10
} in
    result
    
            

In this case, the function add is defined locally within the let block, and is then used to calculate the value of result. This allows for encapsulating functionality within the scope of the let expression.

7.6.4 Let and Recursive Functions

The let construct in IvoryScript also supports recursive and mutually recursive function definitions. This enables defining functions that refer to themselves or to each other within the same let block. Here's an example of a mutually recursive function pair:

               
let {
    isEven n = if n == 0 then True else isOdd (n - 1);
    isOdd n = if n == 0 then False else isEven (n - 1)
} in
    isEven 4
    
            

Here, the functions isEven and isOdd call each other, making them mutually recursive. Both are defined within the same let block, allowing for their mutual recursion to be expressed cleanly.

7.6.5 Evaluation Order in 'let'

In IvoryScript, the evaluation order within a let block is strictly sequential. This means that each expression is evaluated in the order it is written, and the result is bound to the corresponding name. If a variable depends on a previous definition, that previous expression must have been fully evaluated before the dependent variable can be used. This ensures a clear and predictable evaluation flow, avoiding potential ambiguities in execution.

However, if a function is defined within the let block and it is not immediately invoked, the function itself is treated as a value that can be evaluated later. This introduces the concept of delayed evaluation, where the function's body is not executed until the function is called. This behavior is particularly important when dealing with recursive or mutually recursive functions, as discussed in the next section.

7.6.6 Delayed Initialization of Closures in Mutually Recursive Functions

A significant challenge with the let construct arises when dealing with mutually recursive functions, especially in terms of the availability and initialisation of closures. When two or more functions are mutually recursive, the closure (the environment capturing the function's variables and references) for each function may not be fully initialised at the time the other functions try to reference it.

In practical terms, this means that while the names of mutually recursive functions are available within the same let block, their closures may be incomplete when one function tries to call the other. This delayed initialisation can result in runtime errors or undefined behavior if a function is invoked before its closure has been fully initialised.

For example, consider this mutually recursive pair of functions:

               
let {
    f x = if x == 0 then 1 else g (x - 1);
    g y = if y == 0 then 0 else f (y - 1)
} in
    f 5
    
            

In this example, f and g are mutually recursive, meaning they call each other. However, if the function g is invoked before its closure has been fully initialised (due to f calling it), it could result in incorrect behavior. To handle this, IvoryScript currently ensures that the names of these mutually recursive functions are available, but the full initialisation of their closures may be delayed until the functions are first invoked.

This is a key consideration when working with mutually recursive functions in IvoryScript. A common workaround is to structure recursive functions carefully, ensuring that any function that relies on a mutually recursive closure is invoked only after it has been fully initialised. Another approach is to split the mutually recursive functions across separate let blocks or modules, ensuring that each function's closure is fully constructed before mutual recursion occurs.

7.7 Conditional expressions

Conditional expressions are written using the if/then/else structure. IvoryScript evaluates the condition and returns the corresponding branch.

Example:

         
let {
    result = if x > 0 then "positive" else "non-positive"
} in
    result
    
      

The condition must reduce to a boolean, and the result depends on whether the condition holds true.

7.8 Case expressions

Case expressions allow for pattern matching against different values. They are useful for deconstructing data types and handling multiple cases explicitly.

Example:

         
let {
    describeList lst = case lst of {
        [] -> "Empty list";
        _ :+ _ -> "Non-empty list"
    }
} in
    describeList [1, 2, 3]  -- Matches the non-empty list case
    
      

Pattern matching with case expressions is flexible, allowing behavior to be defined for various structures of data.

7.9 Operator expressions

Operators like +, -, *, and == are used to perform arithmetic or logical operations.

Example:

         
let {
    sum = 1 + 2;
    isEqual = 5 == 5
} in
    (sum, isEqual)  -- Returns (3, True)
    
      

Operator expressions are syntactic sugar for function applications. For example, x + y is equivalent to add x y where add is the operator function.

7.10 Conclusion

The expression forms in IvoryScript provide the building blocks for defining logic, performing calculations, and manipulating data. From simple literals and variable bindings to more advanced constructs like pattern matching and user-defined types, expressions form the core of writing functional programs in IvoryScript.

8 Names and properties

IvoryScript provides a first-class Name type representing interned identifiers. Names are a core-language concept: the # literal syntax and the Eq Name instance are defined in the core. The Property type — a name paired with a dynamically-typed value — and the collection types built on it (PropertySet, NameAnyMap, TransientStore) are all module-provided and are covered fully in the Name/property collections chapter in Part 2. This chapter introduces the core concepts that those modules depend on.

8.1 The Name type

A Name value is an interned identifier. Two names are equal if and only if they represent the same identifier string. The # prefix introduces a name literal:

#alice
#totalAmount
#x
      

Name literals may appear wherever any other value appears. They can be stored in collections, passed as function arguments, returned as results, and compared using the Eq Name instance:

#alice = #alice    -- True
#alice = #bob      -- False
      

The underlying representation is a compact integer index rather than a character string, so equality comparison is O(1) in the common case. The performance characteristics of name comparison across different environments — and how to guarantee O(1) comparison in all cases using built-in name registration — are discussed in the following chapter.

8.2 Property

A Property, defined in the Property module, is a name/value pair where the value is of type Any. Any is a uniform wrapper that preserves the runtime type of any IvoryScript value; it is defined in the Any module and described fully in Part 2. The #-syntax name is the key; any expression may appear as the value.

The colon shorthand constructs a Property with the identifier on the left coerced to a Name literal. These two forms are equivalent:

-- Shorthand (colon notation)
alice: 42
bob:   \x :: Int -> x * x

-- Explicit equivalent
Property #alice 42
Property #bob   (\x :: Int -> x * x)
      

The colon notation is only meaningful as a Property constructor — it is not a general key/value separator in other contexts. The full collection types built on Property are introduced in Part 2.

8.3 The Select class and dot notation

Dot selection is not a built-in syntactic form in IvoryScript. It is defined entirely through the Select class, which is part of the Prelude:

class Select a, b where {
   (.) :: a -> Name -> b
};
      

An expression such as store.result desugars to an application of (.) with the right-hand identifier coerced to a Name value. Any type can participate in dot selection by providing a Select instance; there is no privileged treatment of any particular collection type.

Selection chains associate left: contact.address.city is parsed as (contact.address).city, first selecting address from contact, then selecting city from the result. Each step is a separate application of (.), potentially dispatching to a different Select instance.

The Select class is defined in the Prelude; instances for PropertySet, NameAnyMap, and TransientStore are provided by their respective modules and introduced in Part 2.

9 Built-in name optimisation

All identifiers are represented internally as compact integer indices, so equality is resolved by comparing those indices rather than by examining identifier text. Built-in names form an initial collection whose indices are fixed and shared across every environment. Names introduced within a script receive indices allocated in that script's environment but participate in comparison in the same way.

When a built-in identifier appears within a script, a corresponding local entry is created so that persistent data remains independent of any particular built-in set. The association with the built-in index is retained, ensuring that constant-time comparison continues to apply.

Name/value collections such as PropertySet and NameAnyMap are intended to be used throughout IvoryScript in place of the static record structures commonly found in other languages. Dynamic collections can be created, inspected, and combined without loss of efficiency, since all identifier comparisons are index-based. An example is shown below:

PropertySet [
   alice: 42,
   mary:  "Hello, world",
   bob:   \x :: Int -> x * x
]
   

Such structures can contain identifiers of any length, yet operations on them rely on index comparison rather than string matching.

9.1 Names across environments

A name/value access such as .store.p may involve values drawn from distinct environments: the environment holding the store variable, and the environment associated with each retrieved value. When two names originate from different environments their integer indices are only guaranteed to be equal if both refer to the same built-in name. Otherwise a string comparison is required to establish equality.

For a commonly used identifier this cross-environment string comparison can be avoided entirely by registering the identifier as a built-in name. In a C++ application file this takes the form:

static Name p_name = builtInName(p);
      

After registration, any occurrence of the corresponding name literal #p in a script, and any stored name that resolves to the same identifier, will carry the fixed built-in index. Equality comparisons involving #p then reduce to a single integer comparison regardless of how many environment boundaries are crossed.

9.2 When to register built-in names

Registration is most beneficial for identifiers that form a stable, application-wide vocabulary — property names accessed repeatedly across multiple stores or passed through many function calls. Identifiers that are purely local to a single script or that appear only rarely do not warrant registration.

Additional built-in identifiers may be defined by an application when predictable performance characteristics are required across all environments that the application creates.

10 Coercion and type casting

10.1 Introduction

In IvoryScript, coercion and type casting are essential mechanisms that ensure expressions conform to expected types, with a focus on small steps of graph reduction rather than full evaluation. Coercion is applied automatically when values are transferred between contexts, such as when passing function arguments or defining names.

When an expression requires type adaptation, IvoryScript represents it as:

(COERCE, expr)

During type checking, IvoryScript attempts to match the type of expr with the expected type. If the types align, the coercion is no longer needed, and the expression is used directly. Otherwise, the expression is transformed using a cast:

(!)(castexpr)

This transformation introduces the necessary graph reduction steps for cases where type adaptation is required, ensuring the expression can be used in the given context.

10.2 The General Cast Class

IvoryScript uses the Cast class as the general mechanism for handling all type conversions. Instances of the Cast class define how an expression of one type can be cast to another. While there are many types of casts, the most common involves graph reduction.

The instance that deals with reductions is defined as follows:

         
instance Cast Exp b, b where { 
   inline cast x = (!)x
};
    
      

This instance specifically handles the conversion from an expression of type Exp b to type b; a single step of graph reduction to the denoted value of x, ensuring that the expression is reduced to a value before it is used. Given that function application and constants are lazy, this cast is frequently encountered in practice.

10.3 Strict type casts

Casting between values of different strict types (where appropriate), such as from Int to Double, is also handled by the general Cast class. However, for strict types, the transformation is direct and does not involve reduction. For example, casting from Int to Double is achieved using the following instance of StrictCast:

         
primitive fromIntDouble :: Int -> Double;

instance StrictCast Int, Double where {
   inline strictCast = fromIntDouble;
   inline fromInt = fromIntDouble
};
    
      

In this case, the StrictCast instance uses a primitive function, fromIntDouble, to cast the value directly. This transformation occurs without coercion or reduction, as the cast involves only a straightforward type conversion between two strict types.

10.4 Transforming strict values to lazy expressions

IvoryScript also supports casting from strict values to lazy expressions, such as from a strict type a to Exp a. This conversion is handled using the general Cast class and is performed via instances of StrictCast that convert strict values into lazy expressions.

         
-- mkExpr :: a -> Expr a;
instance StrictCast a, Exp a | a ¬= Void where {
    inline strictCast x = mkExpr x
}
    
      

In this instance, the strict value is wrapped in a lazy expression of type Exp a using the mkExpr function. This cast is essential for supporting deferred computation and graph reduction in IvoryScript. The qualifier a¬= Void ensures that this transformation is valid for types that have representable values, excluding Void.

10.5 Summary

Coercion and type casting in IvoryScript are critical for managing type transformations while ensuring that expressions conform to expected types. The Cast class serves as the general mechanism for all conversions, and reduction is the most common type of cast due to the laziness of function application and constants:

• The Cast class handles all type conversions, including strict type casts and transformations between expressions and values.

• Instances of Cast (Exp a), a involve reduction using the reduceCast function, which reduces the expression step by step as required. This is the most frequently encountered cast.

• Casting between strict types, such as from Int to Double, involves direct conversion without reduction.

• Strict values can be transformed into lazy expressions, enabling deferred computation and graph reduction, through instances of StrictCast.

11 Tail recursion

11.1 How it works

If the result of a function is a direct function application, then the function call can be made simply by adjusting the stack appropiately and transferring control. This is referred to as a tail call and, if the call is recursive, then such a function is termed tail recursive.

Thus tail recursion allows the compiler to simply discard the current function’s stack frame, which results in efficient recursion with constant memory usage, similar to iterative loops.

Consider:

let sum n = 
  if n = 0 then 0 else n + sum (n - 1)
         

This function is not tail recursive because the recursive call sum (n - 1) is combined with the addition of n. The function cannot return directly from the recursive call; the addition must first be completed.

11.2 Example

To make the function tail recursive, the operation must be included in the recursive call. This is often done by introducing an accumulator:

let sum_tail n acc = 
  if n = 0 then acc else sum_tail (n - 1) (acc + n)

In this version, thesum_tail, making it tail recursive. The accumulator acc carries the result forward, allowing the function to be optimised and avoid the need for new stack frames.

11.3 Restructuring functions

To achieve tail recursion, a function may need to be restructured. The goal is to ensure that the recursive call is the final action in the function. This often involves introducing an additional parameter, such as an accumulator, to carry intermediate results. Here’s how a typical non-tail-recursive function might be restructured:

let factorial n = 
  if n = 0 then 1 else n * factorial (n - 1)

This function can be restructured into a tail-recursive version by using an accumulator:

let factorial_tail n acc = 
  if n = 0 then acc else factorial_tail (n - 1) (acc * n)

In this tail-recursive version, the recursive call is the final operation, and the intermediate result is carried through the accumulator.

11.4 Inline polymorphic functions

Inline polymorphic functions require special handling to achieve tail recursion. Because they are inlined directly, a tail-recursive function may need to delegate to a non-inline helper function which performs the actual recursion.

An example of this can be seen in the lengthList function, which computes the length of a list:

lengthList :: [a] -> Int;
inline lengthList l =
   let { lengthList_ :: [a] -> Int -> Int;
         lengthList_ l acc =
            case l of {
               [] ->        acc;
               _ :+ xs ->   lengthList_ ((!)xs) (acc + 1)
            }
   } in
      lengthList_ l 0;

The function lengthList is an inline polymorphic function, but the actual recursion occurs in the helper function lengthList_, which uses an accumulator to make the recursion tail recursive. The recursive call lengthList_ ((!)xs) (acc + 1) is the final operation, ensuring that the function remains efficient and tail-recursive while allowing the polymorphic behavior to be maintained.

11.5 Advantages

Tail recursion offers several key advantages:

• Improved performance: Reusing the stack frame for each recursive call reduces the overhead of recursion and leads to faster execution.

• Avoidance of stack overflow: Tail-recursive functions can handle deep recursion without risking stack overflow, making them more reliable for large data sets.

• Memory efficiency: Since tail-recursive functions do not accumulate stack frames, they use constant memory, even for complex or deeply recursive operations.

11.6 Future steps

While tail recursion must currently be implemented manually in IvoryScript, future enhancements are planned to introduce program analysis that can automatically detect and transform functions into their tail-recursive equivalents. This would allow the compiler to optimise recursion without requiring explicit restructuring by the developer.

For now, restructuring functions for tail recursion requires manual refactoring to ensure the recursive call is the final action, using accumulators or helper functions as necessary.

12 Garbage collection

12.1 'Mark_GC' class

The Mark_GC class is provided to support garbage collection of expression, function and Ptr values. There must be an instance of Mark_GC for every type, although the essential method mark_GC need only be defined if the representation of any constructor includes one or more primitive expression, function or Ptr values, or a value of nested type similarly.

The mark_GC method ensures that all reachable objects of the specified type are correctly processed by the garbage collector. This method traverses the structure of the object, marking any heap allocated values, which is essential for preventing the reclamation of live objects during memory management.

class Mark_GC a where
    mark_GC :: a -> ()

instance Mark_GC MyType where
    mark_GC v = case v of
        MyType x y -> {
            mark_GC x;
            mark_GC y;
        }

In this example, the mark_GC function uses a case expression to deconstruct MyType, marking each of its fields to ensure all components are handled during garbage collection.

12.2 Future derivation

Although every custom type currently requires an explicit Mark_GC instance with an appropriate definition of mark_GC, future developments are expected to allow automatic derivation of mark_GC based on the data constructor(s) representation function(s). This simplification would enable the compiler automatically to generate the necessary marking logic for most types without manual intervention and reduce the need for repetitive code in custom type definitions.

13 Embedded applications with limited resources

13.1 Introduction

IvoryScript is designed to underpin the Ivory System declarative event-driven model, which can be specifically tailored for resource-constrained applications. By natively supporting multiple environments and persistence, it is particularly suited for embedded systems operating under constraints on memory and communication bandwidth. This chapter explains how resource usage can be minimised by separating the compilation process, using remote bytecode interpretation and dynamic linking to the runtime system. It also explores how diverse applications can evolve dynamically and run non-stop with remote management for inspection and modification, enabling flexible deployment and control of a wide range of mobile units including vehicles, drones and spacecraft etc.

13.2 Lightweight deployment

In embedded systems where resources are constrained, the compiler can be omitted entirely. This reduces the memory footprint and computational overhead, as the only necessary components are the bytecode interpreter and runtime system.

• Local compilation and bytecode preparation: Order scripts are prepared on a local host with full compiler support. The resulting bytecode, which is compact and efficient, is then ready for transmission to the embedded device.

• Remote bytecode transmission and interpretation: The compiled bytecode is transmitted to the remote platform and dynamically linked to the runtime system. The lightweight interpreter on the device then executes the bytecode with no need to reboot or restart processes to execute new Order scripts - allowing updates and inspection to happen seamlessly in real-time.

13.3 Benefits

Applications can be updated dynamically as new operational requirements arise, all while minimising communication bandwidth and avoiding re-deployment. This flexibility ensures that systems can evolve over time and remain in continuous operation.

14 The Prelude

Part 2 of this guide covers the modules that form the practical foundation of IvoryScript programming. The first and most important of these is the Prelude.

The Prelude is loaded automatically for every IvoryScript script. It extends the core language with the classes and types that give the language its day-to-day expressive power, without which even basic operations such as addition or list traversal would be unavailable. Understanding the Prelude is therefore a prerequisite for reading the subsequent module chapters.

14.1 Foundational types

The Prelude defines several types that are used pervasively across all other modules:

14.2 Core classes

The Prelude defines all fundamental classes. Each is summarised below; instances for the built-in and module-defined types are described in the relevant module chapters.

14.2.1 Env

Every heap-allocated value in IvoryScript carries a reference to the environment (managed memory space) that owns it. The Env class abstracts over this relationship:

class Env a where {
   needsEnv  :: a -> Bool;
   mapToEnv  :: a -> Env -> a;
   copyToPtr :: a -> Ptr a -> Env -> Void
};
         

Scalar types (Int, Float, Double, Char) receive the default implementations — needsEnv returns False and mapToEnv is the identity. Heap-allocated types override needsEnv to return True and provide a mapToEnv that remaps all internal pointers to the destination environment. This mechanism underpins both garbage collection and persistence.

14.2.2 Eq and Ord

class Eq a where {
   (=)  :: a -> a -> Bool;
   (/=) :: a -> a -> Bool
};

class Eq a => Ord a where {
   (<)     :: a -> a -> Bool;
   (<=)    :: a -> a -> Bool;
   (>)     :: a -> a -> Bool;
   (>=)    :: a -> a -> Bool;
   compare :: a -> a -> Ordering;
   max     :: a -> a -> a
};
         

Instances are provided for all primitive types (Int, Float, Double, Char, Bool, Name) and for most module-defined types.

14.2.3 Num and FractionalNum

class Num a where {
   zero   :: a;
   succ   :: a -> a;
   pred   :: a -> a;
   (+)    :: a -> a -> a;
   (-)    :: a -> a -> a;
   (*)    :: a -> a -> a;
   (mod)  :: a -> a -> a;
   negate :: a -> a
};

class FractionalNum a where {
   (/) :: a -> a -> a
};
         

Because arithmetic is class-based, the operators +, -, * and mod work uniformly for any type that provides a Num instance. The primitive types Int, Float and Double all provide instances; Float and Double additionally provide FractionalNum.

14.2.4 Select

class Select a, b where {
   (.) :: a -> Name -> b
};
         

Dot-notation selection is an ordinary class method, not a built-in syntactic form. Any type providing a Select instance supports the dot operator. The right-hand identifier is coerced to a Name at the call site. Instances for PropertySet, NameAnyMap and TransientStore are provided by their respective modules.

14.2.5 Seq, IndexableSeq and ReverseableSeq

class Seq a where {
   length  :: a -> Int;
   tail    :: a -> a;
   take    :: Int -> a -> a;
   drop    :: Int -> a -> a;
   (++)    :: a -> a -> a
};

class Seq a => ReverseableSeq a where {
   reverse :: a -> a
};

class IndexableSeq a, b where {
   getAt :: a -> Int -> b;
   putAt :: a -> Int -> b -> Void;
   head  :: a -> b
};
         

These three classes define the standard sequence interface. Seq covers length, slicing, and concatenation. IndexableSeq adds random access and head extraction. ReverseableSeq adds reversal. Instances are provided by the List, Vector, and String modules.

14.2.6 Textual and Show

class Textual a where {
   format    :: a -> String -> String;
   stringify :: a -> String
};

class Show a where {
   show            :: a -> Void;
   showWithNewline :: a -> Void
};
         

Textual converts a value to a String, optionally via a format string. Show writes a value to the standard output stream; the default implementation calls stringify and then the Show String instance, so providing a Textual instance is usually sufficient.

14.2.7 Cast, StrictCast and Eval

class Cast a, b where {
   cast :: a -> b
};

class StrictCast a, b where {
   strictCast :: a -> b
};

class Eval a, b where {
   eval :: a -> b
};
         

These classes provide the type-directed coercion and evaluation mechanisms described in the Coercion chapter. Cast performs a lazy coercion; StrictCast forces evaluation before coercing; Eval reduces an expression to its value. The default Eval Exp b, b instance applies reduce.

14.3 Utility functions

In addition to class definitions, the Prelude provides a set of general utility functions available in every script:

15 Sequences

The Seq, IndexableSeq, and ReverseableSeq classes from the Prelude define common methods for working with ordered, traversable collections — length, head, take, drop, ++, and more. Any type that implements these classes can be used wherever a sequence is expected, regardless of its underlying structure. IvoryScript provides four such types in the standard library: lazy lists, suited to recursive functional processing, vectors for efficient mutable random-access storage, character lists and strings for text and binary trees for hierarchical data.

15.1 The sequence classes

The three sequence classes from the Prelude are declared as follows:

class Seq a where {
   length  :: a -> Int;
   tail    :: a -> a;
   take    :: Int -> a -> a;
   drop    :: Int -> a -> a;
   (++)    :: a -> a -> a
};

class Seq a => ReverseableSeq a where {
   reverse :: a -> a
};

class IndexableSeq a, b where {
   head  :: a -> b;
   getAt :: a -> Int -> b;
   putAt :: a -> Int -> b -> Void
};
      

Any instance defintion for a data type (which may be polymorphic) implements one or more of the type class methods; examples are the List, Vector, and StringList modules.

15.2 Lists

The list type [a], defined in the List module, is an important sequence type. A list is either empty or a pairing of a head value (which may be lazy) with a lazy list tail; making it natural for recursive processing and well-suited to sequences whose length is not known in advance. List literals use bracket syntax:

[1, 2, 3, 4, 5]           -- [Int]
['a', 'b', 'c']           -- [Char]
[[1, 2], [3, 4], [5, 6]]  -- [[Int]]
[]                         -- empty list (any element type)
      

let xs = [10, 20, 30, 40, 50]

head xs          -- 10
tail xs          -- [20, 30, 40, 50]
length xs        -- 5
take 3 xs        -- [10, 20, 30]
drop 3 xs        -- [40, 50]
xs ++ [60, 70]   -- [10, 20, 30, 40, 50, 60, 70]
reverse xs       -- [50, 40, 30, 20, 10]
getAt xs 3       -- 40
      

Sorting and merging ordered lists are provided by the List module directly:

mergeSort [5, 2, 8, 1, 9, 3]        -- [1, 2, 3, 5, 8, 9]
merge [1, 3, 5, 7] [2, 4, 6, 8]    -- [1, 2, 3, 4, 5, 6, 7, 8]
      

15.3 Ranges and infinite lists

The List module provides several constructors for arithmetic sequences and infinite lists. Because list tails are lazy, infinite lists are perfectly valid — only the portion actually consumed is ever evaluated.

-- Bounded arithmetic sequence: listFromTo start step end
listFromTo 1 1 5         -- [1, 2, 3, 4, 5]
listFromTo 0 2 10        -- [0, 2, 4, 6, 8, 10]
listFromTo 10 (-1) 1     -- [10, 9, 8, 7, 6, 5, 4, 3, 2, 1]

-- Infinite arithmetic sequence: listFrom start step
take 6 (listFrom 1 1)    -- [1, 2, 3, 4, 5, 6]
take 5 (listFrom 0 3)    -- [0, 3, 6, 9, 12]

-- Constant repetition
take 4 (repeat 42)                 -- [42, 42, 42, 42]
replicate 0 5                      -- [0, 0, 0, 0, 0]

-- Cyclic repetition
take 8 (cycle [1, 2, 3])           -- [1, 2, 3, 1, 2, 3, 1, 2]
      

The most powerful range constructor is iterate, from the ListFunction module. It generates an infinite list by repeatedly applying a function to a seed value. Powers of two:

take 8 (iterate ((*) 2) 1)    -- [1, 2, 4, 8, 16, 32, 64, 128]
      

A more expressive use of iterate carries a pair of state values through each step. The Fibonacci sequence, for example, advances a pair (a, b) to (b, a+b) at each step, with mapf fst extracting the leading value:

mapf fst (take 10 (iterate (\p -> (snd p, fst p + snd p)) (0, 1)))
-- [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
      

15.4 Higher-order list processing

The ListFunction module provides the higher-order functions that form the backbone of functional list processing.

15.4.1 Mapping and filtering

mapf applies a function to every element of a list, producing a new list of results:

mapf ((*) 3) [1, 2, 3, 4, 5]             -- [3, 6, 9, 12, 15]
mapf length ["one", "two", "three"]      -- [3, 3, 5]
         

filter retains only those elements satisfying a predicate:

filter (\n -> n mod 2 = 0) [1, 2, 3, 4, 5, 6]    -- [2, 4, 6]
filter (\n -> n * n < 20) [1, 2, 3, 4, 5]         -- [1, 2, 3, 4]
         

flatMap maps a function returning a list over every element, then flattens the results into a single list. It is equivalent to applying flatten after mapf:

flatMap (\x -> [x, x * x]) [1, 2, 3, 4]
-- [1, 1, 2, 4, 3, 9, 4, 16]

-- Expand each word into its characters, inserting a separator
quotedChars(flatten (mapf (\w -> #!(chars w) ++ [' ']) ["red", "green", "blue"]))
-- ['r','e','d',' ','g','r','e','e','n',' ','b','l','u','e',' ']
         

15.4.2 Folding

foldl reduces a list to a single value by applying a combining function left-to-right, carrying an accumulator:

foldl (+) 0 [1, 2, 3, 4, 5]      -- 15  (sum)
foldl (*) 1 [1, 2, 3, 4, 5]      -- 120 (product; 5!)
foldl max 0 [3, 1, 9, 1, 5, 4]   -- 9   (running maximum)
         

foldr applies the combining function right-to-left. It is the natural choice when the result shares the list's structure, since it can operate lazily on infinite inputs where foldl cannot:

-- Count elements satisfying a predicate using foldr
let inline count p xs = foldr (\x acc -> if p x then acc + 1 else acc) 0 xs
in count (\n -> n mod 2 = 0) [1, 2, 3, 4, 5, 6]    -- 3
         

15.4.3 Splitting and grouping

span splits a list at the first element that fails a predicate, returning the prefix and the remainder as a pair:

span ((>) 5) [1, 3, 5, 7, 2]           -- ([1, 3], [5, 7, 2])
span (\c -> not (c = ':')) (chars "host:8080")
-- (['h','o','s','t'], [':','8','0','8','0'])
         

The second example shows a common pattern: splitting a string at a delimiter character. Combined with string, this gives a simple field extractor:

let {
   splitAt :: Char -> String -> (String, String);
   splitAt delim s =
   case span (\c -> c ¬= delim) (chars s) of {
      (before, rem) -> (string before, string (tail rem))
   }
} in splitAt ':' "Content-Type: text/plain"
         

groupBy partitions a list into runs of consecutive elements satisfying a binary predicate. Two adjacent elements are placed in the same group when the predicate returns True:

groupBy (=) [1, 1, 2, 3, 3, 3, 1]
-- [[1,1],[2],[3,3,3],[1]]

-- Group by equal parity
groupBy (\x y -> x mod 2 = y mod 2) [2, 4, 1, 3, 6, 5]
-- [[2,4],[1,3],[6],[5]]
         

Note that groupBy only compares adjacent elements; it does not sort. To collect all globally equal elements together, sort the list first:

-- Frequency table: sort then group, count each run

let {
   frequencies :: [Int] -> [(Int, Int)];
   frequencies xs =
      mapf (\grp -> (head grp, length grp))
           (groupBy (=) (mergeSort xs))
} in
   frequencies [3, 1, 4, 1, 5, 9, 2, 6, 5, 3, 5] -- [(1,2),(2,1),(3,2),(4,1),(5,3),(6,1),(9,1)]
         

15.5 Strings and character lists

IvoryScript distinguishes between the primitive String type — an efficient non-decomposable value — and the (lazy) character list [Char] which supports all sequence methods. The StringList module bridges the two.

chars converts a String to a [Char], making the full list API available:

chars "Hello"                           -- [H,e,l,l,o]
length (chars "Hello")                  -- 5
reverse (chars "Hello")                 -- [o,l,l,e,H]
filter (\c -> c ¬= 'l') (chars "Hello")   -- [H,e,o]
      

string converts back from [Char] (or Exp [Char]) to String:

string (reverse (chars "stressed"))     -- "desserts"
string (filter (\c -> c = 'a' | c = 'e' | c = 'i' | c = 'o' | c = 'y')
               (chars "IvoryScript"))   -- "oyi"
      

A String can also be coerced directly to Exp [Char] through the StrictCast instance, and a strict [Char] coerces back to String in the same way.

quotedChars converts a character list to a list of strings, each string containing the character in single-quote notation. This is primarily useful for displaying character groups to match IvoryScript syntax:

quotedChars ['H', 'i']    -- ['H','i']
      

15.6 Vectors

Vector a, defined in the Vector module, is a dynamic-length mutable array implemented as a segmented slice structure. Unlike a list, a vector supports O(1) random access and in-place update via getAtVector and putAtVector. The constructor takes an initial element list and a slice size:

let scores = Vector [85, 92, 78, 95, 60] 8

length scores           -- 5
getAt scores 2          -- 78
      

Updates are performed in place. The vector itself is a mutable heap object, so changes are immediately visible through the same binding:

putAtVector scores 2 100
getAtVector scores 2          -- 100

findVectorIndex scores 95     -- 3  (returns -1 if not found)
      

The Seq instance provides length; the IndexableSeq instance provides head, getAt, and putAt. There is no ReverseableSeq instance — reversal of a mutable array is destructive and is left to application code.

Vectors are the appropriate choice when elements need to be looked up or updated by index, when the collection is constructed once and then read many times, or when interoperating with C++ code via pointer arrays.

15.7 Trees

The Tree module provides a binary tree type suited to hierarchical data such as expression trees, decision trees, and priority structures. A tree is one of three cases: an empty node, a leaf holding a single value, or a branch with two lazy subtrees:

type Tree a = Empty | Leaf a | Branch (Exp (Tree a)) (Exp (Tree a))
      

Because subtrees are lazy, arbitrarily large trees can be defined without evaluating every node:

let t :: Exp (Tree Int)= Branch (Branch (Leaf 4) (Leaf 7))
                        (Branch (Leaf 2) (Branch (Leaf 9) (Leaf 1)))
in
...
depthTree t    -- 4
maxTree t      -- 9
sumTree t      -- 23
      

Trees are naturally described by recursive functions. A depth-first in-order traversal collecting all leaf values into a list:

let inorder :: Exp (Tree a) -> [a];
    inorder tr =
        case #!tr of {
            Empty         -> [];
            Leaf x        -> [x];
            Branch lt rt  -> inorder lt ++ inorder rt
        }

inorder t    -- [4, 7, 2, 9, 1]
      

Combining a tree traversal with list processing shows how the two modules complement each other. Collecting and sorting the leaf values:

mergeSort (inorder t)    -- [1, 2, 4, 7, 9]
      

15.8 Worked example: run-length analysis of a string

The following example draws together lists, character lists, higher-order functions, and the quotedChars display utility to perform a run-length analysis of a string — identifying runs of consecutive identical characters.

Step 1. Decompose the string into a character list:

chars "Mississippi"
-- [M,i,s,s,i,s,s,i,p,p,i]
      

Step 2. Apply groupBy (=) to partition the list into runs of consecutive equal characters. Because groupBy only compares adjacent elements, the original order of the string is preserved:

groupBy (=) (chars "Mississippi")
-- [[M],[i],[s,s],[i],[s,s],[i],[p,p],[i]]
      

Step 3. The run-length encoding follows directly. Each group is a list of identical characters; head extracts the representative character and length counts the repetitions:

let rle s = mapf (\grp -> (head grp, length grp))
                 (groupBy (=) (chars s))
in
   rle "Mississippi" -- [(M,1),(i,1),(s,2),(i,1),(s,2),(i,1),(p,2),(i,1)]
      

Step 4. quotedChars converts each character group to a list of single-quoted strings. Applying it via mapf across the outer list:

mapf quotedChars (groupBy (=) (chars "Mississippi"))
-- [['M'],['i'],['s','s'],['i'],['s','s'],
--  ['i'],['p','p'],['i']]
      

The complete expression is a single pipeline: each stage is a pure function applied to the output of the previous one. No intermediate bindings are required unless readability demands them, and no mutation occurs at any point. This compositional style — building complex transformations from small, independently testable pieces — is characteristic of idiomatic IvoryScript.

Extending rle to reconstruct the original string from its encoding confirms the round-trip:

let expand pairs =
   string (flatten (mapf (\p -> replicate (fst p) (snd p)) pairs))

expand (rle "Mississippi")    -- "Mississippi"
      

Here replicate produces a list of n copies of a character, mapf applies this to every run-length pair, flatten concatenates the resulting list of lists into a single character list, and string converts it back to a String.

16 Name/property collections

Name/Property collections differ from the fixed-schema records of statically-typed languages. Rather than declaring a named type with a fixed set of typed fields, a script assembles collections of named values at runtime, with each value carrying its type at the Any level. The collection can be extended, queried by name, and passed through generic code without any schema declaration.

This chapter covers the module stack that enables this: Any as the uniform dynamic-type wrapper; Binding as the name/value pair; Property and its colon-notation shorthand; PropertySet for fixed, vector-backed collections; and NameAnyMap for mutable, hash-backed maps. The lower-level HashTable type is also described for situations where a custom key type is required.

16.1 The Any type

Any, defined in the Any module, is a uniform wrapper that holds a value of any IvoryScript type together with its runtime type descriptor. Internally it is the pair (Type, Ptr a): the first component identifies the concrete type; the second points to the stored value.

Any concrete value is wrapped as Any via the subordinate StrictCast instance, which forces evaluation and records the type. A type annotation on the binding site unwraps the value again:

let a1 = Any #42 in ...        -- Any holding an Int
let a2 = Any #"Hello" in ...   -- Any holding a String
let a3 = Any #!True in ...     -- Any holding a Bool
...
typeOfAny a1    -- the runtime Type descriptor for Int

let n :: Int    = ::a1 in n   -- 42
let s :: String = ::a2 in s   -- Hello
let b :: Bool   = ::a3 in b   -- True
      

Any supports environment remapping, garbage-collection marking, and byte-stream serialisation — all dispatched through the type descriptor for the type. This means Any values appear freely in persistent stores and environments without special handling.

16.2 Bindings and the BindingSet class

A Binding a b, defined in the Binding module, is a pair of typed values: a key of type a and a value of type b. The single data constructor is Bind:

Bind #host "localhost"   -- Binding Name String
Bind 42    "the answer"  -- Binding Int String
      

The BindingSet class abstracts over any mutable collection that supports keyed insertion, lookup, and removal:

class BindingSet a, b where {
   addBinding    :: a -> b -> c -> Void;
   hasBinding    :: a -> b -> Bool;
   removeBinding :: a -> b -> Void
};
      

Both NameAnyMap and HashTable provide BindingSet instances, so a function written against the class works with either collection type.

16.3 Property

A Property, defined in the Property module, is an association between a key of type Any and a value of type Any. It is the element type of PropertySet.

Properties are most commonly written using the colon shorthand: an identifier followed by : and a fully evaluated value expression.

-- Shorthand notation
name:   "Alice"
age:    30
active: True

-- Equivalent explicit form
Property #name   !"Alice"
Property #age    !30
Property #active !True
      

The runtime type of the value is preserved inside the Any wrapper and can be recovered at any point:

let p :: Property = age: 30
in
   typeOfProperty p    -- Int
      

The colon notation is valid wherever a Property value is expected, most commonly in the list literal passed to the PropertySet constructor.

16.4 PropertySet

PropertySet, defined in the PropertySet module, is an ordered, fixed collection of Property values backed by a contiguous BasicVector. The constructor accepts a list of properties, typically written with the colon shorthand:

let person = PropertySet [
   name:    "Alice",
   age:     30,
   active:  True
] in
   person
      

Values are retrieved by name through the dot operator, which dispatches via the Select PropertySet, Any instance. The result type is always Any; a type annotation on the binding site or an explicit cast extracts the concrete value:

let name :: String = person.name in name  -- "Alice"
let age :: Int     = person.age in age    -- 30
      

Dot expressions associate left and chain naturally through nested sets:

let contact = PropertySet [name: "Alice", address: PropertySet [city: "London", postCode: "EC1A"]]
in
   contact.address.city  -- London
      

Because the reprentatious is contiguous, access is a linear scan over the property vector. The built-in name optisation (chapter 9). reduces each comparison to a single integer test for registered names, making PropertySet very efficient for most typical objects.

16.4.1 Example: application event record

A common use of PropertySet is to model a structured event or message where the fields are known in advance. A UI click event carrying positional and user information:

let {
   mkClickEvent :: String -> Int -> Int -> PropertySet;
   mkClickEvent user x y  = PropertySet [
      type_:     "click",
      user:      "foo",
      position:  PropertySet [x: x, y: y]
   ]
} in
...
let ev = mkClickEvent "alice" 320 240
in
...
let evType :: String = ev.type_        -- click
let evUser :: String = ev.user         -- alice
let evX    :: Int    = ev.position.x   -- 320
         

A handler function can pattern-match on the event type and then read further properties as needed, without requiring a pre-declared record type.

16.5 NameAnyMap

NameAnyMap, defined in the NameAnyMap module, is a mutable hash-table-backed map from Name keys to Any values. The constructor takes the number of hash table slots; typically a prime number:

let config :: NameAnyMap = NameAnyMap 47
      

Entries are managed through the BindingSet interface. Values are generally implicitly coerced to Any.

addBinding config #host    "localhost"
addBinding config #port    8080
addBinding config #timeout 30
addBinding config #debug   False
      

Retrieval uses the same dot syntax as PropertySet:

let h :: String = config.host    -- "localhost"
let p :: Int    = config.port    -- 8080
      

Entries may be removed and replaced across the map lifetime.

removeBinding config #debug
addBinding    config #debug True     -- overwrite with updated value
      

16.5.1 Folding and inspection

The NameAnyMapFunction module provides higher-order operations. foldNameAnyMap visits every entry and accumulates a result; entry order follows the internal hash slot ordering, not insertion order:

-- Count the number of entries
let mapSize m = foldNameAnyMap (\k v acc -> acc + 1) 0 m
mapSize config    -- 4

-- Collect all keys
let mapKeys m = 
   mergeSort (foldNameAnyMap (\k v acc -> (stringify k) :+ (::acc)) [] m)
mapKeys config    -- e.g. [#debug, #timeout, #port, #host]
         

nameAnyMapNames returns a formatted diagnostic string listing all names whose value has a given runtime type:

nameAnyMapNames config #::String    -- ["host"]
nameAnyMapNames config #::Int       -- ["port", "timeout"]
         

16.6 HashTable

HashTable keyType valType, defined in the HashTable module, is the underlying implementation used by NameAnyMap. It is available directly for situations where a key type other than Name is needed, or where explicit control over slot allocation is required.

Unlike NameAnyMap, the HashTable API requires the caller to supply the slot index (the hash of the key modulo the table size) explicitly for every operation. The table size is set at construction time:

let ht = HashTable 32   -- 32 slots, key: Int, value: String

let slot k = k mod (nHashTableSlots ht)

hashTableAdd    ht (slot 42) 42 "forty-two"
hashTableAdd    ht (slot 99) 99 "ninety-nine"

hashTableGet    ht (slot 42) 42   -- Just "forty-two"
hashTableGet    ht (slot 99) 99   -- Just "ninety-nine"
hashTableGet    ht (slot  7)  7   -- Nothing

hashTableRemove ht (slot 42) 42
hashTableGet    ht (slot 42) 42   -- Nothing
      

Each slot holds a linked chain of entries (HashTableEntry nodes), allowing multiple keys to share a slot when collisions occur. findInHashTableSlot searches a single chain; the higher-level hashTableGet combines hashing and chain traversal in one call.

The HashTableFunction module provides foldHashTable, which visits every key/value entry across all slots and is the basis for foldNameAnyMap:

-- Sum all integer values in a HashTable Name Int
let total ht = foldHashTable (\k v acc -> acc + v) 0 ht
      

16.7 Worked example: a dynamic request context

The following example illustrates the complementary use of PropertySet and NameAnyMap in a request-handling scenario. The static server configuration is expressed as a PropertySet — its schema is fixed and known at startup. The per-request context is a NameAnyMap — it is assembled incrementally as middleware layers process the request.

-- Static configuration: fixed schema, constructed once
let serverConfig = PropertySet [
   host:       "0.0.0.0",
   port:       8443,
   maxClients: 100,
   debug:      False
]

let cfgPort  :: Int  = serverConfig.port    -- 8443
let cfgDebug :: Bool = serverConfig.debug   -- False
      

The per-request context starts empty and accumulates entries as routing and authentication middleware add information:

let handleRequest rawPath userId =
   let ctx = NameAnyMap 16 in {

      -- Routing layer
      addBinding ctx #method "GET";
      addBinding ctx #path   rawPath;

      -- Authentication layer
      addBinding ctx #userId   userId;
      addBinding ctx #authed   True;

      -- Handler reads from context
      let method :: String = ctx.method;
      let path   :: String = ctx.path;
      let uid    :: Int    = ctx.userId;

      -- Log if debug mode is on
      if cfgDebug then {
         showWithNewline method;
         showWithNewline path
      };

      processRequest ctx
   }
      

Neither layer needs to know about the fields added by the other. foldNameAnyMap can enumerate the full context for diagnostics without any advance knowledge of which keys are present:

let logContext ctx =
   foldNameAnyMap
      (\k v acc -> { showWithNewline k; acc })
      Void
      ctx
      

This pattern — a fixed configuration object plus a dynamic per-invocation context — recurs throughout the Ivory System and is the natural way to model application objects in IvoryScript.

17 I/O and streams

IvoryScript provides I/O at two levels. At the lower level, the built-in FileHandle type gives direct byte-by-byte access to files. At the higher level, the Streams module wraps file handles in typed InputStream and OutputStream values and provides uniform serialisation and deserialisation for all standard types through the StreamInsert and StreamExtract classes. The Dir module adds directory listing as a lazy list.

17.1 File handles and I/O primitives

Files are opened with the built-in openFile, which takes a path and an IO_Mode and returns a FileHandle:

type IO_Mode = IO_Read | IO_Write | IO_Append | IO_ReadWrite
      

let fh = openFile "data.bin" IO_Write
writeFileByte fh 72    -- write byte 0x48
writeFileByte fh 101   -- write byte 0x65
closeFile fh
      

Reading is symmetric:

let fh = openFile "data.bin" IO_Read
let b1 :: Byte = readFileByte fh
let b2 :: Byte = readFileByte fh
closeFile fh
      

Direct byte access is the appropriate level for custom binary protocols or when interoperating with external C++ code that manages file handles directly. For structured IvoryScript values, the typed stream interface described in the next section is more convenient and handles all framing automatically.

17.2 Typed byte streams

The Streams module wraps FileHandle in two typed stream types:

type InputStream  a = InputStream  (Exp a)      (Exp Void)
type OutputStream a = OutputStream (a -> Void)  (Exp Void)
      

Each stream packages its I/O action (or function) alongside a close action. File-backed byte streams are created with:

fileInputByteStream  :: String -> InputStream  Byte
fileOutputByteStream :: String -> OutputStream Byte
      

Values are written and read through the StreamInsert and StreamExtract class methods:

insert  :: OutputStream Byte -> b -> Void     -- serialise b to stream
extract :: InputStream  Byte -> b             -- deserialise b from stream
      

The concrete type of b is resolved from the context at the call site. Writing a structured record to a file:

let os = fileOutputByteStream "record.bin"
insert os 42
insert os "Alice"
insert os True
#!(streamCloseOutputAction os)
      

Reading it back requires type annotations to tell the compiler which instance of extract to use at each step:

let is              = fileInputByteStream "record.bin"
let n  :: Int       = extract is
let s  :: String    = extract is
let b  :: Bool      = extract is
#!(streamCloseInputAction is)
      

The close actions are lazy expressions; they must be forced with #! to actually close the underlying file. Forgetting to close a stream is a resource leak.

Stream accessor functions are available when the stream object needs to be passed to lower-level code:

streamInputAction      :: InputStream  a -> Exp a
streamOutputFn         :: OutputStream a -> (a -> Void)
streamCloseInputAction :: InputStream  a -> Exp Void
streamCloseOutputAction:: OutputStream a -> Exp Void
      

17.3 Standard serialisation instances

The Streams module provides StreamInsert and StreamExtract instances for all primitive and structural types:

All module-defined collection types — [a], Vector a, Tree a, HashTable k v, NameAnyMap, PropertySet, and others — provide their own instances in their respective modules, so insert and extract work uniformly across the full type system.

17.4 The Serialisable class

The Serialisable module defines a format-neutral abstraction over round-trip conversion:

class Serialisable a, b where {
   serialise   :: a -> b;
   deserialise :: b -> a
};
      

Where StreamInsert and StreamExtract prescribe a specific binary wire format, Serialisable a b expresses only the round-trip contract — that values of type a can be converted to and from type b losslessly. The serialised form b might be a String (for a text representation), a PropertySet (for a structured object representation), or any other intermediate form.

User-defined types that need format flexibility — for example an API type that must be representable as both binary and JSON — can provide separate Serialisable instances for each target, keeping the representation concern separate from the data model.

17.5 Directory listing

The Dir module provides a single high-level function:

dir :: String -> Exp [String]
      

dir opens the directory at the given path, reads all entries lazily using the underlying openDir, nextDirEntry, and dirEntryName primitives, and returns a lazy list of filename strings. An empty list is returned if the path does not exist or is not a directory.

Because the result is an Exp [String], the full suite of list operations from the List and ListFunction modules applies directly:

-- List all entries in a directory
dir "/home/alice/scripts"

-- Count entries
length (dir "/tmp")

-- Filter to IvoryScript source files only
filter (\name -> length name > 3 &&
                 drop (length name - 3) name = ".is")
        (dir "/home/alice/scripts")

-- Sorted listing
mergeSort (dir "/home/alice/data")
      

The underlying primitives are available for lower-level directory traversal where explicit control over the stream lifecycle is needed:

openDir    :: String   -> Maybe Dir
nextDirEntry :: Dir    -> Maybe DirEntry
dirEntryName :: DirEntry -> String
closeDir   :: Dir      -> Void
      

The Dir module implementation uses these directly and shows the idiomatic pattern: open the directory, recurse with nextDirEntry until it returns Nothing, then rely on garbage collection to release the handle. Application code should prefer dir and reserve the primitives for cases where filtering or early exit needs to happen inside the traversal loop.

17.6 Worked example: persisting a list of records

The following example writes a list of name/score pairs to a binary file and reads them back. Because the length of the list is not encoded by the stream format itself, it is written as an Int prefix so that the reader knows how many records to extract.

-- Write a list of (String, Int) records to a binary file
let writeScores filename scores =
   let os = fileOutputByteStream filename in {
      insert os (length scores);
      mapf (\p -> { insert os (fst p); insert os (snd p) }) scores;
      #!(streamCloseOutputAction os)
   }

-- Read them back
let readScores filename =
   let is = fileInputByteStream filename;
       n  :: Int = extract is;
       readOne   = \_ -> let name  :: String = extract is;
                             score :: Int    = extract is
                         in (name, score)
   in {
      let scores = mapf readOne (listFromTo 1 1 n);
      #!(streamCloseInputAction is);
      scores
   }
      

Usage:

let scores = [("Alice", 95), ("Bob", 82), ("Carol", 91)]
writeScores "scores.bin" scores
readScores  "scores.bin"
-- [("Alice",95),("Bob",82),("Carol",91)]
      

Combining I/O with the Dir module, all score files in a directory can be aggregated into a single sorted leaderboard:

let loadAll dir_ =
   flatten (mapf readScores
                 (filter (\n -> drop (length n - 4) n = ".bin")
                         (dir dir_)))

mergeSort (mapf snd (loadAll "/data/scores"))
-- all scores from all .bin files, sorted ascending
      

18 Numeric and mathematical

Four modules cover the numeric and mathematical areas of the standard library. MathConst provides a set of high-precision floating-point constants. Arith supplies general-purpose arithmetic functions — factorial, GCD, LCM, and the Fibonacci recurrence. NumAggregate defines aggregate classes (SumAggregate, MeanAggregate) with instances for Int and Double lists, and a polymorphic median function. IntSeq rounds out the group with a collection of infinite lazy integer sequences — factorials, Fibonacci numbers, primes, and prime gaps — that integrate naturally with the sequence combinators covered in Chapter 15.

18.1 Mathematical constants

The MathConst module exports six named Double constants at full IEEE 754 double precision:

A circle's area and circumference follow directly:

circleArea :: Double -> Double
circleArea r = pi * r * r

circumference :: Double -> Double
circumference r = 2.0 * pi * r
      

Continuous compound interest uses e:

-- compoundValue principal rate years
compoundValue :: Double -> Double -> Double -> Double
compoundValue p r t = p * (exp (r * t))
      

The golden ratio satisfies phi² = phi + 1; a quick verification illustrates direct use of the constant:

let diff :: Double = phi * phi - phi - 1.0
-- diff is approximately 0.0 (within floating-point rounding)
      

18.2 Arithmetic functions

The Arith module provides four standard functions:

18.2.1 Factorial

fact is implemented with a tail-recursive helper accumulating the product:

fact 0   -- 1
fact 5   -- 120
fact 10  -- 3628800
         

Because the loop is tail-recursive, arbitrarily large factorials do not grow the stack.

18.2.2 GCD and LCM

gcd uses the Euclidean algorithm and handles the case where either argument is zero:

gcd 54 24   -- 6
gcd 0  17   -- 17
lcm 4  6    -- 12
lcm 54 24   -- 216
         

A common use is simplifying a fraction before display:

simplify :: (Int, Int) -> (Int, Int)
simplify (n, d) =
   let g :: Int = gcd n d
   in  (n div g, d div g)

simplify (36, 48)   -- (3, 4)
         

18.2.3 Fibonacci

fibonacci n returns the nth Fibonacci number (0-indexed: fibonacci 0 = 0, fibonacci 1 = 1):

fibonacci 10   -- 55
fibonacci 20   -- 6765
         

For a list of successive Fibonacci values, fibonaccis from IntSeq (§18.4) is more efficient as it shares intermediate results. fibonacci is the appropriate choice when a single value at a given index is required.

18.3 Numeric aggregates

NumAggregate defines two multi-parameter classes and a standalone median function.

18.3.1 SumAggregate and MeanAggregate

The class declarations follow the two-parameter style used throughout the library:

class SumAggregate a, b where {
   sum :: a -> b
}

class MeanAggregate a, b where {
   mean :: a -> b
}
         

The provided instances cover integer and floating-point lists:

Basic descriptive statistics over a sample:

let scores :: [Int] = [72, 85, 91, 68, 78, 95, 60, 83]

let total  :: Int    = sum scores      -- 632
let avg    :: Double = mean scores     -- 79.0
         

18.3.2 Median

median is a polymorphic function, not a class method. It sorts its argument with mergeSort and returns the middle element for odd-length lists, or the arithmetic mean of the two middle elements for even-length lists:

median :: [a] -> a

median [3, 1, 4, 1, 5, 9, 2, 6]   -- 3 (integer truncation for even-length Int list)
median [3.0, 1.0, 4.0, 1.0, 5.0]  -- 3.0
         

For an [Int] list of even length, the two central elements are summed and divided with integer division, so the result is truncated. When exact midpoints are required for integer data, converting to [Double] first is advisable.

median calls error on an empty list.

18.4 Integer sequences

IntSeq provides a collection of named infinite lazy sequences, each stored as a shared Exp [Int] (or Exp [(Int,Int)]) value created with variable. All sequences are forced with the #! operator before use, and standard list combinators such as take, filter, and mapf apply directly.

18.4.1 positiveInts, factorials, fibonaccis

Three foundational sequences are pre-built:

positiveInts :: Exp [Int]   -- 1, 2, 3, 4, ...
factorials   :: Exp [Int]   -- 1, 1, 2, 6, 24, 120, ...
fibonaccis   :: Exp [Int]   -- 0, 1, 1, 2, 3, 5, 8, 13, ...
         

Taking a prefix from each:

take 5 (#!positiveInts)   -- [1, 2, 3, 4, 5]
take 7 (#!factorials)     -- [1, 1, 2, 6, 24, 120, 720]
take 8 (#!fibonaccis)     -- [0, 1, 1, 2, 3, 5, 8, 13]
         

Because each sequence is a shared variable, the underlying list is computed at most once regardless of how many times it is accessed from different call sites.

18.4.2 Prime sequences

The primes sequence is constructed with a lazy Sieve of Eratosthenes. A sieve helper strips multiples of the current head from its tail, and iterate sieve applies the sieve at each step. Taking the head of each successive filtered list — with mapf hd — yields the prime sequence:

primes :: Exp [Int]   -- 2, 3, 5, 7, 11, 13, ...

take 10 (#!primes)   -- [2, 3, 5, 7, 11, 13, 17, 19, 23, 29]
         

Two convenience functions query the sequence by value or position:

nthPrime :: Int -> Int
-- Returns the nth prime (1-indexed).
nthPrime 1    -- 2
nthPrime 5    -- 11
nthPrime 10   -- 29

maxPrime :: Int -> Int
-- Returns the largest prime strictly less than n.
maxPrime 20   -- 19
maxPrime 100  -- 97
         

18.4.3 Prime gaps

primeGaps pairs each prime with the gap to the next prime. maximalPrimeGaps filters this to the subsequence of pairs where each gap exceeds all previous gaps — the record-setting gaps:

primeGaps        :: Exp [(Int, Int)]   -- (prime, gap to next prime)
maximalPrimeGaps :: Exp [(Int, Int)]   -- record-setting (prime, gap) pairs

take 6 (#!primeGaps)
-- [(2,1),(3,2),(5,2),(7,4),(11,2),(13,4)]

take 6 (#!maximalPrimeGaps)
-- [(2,1),(3,2),(7,4),(23,6),(89,8),(113,14)]
         

The first maximal gap is 1 (between 2 and 3). The first gap exceeding that is 2 (between 3 and 5). The first gap exceeding 2 is 4 (between 7 and 11), and so on.

18.5 Worked example: analysing a prime distribution

This example combines all four modules. Given a range of integers up to some limit, the code reports the number of primes in that range, their sum and mean, and the largest prime gap encountered.

-- Collect all primes up to (but not including) n.
primesUpTo :: Int -> [Int]
primesUpTo n = takeWhile (\p -> p < n) (#!primes)

-- Summary statistics for primes up to n.
primeStats :: Int -> Void
primeStats n =
   let {
      ps    :: [Int]   = primesUpTo n;
      count :: Int     = length ps;
      total :: Int     = sum ps;
      avg   :: Double  = mean ps;
      med   :: Double  = median (mapf (:: :: Int -> Double) ps)
   } in {
      show "Primes below "; show n; show ": "; show count; show "\n";
      show "Sum:    "; show total; show "\n";
      show "Mean:   "; show avg;   show "\n";
      show "Median: "; show med;   show "\n"
   }
      

Running primeStats 100 produces:

Primes below 100: 25
Sum:    1060
Mean:   42.4
Median: 41.0
      

The largest prime gap up to the same limit can be extracted from maximalPrimeGaps:

largestGapBelow :: Int -> (Int, Int)
largestGapBelow n =
   let gaps :: [(Int, Int)] = takeWhile (\g -> fst g < n) (#!maximalPrimeGaps)
   in  last gaps

largestGapBelow 100   -- (89, 8)
      

The largest maximal prime gap with first prime below 100 starts at 89, with the next prime (97) being 8 away. A circle whose radius is the mean prime (42.4) has an area of approximately pi * 42.4 * 42.4, which can be woven in directly using MathConst.pi:

let r :: Double = mean (primesUpTo 100)   -- 42.4
let area = pi * r * r                     -- ~5647.0 (using MathConst.pi)
      

19 Persistence

19.1 Introduction

IvoryScript provides native persistence: any IvoryScript value can be written to a binary stream and later restored with full type fidelity, without any external schema, serialisation format, or configuration file. The mechanism is a natural consequence of the language's typed environment model, and the same compiler infrastructure that supports persistence also underpins garbage collection and environment migration. Where other languages require a separate serialisation layer (JSON, XML, Protocol Buffers), IvoryScript values carry all the structural information needed to reconstruct them, and the runtime handles the rest transparently.

The primary user-facing types are TransientStore (a named key/value store backed by a NameAnyMap) and the lower-level TransientDataStore a (a generic store whose root may be any IvoryScript type). For in-process global state that does not require serialisation, SimpleRoot provides a pre-allocated NameAnyMap accessible as the global binding root.

19.2 Environments

The foundation of persistence is the Env: an isolated managed memory space that owns the heap storage for all values allocated within it. Every heap-allocated value — a Name, a String, a function closure, or any algebraic data type that contains them — carries a reference to its owning environment alongside the value itself. Scalar types (Int, Float, Double, Char) are represented directly by value and require no environment reference.

This classification is enforced at compile time. During the location generation pass the compiler tracks heap-allocated values as (value, environment) pairs: each such value occupies two stack slots, one for the value and one for its environment pointer. Operands that require an environment pointer are resolved with representation tag REPR_ENV_PTR. At each heap allocation site a garbage collection stub records the stack positions of all live (value, env) pairs in a frame descriptor, giving the collector precise information to traverse the live set without a conservative scan.

When a persistent store is serialised, the runtime uses these same per-type methods to walk the environment graph, writing the type tag, the heap contents, and the root value to the output stream. On deserialisation, a fresh environment is allocated and the root value is remapped into the new address space. Because the type structure is embedded in the binary, no external description is required: the stream is self-describing.

19.3 Compiler-generated transfer methods

Every transferable cell in IvoryScript carries a CellInfo descriptor that stores four method labels generated by the compiler. The same four methods are generated for two distinct categories of cell:

• Closure cells, generated within Lambda::gen. For each lambda that captures free variables a closure cell is created and the following methods are generated and stored in the cell's CellInfo:

  • copyClosure (_copyFnLabel) — maps the closure's free variables to a target environment via mapFreeVarToEnv.

  • extractFreeVars (_extractFnLabel) — deserialises the closure's free variables from an InputStream Byte.

  • insertFreeVars (_insertFnLabel) — serialises the closure's free variables to an OutputStream Byte.

  • markFreeVars_GC (_gcFnLabel) — marks the closure's free variables for garbage collection.

  Because lambdas may be nested, Lambda::gen is called recursively, and each nested closure receives its own independent set of methods.

• Algebraic data type cells, generated by Code::genTypeMethods. For each exposed closed type a TypeDescrInstruction is emitted carrying the entry-point labels for the corresponding methods:

  • mapGenPtr :: Ptr a -> Env -> Ptr a — remaps all heap values within the type to a target environment.

  • assignByPtr :: Ptr a -> Ptr a -> Void — copies a value in place from one pointer to another.

  • showGenPtr :: Ptr a -> Void — displays the value.

  • extractGenPtr :: InputStream Byte -> Ptr a — deserialises a heap value from a byte stream.

  • insertGenPtr :: OutputStream Byte -> Ptr a -> Void — serialises a heap value to a byte stream.

  • markGenPtr :: Ptr a -> Void — marks the heap value for garbage collection.

  These are generated by resolving constrained snippets (for example, mapGenPtr constrained to type Ptr a -> Env -> Ptr a) through the standard type class machinery, then calling Lambda::gen on the resulting lambda. The serialisation and GC methods are conditionally compiled under the SERIALISATION and GARBAGE_COLLECTION build flags respectively.

The result is that every value that can cross an environment boundary — whether a closure or an algebraic data value — carries with it all the code needed to copy, serialise, deserialise, and collect it. The persistence types TransientDataStore and TransientStore simply invoke these methods via the StreamInsert and StreamExtract type class instances.

19.4 Transient stores

TransientDataStore a is the primitive persistent container. It allocates an isolated environment and holds a single root value of type a. Any IvoryScript type that has StreamInsert and StreamExtract instances may serve as the root. The binary format begins with the magic header I_SDS_v_0001 followed by the serialised environment and root value.

TransientStore is defined as TransientDataStore NameAnyMap and is the standard user-facing persistence type. A NameAnyMap root provides a named dictionary of heterogeneous Any values, making TransientStore directly equivalent to the configuration files common in other languages, but with typed values and no parser. The key operations are:

• TransientStore — constructs a new store with an isolated environment and a 47-slot hash map.

• addBinding — associates a Name key with an Any value in the store.

• removeBinding — removes a binding by name.

• (.) — retrieves a value by name (the Select instance).

• insert — serialises the store to a file path.

• extract — deserialises a store from a file path.

• destroy — releases the store's environment and all heap storage it owns.

19.5 Persisting data: a worked example

The following example demonstrates the full persistence cycle. A lazy list of prime numbers is computed using the Sieve of Eratosthenes, the first 100 values are stored under the name 'primes, and the store is written to a file. On the next run the store is reloaded and the saved sequence is retrieved without recomputation.

-- Compute and persist the first 100 prime numbers

let {
   sieve :: [Int] -> [Int];
   sieve l =
      case l of {
         []      -> [];
         p :+ xs -> filter (\x -> x mod p /= 0) (::xs)
      };

   primes :: [Int];
   primes = mapf hd (iterate sieve (listFrom 2 1))
} in
let store = TransientStore() in {
   addBinding 'primes (toAny (take 100 primes)) store;
   insert "primes.is" store;
   destroy store
}

To reload the persisted data in a subsequent session:

let store = extract "primes.is" :: TransientStore in
let savedPrimes = fromAny (store . 'primes) :: [Int] in
   show savedPrimes

The type annotation on extract is required because the return type is polymorphic; it tells the runtime which StreamExtract instance to use when reading the file. Once loaded, the value is retrieved by name and cast from Any back to its original type. No schema file, no parser, and no format conversion is involved at any stage. The compiler-generated extractGenPtr and insertGenPtr methods handle the binary encoding of each constituent type automatically.

19.6 In-process global data

For state that must persist across calls within a single process but does not need to be serialised to disk, SimpleRoot provides a pre-allocated NameAnyMap in an isolated environment, accessible via the global binding root.

-- Store an application counter in the global root
addBinding 'counter (toAny (0 :: Int)) root;

-- Retrieve it elsewhere in the same process
let n = fromAny (root . 'counter) :: Int in
   show n

Unlike TransientStore, root is a single shared instance; there is no constructor call and no destroy. It is well suited to lightweight global application state — configuration values, caches, or counters — where the overhead of a separate environment and serialisation round-trip is unnecessary.

20 Dynamic and polymorphic

The chapters so far have operated in IvoryScript's statically typed world, where the type of every value is known at compile time. The Any type (Chapter 16) provides an escape hatch: any value of any type can be wrapped in Any and carried through code that does not need to inspect it. The three modules covered in this chapter extend that capability by providing class instances for Any itself, enabling standard operators and methods — arithmetic, sequence concatenation, dot notation, and binding mutation — to be applied to Any values with full runtime dispatch to the correct concrete implementation.

AnyNum provides a Num instance for Any, dispatching arithmetic to Int or Double based on the runtime type of the operand. AnySeq provides a Seq instance, dispatching sequence concatenation. AnyBindingSet provides both a Select and a BindingSet instance, so that dot notation and binding operations work uniformly over an Any that wraps a PropertySet, a NameAnyMap, or a TransientStore.

20.1 The dynamic dispatch model

IvoryScript resolves class instances at compile time using type inference. When the type is Any — a value whose concrete type is known only at runtime — a second mechanism is needed. The three modules in this chapter use two language features to provide it.

Instance for *. In IvoryScript an instance declaration whose type parameter is * defines behaviour for the Any type. The arithmetic instance in AnyNum is declared as:

instance Num * where { ... }
      

Within the implementation, typeOfAny returns the runtime type tag embedded in the Any wrapper, and a case expression on that tag selects the concrete operation. The type names appear as name literals — #::Int, #::Double, #::String — against which the tag is matched.

Subordinate instances. The subordinate qualifier on an instance declaration lowers its priority in instance selection. A subordinate instance is chosen only when no non-subordinate instance matches. This is the mechanism by which AnyBindingSet provides dot notation for Any: code that holds a PropertySet directly still resolves to the dedicated Select PropertySet, Any instance; code that holds the same value through an Any wrapper resolves to the subordinate Select Any, Any instance, which then unwraps and redispatches.

20.2 AnyNum — arithmetic over Any

AnyNum supplies a Num instance for *. The addition operator inspects the runtime type of its left operand and promotes both operands to that type before performing the operation:

instance Num * where {
   (+) anyX anyY =
      case typeOfAny anyX of {
         #::Int    -> Any #!(((::anyX)::Int)    + ((::anyY)::Int));
         #::Double -> Any #!(((::anyX)::Double) + ((::anyY)::Double))
      }
}
      

The result is always re-wrapped in Any. A heterogeneous list of numeric values can therefore be reduced with a single accumulator, provided all elements share a consistent concrete type at runtime:

let ints :: [Any] = [Any 1, Any 2, Any 3, Any 4, Any 5]
let total :: Any  = foldl (+) (Any (0::Int)) ints
-- total wraps Int 15

let doubles :: [Any] = [Any 1.5, Any 2.5, Any 3.0]
let dtotal  :: Any   = foldl (+) (Any (0.0::Double)) doubles
-- dtotal wraps Double 7.0
      

The dispatch is on the type of the left operand. Mixing Int and Double Any values in a single fold will fail at runtime when the coercion is attempted. For genuinely mixed numeric data, an explicit conversion pass before folding is the recommended approach.

20.3 AnySeq — sequence operations over Any

AnySeq provides a Seq instance for *, currently implementing the concatenation operator (++) for String:

instance Seq * where {
   (++) anyX anyY =
      case typeOfAny anyX of {
         #::String -> Any #!(((::anyX)::String) ++ ((::anyY)::String))
      }
}
      

This allows string fragments carried as Any to be assembled with the familiar (++) operator, without unwrapping and re-wrapping at each step:

let parts :: [Any] = [Any "Hello", Any ", ", Any "world", Any "!"]
let msg   :: Any   = foldl (++) (Any ("" :: String)) parts
-- (::msg)::String  =  "Hello, world!"
      

As with AnyNum, the dispatch is on the left operand. Additional sequence types — lists and vectors — may be added as further typeOfAny cases in future releases.

20.4 AnyBindingSet — uniform dot notation and binding

AnyBindingSet is the most consequential of the three modules. It provides two instances for Any that together allow container-agnostic code to be written against any of the name/property collection types.

20.4.1 Select instance — dot notation

The subordinate Select Any, Any instance unpacks the concrete container type from the Any wrapper and redispatches the dot operation to the appropriate typed instance:

subordinate instance Select Any, Any where {
   (.) any name = case any of {
      #Any (#::PropertySet,    Ptr (ps::PropertySet))   -> (.) ps name;
      #Any (#::NameAnyMap,     Ptr (m::NameAnyMap))     -> (.) m  name;
      #Any (#::TransientStore, Ptr (ts::TransientStore)) -> (.) ts name
   }
}
         

The pattern #Any (#::PropertySet, Ptr (ps::PropertySet)) matches the constructor of the Any type, extracting the runtime type tag and dereferencing the pointer to recover the typed value. The subordinate qualifier ensures that code holding a PropertySet directly is unaffected — the subordinate instance is only selected when the compiler sees Any as the container type.

From the caller's perspective, dot notation simply works:

-- A function that reads a named value from any supported container.
getValue :: Any -> Name -> Any
getValue container name = container.name

let ps  :: PropertySet = [x: 10, y: 20]
let any :: Any         = Any ps

-- Direct access (Select PropertySet, Any instance):
let v1 :: Any = ps.#x         -- Any 10

-- Access through Any (subordinate Select Any, Any instance):
let v2 :: Any = any.#x        -- Any 10
         

Both expressions return the same result; the instance selected differs, but the calling code is identical.

20.4.2 BindingSet instance — addBinding and removeBinding

The BindingSet Any, Name, Any instance dispatches addBinding and removeBinding to NameAnyMap or TransientStore (mutable containers; read-only PropertySet is excluded):

instance BindingSet Any, Name, Any where {
   addBinding any n v =
      case any of {
         #Any (#::NameAnyMap,     Ptr (m::NameAnyMap))      -> addBinding m  n v;
         #Any (#::TransientStore, Ptr (ts::TransientStore)) -> addBinding ts n v
      };
   removeBinding any n =
      case any of {
         #Any (#::NameAnyMap,     Ptr (m::NameAnyMap))      -> removeBinding m  n;
         #Any (#::TransientStore, Ptr (ts::TransientStore)) -> removeBinding ts n
      }
}
         

This enables middleware or plugin code to mutate a context object without committing to whether the backing store is a NameAnyMap or a persistent TransientStore.

20.5 Worked example: a container-agnostic pipeline

A request-handling pipeline often passes a context object through a chain of handlers, each of which reads and writes named values. The backing store for that context might be a lightweight NameAnyMap during unit tests but a TransientStore in production, where values must survive garbage collection. With AnyBindingSet, the handler code need not care which it is.

First, a utility that reads an optional integer from an Any container, returning a default if the name is absent:

getInt :: Any -> Name -> Int -> Int
getInt ctx name dflt =
   let v :: Any = ctx.name
   in  if isNothing (v :: Maybe Int)
       then dflt
       else (::v)::Int
      

A handler that records an attempt counter and timestamps its last activation, regardless of container type:

recordAttempt :: Any -> Int -> Void
recordAttempt ctx timestamp =
   let count :: Int = getInt ctx #attempts 0
   in {
      addBinding ctx #attempts (Any (count + 1));
      addBinding ctx #lastSeen  (Any timestamp)
   }
      

The same handler works with either backing store:

-- In-process test context backed by NameAnyMap:
let testCtx  :: Any = Any (NameAnyMap [])
recordAttempt testCtx 1000
recordAttempt testCtx 1001
-- testCtx.#attempts evaluates to Any 2

-- Production context backed by TransientStore:
let prodCtx :: Any = Any (TransientStore someEnv)
recordAttempt prodCtx 2000
-- prodCtx.#attempts evaluates to Any 1
      

A supervisor that processes a list of contexts and reports those that have exceeded a threshold uses only Any-typed operations throughout:

flagOverLimit :: [Any] -> Int -> [Any]
flagOverLimit contexts limit =
   filter (\ctx -> getInt ctx #attempts 0 > limit) contexts
      

The combination of AnyNum and AnyBindingSet makes it straightforward to write aggregate reporting across a heterogeneous list of contexts. A total attempt count across all containers:

totalAttempts :: [Any] -> Int
totalAttempts contexts =
   foldl (\acc ctx -> acc + getInt ctx #attempts 0) 0 contexts
      

In this last example the (+) call operates on Int values directly — getInt already unwraps. The same accumulator could be written over Any values using AnyNum if the container mix includes unknown numeric types that should be summed without unwrapping at each step.