August 20, 2020

Red/System: New Features

In the past months, many new features were added to Red/System, the low-level dialect embedded in Red. Here is a sum up if you missed them.

Subroutines

During the work on the low-level parts of the new Red lexer, the need arised for intra-function factorization abilities to keep the lexer code as DRY as possible. Subroutines were introduced to solve that. They act as the GOSUB directive from Basic language. They are defined as a separate block of code inside a function's body and are called like regular functions (but without any arguments). So they are much lighter and faster than real function calls and require just one slot of stack space to store the return address. 

The declaration syntax is straightforward:

    <name>: [<body>]

    <name> : subroutine's name (local variable).
    <body> : subroutine's code (regular R/S code).

To define a subroutine, you need to declare a local variable with the subroutine! datatype, then set that variable to a block of code. You can then invoke the subroutine by calling its name from anywhere in the function body (but after the subroutine own definition).

Here is a first example of a fictive function processing I/O events:

    process: func [buf [byte-ptr!] event [integer!] return: [integer!]
        /local log do-error [subroutine!]
    ][
        log: [print-line [">>" tab e "<<"]]
        do-error: [print-line ["** Error:" e] return 1]
    
        switch event [
            EVT_OPEN  [e: "OPEN"  log unless connect buf [do-error]]
            EVT_READ  [e: "READ"  log unless receive buf [do-error]]
EVT_WRITE [e: "WRITE" log unless send buf [do-error]]
EVT_CLOSE [e: "CLOSE" log unless close buf [do-error]]
default [e: "<unknown>" do-error] ] 0 ]

This second example is more complete. It shows how subroutines can be combined and how values can be returned from a subroutine:

    #enum modes! [
    	CONV_UPPER
    	CONV_LOWER
    	CONV_INVERT
    ]

    convert: func [mode [modes!] text [c-string!] return: [c-string!]
        /local
            lower? upper? alpha? do-conv [subroutine!]
            delta [integer!]
            s     [c-string!]
            c     [byte!]
    ][
        lower?:  [all [#"a" <= c c <= #"z"]]
        upper?:  [all [#"A" <= c c <= #"Z"]]
        alpha?:  [any [lower? upper?]]
        do-conv: [s/1: s/1 + delta]
        delta:   0
        s:       text

        while [s/1 <> null-byte][
            c: s/1
            if alpha? [
                switch mode [
                    CONV_UPPER  [if lower? [delta: -32 do-conv]]
                    CONV_LOWER  [if upper? [delta: 32 do-conv]]
                    CONV_INVERT [delta: either upper? [32][-32] do-conv]
                    default     [assert false]
                ]
            ]
            s: s + 1
        ]
        text
    ]
    
    probe convert CONV_UPPER "Hello World!"
    probe convert CONV_LOWER "There ARE 123 Dogs."
    probe convert CONV_INVERT "This SHOULD be INVERTED!"

will output:

    HELLO WORLD!
    there are 123 dogs.
    tHIS should BE inverted!

Support for getting a subroutine address and dispatching dynamically on it is planned to be added in the future (something akin computed GOTO). More examples of subroutines can be found in the new lexer code, like in the load-date function.

New system intrinsics


Several new extensions to the system path have been added.

Lock-free atomic intrinsics


A simple low-level OS threads wrapper API has been added internally to the Red runtime as preliminary work on supporting parts of IO concurrency and parallel processing in the future. In order to complement it, a set of atomic intrinsics were added to enable the implementation of lock-free and wait-free algorithms in a multithreaded execution context.

The new atomic intrinsics are all documented here. Here is a quick overview:
  • system/atomic/fence: generates a read/write data memory barrier.
  • system/atomic/load: thread-safe atomic read from a given memory location.
  • system/atomic/store: thread-safe atomic write to a given memory location.
  • system/atomic/cas: thread-safe atomic compare&swap to a given memory location.
  • system/atomic/<math-op>: thread-safe atomic math or bitwise operation to a given memory location (add, sub, or, xor, and).

Other new intrinsics
  • system/stack/allocate/zero: allocates a storage space on stack and zero-fill it.
  • system/stack/push-all: saves all registers to stack.
  • system/stack/pop-all: restores all registers from stack.
  • system/fpu/status: retrieves the FPU exception bits status as a 32-bit integer.

Improved literal arrays

The main change is the removal of the hidden size inside the /0 index slot. The size of a literal array can now only be retrieved using the size? keyword, which is resolved at compile time (rather than run-time for /0 index access).

A notable addition is the support for binary arrays. Those arrays can be used to store byte-oriented tables or embed arbitray binary data into the source code. For example:

    table: #{0042FA0100CAFE00AA}
    probe size? table                      ;-- outputs 9
    probe table/2                          ;-- outputs "B"
    probe as integer! table/2              ;-- outputs 66
The new Red lexer code uses them extensively.

Variables and arguments grouping

It is now possible to group the type declaration for local variables and function arguments. For example:

    foo: func [
        src dst    [byte-ptr!]
        mode delta [integer!]
        return:    [integer!]
        /local
            p q buf  [byte-ptr!]
            s1 s2 s3 [c-string!]
    ]
 

Note that the compiler supports those features through code expansion at compile time, so that error reports could show each argument or variable having its own type declaration.

Integer division handling

Integer division handling at low-level has notorious shortcomings with different handling for each edge case depending on the hardware platform. Intel IA-32 architecture tends to handle those cases in  a slightly safer way, while ARM architecture produces erroneous results silently typically for the following two cases:

  • division by zero
  • division overflow (-2147483648 / -1)

IA-32 CPU will generate an exception, while ARM ones will return invalid results (respectively 0 and -2147483648). This makes it difficult to produce code that will behave the same on both architectures when integer divisions are used. In order to reduce this gap, R/S compiler will now generate extra code to detect those cases for ARM targets and raise a runtime exception. Such extra checkings for ARM are produced only in debug compilation mode. In release mode, priority is given to performance, no runtime exception will occur in such cases on ARM (as the overhead is significant). So, be sure to check your code on ARM platform thoroughly in debug mode before releasing it. This is not a perfect solution, but at least, it makes it possible to detect those cases through testing in debug mode.

Others

Here is a list of other changes and fixes in no particular order:

  • Cross-referenced aliased fields in structs defined in same context are now allowed. Example:

        a!: alias struct! [next [b!] prev [b!]]
        b!: alias struct! [a [a!] c [integer!]]
  • -0.0 special float literal is now supported.
  • +1.#INF is also now supported as valid literal in addition to 1.#INF for positive infinite.
  • Context-aware get-words resolution.
  • New #inline directive to inline assembled binary code.
  • Dropped support for % and // operators on float types, as they were relying on FPU's relative support, the results were not reliable across platforms. Use fmod function instead from now on.
  • Added --show-func-map compilation option: when used, it will output a map of R/S function addresses/names, to ease low-level debugging.
  • FIX: issue #4102: ASSERT false doesn't work.
  • FIX: issue #4038: cast integer to float32 in math expression gives wrong result.
  • FIX: byte! to integer! conversion not happening in some cases. Example: i: as-integer (p/1 - #"0")
  • FIX: compiler state not fully cleaned up after premature termination. This affects multiple compilation jobs done in the same Rebol2 session, resulting in weird compilation errors.
  • FIX: issue #4414: round-trip pointer casting returns an incorrect result in some cases.
  • FIX: literal arrays containing true/false words could corrupt the array. Example: a: ["hello" true "world" false]
  • FIX: improved error report on bad declare argument.

7 comments:

  1. I want to be the first to congratulate for such great work. It's a source of inspiration to have the knowledge to use everything inside.

    ReplyDelete
  2. Thanks for improving Red/System, this is so cool! What I like most is the new lock-free atomic intrinsics for future concurrency and parallel processing developments. So refreshing! :-)

    ReplyDelete
  3. Impressive creative solid work
    Keep on keeping on !

    ReplyDelete
  4. Expecting much better performance now onwards with these changes...

    ReplyDelete
  5. Hello RED team. The last post here was April 20 and on Twitter Red-lang, Oct 20, of last year. Never feel like the teams posts must be somewhat ground breaking or totally exciting to the general reader/supporter. We and new interested parties must see you are still supporting your own vision or like a slow loading web page they'll be forced to move on. Even the dogs get scraps from there masters table when there really hungered.

    ReplyDelete
  6. Honestly one of the things that bugs me the most about this post is ragging on Zortech. You guys really, really don't know what you're talking about and never have. History won't be kind when they look back at this story.

    ReplyDelete
  7. Soundbite again, for emphasis: "Stop blaming the fact that Twitter scales on that you can't code. You sound like Carl."

    ReplyDelete

Fork me on GitHub