Showing posts with label compilation. Show all posts
Showing posts with label compilation. Show all posts

August 20, 2020

Red/System: New Features

In the past months, many new features were added to Red/System, the low-level dialect embedded in Red. Here is a sum up if you missed them.

Subroutines

During the work on the low-level parts of the new Red lexer, the need arised for intra-function factorization abilities to keep the lexer code as DRY as possible. Subroutines were introduced to solve that. They act as the GOSUB directive from Basic language. They are defined as a separate block of code inside a function's body and are called like regular functions (but without any arguments). So they are much lighter and faster than real function calls and require just one slot of stack space to store the return address. 

The declaration syntax is straightforward:

    <name>: [<body>]

    <name> : subroutine's name (local variable).
    <body> : subroutine's code (regular R/S code).

To define a subroutine, you need to declare a local variable with the subroutine! datatype, then set that variable to a block of code. You can then invoke the subroutine by calling its name from anywhere in the function body (but after the subroutine own definition).

Here is a first example of a fictive function processing I/O events:

    process: func [buf [byte-ptr!] event [integer!] return: [integer!]
        /local log do-error [subroutine!]
    ][
        log: [print-line [">>" tab e "<<"]]
        do-error: [print-line ["** Error:" e] return 1]
    
        switch event [
            EVT_OPEN  [e: "OPEN"  log unless connect buf [do-error]]
            EVT_READ  [e: "READ"  log unless receive buf [do-error]]
EVT_WRITE [e: "WRITE" log unless send buf [do-error]]
EVT_CLOSE [e: "CLOSE" log unless close buf [do-error]]
default [e: "<unknown>" do-error] ] 0 ]

This second example is more complete. It shows how subroutines can be combined and how values can be returned from a subroutine:

    #enum modes! [
    	CONV_UPPER
    	CONV_LOWER
    	CONV_INVERT
    ]

    convert: func [mode [modes!] text [c-string!] return: [c-string!]
        /local
            lower? upper? alpha? do-conv [subroutine!]
            delta [integer!]
            s     [c-string!]
            c     [byte!]
    ][
        lower?:  [all [#"a" <= c c <= #"z"]]
        upper?:  [all [#"A" <= c c <= #"Z"]]
        alpha?:  [any [lower? upper?]]
        do-conv: [s/1: s/1 + delta]
        delta:   0
        s:       text

        while [s/1 <> null-byte][
            c: s/1
            if alpha? [
                switch mode [
                    CONV_UPPER  [if lower? [delta: -32 do-conv]]
                    CONV_LOWER  [if upper? [delta: 32 do-conv]]
                    CONV_INVERT [delta: either upper? [32][-32] do-conv]
                    default     [assert false]
                ]
            ]
            s: s + 1
        ]
        text
    ]
    
    probe convert CONV_UPPER "Hello World!"
    probe convert CONV_LOWER "There ARE 123 Dogs."
    probe convert CONV_INVERT "This SHOULD be INVERTED!"

will output:

    HELLO WORLD!
    there are 123 dogs.
    tHIS should BE inverted!

Support for getting a subroutine address and dispatching dynamically on it is planned to be added in the future (something akin computed GOTO). More examples of subroutines can be found in the new lexer code, like in the load-date function.

New system intrinsics


Several new extensions to the system path have been added.

Lock-free atomic intrinsics


A simple low-level OS threads wrapper API has been added internally to the Red runtime as preliminary work on supporting parts of IO concurrency and parallel processing in the future. In order to complement it, a set of atomic intrinsics were added to enable the implementation of lock-free and wait-free algorithms in a multithreaded execution context.

The new atomic intrinsics are all documented here. Here is a quick overview:
  • system/atomic/fence: generates a read/write data memory barrier.
  • system/atomic/load: thread-safe atomic read from a given memory location.
  • system/atomic/store: thread-safe atomic write to a given memory location.
  • system/atomic/cas: thread-safe atomic compare&swap to a given memory location.
  • system/atomic/<math-op>: thread-safe atomic math or bitwise operation to a given memory location (add, sub, or, xor, and).

Other new intrinsics
  • system/stack/allocate/zero: allocates a storage space on stack and zero-fill it.
  • system/stack/push-all: saves all registers to stack.
  • system/stack/pop-all: restores all registers from stack.
  • system/fpu/status: retrieves the FPU exception bits status as a 32-bit integer.

Improved literal arrays

The main change is the removal of the hidden size inside the /0 index slot. The size of a literal array can now only be retrieved using the size? keyword, which is resolved at compile time (rather than run-time for /0 index access).

A notable addition is the support for binary arrays. Those arrays can be used to store byte-oriented tables or embed arbitray binary data into the source code. For example:

    table: #{0042FA0100CAFE00AA}
    probe size? table                      ;-- outputs 9
    probe table/2                          ;-- outputs "B"
    probe as integer! table/2              ;-- outputs 66
The new Red lexer code uses them extensively.

Variables and arguments grouping

It is now possible to group the type declaration for local variables and function arguments. For example:

    foo: func [
        src dst    [byte-ptr!]
        mode delta [integer!]
        return:    [integer!]
        /local
            p q buf  [byte-ptr!]
            s1 s2 s3 [c-string!]
    ]
 

Note that the compiler supports those features through code expansion at compile time, so that error reports could show each argument or variable having its own type declaration.

Integer division handling

Integer division handling at low-level has notorious shortcomings with different handling for each edge case depending on the hardware platform. Intel IA-32 architecture tends to handle those cases in  a slightly safer way, while ARM architecture produces erroneous results silently typically for the following two cases:

  • division by zero
  • division overflow (-2147483648 / -1)

IA-32 CPU will generate an exception, while ARM ones will return invalid results (respectively 0 and -2147483648). This makes it difficult to produce code that will behave the same on both architectures when integer divisions are used. In order to reduce this gap, R/S compiler will now generate extra code to detect those cases for ARM targets and raise a runtime exception. Such extra checkings for ARM are produced only in debug compilation mode. In release mode, priority is given to performance, no runtime exception will occur in such cases on ARM (as the overhead is significant). So, be sure to check your code on ARM platform thoroughly in debug mode before releasing it. This is not a perfect solution, but at least, it makes it possible to detect those cases through testing in debug mode.

Others

Here is a list of other changes and fixes in no particular order:

  • Cross-referenced aliased fields in structs defined in same context are now allowed. Example:

        a!: alias struct! [next [b!] prev [b!]]
        b!: alias struct! [a [a!] c [integer!]]
  • -0.0 special float literal is now supported.
  • +1.#INF is also now supported as valid literal in addition to 1.#INF for positive infinite.
  • Context-aware get-words resolution.
  • New #inline directive to inline assembled binary code.
  • Dropped support for % and // operators on float types, as they were relying on FPU's relative support, the results were not reliable across platforms. Use fmod function instead from now on.
  • Added --show-func-map compilation option: when used, it will output a map of R/S function addresses/names, to ease low-level debugging.
  • FIX: issue #4102: ASSERT false doesn't work.
  • FIX: issue #4038: cast integer to float32 in math expression gives wrong result.
  • FIX: byte! to integer! conversion not happening in some cases. Example: i: as-integer (p/1 - #"0")
  • FIX: compiler state not fully cleaned up after premature termination. This affects multiple compilation jobs done in the same Rebol2 session, resulting in weird compilation errors.
  • FIX: issue #4414: round-trip pointer casting returns an incorrect result in some cases.
  • FIX: literal arrays containing true/false words could corrupt the array. Example: a: ["hello" true "world" false]
  • FIX: improved error report on bad declare argument.

March 26, 2017

0.6.2: LibRed and Macros

It is with great pleasure that we announce the 0.6.2 release of the Red programming language and toolchain. This release is the second heaviest one we have ever made (after the 0.6.0), weighing about 1200 commits! It was intended initially to be a minor one, but the needs for preprocessing support for Red runtime code arose, so it was the right time to make a first iteration into the land of macros. Moreover, the work on libRedRT (described below) gave us the opportunity to fulfill one of the goals we had for Red: become embeddable. This is now a reality, thanks to libRed. So the main features of this release are:

  • Macros and preprocessor support
  • Fast compilation using libRedRT
  • LibRed for embedding Red anywhere

Macros

The Red macros and preprocessing abilities have been the topic of a previous article, so I invite you to discover them (and hopefully enjoy!) if you have not yet done so.

In addition to this article, it is worth mentioning that a feature called pre-load has been implemented since then (you can read about it down below), which can be used as an entry point for user-provided reader macros, and allow easier support for non-standard syntax. This is a minor feature, as Rebol-like languages already come strongly equipped for string parsing and processing, thanks to the Parse dialect.


Development and release modes for compilation

The biggest change introduced by this new version is the splitting of the compilation process in two modes, which are controlled by different command-line options:

  • development: (-c) only user code is compiled (new mode).
  • release: (-r) both user code and Red runtime library are compiled together.

When using -c, the Red runtime library is pre-compiled to a dynamic library called libRedRT ("RT" for RunTime), stored in the hidden red/ folder. When libRedRT is already present, only user code is compiled resulting in drastically shorter compilation times.

When -r option is used, the toolchain compiles user code in release mode, compiling with it all dependencies (including Red runtime library).

Here are a couple of simple benchmarks to show the (huge) difference in performance, using hello.red and view-test.red scripts compiled in both modes:


Another benchmark is compiling the Red tests suite (~18000 tests), unified way combines the tests into the minimum number of compilation units, while split way compiles each test file separately:


Measured on a Corei7 4Ghz Win7/64-bit machine.

Such performance results in significant daily productivity gains, both for the Red team, and Red users. It was worth the time and effort it took to properly convert the runtime library into a shared library. Though, the full support for modular compilation will come in 0.8.0, which will result in drastic cuts for the compilation time in release mode too.

Such fast compilation mode also works for Red scripts that embed Red/System code. Two cases are possible:
  1. Red/System code does not contain any call to Red runtime library.
  2. Red/System code contains one or several calls to Red runtime library.
In the first case, nothing needs to be done, the usage rules described above apply.

In the second case, a custom version of libRedRT is required, but the toolchain will take care of the process, it just requires the user to compile once using -u option, then simply use -c for next compilations. If new functions from the runtime library are called, then a new custom library needs to be rebuilt.

When compiling a custom run-time library, cleaning libRedRT files is needed to get rid of any outdated versions. The toolchain provides a red clear command which will remove all current libRedRT related files. Note that when upgrading the red binary to a newer version, it will automatically upgrade libRedRT on first invocation.

Any Red user code can be compiled in development mode, with the exception of objects with multiple inheritance, which is not supported by libRedRT (so can only be compiled in release mode).


Embedding Red using libRed

LibRed is the embeddable version of the Red interpreter + runtime library. It includes Parse dialect, reactive programming and our GUI system (View engine, VID dialect, Draw dialect). It exposes an interface to the host language through an API, which is C-compatible, so that any language that can embed C shared libraries, can also embed libRed.

The libRed API has been designed to be as simple and as straightforward as possible. See for yourself in the API documentation. The role of that API is to provide the required hooks for interfacing the Red runtime and interpreter with the host language. A high-level binding can eventually be built on top of the libRed C-level API, that will best map Red features, to the host language features.

A libRed HelloWorld in C:
    #include "red.h"

    int main() {
        redOpen();
        redPrint(redString("Hello World!"));
        redClose();
        return 0;
    }
A libRed graphic HelloWorld in C:
    #include "red.h"

    int main() {
        redOpen();
        redDo("view [text {Hello World!}]");
        redClose();
        return 0;
    }

In addition to the C-level API, we provide also a binding for VisualBasic for Applications, which can be used to embed Red into Microsoft Office applications!

Here is a demo showing side-by-side an Excel/VB window and a Red window playing Pong game:


All the code for this demo fits in a single page of VB code! You can get the required files from here. In that same Excel file, you will find two other simple examples of integration of Red with Excel sheets, which look like this:



Building libRed is straightforward:
    red build libred
That will build libRed with the cdecl ABI, suitable for integration with any C-compatible host. For VBA and MS Office, the stdcall ABI is required, which is achieved using:
    red build libred stdcall
The building process will also result in creating a libRed/ directory locally, expanding some extra files required for linking libRed properly.

Currently, full bindings are provided for C and VisualBasic. Experimental bindings are also available for RustAdobe AIR and C#.

This is a first iteration for libRed, we have plans for improving it, and for providing proper multi-instances support (currently limited at one instance per process).


Changes in 0.6.2

Core language
  • New datatypes: tag!, email!
  • New action: to
  • New natives: as, call
  • New functions: tag?, email?, hex-to-rgb, sqrt, class-of, face?, distance?, expand-directives, to-*, rejoin
  • Adds integers auto-promotion to floats on loading and on some integer operations.

If you need to preprocess the input to LOAD, you can now do it easily by plugging a function into system/lexer/pre-load. This feature is mostly meant for pre-processing the console's input, though it could also be used for changing some Red syntactic rules. For example:
    system/lexer/pre-load: func [src part][replace/all src comma space]

    >> [1,2,3,abd,"hello"]
    == [1 2 3 abd "hello"]
Another usage could be to translate words on-the-fly in the console:
    system/lexer/pre-load: function [src part][
        parse src [
            any [
                s: [
                    "affiche"       (new: "print")
                    | "si"          (new: "if")
                    | "tant que"    (new: "while")
                    | "pair?"       (new: "even?")
                    | "impair?"     (new: "odd?")
                ] e: (s: change/part s new e) :s
                | skip
            ]
        ]
    ]

    >> i: 10 tant que [i > 0][si impair? i [affiche i] i: i - 1]
    9
    7
    5
    3
    1

In addition to several small fixes, load now offers a /trap refinement, which enables manual error management when loading a string series. Instead of raising an error, load/trap will just stop and return a block made of:
  • a block of successfully loaded values.
  • the input string at the position where the lexer failed.
  • an error! value (or none! value if the tail of the string has been reached).

Command-line argument processing has been mostly rewritten to provide a consistent experience across platforms and type of binaries (red executables, console executables, compiled user scripts and, to a lesser extent, Red/System executables). The new features are:
  • system/options/script refers to the script name (string! or none!).
  • system/options/args refers to a list of tokenized arguments (block! or none!).
  • system/script/args refers to the original command-line (string! or none!).
  • full Unicode support for red executable's arguments on Windows.
  • single-quoted arguments are accepted on all platforms (same as double quotes)
  • multiple nested quotes are treated as just one level of quoting when splitting the command-line.

Datatypes conversions are now fully supported in Red! The to action is now officially supported, and make action has been completed. The usual to-<type> helper functions you can find in Rebol, are also now defined. All the conversion rules are documented in an Excel matrix for now.

Call

Calling external applications is also now possible thanks to the contributed code for call function by Bruno Anselme, which has now been integrated into the runtime library. Use help call to explore all the options offered. Note that the /console option, which redirects I/O from child process, is not supported currently by the Red GUI console (it works fine from within the Red CLI console).

View and VID dialect

A number of small changes and fixes have been provided both in View and VID, among them:
  • box, h1 to h5 styles added to VID.
  • Colors in VID can now be specified as hex values, using #rgb or #rrggbb formats.
  • Adds support for no-border flag to area and field face types.
  • Adds now option to rate keyword in VID, to fire on-time actor at once.
  • The wheel event  and on-wheel actor are now available for handling mouse wheel events.
  • Default tab size for area changed to 4 spaces.
  • View now uses DirectWrite to draw text in base face (except WindowsXP).
  • Better handling of default fonts.
  • Enhanced GUI console, with new settings window with pre-selected colors picker.
  • A hint text can now be specified in a field options block and a hint keyword has been added to VID. For example:



Draw dialect

Big additions were made to Draw, most thanks to massive contributions from Zamlox:

  • Matrix operations support: matrix, invert-matrix, reset-matrix, matrix-order, push, rotate, scale, translate, skew, transform.
  • A new clip command is available for defining a clipped drawing/filling region.
  • A Shape sub-dialect has been added for more complex shapes drawing and filling.
  • A crop option is now available for image command.
  • pen and fill-pen have been vastly extended to allow drawing and filling with gradients, patterns, arbitrary shapes and images. Have a look at some of the new capabilities (source):



Parse dialect

In addition to some fixes, a few new features were added:
  • insert command now also support position argument (like change).
  • added pick option to keep, so user can control how keep captures the matched input:
    • keep collects matched values as a series if many, or as a value if only one.
    • keep pick collects all the matched values separately in a block.
    • keep copy <word> collects all the matched values as a single series (of same type as input).

Red/System dialect

  • Support for float! / float32! conversions from/to integer!.
  • Adds system/cpu/overflow? field for reading CPU's integer overflow state.
  • Adds support for importing variables from shared libraries.
  • Allows loading of libraries from current folder and PATH environment variable on macOS.
  • Now #call directive supports function calls with refinements.
  • Default ABI for exported functions is now settable through export-ABI entry in config object.
  • Renamed log and log10 imports from libC, respectively to log-2 and log-10.
  • Now size? accepts a context path argument.
  • Improved error reporting for "missing argument" errors.

Other changes

  • A prototype Red/.Net bridge has been introduced.
  • New --config [...] command-line option, for passing a block of compiler settings.
  • Added -s and --show-expanded command-line options to output expanded version of compiled source code.
  • Added /target option to react?.
  • Added /seek and /lines option to write.
  • Added /expand refinement to do for preprocessor invocation.
  • Added math function for evaluating code using math precedence rules.
  • CTRL-L key combination can now clear the GUI console's screen.
  • Checksum function can now trigger object on-deep-change event.
  • Now keep returns its argument (collect function).
  • Added temporary rejoin function.
  • Added 'class reflective property to objects.
  • Added class-of accessor (only objects).
  • Nicer handling of line breaks in molded image! binary buffer.
  • Now #include is converted to do in interpreted code (using macros).
  • Zero? function is now a native and supports time! values.

Also, more than 150 issues have been fixed (and wishes granted) during the last months, 22 issues marked as bugs are left open.

Last but not least, our documentation on Gitbook (which is an ongoing work) has been moved to Asciidoc format now, thanks to the efforts of Tovim. That new format will provide us better options for a more accurate control of the styles and layout.

A big thank goes to all contributors who pushed code or who opened tickets clearly identifying issues.

What's Next?

Since 0.6.1, we have adjusted the Red roadmap to work on two releases in parallel. This means that while 0.6.2 was progressing, 0.6.3 was advancing at the same time, is now almost ready, and will be merged into master in a couple of days (if you happen to have a Mac, it contains the macOS GUI backend for Red!). As soon as 0.6.3 is out, 0.7.0 will start (full async I/O), while 0.6.4 (Android) is being worked on. That should provide us a higher number of new releases this year, while still implementing large new features. So far, such approach has worked pretty well.

In the meantime, and as usual, enjoy playing with Red!

April 26, 2015

0.5.3: Faster compilation and extended vector! support

The main point of this minor release is to speed up compilation time by introducing a new way for the compiler to store Red values required for constructing the environment during the runtime library startup.

Introducing Redbin

Red already provides two text-oriented serialization formats, following the base Rebol principles. Here are the available serialization formats now in Red with some pros/cons:
  • MOLD format
    • provides a default readable text format, very close to the source code version
    • cannot properly encode many values
  • MOLD/ALL format
    • can encode series offsets
    • some values with literal forms that rely on words that can be natively encoded (none, true/false, objects, ...)
    • human-readable, but not always nice-looking
  • Redbin format
    • can encode any value accurately
    • supports words binding
    • can encode contexts efficiently
    • supports cycles in blocks
    • can encoded name/value pairs in any context
    • extremely fast loading time
    • very small storage space used when compressed
    • non human-readable
So far, the existing environment source code (mostly block values) was converted to pure Red/System construction code which was pretty simple and straightforward to implement, but was generating thousands of extra lines of code, slowing down the native compilation process. The right solution for that was to introduce a new binary serialization format for Red values called Redbin (very inspired by Carl's REBin proposal).

Redbin's specification focuses on optimizing the loading time of encoded values, by making their stored representation very close to their memory representation, bypassing the parsing and validation stages. Moreover, the Redbin payload is compressed using the Crush algorithm (that Qtxie ported to Red/System), which features one of the fastest decompressors around while having a general compression ratio very close to the deflate algorithm (but compression speed is about an order of magnitude slower). This fits perfectly the needs for our Redbin use-case.

So the gains compared to pre-0.5.3 version are:
  • compilation time of empty Red program is ~40% faster!
  • generated executable of empty Red program is about 100KB smaller (278KB only on Windows now).
  • faster startup time, as the Redbin decoding process is much faster than the previous Red-stack-oriented construction approach.
Those benefits also extend to user code, your static series will be saved in Redbin format as well.

Redbin format is currently emitted by the compiler and decoded by the Red runtime, but there is no encoder yet in the runtime that would allow user code to emit Redbin format. We will provide that support in a future version, it is not high priority for now. A "compact" version of the encoding format will also be added, so that Redbin can also be a good choice for remote data exchange.

Compilation from Rebol console

For those using Red toolchain from Rebol2 console, a new rc function is introduced to avoid reloading the toolchain on each run. Typical session looks like this:

    >> do %red.r
    >> rc "-c tests\demo.red"

    -=== Red Compiler 0.5.3 ===-

    Compiling /C/Dev/Red/tests/demo.red ...
    ...compilation time : 416 ms

    Compiling to native code...
    Script: "Red/System PE/COFF format emitter" (none)
    ...compilation time : 12022 ms
    ...linking time     : 646 ms
    ...output file size : 284160 bytes
    ...output file      : C:\Dev\Red\demo.exe
 
    >> call/output "demo.exe" s: make string! 10'000
    == 0
    >> print s

      RedRed              d
      d     d             e
      e     e             R
      R     R   edR    dR d
      d     d  d   R  R  Re
      edRedR   e   d  d   R
      R   e    RedR   e   d
      d    e   d      R   e
      e    R   e   d  d  dR
      R     R   edR    dR d

Collation tables

Since 0.5.2, Red provides collation tables for more accurate case folding support. Those tables can now be accessed by users using these paths:
    system/locale/collation/upper-to-lower
    system/locale/collation/lower-to-upper
Each of these tables is a vector of char! values which can be freely modified and extended by users in order to cope with some specific local rules for case folding. For example, in French language, the uppercase of letter é can be E or Ã‰. There is a divide among French people about which one should be used and in some cases, it can just be a typographical constraint. By default, Red will uppercase é as Ã‰, but this can be easily changed if required, here is how:

    uppercase "éléphant"
    == "ÉLÉPHANT"
    
    table: system/locale/collation/lower-to-upper
    foreach [lower upper] "àAéEèEêEôOûUùUîIçC" [table/:lower: upper]

    uppercase "éléphant"
    == "ELEPHANT"

Extended Vector! datatype

Vector! datatype now supports more actions and can store more datatypes with different bit-sizes. For integer! and char! values, you can store them as 8, 16 or 32 bits values. For float!, it is 32 or 64 bits. Several syntactic forms are accepted for creating a vector:
    make vector! <slots>
    make vector! [<values>]
    make vector! [<type> <bit-size> [<values>]]
    make vector! [<type> <bit-size> <slots> [<values>]]

    <slots>    : number of slots to preallocate (32-bit slots by default)
    <values>   : sequence of values of same datatype
    <type>     : name of accepted datatype: integer! | char! | float!
    <bit-size> : 8 | 16 | 32 for integer! and char!, 32 | 64 for float!
The type of the vector elements can be inferred from the provided values, so it can be omitted (unless you need to force a bit-size different from the values default one). If a value with a bit-size greater than the vector elements one, is inserted in the vector, it will be truncated to the bit-size of the vector.

For example, creating a vector that contains 1000 32-bit integer values:
    make vector! 1000
Or if you want to specify the bit-size of the vector element:
    make vector! [char! 16 1000]
    make vector! [float! 64 1000]
You can also initialize a vector from a block as below:
    make vector! [1.1 2.2 3.3 4.4]
Again you can also specify the bit-size of the vector element:
    make vector! [integer! 8 [1 2 3 4]]
For integer! and char! vectors, you can use all math and bitwise operators now.
    x: make vector! [1 2 3 4]
    y: make vector! [2 3 4 5]
    x + y
    == make vector! [3 5 7 9]
In case of different bit-sizes, the resulting vector will be using the highest bit-size. If a math operation is producing a result that does not fit the bit-size, the result is currently truncated to the bit-size (using a AND operation). Ability to read and change the bit-size of a vector will be added in future releases.

The following actions are added to vector! datatype: clear, copy, poke, remove, reverse, take, sort, find, select, add, subtract, multiply, divide, remainder, and, or, xor.

The vector! implementation is not yet final, some of its actions will get optimized for better performances and, in future, rely on SIMD for even faster operations. For multidimensional support, it will be implemented as a new matrix! datatype in the near future, inheriting from vector!, so the additional code required will be kept minimal.

Bugfixing

This was a short-term release, but we managed to fix a few bugs anyway.

What's next

Another minor release will follow with many runtime library additions and new toolchain improvements. See the planned features for 0.5.4 on our Trello board.

The 0.6.0 release will also most probably be split in two milestones, one for GUI and another for Android support.

In the meantime, enjoy this new release! :-)

September 27, 2013

0.4.0: Red goes binary!

What's that?!

As we are getting closer to the end of the alpha period, we are now moving to a more convenient way to use and distribute the Red toolchain. So far, you needed to download a Rebol interpreter and the sources from Github separately, and run it using, a bit verbose, command-lines. This is fine for developping Red  with contributors that are interested in the inner workings of the toolchain, but for the end users, the goal has always been to provide a simpler and much more convenient way, like Rebol teached us in the past.

So, from now, you can get Red as a single binary (< 1 MB) from the new Download page. Yes, all of Red toolchain and runtime is packed in that small binary, even the REPL is built-in!

The Red repo landing page has been reworked to show Red usage in binary form, all previous command-line options are present, a new one (-c) has been introduced. Here is a short overview of the main options:

Launch the REPL:
    $ red 
Run a Red script directly (interpreted):
    $ red <script>
Compile a script as executable with default name in working path:
    $ red -c <script>
Compile a script as shared library with default name in working path:
    $ red -dlib <script>
Compile a script as executable with alternative name:
    $ red -o <new> <script>
Compile a script as executable with default name in other folder:
    $ red -o <path/> <script>
Compile a script as executable with new name in other folder:
    $ red  -o <path/new> <script>
Cross-compile to another platform:
    $ red -t Windows <script>
Display a description of all possible options:
    $ red -h
Notice that -c option is implied when -o or -t are used. It is designed to keep command-lines as simple and short as possible.

Moreover, for standalone Red/System programs, the Red binary will be able to compile them directly, no special option needed, they will be recognized automatically.

Thanks very much to Tamás Herman for helping with setting up the build farm and providing the Mac OSX machine, and thanks to the HackJam hackerspace group from Hong-Kong for the hosting!

Other changes

  • In addition to that new binary form, 17 issues have been fixed since the 0.3.3 release about a month ago (not counting regression tickets).
  • The work on objects support is half-done, objects are working fine with advanced semantics on the interpreter (see object branch), now the focus will be to support them at the Red compiler level.

What's next?

As we are moving closer to the beta state, version numbers will increase faster, e.g., once objects will be done, the release will be the 0.5.0, while 0.6.0 will bring full I/O support. Between these major versions, smaller versions should be still possible, this means that the release cycle should accelerate with at least one release each month from now on. So, what you should expect in the next ones?

0.4.x
  • Simple I/O support: (just read, write and exists? on files)
  • PARSE support
  • Pre-compiled runtime (much faster compilation times)
0.5.0
  • Full object support
0.5.x
  • VID-like cross-platform dialect binding to native widgets.
  • Mezzanine functions additions
  • Redbin (accurate Red values serialization in binary format)
  • Full errors management
  • Red-level exceptions handling

Enjoy!


September 21, 2011

Red/System v0.2.2 released

This release is mainly a bugfix release that solves several old issues. It is also now correctly synchronized with all the current bindings done for Red/System. The main changes are:
  • Internal compiler refactoring of: expressions compilation, type casting and ANY/ALL support.
  • Greatly improved runtime error reporting: now it reports both source line number and source file name where the error occured. It works in debug mode only (-g command-line option).
  • Aliased struct names can now be tested separately in typed (RTTI) functions.
  • Callback function attribute removed. It is no more needed and any function can now be used as a callback. In addition a new cdecl attribute is now accepted to allow the switch to C calling convention, when passing a function as argument to an imported C function.
  • 21 issue reports closed.
  • More than 2000 new unit tests were added (mostly generated using scripts wrote by Peter WA Wood) for a total of now 8613 tests.

May 14, 2011

Red/System compiler overview

Source Navigation

As requested by several users, I am giving a little more insights on the Red/System compiler inner workings and a map for navigating in the source code.

Current Red/System source tree:
 red-system/
%compiler.r ; Main compiler code, loads everything else
%emitter.r ; Target code emitter abstract layer
%linker.r ; Format files loader
%rsc.r ; Compiler's front-end for standalone usage

formats/ ; Contains all supported executable formats
%PE.r ; Windows PE/COFF file format emitter
%ELF.r ; UNIX ELF file format emitter

library/ ; Third-party libraries

runtime/ ; Contains all runtime definitions
%common.reds ; Cross-platform definitions
%win32.r ; Windows-specific bindings
%linux.r ; Linux-specific bindings

targets/ ; Contains target execution unit code emitters
%target-class.r ; Base utility class for emitters
%IA32.r ; Intel IA32 code emitter

tests/ ; Unit tests

Once the compiler code is loaded in memory, the objects hierarchy looks like:
 system/words/          ; global REBOL context

system-dialect/ ; main object
loader/ ; preprocessor object
process ; preprocessor entry point function
compiler/ ; compiler object
compile ; compiler entry point function

emitter/ ; code emitter object
compiler/ ; short-cut reference to compiler object
target/ ; reference the target code emitter object
compiler/ ; short-cut reference to compiler object

linker/ ; executable file emitter
PE/ ; Windows PE/COFF format emitter object
ELF/ ; UNIX ELF format emitter object


Note: the linker file formats are currently statically loaded, this will be probably changed to a dynamic loading model.

Compilation Phases

The compilation is a process that goes through several phases to transform a textual source code to an executable file. Here is an overview of the process:


1) Source loading

This is a preparatory phase that would convert the text source code to its memory representation (close to an AST). This is achieved in 3 steps:
  1. source is preprocessed in its text form to make the syntax REBOL-compatible
  2. source is converted to a tree of nested blocks using the REBOL's LOAD function
  3. source is postprocessed to interpret some compiler directives (like #include and #define)
2) Compilation

The compiler walks through the source tree using a recursive descent parser. It attempts to match keywords and values with its internal rules and emits:
  • the corresponding machine code in the code buffer
  • the global variables and complex data (c-string! and struct! literals) in the data buffer
The internal entry point function for the compilation is compiler/comp-dialect. All the compiler/comp-* functions are used to recursively analyze the source code and each one matches a specific semantic rule (or set of rules) from the Red/System language specification.

The production of native code is direct, there is no intermediary representation, machine code is generated as soon as a language statement or expression is matched. This is the simplest way to do it, but code cannot be efficiently optimised without a proper Intermediate Representation (IR). When Red/System will be rewritten in Red, a simple IR will be introduced to enable the full range of possible code optimisations.

As you know, a Red/System program entry point is at the beginning of the source code. During the compilation, the source code in the global context is compiled first and all functions are collected and compiled after all global code. So the generated code is always organised the same way:
  • global code (including proper program finalization)
  • function 1
  • function 2
  • ...
The results of the compilation process are:
  • a global symbol table
  • a machine code buffer
  • a global data buffer
  • a list of functions from external libraries (usually these are OS API mappings)
The compiler is able to process one or several source files this way before entering the linking phase.

3) Linking

The linking process goal is to create an executable file that could be loaded by the target platform. So the executable file needs to conform to the target ABI for that, like PE for Windows or ELF for Linux.

During the linking, the global symbol table is used to "patch" the code and data buffer (see linker/resolve-symbol-refs), inserting the final memory address for the pointed resources (variable, function, global data). The different parts to assemble are grouped into so-called "sections", that can be themselves be grouped into "segments" (as, e.g., in ELF).

Finally, some headers describing the file and its sections/segments are inserted to complete the file. The file is then written down on disk, marking the end of the whole process.

Static linking of external libraries (*.lib, *.a,...) will be added at some point in the future (when the need for such feature will appear).


I hope this short description gives you a better picture on how Red/System compiler works, even if it is probably obvious for the most experienced readers. Feel free to ask for more in the comments, or better, on the Google Groups mailing-list.

Fork me on GitHub