July 17, 2017

0.6.3: macOS GUI backend

This new Red release contains ~830 commits, closes 86 issues, and brings a large number of  new features. The main one is official support for the macOS GUI, through a new backend for the View engine.


For readers not familiar with our GUI system, you can get a quick overview in a previous article.

In a nutshell, it is a dynamically-built reactive object tree, with:
  • pluggable backends (full support for Windows and prototype for GTK and Android)
  • full 2D vector graphics support (a large subset of SVG)
  • direct reactive programming support (not FRP style for now)
  • a declarative DSL for GUI building and layout management
  • a declarative DSL for 2D graphics

The macOS backend supports all the same features as the Windows backend, except the following, yet to be implemented:
  • fill-pen pattern, fill-pen bitmap and diamond gradient in Draw.
  • matrix transformations on pen and fill-pen in Draw.

The GUI console for macOS will be available in the 0.6.4 branch, so is not part of this release. The View module, though, is compiled in the CLI console by default on macOS, so no show-stopper there.

Here are a few short examples, starting with a GUI Hello World!:
    view [text "Hello Mac World!"]

A tiny demo script (hopefully, it should get us an office at Station F, right? ;-)):
    view [
        t: text "French Touch! " on-time [move t/text tail t/text]
        base 21x15 draw [
            pen off
            fill-pen blue  box 0x0  6x14
            fill-pen white box 7x0  14x14
            fill-pen red   box 14x0 20x14
        ] return
        button "Start" [t/rate: 10]
    ]

A small reactive text size measuring tool (one of our test scripts):
    view [
        style txt: text right
        txt "Text" f: area 200x80 
            font [name: "Comic Sans MS" size: 15 color: black]
            return

        txt "Size in pixels" text "0x0"
            react [[f/text f/font] face/text: form size-text f]
            return
 
        txt "Font name" drop-list 120
            data  ["Arial" "Consolas" "Comic Sans MS" "Times"]
            react [f/font/name: pick face/data any [face/selected 3]]
            return
  
        txt "Font size" s: field "15" react [f/font/size: s/data]
        button "+" bold 40 [s/data: s/data + 1]
        button "-" bold 40 [s/data: max 1 s/data - 1]
        return
    ]


Generating macOS bundle apps is made simple by our toolchain, just use the -t macOS target when compiling, for example using our SVG Tiger demo:
   $ red -o ~/Desktop/ -t macOS tiger.red
This will produce a tiger.app bundle on the Desktop:



Cross-Platform GUI Metrics

In order to cope with different UI guidelines across GUI platforms, this release also introduces an experimental rule-oriented GUI rewriting engine, capable of modifying a face tree dynamically according to pre-set rules. It is integrated as a last stage in VID (Visual Interface Dialect) processing. The currently implemented rules are per-platform for now (more general, cross-platform rules are also planned):

Windows rules:
  • color-backgrounds: color the background of some colorless faces to match their parent's color
  • color-tabpanel-children: Like color-backgrounds, but tab-panel specific
  • OK-Cancel: buttons ordering rule, puts Cancel/Delete/Remove buttons last
macOS rules:
  • adjust-buttons: use standard button sub-classes when buttons are narrow enough
  • capitalize: capitalize widget text according to macOS guidelines
  • Cancel-OK: buttons ordering rule, puts Ok/Save/Apply buttons last

To give you a quick grasp about why that feature matters and what it is capable of, here is a simple example, which leverages the buttons ordering and capitalization rules:
    view [
        text "Name" right 50 field return
        text "Age"  right 50 field return
        button "ok" button "cancel"
    ]

You can see, side-by-side, the macOS and Windows generated forms. Notice the button text and ordering, yes, they differ from the source code! The GUI rules have ensured that:
  • the buttons are ordered according to each platform's guidelines, "Ok" last on macOS, "Cancel" last on Windows.
  • the button's labels are properly capitalized on macOS.

This small example just demonstrates the concept, but actually, there is no limit on how much it could alter the UI and how far it could go with the post-processing.

As it is still experimental, if it causes you any trouble, you can easily disable it using:
    system/view/VID/GUI-rules/active?: no
You can also remove rules selectively, by modifying the content of the following lists:
    system/view/VID/GUI-rules/OS/Windows
    == [
        color-backgrounds
        color-tabpanel-children
        OK-Cancel
    ]

    system/view/VID/GUI-rules/OS/macOS
    == [
        adjust-buttons
        capitalize
        Cancel-OK
    ]
This allows you total control where needed, but also helps you conform to UI guidelines with no effort, saving everyone time. Not only do you not have to tweak your code for each platform when you write it, when a new platform is added, you won't have to change your code to support it. In a world where getting one detail wrong may keep you out of an app store, or prevent you from being a "certified" app, this is huge. It should also be possible to use the rule engine to write a guideline linter, where you could get a report of what rules will be applied to your VID code on each platform. By "report", we don't mean just a text listing, but it could highlight elements in the UI that were processed by rules. Having a system that helps you is great. Having a system that tells you how it helped you is even better.

There are tons of possible rules we could implement, if you have ideas for great ones, please let us know by joining us in our online chat room on Gitter or, if you prefer, by posting on our mailing-list.


Other changes in 0.6.3

Core language

The biggest addition to the core language in this release is the date! datatype, which supports all Rebol's date! datatype features and more. The additions include support for a large range of ISO 8601 date formats. About 350 unit tests have been written so far. The full documentation is available here.

Here are a few examples of accepted input formats:
    >> 1999-10-5
    == 5-Oct-1999

    >> 5-October-1999
    == 5-Oct-1999

    >> 5/9/2012/6:00
    == 5-Sep-2012/6:00:00

    >> 4/Apr/2000/6:00+8:00
    == 4-Apr-2000/6:00:00+08:00

    >> 2017-07-07T08:22:23Z
    == 7-Jul-2017/8:22:23

    >> 2017-W23-5
    == 9-Jun-2017

    >> 2017-153T105000-4:00
    == 2-Jun-2017/10:50:00-04:00
Math and other operations are fully supported:
    >> d: now
    == 14-Jul-2017/16:04:07+08:00

    >> d + 10
    == 24-Jul-2017/16:04:07+08:00

    >> d + 100
    == 22-Oct-2017/16:04:07+08:00

    >> d - 24:00
    == 13-Jul-2017/16:04:07+08:00

    >> d - 01/01/2010
    == 2751

    >> d/month: d/month - 5
    >> d
    == 14-Feb-2017/16:04:07+08:00

    >> d/yearday: random 365
    >> d
    == 1-Aug-2017/16:04:07+08:00

    >> sort [1/2/2018 1999-5-3/10:30:28 29-2-1980]
    == [29-Feb-1980 3-May-1999/10:30:28 1-Feb-2018]

    >> random 01/01/2018
    == 15-Dec-1248
Red also adds three new accessors:
  • /timezone: changes the zone and adjusts time accordingly (UTC-invariant)
  • /week: gets/sets the week of the year, starting on Sunday.
  • /isoweek: gets/sets the ISO week of the year, starting on Monday.

Conversions to/from Unix epoch time are also built-in:
    >> to-integer 1/1/1970
    == 0

    >> to-integer now
    == 1500020743

    >> to-date 1000000000
    == 9-Sep-2001/1:46:40

Other Core Additions

New functions were added to the existing simple IO API:
  • delete: deletes a file or an empty folder.
  • size?: returns the size in bytes of a file.

The do native now accepts a URL argument. For example:
   do https://raw.githubusercontent.com/red/code/master/Showcase/eve-clock.red
Run-time error reports now include a compact call stack trace:
    foo: does [1 / 0]
    bar: does [foo]

    bar
    *** Math Error: attempt to divide by zero
    *** Where: /
    *** Stack: bar foo

A set of new functions for disk-cached remote file access is now available: do-thru, exists-thru?, load-thru, read-thru, path-thru. They work in the same way as their normal counterparts, except that the retrieved file is cached locally, so on their next access, the locally cached copy will be used.

The system object has been extended with:
  • system/options/cache: points to the cache folder used by the Red toolchain.
  • system/options/thru-cache: points to the cache folder used by *-thru functions.
  • system/catalog/accessors: Path accessors available, per datatype.

The browse native can now work on macOS (it opens the default browser on a given URL).

Rudolf Meijer did a huge job, gathering currently implemented rules for math operations on mixed datatypes. This has resulted in documenting and improving those rules, better defining the resulting datatype for all math operations.

More changes:
  • Now is now fully operational and can return current date and time.
  • Wait now accepts a time! value argument
  • Improved make and to url! construction when the spec argument is a block.
  • Auto-detect older Intel CPU on Windows platform when building the console.
  • Even? and odd? now support time! values as argument.
  • R/S compiler can now output a function address map for verbosity >= 3
  • Improved min and max natives support for pair! and tuple!(now min/max is applied per dimension to pair! and tuple!, and can work with a number! as second argument).
  • -t compilation option does not force release mode anymore if target is the same OS.
  • Source file name in now displayed on syntax errors at compile-time.

LibRed

LibRed has been improved, thanks to the work of Joa-quim on a libRed binding for the Julia language, providing Julia with the full power of Red's native GUI system. Several new functions were added to libRed API:
  • redBinary(): constructs a binary! value from an external byte array.
  • redImage(): constructs an image! value from various possible external byte arrays.
  • redGetField(): gets the value of an object's word.
  • redSetField(): sets the value of an object's word.
There is more info about them in the libRed documentation.

LibRed's robustness has also been vastly improved. LibRed API functions can now catch null pointer arguments, and even invalid pointers passed to the libRed API. In these situations, libRed returns a Red error! value.

View engine

In addition to the macOS backend, a new test backend is included in this release. This backend works by simulating a GUI backend for testing purposes. It is still experimental but works fine so far. We should be able to write a whole suite of unit tests for the View engine and VID dialect now and run them on a headless Linux server (like our Travis CI). In the same way, this backend enables Red users to unit test their GUI apps with minimal effort (though more tooling on top of the current backend is welcome).

The working principle is simple: use view/no-wait to produce your GUI without starting the event loop, then form your tests using the following model:
  • Set the focus on the face where you want to simulate an event (using set-focus).
  • Trigger an event (using do-event).
  • Test if the result of the event conforms to your expectations.

Here is an extract from a short test script:
    view/no-wait [
        hello: button "Hello" [print "ok"]
        name: field
    ]

    set-focus hello
    do-event 'click

    set-focus name
    do-event/with 'key #"4"
    do-event/with 'key #"2"
    do-event/with 'key 'enter

    probe name/text
    probe name/data
Using this test backend requires setting some options in the header (see the linked script's header), and compilation in release mode.

Another notable addition is the foreach-face function, which powers our GUI rewriting rules engine. It allows easy traversal of a face tree (depth-first), selecting only the faces you need to process. The syntax is:
    foreach-face <root> [<actions>]
    foreach-face/with <root> [<actions>][<conditions>]
    
    <root>       : root face of the tree.
    <actions>    : actions to perform on matched faces.
    <conditions> : optional conditions for selecting faces.
The first version applies the actions block to all faces, while the second one (/with) will select only faces matching the conditions block (which needs to return a true value to act on the current face). Both the actions and conditions blocks have an implicit face word defined, which refers to the current face in the tree. For example:
    do %tests/vid.red
    collect [foreach-face/with win [keep face/text][find face/text #"i"]]
will return:
    == ["China" "in panel" "option 1" "option 2" "option 3" "option 4"]
Here is a more sophisticated example using foreach-face to dynamically localize labels and adjust buttons size and position accordingly, so that they fit the text nicely:

    langs:  ["English" "French" "中文"]
    labels: [
        ["Name" "Age" "Phone #" "Cancel" "Submit"]
        ["Nom" "Age" "Tél." "Abandon" "Envoyer"]
        ["名字" "年龄" "电话" "取消" "提交"]
    ]
    set-lang: function [f event][
        root: f/parent
        condition: [all [face/text face/type <> 'drop-list]]

        list: collect [foreach-face/with root [keep face/text] condition]
        forall list [append clear list/1 labels/(f/selected)/(index? list)]

        foreach-face/with root [
            pads: any [metrics?/total face 'paddings 'x 0]
            prev: face/size/x
            face/size/x: pads + first size-text face
            face/offset/x: face/offset/x + ((prev - face/size/x) / 2)
        ][face/type = 'button]
    ]
    view [
        style txt: text right 45
        drop-list data langs select 1 on-change :set-lang return
        group-box [
            txt "Name"  field return
            txt "Age"   field return
            txt "Phone" field
        ] return
        pad 15x0 button "Cancel" button "Submit"
    ]

You can also detect window resizing events, and use the current window size to resize and position faces dynamically.

It is now also possible to change the mouse cursor shape, using pre-defined values, setting a custom image to face/options/cursor, or using the cursor keyword in VID, followed by an image! value or one of the following keywords: arrow, I-beam, hand, cross.

Here is an example:

Other changes:
  • Lit-words are now accepted in flags facet blocks.
  • New Shape dialect command: close.
  • Changed default background color to white in tab-panel (Windows).
  • Added system/view/metrics (used mostly by VID, for more accurate sizing/positioning).
  • Added accelerated? option in face, when set to true, for faster/smoother face rendering (only base faces). Z-ordering is then only honored between accelerated faces.
  • When a button has the focus, pressing enter key will trigger a click event.
  • Renamed enable? facet to the more correct enabled?.
  • New function: set-focus (sets the focus on a given face, important for GUI testing).
  • New class option in face/options, allows setting a sub-class of a native control.
  • text-list now accepts a data keyword and any-string! values as input. This lets you to feed the list with values from VID. For example:
    view [
        text-list data parse
            read https://api.github.com/repos/red/red/events
            [collect [any [thru "message" 3 skip keep to ["\n" | {"}]]]]
    ]

The above script is just a single expression using two DSLs. I let you meditate on that. ;-)


VID dialect

VID has received many enhancements too. The most significant are:
  • more accurate sizing and positioning of native widgets using OS metrics
  • addition of alignment control, extending across/below layout modes.

So, it is now possible to add optional alignment keywords for the two linear layout modes:
  • across: top, middle, bottom
  • below: left, center, right
Those keywords can also follow the return command, if changing the alignment mode of the next row/column is required. The defaults are top and left. Here is a short example:
    view [
        style chk: check "Option" data yes
        style vline: base 2x60 red
        across top    vline button "Ok" text "Text" field chk return
        across middle vline button "Ok" text "Text" field chk return
        across bottom vline button "Ok" text "Text" field chk
    ]

Other changes:
  • New on-created event, triggered just after a face has been shown for the first time.
  • New strike option for faces: sets the strikethrough font style.
  • New init [<body>] styling option allows you to define a block of code to run when the style is instantiated.
  • later option added to react keyword.
  • Allows data and at keyword argument to be an arbitrary Red expression (see the above text-list example).
  • More robust face options parsing.
  • The default actor for h1 to h5 and text widgets is now on-down.
  • VID now saves the style name for each face in face/options/style.

Parse dialect

A new case command has been added to Parse dialect. Case temporarily changes the case matching mode, for the next rule only.

Syntax:
  case <mode> <rule>

  <mode>: TRUE (case-sensitive matching) or FALSE (case-insensitive) (logic!, word!)
  <rule>: rule on which the local case matching mode is applied to.

Red/System dialect

For newcomers, Red/System is Red's built-in system programming dialect, offering a Red-like syntax with C-level semantics (and many improvements over C, like namespaces, exceptions, RTTI,...).

This release brings an important improvement: structures can now be passed and returned by value on internal and external function calls. They can also now be inlined in other structures. This is achieved by putting the value keyword after a struct! type declaration:
 name [struct! [<fields>] value]
Passing structs by value is sadly not formally specified in C language, resulting in incompatible compiler-specific ABIs. Moreover, those ABIs are not documented, or only partially, so gathering all the right information was painful and time consuming. Just to illustrate how bad the situation is, after spending days researching all the various C compilers ABI, I was able to answer a stackoverflow question about returning C structs by value, which was left unanswered for 6 years... I might write a synthetic article about those C ABIs someday, from my notes, as I could not find such documentation online. Anyway, the hard work has been done for you. Red/System now implements the most common ABI for each platform:
  • Windows: Microsoft Visual C ABI
  • Linux: gcc/Clang ABI (with support for ARM-specific ABI) 
  • macOS: Clang ABI

In addition, more stack-oriented features are now supported:

The new use feature allows you to create local variables anywhere in a function body. It is useful for splitting long functions and creating local variables in macros.

The ARM backend has also been enhanced significantly in this new release:
  • fixes and improvements to soft integer division routine.
  • fixes non-passing unit tests on floats.
  • better AAPCS conformity and various bug fixes.
  • improved accuracy of overflow detection in multiplication.

Console

ANSI escaping sequences are now supported in the Red CLI console, thanks to Oldes. See this pull request for more info and a screenshot.

Other console changes:
  • improved auto-completion, now completes the longest common substring.
  • auto-detect older Intel CPU (non-SSE3) on Windows platform when building console.
  • system/console/size now provides the size of the console in columns and rows

What's Next?

The 0.6.4 version will mostly focus on the GUI console, not only bringing it to macOS, but also extending it greatly. It should be merged into the master branch in the next few days, as the new GUI console is almost finished.

The 0.6.5 version is for Android, and is already being worked on in a separate, private repo. We will merge it with the red/red public repo once it's ready, because we need control on the PR timing and the Android release. It will be a big one, unlike what you've seen so far. ;-)

In the meantime, and as usual, enjoy and have fun!

March 26, 2017

0.6.2: LibRed and Macros

It is with great pleasure that we announce the 0.6.2 release of the Red programming language and toolchain. This release is the second heaviest one we have ever made (after the 0.6.0), weighing about 1200 commits! It was intended initially to be a minor one, but the needs for preprocessing support for Red runtime code arose, so it was the right time to make a first iteration into the land of macros. Moreover, the work on libRedRT (described below) gave us the opportunity to fulfill one of the goals we had for Red: become embeddable. This is now a reality, thanks to libRed. So the main features of this release are:

  • Macros and preprocessor support
  • Fast compilation using libRedRT
  • LibRed for embedding Red anywhere

Macros

The Red macros and preprocessing abilities have been the topic of a previous article, so I invite you to discover them (and hopefully enjoy!) if you have not yet done so.

In addition to this article, it is worth mentioning that a feature called pre-load has been implemented since then (you can read about it down below), which can be used as an entry point for user-provided reader macros, and allow easier support for non-standard syntax. This is a minor feature, as Rebol-like languages already come strongly equipped for string parsing and processing, thanks to the Parse dialect.


Development and release modes for compilation

The biggest change introduced by this new version is the splitting of the compilation process in two modes, which are controlled by different command-line options:

  • development: (-c) only user code is compiled (new mode).
  • release: (-r) both user code and Red runtime library are compiled together.

When using -c, the Red runtime library is pre-compiled to a dynamic library called libRedRT ("RT" for RunTime), stored in the hidden red/ folder. When libRedRT is already present, only user code is compiled resulting in drastically shorter compilation times.

When -r option is used, the toolchain compiles user code in release mode, compiling with it all dependencies (including Red runtime library).

Here are a couple of simple benchmarks to show the (huge) difference in performance, using hello.red and view-test.red scripts compiled in both modes:


Another benchmark is compiling the Red tests suite (~18000 tests), unified way combines the tests into the minimum number of compilation units, while split way compiles each test file separately:


Measured on a Corei7 4Ghz Win7/64-bit machine.

Such performance results in significant daily productivity gains, both for the Red team, and Red users. It was worth the time and effort it took to properly convert the runtime library into a shared library. Though, the full support for modular compilation will come in 0.8.0, which will result in drastic cuts for the compilation time in release mode too.

Such fast compilation mode also works for Red scripts that embed Red/System code. Two cases are possible:
  1. Red/System code does not contain any call to Red runtime library.
  2. Red/System code contains one or several calls to Red runtime library.
In the first case, nothing needs to be done, the usage rules described above apply.

In the second case, a custom version of libRedRT is required, but the toolchain will take care of the process, it just requires the user to compile once using -u option, then simply use -c for next compilations. If new functions from the runtime library are called, then a new custom library needs to be rebuilt.

When compiling a custom run-time library, cleaning libRedRT files is needed to get rid of any outdated versions. The toolchain provides a red clear command which will remove all current libRedRT related files. Note that when upgrading the red binary to a newer version, it will automatically upgrade libRedRT on first invocation.

Any Red user code can be compiled in development mode, with the exception of objects with multiple inheritance, which is not supported by libRedRT (so can only be compiled in release mode).


Embedding Red using libRed

LibRed is the embeddable version of the Red interpreter + runtime library. It includes Parse dialect, reactive programming and our GUI system (View engine, VID dialect, Draw dialect). It exposes an interface to the host language through an API, which is C-compatible, so that any language that can embed C shared libraries, can also embed libRed.

The libRed API has been designed to be as simple and as straightforward as possible. See for yourself in the API documentation. The role of that API is to provide the required hooks for interfacing the Red runtime and interpreter with the host language. A high-level binding can eventually be built on top of the libRed C-level API, that will best map Red features, to the host language features.

A libRed HelloWorld in C:
    #include "red.h"

    int main() {
        redOpen();
        redPrint(redString("Hello World!"));
        redClose();
        return 0;
    }
A libRed graphic HelloWorld in C:
    #include "red.h"

    int main() {
        redOpen();
        redDo("view [text {Hello World!}]");
        redClose();
        return 0;
    }

In addition to the C-level API, we provide also a binding for VisualBasic for Applications, which can be used to embed Red into Microsoft Office applications!

Here is a demo showing side-by-side an Excel/VB window and a Red window playing Pong game:


All the code for this demo fits in a single page of VB code! You can get the required files from here. In that same Excel file, you will find two other simple examples of integration of Red with Excel sheets, which look like this:



Building libRed is straightforward:
    red build libred
That will build libRed with the cdecl ABI, suitable for integration with any C-compatible host. For VBA and MS Office, the stdcall ABI is required, which is achieved using:
    red build libred stdcall
The building process will also result in creating a libRed/ directory locally, expanding some extra files required for linking libRed properly.

Currently, full bindings are provided for C and VisualBasic. Experimental bindings are also available for RustAdobe AIR and C#.

This is a first iteration for libRed, we have plans for improving it, and for providing proper multi-instances support (currently limited at one instance per process).


Changes in 0.6.2

Core language
  • New datatypes: tag!, email!
  • New action: to
  • New natives: as, call
  • New functions: tag?, email?, hex-to-rgb, sqrt, class-of, face?, distance?, expand-directives, to-*, rejoin
  • Adds integers auto-promotion to floats on loading and on some integer operations.

If you need to preprocess the input to LOAD, you can now do it easily by plugging a function into system/lexer/pre-load. This feature is mostly meant for pre-processing the console's input, though it could also be used for changing some Red syntactic rules. For example:
    system/lexer/pre-load: func [src part][replace/all src comma space]

    >> [1,2,3,abd,"hello"]
    == [1 2 3 abd "hello"]
Another usage could be to translate words on-the-fly in the console:
    system/lexer/pre-load: function [src part][
        parse src [
            any [
                s: [
                    "affiche"       (new: "print")
                    | "si"          (new: "if")
                    | "tant que"    (new: "while")
                    | "pair?"       (new: "even?")
                    | "impair?"     (new: "odd?")
                ] e: (s: change/part s new e) :s
                | skip
            ]
        ]
    ]

    >> i: 10 tant que [i > 0][si impair? i [affiche i] i: i - 1]
    9
    7
    5
    3
    1

In addition to several small fixes, load now offers a /trap refinement, which enables manual error management when loading a string series. Instead of raising an error, load/trap will just stop and return a block made of:
  • a block of successfully loaded values.
  • the input string at the position where the lexer failed.
  • an error! value (or none! value if the tail of the string has been reached).

Command-line argument processing has been mostly rewritten to provide a consistent experience across platforms and type of binaries (red executables, console executables, compiled user scripts and, to a lesser extent, Red/System executables). The new features are:
  • system/options/script refers to the script name (string! or none!).
  • system/options/args refers to a list of tokenized arguments (block! or none!).
  • system/script/args refers to the original command-line (string! or none!).
  • full Unicode support for red executable's arguments on Windows.
  • single-quoted arguments are accepted on all platforms (same as double quotes)
  • multiple nested quotes are treated as just one level of quoting when splitting the command-line.

Datatypes conversions are now fully supported in Red! The to action is now officially supported, and make action has been completed. The usual to-<type> helper functions you can find in Rebol, are also now defined. All the conversion rules are documented in an Excel matrix for now.

Call

Calling external applications is also now possible thanks to the contributed code for call function by Bruno Anselme, which has now been integrated into the runtime library. Use help call to explore all the options offered. Note that the /console option, which redirects I/O from child process, is not supported currently by the Red GUI console (it works fine from within the Red CLI console).

View and VID dialect

A number of small changes and fixes have been provided both in View and VID, among them:
  • box, h1 to h5 styles added to VID.
  • Colors in VID can now be specified as hex values, using #rgb or #rrggbb formats.
  • Adds support for no-border flag to area and field face types.
  • Adds now option to rate keyword in VID, to fire on-time actor at once.
  • The wheel event  and on-wheel actor are now available for handling mouse wheel events.
  • Default tab size for area changed to 4 spaces.
  • View now uses DirectWrite to draw text in base face (except WindowsXP).
  • Better handling of default fonts.
  • Enhanced GUI console, with new settings window with pre-selected colors picker.
  • A hint text can now be specified in a field options block and a hint keyword has been added to VID. For example:



Draw dialect

Big additions were made to Draw, most thanks to massive contributions from Zamlox:

  • Matrix operations support: matrix, invert-matrix, reset-matrix, matrix-order, push, rotate, scale, translate, skew, transform.
  • A new clip command is available for defining a clipped drawing/filling region.
  • A Shape sub-dialect has been added for more complex shapes drawing and filling.
  • A crop option is now available for image command.
  • pen and fill-pen have been vastly extended to allow drawing and filling with gradients, patterns, arbitrary shapes and images. Have a look at some of the new capabilities (source):



Parse dialect

In addition to some fixes, a few new features were added:
  • insert command now also support position argument (like change).
  • added pick option to keep, so user can control how keep captures the matched input:
    • keep collects matched values as a series if many, or as a value if only one.
    • keep pick collects all the matched values separately in a block.
    • keep copy <word> collects all the matched values as a single series (of same type as input).

Red/System dialect

  • Support for float! / float32! conversions from/to integer!.
  • Adds system/cpu/overflow? field for reading CPU's integer overflow state.
  • Adds support for importing variables from shared libraries.
  • Allows loading of libraries from current folder and PATH environment variable on macOS.
  • Now #call directive supports function calls with refinements.
  • Default ABI for exported functions is now settable through export-ABI entry in config object.
  • Renamed log and log10 imports from libC, respectively to log-2 and log-10.
  • Now size? accepts a context path argument.
  • Improved error reporting for "missing argument" errors.

Other changes

  • A prototype Red/.Net bridge has been introduced.
  • New --config [...] command-line option, for passing a block of compiler settings.
  • Added -s and --show-expanded command-line options to output expanded version of compiled source code.
  • Added /target option to react?.
  • Added /seek and /lines option to write.
  • Added /expand refinement to do for preprocessor invocation.
  • Added math function for evaluating code using math precedence rules.
  • CTRL-L key combination can now clear the GUI console's screen.
  • Checksum function can now trigger object on-deep-change event.
  • Now keep returns its argument (collect function).
  • Added temporary rejoin function.
  • Added 'class reflective property to objects.
  • Added class-of accessor (only objects).
  • Nicer handling of line breaks in molded image! binary buffer.
  • Now #include is converted to do in interpreted code (using macros).
  • Zero? function is now a native and supports time! values.

Also, more than 150 issues have been fixed (and wishes granted) during the last months, 22 issues marked as bugs are left open.

Last but not least, our documentation on Gitbook (which is an ongoing work) has been moved to Asciidoc format now, thanks to the efforts of Tovim. That new format will provide us better options for a more accurate control of the styles and layout.

A big thank goes to all contributors who pushed code or who opened tickets clearly identifying issues.

What's Next?

Since 0.6.1, we have adjusted the Red roadmap to work on two releases in parallel. This means that while 0.6.2 was progressing, 0.6.3 was advancing at the same time, is now almost ready, and will be merged into master in a couple of days (if you happen to have a Mac, it contains the macOS GUI backend for Red!). As soon as 0.6.3 is out, 0.7.0 will start (full async I/O), while 0.6.4 (Android) is being worked on. That should provide us a higher number of new releases this year, while still implementing large new features. So far, such approach has worked pretty well.

In the meantime, and as usual, enjoy playing with Red!

December 17, 2016

Incursion into "explorable explanations" lands

A few days ago, Nicky Case released an amazing piece of work in form of an interactive guide to alternative voting systems. This work follows the track of Brett Victor's famous research on explorable explanations and immediately caught our eyes because it is a very good match for the capabilities of our reactive native GUI system. We decided to replicate some of Nicky's interactive explanations in Red and see how it goes. The results are pretty amazing, as it was not only relatively easy to design and code (about 8 hours in total and ~280 LOC) but also great fun! Here is the result:


You can, of course, try it yourself by using the latest build from the master (or macGUI) branch or, preferably, one of the prebuilt consoles:
Put the console executable in the source code folder, run it and from the prompt, just type:
    red>> do %ballots.red
For Linux folks, sorry guys, the GUI support is not yet ready for prime time, though in the meantime, you could use Wine, Red GUI usually runs just fine on it.

Notes for macOS users:
  • You need to `chmod +x` the binary before running it from the terminal.
  • When dragging a face quickly, it can lag behind. The macOS GUI backend is still in a development branch, and still need some work to optimize animation latencies.

Implementation notes

The Red version implements only the "model1" and "ballot1" to "ballot4" interactive explanations. The rest could also be easily done in the same way, though we used up all our spare time for that as we are now focussing on the Red 0.6.2 release.

The colored shapes are images in the web version, but instead we chose to draw them using our 2D vector DSL in the Red version. Overall, the Red version is really small, about 25-45 LOC per screen, 85 LOC for the helper code and 2D shapes. All things considered, it is significantly smaller than the web version:
This is an interesting result, as the web version uses only plain JS, no third-party framework (with the exception of the almost weightless minipubsub). The counted JS code could maybe be shortened (excluding minification) though it already looks pretty simple to me, containing very few comments, doing mostly calculations and canvas drawing. I doubt it could be reduced significantly (any thoughts from expert JS readers about that?).

Final thoughts

This fun experiment shows that Red has the potential to be a great match for explorable explanations and other similar reactive interface needs. We hope that this demo will inspire the Red community to dig more in that direction and see what gems are lying there. For example, learning Red using such interactive visual approach would be great, don't you think so? ;-)

December 2, 2016

Entering the World of Macros

In Rebol languages family, macros have been a topic every now and then, and despite one implementation offered, they never became mainstream. Rebol language, being homoiconic, comes already well-equipped for code transformations at run-time with minimal effort. For example, this short script:
    code: [print "hello"]
    code: reduce [code]
    insert code [loop 2]
    probe code
    do code
when evaluated will output:
    [loop 2 [print "hello"]]
    hello
    hello
For a more sophisticated example, see our GUI live-coding in just a few lines.

Moreover, Rebol being interpreted, the cost of transformations is still paid at run-time.

But with Red, it is a different story. Being compiled, in addition to being interpreted, weighs heavily in favor of a preprocessing facility, in order to leverage compile-time code transformations. It is available now (in the master branch), in the form of a preprocessor with macro capabilities.

Let me restate it for the readers not familiar with macros and source preprocessing:

    The goal is to shift some data and code transformations from run-time to compile-time.

This is the key point of the new features in Red, described below.

Design and implementation

The preprocessing (including macro-time) happens between load and compile phases. It has been designed as a separate phase, with a separate execution context (to the extent allowed by current Red semantics, until we get the module! type implemented). Among other benefits, this enforces hygiene in Red macros. The preprocessor directives syntax relies on the familiar `#` prefixed forms (issue! values), meant to visually stand out from regular code, enforcing the idea of a separate layer of processing.

As Red strives to keep the source code accepted by the compiler and interpreter as interchangeable as possible, the preprocessor can also be run by the interpreter, between load and eval phases, supporting exactly the same features as the compiler version. Actually, they even share the same implementation, a unique file, with a double Rebol/Red header, run by Rebol2 for the toolchain and by Red for the interpreter. Though the preprocessor has to preprocess itself and a few core files used by the compiler setup sequence. In order to solve this chicken and egg problem, some directives have been hardcoded in the preprocessor instead of been implemented as macros.

The preprocessing (including macro expansion) is applied to the whole source fed to the compiler (or interpreter) after the load phase, so it is applied on a parse tree (similar to an AST). This means that no information about objects or other contexts built at run-time is available, it is just a big tree of Red values. Further research is planned in future releases to improve that, if possible. An alternative approach relying on interleaving macro expansion with compile/eval phases (late-time expansion) was considered, but not adopted, due to much higher implementation costs and potentially confusing inter-mixed behaviors, making it hard to grasp and be properly used by a majority of users. As we value simplicity utmostly, we will stick to the current simpler model for now, to ensure the broadest possible usage.

Simple macros

Documentation about preprocessing directives like #if, #either, #switch, #case, ... is available, and should feel familiar to Rebol SDK users, so this article will just focus on the real novel part: macro support.

Macros are basically functions, taking source code references as arguments in order to transform that source code. Most of the time, the returned value from the macro is replacing the macro name and argument at the call site in the source code. Though, this default mode can be overriden, and more control can be given to the macro through pattern-matching macros, explained later below.

For sake of brevity, the Red header is omitted from following code examples. You can copy/paste those code snippets in the console, wrapping them in a do expand [...] call or compile them (just add a Red [] header) using a `red -c -s <script.red>` line.

Simple macros can be defined in a similar way to regular functions, though they need to be preceeded by a #macro directive:
    #macro as-KB: func [n][n * 1024]
    print as-KB 64
When the macro is expanded by the preprocessor, the above source code will result in:
    print 65536
This kind of simple macro is called a named macro in Red's jargon. Once the preprocessor has finished his work, no macro definition or call, nor any preprocessor directive remains in the source code (compiler directives defined as issue! values, are still there though).

Remember that macros take source code values as arguments, without evaluating them, so passing unquoted words or paths is fine:
    #macro capitalize: function [value][
        s: uppercase/part form value 1
        either path? value [to-lit-path s][to-lit-word s]
    ]
    print capitalize hello
    print capitalize hello/macro/world
will result in:
    print 'Hello
    print 'Hello/macro/world
As you can see both func and function constructors are accepted for declaring a macro.

One possible use of macros can then be word-aliasing, for example translating words at compile-time (using French words here):
    #macro si:      func []['if]
    #macro affiche: func []['print]
    #macro vrai:    func []['true]

    si vrai [affiche "Vive Red !"]
will result in:
    if true [print "Vive Red !"]
Now, let's get back to the first example and expand (no pun intended) on it:
    #macro make-KB: func [n][n * 1024]
    #macro make-MB: func [n][make-KB make-KB n]
    print make-MB 1
will result in:
    print 1048576
So macros can freely call other macros, the same way functions would. This is possible because macros are running in a special (hidden) context, which can be accessed and extended (indirectly) by the user using the #do directive:
    #do [kilo: 1024]
    #macro make-KB: func [n][n * kilo]
The #do directive accepts arbitrary code, so local words and functions can be defined there freely, and accessed from macros. This gives a pretty powerful programming layer for the preprocessor, though, it is possible to go even further.

Pattern-matching macros

For Red users, the logical next step once having macros (as defined above), is to be able to combine them with one of the Rebol world's jewels: the Parse DSL. Well that is what pattern-matching macros are. ;-) Instead of matching a word as trigger for the macro, you can replace it with a valid Parse rule (which can call sub-rules defined in #do expressions), defining a pattern that will trigger the macro.

We can now improve our `make-KB` macro with a nicer looking form:
    #macro [number! 'KB] func [s e][to-integer s/1 * 1024]
    print 64 KB
    print 2.5 KB
would result in:
    print 65536
    print 2560
In that example, the macro gets called each time a number! followed by the word! KB is encountered in the source code. They will then both get replaced by the macro returned value. The arguments of pattern-matching macros are always the same: one reference to the starting position (s) of the matched pattern, and one reference to the ending position (e).

Here are a few more examples, showing the true power of Red macros:

Variable-arguments macro
    #macro ['max some [integer!]] func [s e][
        first maximum-of copy/part next s e
    ]
    print max 4 2 3 8 1
would result in:
    print 8

Loop macro

A simple loop macro, extending the existing loop function:
    #macro ['loop [integer! | block!] block!] function [[manual] s e][
        set [spec body] next s
        if integer? spec [return e]    ;-- return position past the pattern
        low: high: none
        step: 1

        unless parse spec [
            word! 
            opt '= set low integer!
            opt [opt 'to set high integer!]
            opt [opt 'step set step integer!]
        ][
            print ["*** LOOP syntax error:" spec]
        ]
        new: reduce either high [
            set-var: to-set-word var: spec/1
            cond: compose [(var) <= (high)]
            repend body [set-var var '+ step]
            [set-var low 'while cond body]
        ][
            ['repeat spec/1 low body]
        ]      
        change/part s new e
    ]

    loop 2 [print "x"]
    loop [i 3][print i]
    loop [i 5 8][print i]
    loop [i = 5 to 10 step 2][print i]
would result in:
    loop 2 [print "x"]
    repeat i 3 [print i]
    i: 5 while [i <= 8] [print i i: i + 1]
    i: 5 while [i <= 10] [print i i: i + 2]
This pattern-matching macro relies on the manual mode ([manual] attribute), where the replacement is done by user code and the returned value needs to be the position in the source code where the expansion process resumes. In this case, loop is also an existing function, so when the argument is an integer, the source is left untouched. With a normal macro, the resuming point would have been the beginning of the pattern, resulting in an infinite loop (again, no pun intended). ;-)


Math expressions folding macro

A powerful way to pre-calculate constant math expressions from your source code could be to use a macro like this one:
    #do [
        p:     none
        op:    ['+ | '- | '* | '** | slash]
        paren: [p: paren! :p into expr]
        fun:   [['sine | 'cosine | 'square-root] expr]
        expr:  [[number! | paren] op expr | number! | paren | fun]
    ]

    #macro [expr] func [[manual] s e][
        if all [e = next s number? s/1][return e]  ;-- skip single number
        change/part s do (copy/part s e) e
        s
    ]

    a: 3 + 2 - 8
    print (3 + 4) * 6
    edge: 100 * cosine 60
    hypotenuse: square-root (3 ** 2) + (4 ** 2)
would result in:
    a: -3
    print 42
    edge: 50.0
    hypotenuse: 5.0


HTML validating macro

HTML tags are a first-class datatype in Red so they can be embedded and manipulated by the core language (like string values). The Red lexer will check the syntax but wouldn't it be nice to also have a minimal check the semantics at compile-time? Let's roll a macro for that:
    #do [
        error: function [pos [block! paren!] msg [block!]][
            print [
                "*** HTML error:" reduce msg lf
                "*** at:" copy/part pos 4
            ]
            halt
        ]
    ]

    #macro tag! function [[manual] s e][
        stack: []
        tag: s/1

        either slash = tag/1 [
            last?: (name: next tag) = last stack
            all [
                not last?
                find stack name
                error s ["overlapping tag" tag]
            ]
            if any [empty? stack not last?][
                error s ["no opening tag for" tag]
            ]
            take/last stack
        ][
            if slash <> last tag [    
                if pos: find tag " " [tag: copy/part tag pos]

                unless find s head insert copy tag slash [
                    error s ["no closing tag for" tag]
                ]            
                append stack tag
            ]
        ]
        next s
    ]

    data: [<html><br/><b>msg<b></html>]
    msg: "Red rocks!"
    print data
The embedded HTML above has an error, the <b> tag is not closed. This macro will catch that error and report it properly during the compilation.


A DSL compiler

Red is already very well-suited for DSL creation, though, the cost of interpreting or compiling DSL has always been paid at run-time so far. With macros, it can be moved to compile-time, when suitable. Here is a simple subset of BASIC language, partially compiled to Red code using a single macro:
    #macro do-basic: function [src /local math][
        output: clear []
        lines:  clear []
        value!: [integer! | string! | word!]
        op:     ['+ | '- | '* | slash]
        comp:   ['= | '<> | '< | '> | '<= | '>=]
        line:   [p: integer! (repend lines [p/1 index? tail output])]
        cmd:    [
            token: 'let word! '= value! opt [copy math [op value!]] (
                emit reduce [to-set-word token/2 token/4]
                if math [emit/part math 2]
            )
            | 'if value! comp value! 'then (
                emit/part token 4
                emit/only make block! 1
                parent: output
                output: last parent
            ) cmd (output: parent)
            
            | 'print value! (emit/part token 2)
            | 'goto integer! (
                line: select/skip lines token/2 2
                emit compose [jump: (line)]
            )
        ]
        emit: function [value /only /part n [integer!]][
            if part [value: copy/part value n]            
            either only [append/only output value][append output value]
        ]
        
        unless parse src [some [line cmd]][
            print ["*** BASIC Syntax error at:" mold token]
        ]
        compose/deep [
            (to-set-word 'eval-basic) function [pc [block!]][
                bind pc 'pc
                while [not tail? pc][
                    do/next pc 'pc
                    if jump [
                        pc: at head pc jump
                        jump: none
                    ]
                ]
            ]
            eval-basic [(output)]
        ]
    ]

    do-basic [
        10 LET A = "hi!"
        20 LET N = 3
        30 PRINT A
        40 LET N = N - 1
        50 IF N > 0 THEN GOTO 30
    ]
will result in:
    eval-basic: function [pc [block!]][
        bind pc 'pc
        while [not tail? pc][
            do/next pc 'pc
            if jump [
                pc: at head pc jump
                jump: none
            ]
        ]
    ]
    eval-basic [
        A: "hi!"
        N: 3
        PRINT A
        N: N - 1
        IF N > 0 [jump: 5]
    ]
Note: I tried to keep this example short, so it ends up as a BASIC source code compiler to Red, but fed to a custom interpreter. Fully compiling that DSL to Red code would be possible, but would require more complex constructs, in order to deal with a GOTO able to arbitrarily jump anywhere, and that is beyond the scope of this article.

Final thoughts

The preprocessor and macros bring great new possibilities to the Red compiler, while still being able to run the same code with the interpreter. Though, as the saying goes, with great power comes great responsibility, so keeping in mind some drawbacks would be wise:
  • As there is no difference in Red between code and data, both can be transformed by the same macros, which is not always desirable. Some mechanisms for limiting the application scope exist in the preprocessor, though there is no guarantee they can cover all use-cases. Stay alert, especially with pattern-matching macros.
  • It is not always easy to reason about and debug macros, unless you have some existing experience. I would suggest not using them until you have a good grasp of Red's toolchain, semantics and fundamental concepts (like homoiconicity).
  • As the Red toolchain is currently run by a Rebol2 interpreter, Rebol is running the compile-time macros, so keep that in mind when writing them. If you want them to run equally well on the interpreter, you need to use only the common subset between Rebol2 and Red. Sooner or later we should move compile-time preprocessing to use a Red engine (thanks to libRed), so this concern is temporary.

Last but not least, for those wondering about syntactic macros (aka readers macros) inclusion in Red, as for AST-macros, they are not strictly necessary, as the Parse DSL already provides us a powerful tool to implement pre-load-time processing. Though they could still bring some extra benefits (like embedding the processing logic within the source code), but could also go against the Red-as-data-format principle, or wreak havoc in IDE support (like messing up syntax coloring and step by step debugging). We need more time to go through each concern and see how to deal with them before adding such a feature.

I hope this long article was useful for those of you who had no past experience with macros and entertaining for those who have. Have fun with this brand new toy and let us know what you think about it on Gitter. Cheers!

See comments on /r/programming.
Fork me on GitHub