December 25, 2017

Leaping into the future: Red goes blockchain!

Here we are, ready to leap into the future. This is a new page opening in the Red history book. It is a great day for us to finally be able to share with everyone some great news for Red and its community. Fasten your seat belts and enjoy the ride! ;-)



Why the blockchain?

Unless you just woke up from prolonged hibernation under Artic ice crust, you cannot miss the hype around blockchains and their related coins. If you are new to cryptocurrencies and blockchains, let me give you the simple bird view on that: speculation on cryptocurrencies is currently very high, but whether it is a bubble or not, bursting soon or never, is not really relevant. The important part, shadowed by the coins market hype, is that the blockchain technologies are a new infrastructure that should help solve many issues we have. It is a missing piece that we need in order to rebalance the modern Internet landscape, and bring back the decentralization that was once promised in the early Internet days.

Though, the impact of such disruptive change goes way beyond just Internet infrastructure (for example, IPFS or Storj), it has the potential to reshape many industries (see the Brave browser and its Basic Attention Token, bringing a saner alternative to online advertisement) and human activities, including in our daily lives. We are currently still just seeing the first generation of blockchains. Those pioneers have inefficiencies, like scalability and power consumption issues, though, after following and studying the evolution of blockchains during a year, I am now confident enough that those issues will be solved in a not-too-distant future by the next wave of blockchains (or upgraded versions of existing ones). For example, zero-knowledge proof and especially the non-interactive variant are really impressive and promising approaches.

So, how does Red fit there?

As Red aims to be a fullstack programming solution, we naturally look at what are the potential next big platforms we should support. Since last year, we are watching and studying the various blockchains, especially focusing on the ones supporting smart contracts, Ethereum being the leading chain in that domain. Basically, it is like a distributed computer, where you can run arbitrary sandboxed code, using the distributed ledger as its database. There are a few programming languages which can be used to implement those smart contracts, but Solidity is the default one now for such tasks. If you are a Red follower, you know how much we care about simplicity and efficiency in our programming tools. Well, those concerns do not seem to be shared by people who created the existing tools for smart contracts coding and testing. Let's put it bluntly, the current state is a nightmare. It takes several hours, up to a full day to just figure out and set up a development environment the first time, even for a senior developer. The sheer complexity in such toolchains is really high. The Remix initiative is still young, but it is a lifesaver compared to other existing toolchains. Still, you have to put up with a language that has many questionable design choices, for coding contracts which could handle billions of dollars worth of assets. Guess what happens under such conditions.

So, there is a huge need for a better, much simpler and safer contract development tool. We know that Red can deliver such solution, because that's our fight and our mission since the first release of Rebol 20 years ago. Still, people who work with me know how skeptical I am. So it took me several months of studying and discussions to figure out where and how Red could help in that domain.

Why now?

We are at an early point of a new revolution.We do know that much better development tools are needed, and the current offer is poor and disappointing to say the least. We know that we can provide a great solution based on Red, in an acceptable timeframe. This is an amazing opportunity, like we haven't seen since late 90's and the rise of the world wide web. This is our time now.

So this is what we will build, in a new branch of the Red project:

Red/C3

A new Red dialect (DSL) whose code-name is Red/CCC (Cross Chain Code) or just Red/C3, for smart contracts programming. Such dialect will compile to the Ethereum VM (EVM) bytecode directly as first target, and more backends will be added later to support other chains, like NEO. We are at the design stage, the first alpha (prototype) is planned for end of Q1 2018. Our current design involves a combination of two small languages:
  • A very high-level layer: a symbolic and highly declarative language with restricted computing semantics and restricted control flow (turing-incomplete), reading mostly like natural language (compared to mainstream languages). Coarse-grained code produced at this level by the compiler will exhibit higher safety, close to the level provided by templates, while still providing much greater flexibility. An intelligent visual program builder will also be considered for pre-generating such code.
  • A lower-level layer: a general-purpose, statically typed, subset of Red language.

Moreover, for testing such contracts, we plan to create:
  • a simulator for the EVM (and later for other chains VM) coded in Red/System for fast performances.
  • a generative testing environment, for auto-testing user contracts extensively in addition to manually written tests using a simple testing framework we will provide too.
  • a GUI interface to handle all that in an easy way, without requiring deep knowledge of the blockchain infrastructure.
The toolchain will be an extension of the current Red toolchain, so it will be a standalone binary of 1-2MB.

Red Dapps

Smart contracts are used to power what is called Decentralized Apps (Dapps)-- Basically a user-facing application operating on blockchains through smart contracts. Currently such Dapps are built using web stacks, but still need to connect to a blockchain node in order to interact with it. Light nodes could help solve that, but they break the decentralization model. So, in order to facilitate the deployment of the required blockchain nodes, such apps are often wrapped in a standalone native installer using an Electron framework. This results in big downloads (50-150MB) and big memory usages inherent to web stacks.

The Red stack, with its ~1MB runtime (~300KB if compressed) can provide a much better alternative, and Red GUI capabilities can lower drastically the entrance barrier and costs for coding apps and Dapps. In order to achieve that, we need to add a blockchain node connector in our runtime library, preferably wrapping a node in C library form, though using JSON-RPC is also an option.

For old Rebolers, maybe you remember Rebol/IOS in early 2000's, for handling the X-Internet? Well, guess what? Carl was right all along (as usual). ;-) You can look at the Red Dapps as a modern blockchain-powered incarnation of that concept!

What about the existing Red roadmap?

The Red core language development continues with the same roadmap, we are just delaying a bit the 0.6.4 and 0.6.5 releases, in order to pass the big wave of changes in the incoming weeks. In order to achieve that, while opening a new branch of development, we need to grow bigger and structure the whole thing differently. Here are the plans to move the whole project to the next stage:

Red Foundation

We were thinking about that since last year, but now is the right time to do it for many reasons. We need a dedicated organization to handle the community and contributors' efforts for the work on Red core language. We also have plans for business activities, and it would be easier to separate the teams working on the open-source project and the ones working on commercial products and services. It is not an unseen set up, the Clojure project structured itself in a similar way.

So, we have filed an application to set up a Red Foundation in France, and will transfer to it the rights and governance for the open source parts of Red. The managers of that structure will be: Gregg Irwin, Peter W A Wood, François Jouen and myself, for starting. The foundation has a complex job and will require resources to operate, hence:

Red community 2.0: RCT token

We are embracing the blockchain philosophy fully and will empower the Red users and community using a decentralized organization (DAO), supported by a new ERC20 token called Red Community Token (RCT). There is no other programming language or tool that I have heard of, attempting to move their existing community to a DAO, so we will be exploring uncharted territories there, and setting an example for others to follow. The foundation will be setting up the rules with the help of the community, for the usages of the RCT token. Some of the target usages for the token holders, will include:
  • voting rights: influence the Red roadmap, vote for features and issue tickets.
  • tipping: useful chat posts, code contributions, learning materials providers, etc.
  • intra-community cryptoeconomics (or rather tokenomics): selling/buying services from other community members (coding tasks, consulting, learning help, bug fixing, decentralized gaming, etc.)
  • paid Dapps, or in-app purchases.

In addition to that, the foundation will hold a significant amount of RCT, which will be used for rewarding:
  • code contributions
  • Red-related online learning or presenting materials (blog, documentations, etc.)
  • promotional actions (presenting Red at a conference)
  • any other actions that will help spread Red and make the community grow up. 

There will be an initial distribution of RCT tokens to the existing Red contributors (once the foundation defines the rules and allocation amounts), that will retro-actively cover existing contributions (like merged PR, tickets resulting in code changes, past donations, etc.).

Last but not least, the RCT might be used in the future as a gas token, to pay for cross-chain different gas currencies, or to power a future Red's own blockchain.

Those are just scratching the surface of the possibilities that such token and DAO (powered by smart contracts) could open. This gives you an idea of the disruptive potential of blockchains, when rightly used. It is not yet clear how far we can go into the decentralized model, it is something we will explore all together (we'll set up dedicated chat rooms on Gitter and other sites for that).

This will be our greatest challenge, but I trust we have enough brain power in the Red foundation and community to figure out the best rules for a healthy organization.

Initial coin offering

In order to kickstart the RCT token, and fund all the resources needed to accomplish the above tasks and roadmap, we need to raise some money. Initial Coin Offering (ICO) are the new way to achieve that. I have been reluctant to do that for a while, given how young and controversial it can be (the amount of scams in that domain is really high), but after studying many other ICO, I am convinced now that this is not just a short-lived fad, but a real new model to fund projects, and especially open-source projects related to blockchains.

We have produced a whitepaper earlier this year in October (co-authored by Tamas Herman and myself), to describe with more details and context what we are building. We will release it publicly in a few days.

The ICO itself should be held in early January 2018, starting with an angel round, then pre-sale and open rounds following right after. A new website will be available in a few days, giving the full information required for the ICO.

The ICO should provide us with enough funds to cover all the needs for this ambitious roadmap. A job list will be available after it, as we hope to hire the best profiles from the Red community, so update your CV and clean-up the source code of your Red apps. ;-)

Final words

These are intense and exciting days for us and the Red community, I wish you all a Merry Christmas, and have a good beer, maybe someday you'll pay it on your smartphone with a Red Dapp powered by RCT tokens. ;-)

Cheers!

July 17, 2017

0.6.3: macOS GUI backend

This new Red release contains ~830 commits, closes 86 issues, and brings a large number of  new features. The main one is official support for the macOS GUI, through a new backend for the View engine.


For readers not familiar with our GUI system, you can get a quick overview in a previous article.

In a nutshell, it is a dynamically-built reactive object tree, with:
  • pluggable backends (full support for Windows and prototype for GTK and Android)
  • full 2D vector graphics support (a large subset of SVG)
  • direct reactive programming support (not FRP style for now)
  • a declarative DSL for GUI building and layout management
  • a declarative DSL for 2D graphics

The macOS backend supports all the same features as the Windows backend, except the following, yet to be implemented:
  • fill-pen pattern, fill-pen bitmap and diamond gradient in Draw.
  • matrix transformations on pen and fill-pen in Draw.

The GUI console for macOS will be available in the 0.6.4 branch, so is not part of this release. The View module, though, is compiled in the CLI console by default on macOS, so no show-stopper there.

Here are a few short examples, starting with a GUI Hello World!:
    view [text "Hello Mac World!"]

A tiny demo script (hopefully, it should get us an office at Station F, right? ;-)):
    view [
        t: text "French Touch! " on-time [move t/text tail t/text]
        base 21x15 draw [
            pen off
            fill-pen blue  box 0x0  6x14
            fill-pen white box 7x0  14x14
            fill-pen red   box 14x0 20x14
        ] return
        button "Start" [t/rate: 10]
    ]

A small reactive text size measuring tool (one of our test scripts):
    view [
        style txt: text right
        txt "Text" f: area 200x80 
            font [name: "Comic Sans MS" size: 15 color: black]
            return

        txt "Size in pixels" text "0x0"
            react [[f/text f/font] face/text: form size-text f]
            return
 
        txt "Font name" drop-list 120
            data  ["Arial" "Consolas" "Comic Sans MS" "Times"]
            react [f/font/name: pick face/data any [face/selected 3]]
            return
  
        txt "Font size" s: field "15" react [f/font/size: s/data]
        button "+" bold 40 [s/data: s/data + 1]
        button "-" bold 40 [s/data: max 1 s/data - 1]
        return
    ]


Generating macOS bundle apps is made simple by our toolchain, just use the -t macOS target when compiling, for example using our SVG Tiger demo:
   $ red -o ~/Desktop/ -t macOS tiger.red
This will produce a tiger.app bundle on the Desktop:



Cross-Platform GUI Metrics

In order to cope with different UI guidelines across GUI platforms, this release also introduces an experimental rule-oriented GUI rewriting engine, capable of modifying a face tree dynamically according to pre-set rules. It is integrated as a last stage in VID (Visual Interface Dialect) processing. The currently implemented rules are per-platform for now (more general, cross-platform rules are also planned):

Windows rules:
  • color-backgrounds: color the background of some colorless faces to match their parent's color
  • color-tabpanel-children: Like color-backgrounds, but tab-panel specific
  • OK-Cancel: buttons ordering rule, puts Cancel/Delete/Remove buttons last
macOS rules:
  • adjust-buttons: use standard button sub-classes when buttons are narrow enough
  • capitalize: capitalize widget text according to macOS guidelines
  • Cancel-OK: buttons ordering rule, puts Ok/Save/Apply buttons last

To give you a quick grasp about why that feature matters and what it is capable of, here is a simple example, which leverages the buttons ordering and capitalization rules:
    view [
        text "Name" right 50 field return
        text "Age"  right 50 field return
        button "ok" button "cancel"
    ]

You can see, side-by-side, the macOS and Windows generated forms. Notice the button text and ordering, yes, they differ from the source code! The GUI rules have ensured that:
  • the buttons are ordered according to each platform's guidelines, "Ok" last on macOS, "Cancel" last on Windows.
  • the button's labels are properly capitalized on macOS.

This small example just demonstrates the concept, but actually, there is no limit on how much it could alter the UI and how far it could go with the post-processing.

As it is still experimental, if it causes you any trouble, you can easily disable it using:
    system/view/VID/GUI-rules/active?: no
You can also remove rules selectively, by modifying the content of the following lists:
    system/view/VID/GUI-rules/OS/Windows
    == [
        color-backgrounds
        color-tabpanel-children
        OK-Cancel
    ]

    system/view/VID/GUI-rules/OS/macOS
    == [
        adjust-buttons
        capitalize
        Cancel-OK
    ]
This allows you total control where needed, but also helps you conform to UI guidelines with no effort, saving everyone time. Not only do you not have to tweak your code for each platform when you write it, when a new platform is added, you won't have to change your code to support it. In a world where getting one detail wrong may keep you out of an app store, or prevent you from being a "certified" app, this is huge. It should also be possible to use the rule engine to write a guideline linter, where you could get a report of what rules will be applied to your VID code on each platform. By "report", we don't mean just a text listing, but it could highlight elements in the UI that were processed by rules. Having a system that helps you is great. Having a system that tells you how it helped you is even better.

There are tons of possible rules we could implement, if you have ideas for great ones, please let us know by joining us in our online chat room on Gitter or, if you prefer, by posting on our mailing-list.


Other changes in 0.6.3

Core language

The biggest addition to the core language in this release is the date! datatype, which supports all Rebol's date! datatype features and more. The additions include support for a large range of ISO 8601 date formats. About 350 unit tests have been written so far. The full documentation is available here.

Here are a few examples of accepted input formats:
    >> 1999-10-5
    == 5-Oct-1999

    >> 5-October-1999
    == 5-Oct-1999

    >> 5/9/2012/6:00
    == 5-Sep-2012/6:00:00

    >> 4/Apr/2000/6:00+8:00
    == 4-Apr-2000/6:00:00+08:00

    >> 2017-07-07T08:22:23Z
    == 7-Jul-2017/8:22:23

    >> 2017-W23-5
    == 9-Jun-2017

    >> 2017-153T105000-4:00
    == 2-Jun-2017/10:50:00-04:00
Math and other operations are fully supported:
    >> d: now
    == 14-Jul-2017/16:04:07+08:00

    >> d + 10
    == 24-Jul-2017/16:04:07+08:00

    >> d + 100
    == 22-Oct-2017/16:04:07+08:00

    >> d - 24:00
    == 13-Jul-2017/16:04:07+08:00

    >> d - 01/01/2010
    == 2751

    >> d/month: d/month - 5
    >> d
    == 14-Feb-2017/16:04:07+08:00

    >> d/yearday: random 365
    >> d
    == 1-Aug-2017/16:04:07+08:00

    >> sort [1/2/2018 1999-5-3/10:30:28 29-2-1980]
    == [29-Feb-1980 3-May-1999/10:30:28 1-Feb-2018]

    >> random 01/01/2018
    == 15-Dec-1248
Red also adds three new accessors:
  • /timezone: changes the zone and adjusts time accordingly (UTC-invariant)
  • /week: gets/sets the week of the year, starting on Sunday.
  • /isoweek: gets/sets the ISO week of the year, starting on Monday.

Conversions to/from Unix epoch time are also built-in:
    >> to-integer 1/1/1970
    == 0

    >> to-integer now
    == 1500020743

    >> to-date 1000000000
    == 9-Sep-2001/1:46:40

Other Core Additions

New functions were added to the existing simple IO API:
  • delete: deletes a file or an empty folder.
  • size?: returns the size in bytes of a file.

The do native now accepts a URL argument. For example:
   do https://raw.githubusercontent.com/red/code/master/Showcase/eve-clock.red
Run-time error reports now include a compact call stack trace:
    foo: does [1 / 0]
    bar: does [foo]

    bar
    *** Math Error: attempt to divide by zero
    *** Where: /
    *** Stack: bar foo

A set of new functions for disk-cached remote file access is now available: do-thru, exists-thru?, load-thru, read-thru, path-thru. They work in the same way as their normal counterparts, except that the retrieved file is cached locally, so on their next access, the locally cached copy will be used.

The system object has been extended with:
  • system/options/cache: points to the cache folder used by the Red toolchain.
  • system/options/thru-cache: points to the cache folder used by *-thru functions.
  • system/catalog/accessors: Path accessors available, per datatype.

The browse native can now work on macOS (it opens the default browser on a given URL).

Rudolf Meijer did a huge job, gathering currently implemented rules for math operations on mixed datatypes. This has resulted in documenting and improving those rules, better defining the resulting datatype for all math operations.

More changes:
  • Now is now fully operational and can return current date and time.
  • Wait now accepts a time! value argument
  • Improved make and to url! construction when the spec argument is a block.
  • Auto-detect older Intel CPU on Windows platform when building the console.
  • Even? and odd? now support time! values as argument.
  • R/S compiler can now output a function address map for verbosity >= 3
  • Improved min and max natives support for pair! and tuple!(now min/max is applied per dimension to pair! and tuple!, and can work with a number! as second argument).
  • -t compilation option does not force release mode anymore if target is the same OS.
  • Source file name in now displayed on syntax errors at compile-time.

LibRed

LibRed has been improved, thanks to the work of Joa-quim on a libRed binding for the Julia language, providing Julia with the full power of Red's native GUI system. Several new functions were added to libRed API:
  • redBinary(): constructs a binary! value from an external byte array.
  • redImage(): constructs an image! value from various possible external byte arrays.
  • redGetField(): gets the value of an object's word.
  • redSetField(): sets the value of an object's word.
There is more info about them in the libRed documentation.

LibRed's robustness has also been vastly improved. LibRed API functions can now catch null pointer arguments, and even invalid pointers passed to the libRed API. In these situations, libRed returns a Red error! value.

View engine

In addition to the macOS backend, a new test backend is included in this release. This backend works by simulating a GUI backend for testing purposes. It is still experimental but works fine so far. We should be able to write a whole suite of unit tests for the View engine and VID dialect now and run them on a headless Linux server (like our Travis CI). In the same way, this backend enables Red users to unit test their GUI apps with minimal effort (though more tooling on top of the current backend is welcome).

The working principle is simple: use view/no-wait to produce your GUI without starting the event loop, then form your tests using the following model:
  • Set the focus on the face where you want to simulate an event (using set-focus).
  • Trigger an event (using do-event).
  • Test if the result of the event conforms to your expectations.

Here is an extract from a short test script:
    view/no-wait [
        hello: button "Hello" [print "ok"]
        name: field
    ]

    set-focus hello
    do-event 'click

    set-focus name
    do-event/with 'key #"4"
    do-event/with 'key #"2"
    do-event/with 'key 'enter

    probe name/text
    probe name/data
Using this test backend requires setting some options in the header (see the linked script's header), and compilation in release mode.

Another notable addition is the foreach-face function, which powers our GUI rewriting rules engine. It allows easy traversal of a face tree (depth-first), selecting only the faces you need to process. The syntax is:
    foreach-face <root> [<actions>]
    foreach-face/with <root> [<actions>][<conditions>]
    
    <root>       : root face of the tree.
    <actions>    : actions to perform on matched faces.
    <conditions> : optional conditions for selecting faces.
The first version applies the actions block to all faces, while the second one (/with) will select only faces matching the conditions block (which needs to return a true value to act on the current face). Both the actions and conditions blocks have an implicit face word defined, which refers to the current face in the tree. For example:
    do %tests/vid.red
    collect [foreach-face/with win [keep face/text][find face/text #"i"]]
will return:
    == ["China" "in panel" "option 1" "option 2" "option 3" "option 4"]
Here is a more sophisticated example using foreach-face to dynamically localize labels and adjust buttons size and position accordingly, so that they fit the text nicely:

    langs:  ["English" "French" "中文"]
    labels: [
        ["Name" "Age" "Phone #" "Cancel" "Submit"]
        ["Nom" "Age" "Tél." "Abandon" "Envoyer"]
        ["名字" "年龄" "电话" "取消" "提交"]
    ]
    set-lang: function [f event][
        root: f/parent
        condition: [all [face/text face/type <> 'drop-list]]

        list: collect [foreach-face/with root [keep face/text] condition]
        forall list [append clear list/1 labels/(f/selected)/(index? list)]

        foreach-face/with root [
            pads: any [metrics?/total face 'paddings 'x 0]
            prev: face/size/x
            face/size/x: pads + first size-text face
            face/offset/x: face/offset/x + ((prev - face/size/x) / 2)
        ][face/type = 'button]
    ]
    view [
        style txt: text right 45
        drop-list data langs select 1 on-change :set-lang return
        group-box [
            txt "Name"  field return
            txt "Age"   field return
            txt "Phone" field
        ] return
        pad 15x0 button "Cancel" button "Submit"
    ]

You can also detect window resizing events, and use the current window size to resize and position faces dynamically.

It is now also possible to change the mouse cursor shape, using pre-defined values, setting a custom image to face/options/cursor, or using the cursor keyword in VID, followed by an image! value or one of the following keywords: arrow, I-beam, hand, cross.

Here is an example:

Other changes:
  • Lit-words are now accepted in flags facet blocks.
  • New Shape dialect command: close.
  • Changed default background color to white in tab-panel (Windows).
  • Added system/view/metrics (used mostly by VID, for more accurate sizing/positioning).
  • Added accelerated? option in face, when set to true, for faster/smoother face rendering (only base faces). Z-ordering is then only honored between accelerated faces.
  • When a button has the focus, pressing enter key will trigger a click event.
  • Renamed enable? facet to the more correct enabled?.
  • New function: set-focus (sets the focus on a given face, important for GUI testing).
  • New class option in face/options, allows setting a sub-class of a native control.
  • text-list now accepts a data keyword and any-string! values as input. This lets you to feed the list with values from VID. For example:
    view [
        text-list data parse
            read https://api.github.com/repos/red/red/events
            [collect [any [thru "message" 3 skip keep to ["\n" | {"}]]]]
    ]

The above script is just a single expression using two DSLs. I let you meditate on that. ;-)


VID dialect

VID has received many enhancements too. The most significant are:
  • more accurate sizing and positioning of native widgets using OS metrics
  • addition of alignment control, extending across/below layout modes.

So, it is now possible to add optional alignment keywords for the two linear layout modes:
  • across: top, middle, bottom
  • below: left, center, right
Those keywords can also follow the return command, if changing the alignment mode of the next row/column is required. The defaults are top and left. Here is a short example:
    view [
        style chk: check "Option" data yes
        style vline: base 2x60 red
        across top    vline button "Ok" text "Text" field chk return
        across middle vline button "Ok" text "Text" field chk return
        across bottom vline button "Ok" text "Text" field chk
    ]

Other changes:
  • New on-created event, triggered just after a face has been shown for the first time.
  • New strike option for faces: sets the strikethrough font style.
  • New init [<body>] styling option allows you to define a block of code to run when the style is instantiated.
  • later option added to react keyword.
  • Allows data and at keyword argument to be an arbitrary Red expression (see the above text-list example).
  • More robust face options parsing.
  • The default actor for h1 to h5 and text widgets is now on-down.
  • VID now saves the style name for each face in face/options/style.

Parse dialect

A new case command has been added to Parse dialect. Case temporarily changes the case matching mode, for the next rule only.

Syntax:
  case <mode> <rule>

  <mode>: TRUE (case-sensitive matching) or FALSE (case-insensitive) (logic!, word!)
  <rule>: rule on which the local case matching mode is applied to.

Red/System dialect

For newcomers, Red/System is Red's built-in system programming dialect, offering a Red-like syntax with C-level semantics (and many improvements over C, like namespaces, exceptions, RTTI,...).

This release brings an important improvement: structures can now be passed and returned by value on internal and external function calls. They can also now be inlined in other structures. This is achieved by putting the value keyword after a struct! type declaration:
 name [struct! [<fields>] value]
Passing structs by value is sadly not formally specified in C language, resulting in incompatible compiler-specific ABIs. Moreover, those ABIs are not documented, or only partially, so gathering all the right information was painful and time consuming. Just to illustrate how bad the situation is, after spending days researching all the various C compilers ABI, I was able to answer a stackoverflow question about returning C structs by value, which was left unanswered for 6 years... I might write a synthetic article about those C ABIs someday, from my notes, as I could not find such documentation online. Anyway, the hard work has been done for you. Red/System now implements the most common ABI for each platform:
  • Windows: Microsoft Visual C ABI
  • Linux: gcc/Clang ABI (with support for ARM-specific ABI) 
  • macOS: Clang ABI

In addition, more stack-oriented features are now supported:

The new use feature allows you to create local variables anywhere in a function body. It is useful for splitting long functions and creating local variables in macros.

The ARM backend has also been enhanced significantly in this new release:
  • fixes and improvements to soft integer division routine.
  • fixes non-passing unit tests on floats.
  • better AAPCS conformity and various bug fixes.
  • improved accuracy of overflow detection in multiplication.

Console

ANSI escaping sequences are now supported in the Red CLI console, thanks to Oldes. See this pull request for more info and a screenshot.

Other console changes:
  • improved auto-completion, now completes the longest common substring.
  • auto-detect older Intel CPU (non-SSE3) on Windows platform when building console.
  • system/console/size now provides the size of the console in columns and rows

What's Next?

The 0.6.4 version will mostly focus on the GUI console, not only bringing it to macOS, but also extending it greatly. It should be merged into the master branch in the next few days, as the new GUI console is almost finished.

The 0.6.5 version is for Android, and is already being worked on in a separate, private repo. We will merge it with the red/red public repo once it's ready, because we need control on the PR timing and the Android release. It will be a big one, unlike what you've seen so far. ;-)

In the meantime, and as usual, enjoy and have fun!

March 26, 2017

0.6.2: LibRed and Macros

It is with great pleasure that we announce the 0.6.2 release of the Red programming language and toolchain. This release is the second heaviest one we have ever made (after the 0.6.0), weighing about 1200 commits! It was intended initially to be a minor one, but the needs for preprocessing support for Red runtime code arose, so it was the right time to make a first iteration into the land of macros. Moreover, the work on libRedRT (described below) gave us the opportunity to fulfill one of the goals we had for Red: become embeddable. This is now a reality, thanks to libRed. So the main features of this release are:

  • Macros and preprocessor support
  • Fast compilation using libRedRT
  • LibRed for embedding Red anywhere

Macros

The Red macros and preprocessing abilities have been the topic of a previous article, so I invite you to discover them (and hopefully enjoy!) if you have not yet done so.

In addition to this article, it is worth mentioning that a feature called pre-load has been implemented since then (you can read about it down below), which can be used as an entry point for user-provided reader macros, and allow easier support for non-standard syntax. This is a minor feature, as Rebol-like languages already come strongly equipped for string parsing and processing, thanks to the Parse dialect.


Development and release modes for compilation

The biggest change introduced by this new version is the splitting of the compilation process in two modes, which are controlled by different command-line options:

  • development: (-c) only user code is compiled (new mode).
  • release: (-r) both user code and Red runtime library are compiled together.

When using -c, the Red runtime library is pre-compiled to a dynamic library called libRedRT ("RT" for RunTime), stored in the hidden red/ folder. When libRedRT is already present, only user code is compiled resulting in drastically shorter compilation times.

When -r option is used, the toolchain compiles user code in release mode, compiling with it all dependencies (including Red runtime library).

Here are a couple of simple benchmarks to show the (huge) difference in performance, using hello.red and view-test.red scripts compiled in both modes:


Another benchmark is compiling the Red tests suite (~18000 tests), unified way combines the tests into the minimum number of compilation units, while split way compiles each test file separately:


Measured on a Corei7 4Ghz Win7/64-bit machine.

Such performance results in significant daily productivity gains, both for the Red team, and Red users. It was worth the time and effort it took to properly convert the runtime library into a shared library. Though, the full support for modular compilation will come in 0.8.0, which will result in drastic cuts for the compilation time in release mode too.

Such fast compilation mode also works for Red scripts that embed Red/System code. Two cases are possible:
  1. Red/System code does not contain any call to Red runtime library.
  2. Red/System code contains one or several calls to Red runtime library.
In the first case, nothing needs to be done, the usage rules described above apply.

In the second case, a custom version of libRedRT is required, but the toolchain will take care of the process, it just requires the user to compile once using -u option, then simply use -c for next compilations. If new functions from the runtime library are called, then a new custom library needs to be rebuilt.

When compiling a custom run-time library, cleaning libRedRT files is needed to get rid of any outdated versions. The toolchain provides a red clear command which will remove all current libRedRT related files. Note that when upgrading the red binary to a newer version, it will automatically upgrade libRedRT on first invocation.

Any Red user code can be compiled in development mode, with the exception of objects with multiple inheritance, which is not supported by libRedRT (so can only be compiled in release mode).


Embedding Red using libRed

LibRed is the embeddable version of the Red interpreter + runtime library. It includes Parse dialect, reactive programming and our GUI system (View engine, VID dialect, Draw dialect). It exposes an interface to the host language through an API, which is C-compatible, so that any language that can embed C shared libraries, can also embed libRed.

The libRed API has been designed to be as simple and as straightforward as possible. See for yourself in the API documentation. The role of that API is to provide the required hooks for interfacing the Red runtime and interpreter with the host language. A high-level binding can eventually be built on top of the libRed C-level API, that will best map Red features, to the host language features.

A libRed HelloWorld in C:
    #include "red.h"

    int main() {
        redOpen();
        redPrint(redString("Hello World!"));
        redClose();
        return 0;
    }
A libRed graphic HelloWorld in C:
    #include "red.h"

    int main() {
        redOpen();
        redDo("view [text {Hello World!}]");
        redClose();
        return 0;
    }

In addition to the C-level API, we provide also a binding for VisualBasic for Applications, which can be used to embed Red into Microsoft Office applications!

Here is a demo showing side-by-side an Excel/VB window and a Red window playing Pong game:


All the code for this demo fits in a single page of VB code! You can get the required files from here. In that same Excel file, you will find two other simple examples of integration of Red with Excel sheets, which look like this:



Building libRed is straightforward:
    red build libred
That will build libRed with the cdecl ABI, suitable for integration with any C-compatible host. For VBA and MS Office, the stdcall ABI is required, which is achieved using:
    red build libred stdcall
The building process will also result in creating a libRed/ directory locally, expanding some extra files required for linking libRed properly.

Currently, full bindings are provided for C and VisualBasic. Experimental bindings are also available for RustAdobe AIR and C#.

This is a first iteration for libRed, we have plans for improving it, and for providing proper multi-instances support (currently limited at one instance per process).


Changes in 0.6.2

Core language
  • New datatypes: tag!, email!
  • New action: to
  • New natives: as, call
  • New functions: tag?, email?, hex-to-rgb, sqrt, class-of, face?, distance?, expand-directives, to-*, rejoin
  • Adds integers auto-promotion to floats on loading and on some integer operations.

If you need to preprocess the input to LOAD, you can now do it easily by plugging a function into system/lexer/pre-load. This feature is mostly meant for pre-processing the console's input, though it could also be used for changing some Red syntactic rules. For example:
    system/lexer/pre-load: func [src part][replace/all src comma space]

    >> [1,2,3,abd,"hello"]
    == [1 2 3 abd "hello"]
Another usage could be to translate words on-the-fly in the console:
    system/lexer/pre-load: function [src part][
        parse src [
            any [
                s: [
                    "affiche"       (new: "print")
                    | "si"          (new: "if")
                    | "tant que"    (new: "while")
                    | "pair?"       (new: "even?")
                    | "impair?"     (new: "odd?")
                ] e: (s: change/part s new e) :s
                | skip
            ]
        ]
    ]

    >> i: 10 tant que [i > 0][si impair? i [affiche i] i: i - 1]
    9
    7
    5
    3
    1

In addition to several small fixes, load now offers a /trap refinement, which enables manual error management when loading a string series. Instead of raising an error, load/trap will just stop and return a block made of:
  • a block of successfully loaded values.
  • the input string at the position where the lexer failed.
  • an error! value (or none! value if the tail of the string has been reached).

Command-line argument processing has been mostly rewritten to provide a consistent experience across platforms and type of binaries (red executables, console executables, compiled user scripts and, to a lesser extent, Red/System executables). The new features are:
  • system/options/script refers to the script name (string! or none!).
  • system/options/args refers to a list of tokenized arguments (block! or none!).
  • system/script/args refers to the original command-line (string! or none!).
  • full Unicode support for red executable's arguments on Windows.
  • single-quoted arguments are accepted on all platforms (same as double quotes)
  • multiple nested quotes are treated as just one level of quoting when splitting the command-line.

Datatypes conversions are now fully supported in Red! The to action is now officially supported, and make action has been completed. The usual to-<type> helper functions you can find in Rebol, are also now defined. All the conversion rules are documented in an Excel matrix for now.

Call

Calling external applications is also now possible thanks to the contributed code for call function by Bruno Anselme, which has now been integrated into the runtime library. Use help call to explore all the options offered. Note that the /console option, which redirects I/O from child process, is not supported currently by the Red GUI console (it works fine from within the Red CLI console).

View and VID dialect

A number of small changes and fixes have been provided both in View and VID, among them:
  • box, h1 to h5 styles added to VID.
  • Colors in VID can now be specified as hex values, using #rgb or #rrggbb formats.
  • Adds support for no-border flag to area and field face types.
  • Adds now option to rate keyword in VID, to fire on-time actor at once.
  • The wheel event  and on-wheel actor are now available for handling mouse wheel events.
  • Default tab size for area changed to 4 spaces.
  • View now uses DirectWrite to draw text in base face (except WindowsXP).
  • Better handling of default fonts.
  • Enhanced GUI console, with new settings window with pre-selected colors picker.
  • A hint text can now be specified in a field options block and a hint keyword has been added to VID. For example:



Draw dialect

Big additions were made to Draw, most thanks to massive contributions from Zamlox:

  • Matrix operations support: matrix, invert-matrix, reset-matrix, matrix-order, push, rotate, scale, translate, skew, transform.
  • A new clip command is available for defining a clipped drawing/filling region.
  • A Shape sub-dialect has been added for more complex shapes drawing and filling.
  • A crop option is now available for image command.
  • pen and fill-pen have been vastly extended to allow drawing and filling with gradients, patterns, arbitrary shapes and images. Have a look at some of the new capabilities (source):



Parse dialect

In addition to some fixes, a few new features were added:
  • insert command now also support position argument (like change).
  • added pick option to keep, so user can control how keep captures the matched input:
    • keep collects matched values as a series if many, or as a value if only one.
    • keep pick collects all the matched values separately in a block.
    • keep copy <word> collects all the matched values as a single series (of same type as input).

Red/System dialect

  • Support for float! / float32! conversions from/to integer!.
  • Adds system/cpu/overflow? field for reading CPU's integer overflow state.
  • Adds support for importing variables from shared libraries.
  • Allows loading of libraries from current folder and PATH environment variable on macOS.
  • Now #call directive supports function calls with refinements.
  • Default ABI for exported functions is now settable through export-ABI entry in config object.
  • Renamed log and log10 imports from libC, respectively to log-2 and log-10.
  • Now size? accepts a context path argument.
  • Improved error reporting for "missing argument" errors.

Other changes

  • A prototype Red/.Net bridge has been introduced.
  • New --config [...] command-line option, for passing a block of compiler settings.
  • Added -s and --show-expanded command-line options to output expanded version of compiled source code.
  • Added /target option to react?.
  • Added /seek and /lines option to write.
  • Added /expand refinement to do for preprocessor invocation.
  • Added math function for evaluating code using math precedence rules.
  • CTRL-L key combination can now clear the GUI console's screen.
  • Checksum function can now trigger object on-deep-change event.
  • Now keep returns its argument (collect function).
  • Added temporary rejoin function.
  • Added 'class reflective property to objects.
  • Added class-of accessor (only objects).
  • Nicer handling of line breaks in molded image! binary buffer.
  • Now #include is converted to do in interpreted code (using macros).
  • Zero? function is now a native and supports time! values.

Also, more than 150 issues have been fixed (and wishes granted) during the last months, 22 issues marked as bugs are left open.

Last but not least, our documentation on Gitbook (which is an ongoing work) has been moved to Asciidoc format now, thanks to the efforts of Tovim. That new format will provide us better options for a more accurate control of the styles and layout.

A big thank goes to all contributors who pushed code or who opened tickets clearly identifying issues.

What's Next?

Since 0.6.1, we have adjusted the Red roadmap to work on two releases in parallel. This means that while 0.6.2 was progressing, 0.6.3 was advancing at the same time, is now almost ready, and will be merged into master in a couple of days (if you happen to have a Mac, it contains the macOS GUI backend for Red!). As soon as 0.6.3 is out, 0.7.0 will start (full async I/O), while 0.6.4 (Android) is being worked on. That should provide us a higher number of new releases this year, while still implementing large new features. So far, such approach has worked pretty well.

In the meantime, and as usual, enjoy playing with Red!
Fork me on GitHub