0.3.2: REPL release

Time has finally come to release the new REPL for Red. A preview version has been pushed last Christmas, since then a lot of work has been done on several fronts (about 310 new commits, excluding merges): interpreter (DO), runtime lexer (LOAD) and console. But before going into details for the REPL, here is an overview of the changes in this new release:

• New datatypes: routine!, issue!, file!
• New natives: do, load, reduce, compose, switch, case, stats, platform?, quit
• New natives: remove
• New mezzanines functions: ??, empty?
• 38 issues from previous releases fixed.
• Unit tests for Red raised to 5602 for a total of 17671 tests (including Red/System ones).
• Notable improvements:
  • Paren! expressions in paths now supported.
  • Mold output for string! and char! deeply improved, special characters are now correctly escaped.
  • Improved and more accurate support for unset! values processing.
  • Prin and print now reduce their arguments.

Red REPL

This is the biggest part of this new release. The REPL has several components:

• The interpreter: it is a full Red interpreter supporting all Red language features, except Red/System code. In the current version, though, exit and return are not yet implemented, they need some special low-level support from Red/System, so couldn't make it for this release. The interpreter can be invoked from compiled code using the do native. It has been developped in Red/System and is about 700 LOC long. All Red compiler tests are run against the interpreter too, all are passing except for the unimplemented yet exit and return (6 tests).

• The runtime lexer: it is the runtime counterpart to the compiler's lexer and, is in charge of loading input string source code into Red and converting it into blocks of values. It can be invoked from compiled code using the load native. The runtime lexer current version only support Latin-1 encoding, a full Unicode lexer will replace it soon (it is a work in progress).

• The console: it is the visible part of the REPL for most users. The current version is minimal but works on most of supported platforms (including the RaspberryPi). It has limited editing abilities, and history doesn't work on Mac OS X, but it supports a Rebol-like multi-line input mode for blocks and strings. We will provide a much better console in the next release, with a cross-platform abstraction layer that feature-wise, will put all platforms on par.

The interpreter and runtime lexer and now part of Red's standard library, so they are bundled with every compiled Red script. The overhead is about 30KB in the final binary, making it almost unnoticeable. The console is a separate script, that can be compiled easily producing a small standalone binary.

An important feature of the Red interpreter is that it is not only meant for the REPL support, but is actually used by the compiler to solve some code patterns that are too dynamic to be statically analyzed and compiled. Moreover, the interpreter has the ability to call pre-compiled code, so as soon as possible, it can drop back to native code execution. Both compiled and interpreted code are deeply collaborating to provide the most flexible language semantics while ensuring the maximum performances. With the future addition of a JIT-compiler, we will be able to reach the optimal internal architecture.

Red Collaborative Execution Model

On the more practical side, to compile the console, from Red root folder, just do:

do/args %red.r "red/tests/console.red"

That will give you a console binary in the working folder. When you run it, you should see:

-=== Red Console alpha version ===-
(only Latin-1 input supported)  

red>> 

This is the Red prompt where you can input any valid Red expression, they will be evaluated on Enter key pressed. Use q or quit to exit the console. Remember that there is no yet proper error management, current error messages are hardcoded, and in some cases, you will be thrown out of the console (that is why it is still an alpha version as stated by the console banner). Note that it is possible to copy/paste Red scripts directly into the console.

Anyway, feel free to experiment.

Routines

In order to more easily interface Red and Red/System, a new function datatype has been added: routine!. A routine is Red/System function defined in a Red program. The routine specification takes Red datatypes as arguments and return value, and the routine will automatically convert them to appropriate Red/System types when called. For example:

increment: routine [
    n       [integer!]
    return: [integer!]
][
    n + 1
]

Here you can see how the Red integer! argument get marshalled forth and back to Red/System integer! datatype for you. For now, routines automatically converts integer! and logic! datatypes this way. Other Red datatypes are passed as their Red/System counterparts as defined in Red's runtime (see %red/runtime/datatypes/structures.reds). Optional arguments are not supported by routines, as such feature does not exist in Red/System for now.

You can now very easily extend Red with routines using the full power of Red/System! Compiled routines can be run from interpreter too.

Project management

We are now moving to Trello for tracking the tasks about Red development. The short-term ToDo list is pretty accurate and updated regularly, you can now follow us more closely. We only started using it a month ago, so not all the tasks (especially the mid-term/long-term ones) are inserted there yet.

Thanks for all the support received for getting this major release out!
Cheers!

Fast forwarding Red history

At the beginning of this new year, I took a moment to look at the work we did since the first public commit of Red. Even if I am a perfectionist that is never satisfied with its own work, I must admit we did a quite incredible job so far. In two years we built not one but two programming languages: Red/System which is in beta state and usable to make real world apps, and Red which is in alpha state and evolving at very fast pace. We built two compilers and one interpreter. We replaced the whole gcc toolchain by our own simple one with incredibly small footprint. We run on ARM platforms since a year now and some experimental Red/System code even runs on Arduino boards. If someone told me two years ago that we will accomplish all that, I am not sure I would have believe it.

So in order to celebrate what we have achieved so far, I offer you an animation showing the history of all 1554 public commits of the project (excluding merge commits) in less than 2 minutes:

This video was made using the gource tool and encoded using ffmpeg. This is my first video uploaded to youtube, so I had to lower the original video quality in order for youtube to process it correctly. You can view it in 720p for best quality.

You can see me and Peter WA Wood flying around the project files making changes and addition. Sometimes Andreas Bolka is popping up to give us a hand, usually on pretty complex bug cases or to update  his ELF format emitter.

I would like to take this opportunity to thank all the people that sent me donations since the beginning, that strongly believe in and support Red project. I would like to especially thank the biggest donators:

Nick Antonaccio

Jerry Tsai

Gregg Irwin

Bas de Lange

Petr Krenzelok

I would have never believed that such level of support from a small community of followers was possible. I am amazed and feel extremely grateful to these people and all the other ones that sent me donations or contributed in a way or another to Red. I would also like to mention and thank two direct contributors to Red project: Kaj de Vos, for his involvement in supporting Red and all the bindings he produced for Red/System, Kaj is still the most prolific Red/System coder so far, and Rudolf Meijer for reverse-engineering Red/System and writing the official language specification document.

To all Red followers, I wish you a Happy New Year! Among many other new features, this year will see the extension of Red to mobile world, with full support for Android and iOS. Let's make it the year where Red will rise even higher than we could have thought possible!

Cheers!

Red v0.3.1: functions support added

This new milestone brings (finally) functions to Red along with other new features and fixes. Let's first have a quick overview of 0.3.1 release content:

• function support
• path get/set notation support
• refinement support for native functions
• expressions in parentheses compilation
• new datatypes: function!, paren!, path!, lit-path!, get-path!, set-path!
• new actions and natives: find, select, copy, reflect, type?, halt,...
• extended mold, form and comparison operators to all new and existing datatypes
• many new mezzanines functions
• modulo and remainder operators on floats implemented for ARM backend
• Quick-Test testing framework ported to Red
• a truckload of new unit tests
• many bugfixes
• about 200 new commits to the Github repository
• updated Red/System formal description document

Functions

They are declared using a similar syntax to REBOL, with some extensions. The specification block looks like this:

    [arg1 [type1] arg2 [type2]...return: [type] /local var1 [type-v1]...]

• Arguments and local variables are declared in the same way as in REBOL
• A return value type can be optionally specified
• Local words can have a type specifier

All type specifiers are optional, adding them will not only allow the compiler to make additional type checkings but also generate faster code in some cases (once the optimizations added to the Red compiler).

Note: argument and return value type checking have not been implemented yet, they need typeset! and error! datatypes to be implemented first.

Functions can be built using several constructors:

• function: automatically collects local variables, like `funct` in REBOL.

    twice: function [a [integer!] /one return: [integer!]][
    c: 2
      a: a * c
    either one [a + 1][a]
    ]

• func: low-level function constructor, everything needs to be manually specified.

     twice: func [a [integer!] /one return: [integer!] /local c][
      c: 2
      a: a * c
      either one [a + 1][a]
    ]

• has: no argument, just lists local words.

    globals: [1 2 3 _]
    
    foobar: has [list][
      if list: find globals '_ [clear list]
    ]

• does: no argument, no local words.

    quit: does [
      print "Goodbye cruel world!"
      halt
    ]

Early exit points are also there in form of exit and return functions.

    foobar: func [a [integer!] b [integer! none!]][
      if none? b [return none]
      if b = 0 [
         print "Error: division by zero"
         exit
      ]
  a / b
    ]
 
Refinements are also fully supported, when not used, their value is set to false (while in REBOL it is set to none).

Still some features are not yet implemented:

• building a function at runtime (requires the ability to JIT-compile source code)
• passing a function as argument (pending)
• tail recursion optimization

Currently functions are using a local context on stack, so their life-time is very short. For indefinite extent support, a closure! type will be added in the next months.

Path support

Path datatypes were added (path!, lit-path!, get-path!, set-path!) to the runtime library and compiler, allowing the use of path notation as syntactic sugar for series access. Getting and setting values using path notation are supported.

Some usage examples:

    list: [a 5 b [c 8] d #"e" name: "John"]
    
    probe list/2 ;-- outputs 5
    probe list/7 ;-- outputs name:
    probe list/a ;-- outputs 5
    probe list/4/2 ;-- outputs 8
    probe list/b/c ;-- outputs 8
    probe list/name ;-- outputs "John"
    
    list/b/c: 0
    probe list/b/c ;-- outputs 0
    
    index: 3
    probe list/:index ;-- outputs b
    list/:index: 'z
    probe list/:index ;-- outputs z
    
Note: notice how words lookups in blocks can work with any word datatypes.

Paren expressions

Paren! datatype and expressions in parentheses compilation support has been added too. They are mainly useful with infix operators in order to force a given execution priority. Example:

    print (1 + 2) * (3 + 4) ;-- outputs 21

New built-in functions

• actions: find, select, copy, reflect

• natives: type?, halt, function, func, does, has, exit, return

• mezzanines: probe, first, second, third, fourth, fifth, last, spec-of, body-of and all <datatype>? type testing functions.

Existing native or action functions were extended with refinements (like mold and form). See the boot.red script for more details.

Testing

The Quick-Test framework from Peter WA Wood was ported to Red and now powers all the current 1800+ unit tests written in Red itself, mostly added by Peter.

About 32 issues/bugs in both Red and Red/System were fixed in this new release.

Red/System formal description

Last but not least, Rudolf updated his Red/System formal description document [PDF, 162KB] to include the additional features like namespaces. Many thanks to him for his remarkable work and support to Red project!

What's next?

Objects support is next priority along with new natives like switch and case. Typesets and error handling will probably be added very soon too. Once that part done, the next focus will be to add I/O support!


Many thanks to all the people that have made significant donations in last months to Red project, allowing me to continue to work full time on it! It is really making a big difference, as shown by Red increased progress speed since this summer. Thanks again!

Cheers!

Red alpha release

We have all waited long for this to happen, but it is finally there: first Red alpha release!

This is the biggest milestone since the initial release of Red System, as it marks the entrance of the higher layer of the Red software stack. There's still another layer pending, the JIT compiler, but that's for 2013, when Red will be self-hosted.

What Red can do so far?

So what happened since the first Red compiler release a little more than a month ago? We have added quite a lot of features actually:

• 15 datatypes: block!, string!, char!, integer!, logic!, none!, word!, lit-word!, get-word!, set-word!, refinement!, action!, native!, op!, unset!.

• 19 actions: make, form, mold, add, divide, multiply, subtract, append, at, back, clear, head, index?, length?, next, pick, poke, skip, tail.

• 21 natives: if, unless, either, any, all, while, until, loop, repeat, foreach, forall, print, prin, equal?, not-equal?, strict-equal?, lesser?, greater?, lesser-or-equal?, greater-or-equal?, not.

• 10 infix operators: +, - , *, /, =, ==, <, >, <=, >=.

String! datatype fully supports Unicode from end to end and in a cross-platform way, according to our plan (see the hello.red demo script).

We also have some unit tests (223 tests right now), but that number should rapidly increase once we will be able to fully port QuickTest framework to Red (in a couple of weeks, once Red will gain functions).

All those features were mostly coded in the last month, some parts are still lacking (e.g. refinements support for actions/natives) but that gives you an idea of the pace at which Red will be further developed.

However, we haven't started yet writting the documentation for Red language, we should decide soon on the kind of documentation we want (hint: you can make propositions about that on the Red mailing-list or FB page). Red API documentation will most probably be generated automatically from source code by extracting doc-strings.

What's not yet there?

A lot of things, but they will be added progressively until the end of the year, where we should have a pretty complete Red (bootstrap) implementation. So, the main pieces to implement are:

• Red runtime lexer (LOAD native)
• Functions and objects support
• I/O support (including networking)
• More complete memory allocator and a garbage collector.
• Concurrency support
• Additional datatypes

Can I try it already?

Red codebase is available at github, just clone it locally or download a packaged version. Some simple instructions to setup and run it can be found here. As we are still at the bootstrap stage, the installation process and usage is more complex than it should be, once self-hosted. Anyway, we should be able to deliver some test binaries soon and simplify the usage for those who want to play with the alpha releases.

How does Red work?

Currently Red scripts are statically compiled using the %red.r front-end wrapper. The Red lexer loads the UTF-8 encoded Red scripts in memory in form of blocks of values. Those blocks are then compiled to a Red/System script, that gets itself compiled to native code and outputs an executable binary. The Red/System compiler has been enhanced to be able to compile source code directly from memory.

Some compilation options are already available, the famous -t option from Red System is also present, allowing Red to cross-compile any script as easily as with Red/System.

This is just the beginning, so for now, whole Red runtime library is compiled with user scripts each time. We will add modular compilation support at some point to get the full compilation speed back.

You can test the few example Red scripts from the repository. Here is what the demo.red script outputs on some of the platforms Red can already run on:

The missing characters in some of the screenshots are Chinese ones as I only have font support for them on Ubuntu.

Red was originally conceived to be statically typed. However, during the early stages of Red implementation, it appeared that an hybrid typing system will be more adequate for a language that will support high-level scripting, without the burden of a sophisticated, but slow, type inference engine. The overhead of such inference engine can be very high (cf Scala slow compilation speed ). The hybrid type system works very simply, typing is optional, but if local variables and function returned values are typed by user, the compiler will be able to generate faster code (more than an order of magnitude faster for, e.g., pure math expressions) and catch more programming errors at compile time.

Red compiler does not do any optimization so far, it basically outputs the equivalent of an unrolled interpreter loop. Some optimizations will be added along the development of the bootstrap version of Red,  but the full extent of possible optimizations will be left for the self-hosted Red.  Firstly because the current compiler code is disposable, so we don't want to invest too much time in it, secondly because the final JIT compiler will allow even deeper optimizations thanks to reflection and profiling information collected at runtime.

Red/System changes

Red/System has been improved significantly during the Red early development:

• function pointer support has been extended and improved, dereferencing is now possible.
• 20 issues have been fixed.
• some long-standing bugs with GTK+ and other bindings on ARM platforms have been fixed.

So, what's coming next?

Red will now mature very quickly, as functions and objects will make their entrance in the next days/weeks. A new Red alpha release is planned very soon, probably for mid-november with a lot of new features.

The work on shared library generation for MacOS and Linux platforms is on-going. Once available, we'll be able to literally "plug" Red in a lot of third-party apps, including other programming languages. Also, this will enable (or make easier) the bridging with some external systems like Java, .Net and objective-c, in order to fully support the main mobile platforms.

As you can see, next months will see some important parts of Red come to life, be sure not to miss them, follow us on our mailing-list, on Twitter, IRC and Facebook.

Cheers!

Plan for Unicode support

Red is growing up fast, even if just born two weeks ago! It is time we implement basic string support so we can do our first, real, hello-word. ;-)

Red strings will natively support Unicode. In order to achieve that in an efficient and cross-platform way, we need a good plan. Here is the list of Unicode native formats used by our main target platforms API:

        Windows       : UTF-16
        Linux         : UTF-8
        MacOSX/Cocoa  : UTF-16
        MacOSX/Darwin : UTF-8
        Java          : UTF-16
        .Net          : UTF-16
        Javascript    : UTF-8
        Syllable      : UTF-8
   
All these formats are variable-width encodings, requiring any indexed access to pay the cost of walking through the string.

Fortunately, there are also fixed-width Unicode encodings that can be used to give us back constant time for indexed accesses. So, in order to make it the most space-efficient, Red strings will internally support only these encoding formats:

        Latin-1 (1 byte/codepoint)
        UCS-2   (2 bytes/codepoint)
        UCS-4   (4 bytes/codepoint)

This is not something new, at least Python 3.3 does it in the same way.

Additionally, UTF-8 and UTF-16 codecs will be supported, in order to deal with I/O accesses on host platforms.

Red will use UTF-8 for exchanging strings with outer world by default, except when accessing a UTF-16 API is necessary. Conversion for input and output strings will be done on-the-fly between one of the internal representation and UTF-8/UTF-16. When reading an input string, Red will select the most space-efficient internal format depending on highest codepoint in the input string. Also users should be able to force the encoding of a string to a given internal format, when possible.

So far, this is the plan for additing Unicode to Red, a prototype implementation will be done quickly, so we can fine-tune it if required.

Comments and suggestions are welcome.

Red is born

Yesterday, I finally got my first real Red program compiling and running:

    Red []
    print 1

This doesn't look like much but it proves that the whole current Red stack is working properly. Red compiler generates Red/System code in memory, that is then compiled to native code and linked in a 14KB executable file (contains the full Red runtime with currently 9 datatypes partially implemented).

The baby needs a few more days in the nursery before I commit the new code to the v0.3.0 branch on Github. Once the existing datatypes will be more complete and once we choose how to deal internally with Unicode strings, we should be able to release the first Red alpha.

The current Red compiler is pretty simple and should remain light in the future. I have finally chosen an hybrid dynamic/static type system, to avoid diving into a complex type inference engine now, as I realized that once we get out of the bootstrap and have the final Red JIT-compiler, it will be much more easier to achieve. Also, I want to pass that bootstrap stage as soon as possible, because it is really limiting the full Red potential.

Stay tuned, the next months will be fun! ;-)

Red/System v0.2.6 released

The need for more structured code support has arisen with the addition of bigger and more sophisticated bindings like GTK one or the (soon to be released) work on alpha version of Red compiler (which emits Red/System code). So, v0.2.6 introduces namespaces support using a very simple, but efficient model.

For example, to encapsulate some variables and functions in a local context:

    b: 1
    a: context [
       b: 2
       inc: func [i [integer!] return: [integer!]][
           i + b
       ]
    ]
    a/inc 5                            ;-- will return 7

Local variables take precedence other global ones with same name. This simple rule also applies to nested contexts, the nearest one has priority, e.g.:

    a: context [
       b: 123

       c: context [
           #enum colors! [red green blue]
           b: "hello"
           foo: func [][print-line b]
       ]

       print-line b                    ;-- will output 123
       c/foo                           ;-- will output hello
    ]

    print-line a/b                     ;-- will output 123
    a/c/foo                            ;-- will output hello
    print-line a/c/b/2                 ;-- will output e
    print-line a/c/blue                ;-- will output 2

As you can see from this example, enumerations can also be defined locally to a context, but not only. Aliases and imported functions can also be defined locally too! With that powerful new feature, you can now modularize your Red/System programs simply and efficiently.

See the namespaces documentation for more features and info.

In this release you will also find several bugfixes, mainly for floats support. Thanks to Rebolek for his aggressive floats testing. ;-)

Enjoy!