Fork me on GitHub

August 3, 2014

0.4.3: Floating point support

After a long time having only partial floating point support in Red/System, it comes now to Red with a broader support, thanks to the terrific work from Qtxie and Oldes, who managed to push Red further while I was busy moving from Europe to China (though, this might be the topic for another blog entry, as requested by many of you).

The new float! datatype implements IEEE-754 64-bit floating point format. It is available with most of the usual math functions support:

  • infix operators: +, -, *, /, **.
  • prefix base functions: add, substract, multiply, divide, power.
  • trigonometric functions:  cosine, sine, tangent, arcsine, arccosine, arctangent.
  • other math functions: log-2, log-10, log-e, exp, square-root, round

Note that these trigonometric functions are taking arguments in degrees, a /radians refinement is provided for input values in radians. However, this can result in extra verbosity for some long math expressions where using only radians, like:
((sine/radians b) * (cosine/radians c)) + ((cosine/radians b) * (sine/radians c))
Some radians-oriented shortcuts to these functions are also provided for convenience: cos, sin, tan, arcsin, arccos, arctan. So the above expression becomes:
((sin b) * (cos c)) + ((cos b) * (sin c))
Here are some code examples from Red console:
red>> 1.23456
== 1.23456
red>> 1e10
== 10000000000.0
red>> 1.23 * 2
== 2.46
red>> 1.23 * 2.0
== 2.46
red>> to integer! 1.23 * 2.0
== 2
red>> cos pi
== -1.0
red>> sin pi
== 0.0
red>> cos pi / 2
== 0.0
red>> cos pi / 3
== 0.5
red>> cosine/radians pi / 3
== 0.5
red>> cosine 60
== 0.5
red>> .1 + .2 + .3
== 0.6
red>> .1 + .2 + .3 = .6
== true
red>> .1 + .2 + .3 - .6
== 1.110223024625157e-16
red>> float? load "0.1"
== true
red>> to float! 1
== 1.0
red>> 1 = to integer! to float! 1
== true
As you can see, Red tries to give you meaningful outputs even when the result is not exact, but this approach has its limits too. Qtxie has ported partially dtoa() functions to Red/System, however, the overhead of the additional code (20-40KB once compiled) is quite costly given how tiny is currently our runtime library (~350KB). So, for now, that implementation has been provided as an optional library for Red/System, and will be modularized for Red, once modules will be supported.

IEEE-754 special values

You might know that standard floating point format supports a few extra special values that are meant to make some calculation possible in edge cases. Those are also supported natively by Red, with the following literal formats:
Not a Number (NaN)        :  1.#NaN
Positive Infinity (+INF)  : +1.#INF (or just 1.#INF)
Negative Infinity (-INF)  : -1.#INF
Positive Zero             : +0.0 (or just 0.0)
Negative Zero             : -0.0
These values are mostly intended for scientific calculations, you do not have to worry about them. They can be produced as results of some math operations on floats, but by default, an error will be thrown instead.

In case, you need to operate with maximum precision, and have all the special float values as results instead of errors, a couple of flags are available for that through the system special access. The syntax is:
system/float-options [spec]

[spec]: block of flags (word! | set-word!) with values (logic! | word!)
Valid flags are:

  • pretty?: enables pretty printing of float numbers when very close to an integer value (default: true)
  • full?: enables math operations on float special values (default: false)

 Examples:
red>> 4.000000000000001e32
== 4.0e32
red>> system/float-options [pretty?: no]
red>> 4.000000000000001e32
== 4.000000000000001e32
Armhf support

So far, Red supported only the armel ABI for ARM backends. Since this release, we fully support now armhf ABI too, through a specific compilation option that can be found in the new RPi compilation target (intended mainly for default OS on RaspberryPi). The main difference between these ABI is the way float values are passed as arguments to functions, armel requires passing them on stack, while armhf requires passing them through FPU registers.

Other changes

  • url! datatype preliminary support: all actions are working, but no path access support yet.
  • New actions: reverse, random, swap, take, to(*), trim
  • New natives: same?, NaN?
  • New mezzanines: float?, routine?
  • Red/System FPU direct access through system/fpu/* options.
  • Help command now displays full help on routines too.
  • Many bug fixes and a few wishes granted.

(*) to is currently limited to integer/float/string conversions only.

What's next?

After the digression in the floating point lands, we go back to our main roadmap, so in the next releases, expect (in no particular order):

  • GUI support for Android / Windows platforms
  • Improved toolchain for Android APK generation
  • Object compilation support
  • New console engine
  • Error! datatype and exceptions handling
  • Typeset! and other new datatypes
  • Redbin format specification and implementation for the compiler
  • Improved compiler performance

Thanks for your patience and support during these last months, we are now back to our cruise development speed, so expect faster changes until the end of the year.

April 29, 2014

0.4.2: Unicode console and FreeBSD support

This long awaited new release is now available. As I have been travelling a lot in the last months, this release has been delayed much more than I wanted. Anyway, we managed to achieve a really big amount of work, as shown by the 500+ commits since previous release and the 75 fixes over 210 new tickets opened. As usual, we strive to keep the number of opened tickets (especially bug reports) as low as possible, achieving 97.5% of closed tickets out of a total of 794 tickets so far! We really do care about bug reports.

New runtime lexer

The first runtime lexer (wrapped by load function) was implemented a year ago, as a quick hack for the console addition to Red. It was coded in Red/System and supported ASCII inputs only. It was not meant to stay more than a few weeks, but as often in the software world, temporary code lifespan exceeds by far the wishes of the author. The addition of Parse dialect in previous release has opened the possibility of rewriting the Red runtime lexer using the Unicode-aware parse function. It turned out to be a great design choice and opens even more interesting future options like I/O streaming support (when parse will support it) or dynamically extending the lexical rules (when loading custom datatypes for example).

Improved console

The new runtime lexer is now powering the Red console, so we finally have proper Unicode input support!

A help system has also been provided, including the following functions: help, what and source. Try them from the console!

From the console code, the line editing features have been extracted in a different source file that can be now included in your Red programs when you need user input support. For that, two new functions have been provided (working like in Rebol): input and ask.

Moreover, a new branch was started in order to provide cross-platform line editing capabilities without the burden of external dependencies that have proved to be problematic and limited. The new vt100 version should work fine, but it is unfinished. Contributors with deep terminal coding experience are welcome to help improve the current code. We are aiming at a cross-platform console engine that could be used both in CLI and GUI apps.

Additional functions
  • New actions: absolute, remainder, complement, power, odd?, even?, and, or, xor, reverse
  • New natives: complement?, positive?, negative?, min, max, shift, to-hex
  • New operators: <<, >>, >>>
A new option was added to the system function: system/interpreted?, that will return true if the code is run by the interpreter (remember that do invokes the interpreter until we get JIT-compilation abilities).

Parse and load have been extended to accept a /part refinement.

Infix operators can now be created from existing functions or routines.

A first call function implementation has been contributed by Bruno Anselme with support for both Red and Red/System.

FreeBSD support

Yes, we got it now! :-) All thanks to Richard Nyberg who kindly provided us with the low-level patches required to make Red work on FreeBSD.

Red/System changes

The Red/System lexer used to be simply the load native from Rebol, which was a great way to speed up the development at the beginning, but was also limitating the syntax to only what Rebol2 accepts. Now the Red/System lexer uses the same code as the Red lexer (the compiler version, not the runtime one), freeing the Red/System syntax from the limitations and making it truly a dialect of Red!

Literal arrays support has been added also in order to facilitate initialization of arrays of value (until we get a first class array! datatype).

CPU registers read/write access has been added. It will be extended in the future to support special registers (like status flags register).

The maximum number of function local variables supported by Red/System was limited to 16, this was limitating also the number of local words that could be used in a Red function. This limitation has now been raised much higher, at least 512 local variables are now allowed.

Work in progress...

Object support is already present in this release, but is it not official yet, as it is supported by the interpreter only and not the compiler. Expect quick progress on this front.

The Android GUI support is also under heavy work in the android branch. In order to implement a proper GUI API, the implementation of a VID-like dialect has started, with Android as first back-end. Windows support should follow shortly, then Linux (most probably using GTK+) and finally MacOSX (once we implement the objective-c bridge).

Gear second!

I am not made of rubber, but I can go gear second too! ;-) You may have not noticed, but the project is rapidly growing up in the last months. It is moving faster and on a larger scale as more contributors are joining. We also get better organized. This is the github stats for just this month:


The most important power-up we got was the addition of Xie Qingtian (aka qtxie) to the Red core team. Xie is an amazingly skilled young programmer from China, who is contributing to Red since more than a year now. But the thing is that he is working full time on Red project now, thanks to the kind sponsoring of CSDN and his CEO, Jiang Tao. Xie is the one who implemented all the new functions listed above and in a short amount of time! So consider that from now on, Red will be advancing twice faster thanks to him! ;-)

In order to organize better the work on Red, we are now using extensively Trello as our main task manager tool. The Red tasks board contains three main lists:

  • "Work in progress": for features we are working on.
  • "Road to 1.0": lists the required features for 1.0 version.
  • "Milestones": helps us organize upcoming releases.

Last but not least, the number of visitors on this site and the github repo has, at least, doubled since new year, thanks to an article on CSDN about Red, our Google Summer of Code campaign and the successful StackOverflow ad campaign (finished earlier this month) run by HostileFork, that brought us more than 10k new visitors who clicked the ad, making it the most clicked ad on SO since the new year! The ad is still visible here.

Big thank to all the people that have contributed to this (big) release. Enjoy it! :-)

February 6, 2014

Project Ideas for the Google Summer of Code 2014

Red has been ramping up its recruiting efforts in 2014.  So far in January, we've done very well with our open-source recruitment campaign on StackOverflow.  Now in February, we are throwing our hat into the ring to be a mentoring organization for Google's Summer of Code:


If you haven't heard about Summer of Code, it pairs college students with mentors who represent an open source project.  The students work remotely and are paid a stipend... and they are expected to complete a measurable goal in the 3 months that they have.  There's an opportunity to start getting involved before the summer, and hopefully the students will have a good experience and stay involved after the summer ends!

Though Nenad will be too busy in the new Beijing Red headquarters :-) to mentor, several members of the Red team have stepped up to volunteer if we are selected.  There is of course around-the-clock support in the [Rebol and Red] chat room, but Google needs commitments from specific people to fill out paperwork and reviews.  As the project "admin" Nenad should only have to be consulted if there was a disagreement between the student and the mentor (which would not happen!)  Of course he would be reviewing any pull request, as with any other code submission.

Google says the best project ideas come from students, which sounds like a good idea.  Still, they require a "starter list" of example projects to originate from the potential mentoring organizations.  So here is the one we came up with...

January 27, 2014

Year of the Horse

In a few days is the Chinese New Year, the year of the Horse. This is what you can find online about its meaning:

The spirit of the horse is recognized to be the Chinese people's ethos – making unremitting efforts to improve themselves. It is energetic, bright, warm-hearted, intelligent and able. Ancient people liked to designate an able person as 'Qianli Ma', a horse that covers a thousand li a day (one li equals 500 meters).
Clever combination of 2,0,1,4 numbers and an artistic horse illustration!

The horse empowers men to reach out far places quickly, Red aims at empowering programmers to accomplish more with less efforts. It looks like the Year of the Horse might be a good one for Red to take off.

I have been travelling through Asia since a month now and will be getting back home in a few days. I went to Hong-Kong, Shanghai and Beijing. This is my first time in Asia, and it felt like accomplish a kid's dream, full of exotic names and places. So, it was full of Marco Polo's stories in mind that I came in China...What a contrast! China is so modern that I must confess Europe looks really behind sometimes... China is also full of opportunities for western people, that's not just a saying, I experienced it first hand.

The purpose of my trip was to get information about the Chinese market for programmers and especially mobile programmers, meet with some startups there and IT people there, present Red (and Rebol) to any group of programmers interested in knowing more about it and get feedback from them.

Meeting with Qtxie, one of the most talented Red contributors.

I got much more than that in fact and I was not even expecting it. Thanks to Jerry Tsai who is supporting Red and Rebol since many years, we could meet with some of the key IT people in China and main startups. I had the opportunity to give an interview at CSDN (the largest developer community in China) and even give a teaching to an audience of beginners and non-programmers using Rebol2 and its GUI DSL.


So, not only, I could gather much more information than I expected, but I could also meet with the #1 incubator in China: InnovationWorks (IW).


IW made me an offer for spending 6 months at their offices in Beijing as a EIR (Entrepreneur In Residence) that I accepted. Basically, I will be able to continue coding on Red project while providing some architecture and tech advices to the incubated startups. This will be a great opportunity for me to study deeply the Chinese programmer's market and prepare for proposing some Red-related products or services in China and worldwide. So I will be moving in Beijing in a few months. This move does not change in any way my work on Red, as I will be able to continue working hard during the whole year on pushing Red closer to a production version.

I was able to work a bit during my trip, mainly on GUI support for Android (reusable for other platforms as well), so the next Red release should feature an alpha version of GUI programming support!

Special thank to Tamas Herman who hosted me during my stay at Hong-Kong and provided me with all I needed for working (including delicious Thai food!). :-)

So, let's start riding the Red horse this year! ;-)

November 30, 2013

0.4.1: Introducing Parse


One of the greatest feature of the Rebol language has always been its parsing engine, simply called Parse. It is an amazing piece of design from Carl Sassenrath, that spared all Rebol users for the last 15 years, from the pain of having to use the famously unmaintainable regexps. Now, Parse is also available to Red users, in an enhanced version!

So, in short, what is Parse? It is an embedded DSL (we call them "dialects" in the Rebol world) for parsing input series using grammar rules. The Parse dialect is an enhanced member of the TDPL family. Parse's common usages are for checking, validating, extracting, modifying input data or even implementing embedded and external DSLs.

The parse function call syntax is straightforward:
    parse <input> <rules>  

    <input>: any series value (string, file, block, path, ...)
    <rules>: a block! value with valid Parse dialect content
Here are a few examples, even if you don't know Red and Parse dialect, you can still "get" most of them, unlike regexps.  You can copy/paste them directly into the Red console.

Some simple examples of string or block input validation using grammar rules:
    parse "a plane" [["a" | "the"] space "plane"]
    parse "the car" [["a" | "the"] space ["plane" | "car"]]

    parse "123" ["1" "2" ["4" | "3"]]
    parse "abbccc" ["a" 2 "b" 3 "c"]
    parse "aaabbb" [copy letters some "a" (n: length? letters) n "b"]

    parse [a] ['b | 'a | 'c]
    parse [hello nice world] [3 word!]
    parse [a a a b b b] [copy words some 'a (n: length? words) n 'b]

How to parse an IPv4 address accurately:
    four:     charset "01234"
    half:     charset "012345"
    non-zero: charset "123456789"
    digit:    union non-zero charset "0"

    byte: [
          "25" half
        | "2" four digit
        | "1" digit digit
        | non-zero digit
        | digit
    ]
    ipv4: [byte dot byte dot byte dot byte]

    parse "192.168.10.1" ipv4
    parse "127.0.0.1"    ipv4
    parse "99.1234"      ipv4
    parse "10.12.260.1"  ipv4

    data: {
        ID: 121.34
        Version: 1.2.3-5.6
        Your IP address is: 85.94.114.88.
        NOTE: Your IP Address could be different tomorrow.
    }
    parse data [some [copy value ipv4 | skip]]
    probe value                      ; will ouput: "85.94.114.88"


A crude, but practical email address validator:
    digit:   charset "0123456789"
    letters: charset [#"a" - #"z" #"A" - #"Z"]
    special: charset "-"
    chars:   union union letters special digit
    word:    [some chars]
    host:    [word]
    domain:  [word some [dot word]]
    email:   [host "@" domain]

    parse "john@doe.com" email
    parse "n00b@lost.island.org" email
    parse "h4x0r-l33t@domain.net" email

Validating math expressions in string form (from Rebol/Core manual):
    expr:    [term ["+" | "-"] expr | term]
    term:    [factor ["*" | "/"] term | factor]
    factor:  [primary "**" factor | primary]
    primary: [some digit | "(" expr ")"]
    digit:   charset "0123456789"
    
    parse "1+2*(3-2)/4" expr        ; will return true
    parse "1-(3/)+2" expr           ; will return false

Creating a simple parser for a subset of HTML:
    html: {
        <html>
            <head><title>Test</title></head>
            <body><div><u>Hello</u> <b>World</b></div></body>
        </html>
    }

    ws: charset reduce [space tab cr lf]

    parse html tags: [
        collect [any [
            ws
            | "</" thru ">" break
            | "<" copy name to ">" skip keep (load name) opt tags
            | keep to "<"
        ]]
    ]

    ; will produce the following tree of blocks as output of parse:
     [
         html [
             head [
                 title ["Test"]
             ]
             body [
                 div [
                     u ["Hello"]
                     b ["World"]
                 ]
             ]
         ]
     ]

The Parse dialect

Parse's core principles are:
  • Advance input series by matching grammar rules until top-level rule failure (returning false) or input exhaustion (returning true). (*)
  • Ordered choices (e.g. in ["a" | "ab"] rule, the second one will never succeed).
  • Rules composability (unlimited).
  • Limited backtracking: only input and rules positions are backtracked, other changes remain.
  • Two modes: string-parsing (for example: external DSL) or block-parsing (for example: embedded DSL).
(*) If collect keyword is used in any rule in its simplest form, a block will be returned by parse no matter if the root rule succeeded or not.


The Parse rules can be made from:
  • keyword : a dialect reserved word (see the tables below).
  • word : word will be evaluated and its value used as a rule.
  • word: : set the word to the current input position.
  • :word : resume input at the position referenced by the word.
  • integer value : specify an iterated rule with a fixed number or a range of iterations.
  • value : match the input to a value
  • | : backtrack and match next alternate rule
  • [rules] : a block of sub-rules
  • (expression) : escape the Parse dialect, evaluate a Red expression and return to the Parse dialect.

The following keywords are currently available in Red's Parse implementation. They can be composed together freely.

Matching

ahead rule : look-ahead rule, match the rule, but do not advance input.
end: return success if current input position is at end.
none: always return success (catch-all rule).
not rule: invert the result of the sub-rule.
opt rule: look-ahead rule, optionally match the rule.
quote value: match next value literally (for dialect escaping needs).
skip: advance the input by one element (a character or a value).
thru rule: advance input until rule matches, input is set past the match.
to rule: advance input until rule matches, input is set at head of the match.

Control flow

break: break out of a matching loop, returning success.
if (expr): evaluate the Red expression, if false or none, fail and backtrack.
into rule: switch input to matched series (string or block) and parse it with rule.
fail: force current rule to fail and backtrack.
then: regardless of failure or success of what follows, skip the next alternate rule.
reject: break out of a matching loop, returning failure.

Iteration

any rule: repeat rule zero or more times until failure or if input does not advance.
some rule: repeat rule one or more times until failure or if input does not advance.
while rule: repeat rule zero or more times until failure regardless of input advancing.

Extraction

collect [rule]: return a block of collected values from the matched rule.
collect set word [rule]: collect values from the matched rule, in a block and set the word to it.
collect into word [rule]: collect values from the matched rule and insert them in the block referred by word.
copy word rule: set the word to a copy of the matched input.
keep rule: append a copy of the matched input to the collecting block.
keep (expr): append the last value from the Red expression to the collecting block.
set word rule: set the word to the first value of the matched input.

Modification

insert only value: insert[/only] a value at the current input position and advance input after the value.
remove rule: remove the matched input.

The core principles mention two modes for parsing. This is necessary in Red (as in Rebol) because of the two basic series datatypes we have: string! and block!. The string! datatype is currently an array of Unicode codepoints (Red will support an array of characters in a future version) while the block! datatype is an array of arbitrary Red values (including other blocks).

In practice, this results in some minor differences in Parse dialect usage. For example, it is possible to define arbitrary sets of characters using the new bitset! datatype, which are useful only for string! parsing in order to match with a large number of characters in one time. Here is an example using only bitsets matching and iterators:
    letter: charset [#"a" - #"z"]
    digit:  charset "0123456789"

    parse "hello 123 world" [5 letter space 3 digit space some letter]

The Bitset! datatype

A bitset value is an array of bits that is used to store boolean values. In the context of Parse, bitsets are very useful to represent arbitrary sets of characters across the whole Unicode range, that can be matched against an input character, in a single operation. As bitset! is introduced in this 0.4.1 release, it is worth having an overview of the features supported. Basically, it is on par with Rebol3 bitset! implementation.

In order to create a bitset, you need to provide one or several characters as base specification. They can be provided in different forms: codepoint integer values, char! values, string! values, a range or a group of previous elements. The creation of a new bitset is done using the following syntax:
    make bitset! <spec>

    <spec>: char!, integer!, string! or block!
For example:
    ; create an empty bitset with places at least for 50 bits
    make bitset! 50

    ; create a bitset with bit 65 set
    make bitset! #"A"

    ; create a bitset with bits 104 and 105 set
    make bitset! "hi"

    ; create and set bits using different values representations
    make bitset! [120 "hello" #"A"]

    ; create a bitset using ranges of values
    make bitset! [#"0" - #"9" #"a" - #"z"]
Ranges are defined using two values (char! or integer! allowed) separate by a dash word.

Bitsets are auto-sized to fit the specification value(s) provided. The size is rounded to the upper byte bound.

A shortcut charset function is also provided for practical usage, so you can write:
    charset "ABCDEF"
    charset [120 "hello" #"A"]
    charset [#"0" - #"9" #"a" - #"z"]

For reading and writing single bits, the path notation is the simplest way to go:
    bs: charset [#"a" - #"z"]
    bs/97     ; will return true
    bs/40     ; will return false
    c: #"z"
    bs/:c     ; will return true
    bs/97: false
    bs/97     ; will return false
(Note: bitset indexing is zero-based.)

Additionally, the following actions are supported by bitset! datatype:
pick, poke, find, insert, append, copy, remove, clear, length?, mold, form

See the Rebol3 bitset documentation for more info about usage of these actions.

In order to cope with the wide range of Unicode characters, bits outside of the bitsets are treated as virtual bits, so they can be tested and set without errors, the bitset size will auto-expand according to the needs. But that is still not enough to deal with big ranges, like for example a bitset for all Unicode characters except digits. For such cases, it is possible to define a complemented bitset that represents the complement range of the specified bits. This makes possible to have large bitsets while using only a tiny memory portion.

Complemented bitsets are created the same way as normal bitsets, but they need to start with the not word and always use a block! for their specification:
    ; all characters except digits
    charset [not "0123456789"]

    ; all characters but hexadecimal symbols
    charset [not "ABCDEF" #"0" - #"9"]

    ; all characters except whitespaces
    charset reduce ['not space tab cr lf]

Set operations are also possible, but only union is currently implemented in Red (it is the most used anyway for bitsets). With union, you can merge two bitsets together to form a new one, which is very useful in practice:
    digit: charset "0123456789"
    lower: charset [#"a" - #"z"]
    upper: charset [#"A" - #"Z"]

    letters:  union lower upper
    hexa:     union upper digit
    alphanum: union letters digit

Parse implementation

Parse dialect has been implemented as a FSM which differs from the Rebol3 implementation that relies on recursive function calls. The FSM approach makes possible several interesting features, like the ability to stop the parsing and resume it later, or even serialize the parsing state, send it remotely and reload it to continue the parsing. Such features could now be implemented with minimal efforts.

Red Parse implementation is about 1300 lines of Red/System code, with a significant part of it spent on optimized iteration loops for common cases. About 770 unit tests have been hand-written to cover the basic Parse features.

The current Parse runs as an interpreter, which is fast enough for most use-cases you will have. For cases where maximum performance is required, work has started on a Parse static compiler to soon provide the fastest possible speed to Parse-intensive Red apps. The generated code is pure Red/System code and should be about a magnitude faster on average than the interpreted version. When Red will be self-hosted, a Parse JIT compiler will be provided to deal with the cases that the static compiler cannot process.

As Red gets more features, Parse will continue to be improved to take advantage of them. Among other future improvements, binary! parsing will be added as soon as binary! datatype is available, and stream parsing will be possible when port! datatype will be there.

The Red Parse also exposes a public event-oriented API in form of an optional callback function that can be passed to parse using the /trace refinement.
    parse/trace <input> <rules> <callback>

    <callback> specification:

    func [
        event   [word!]   ; Trace events
        match?  [logic!]  ; Result of last matching operation
        rule    [block!]  ; Current rule at current position
        input   [series!] ; Input series at next position to match
        stack   [block!]  ; Internal parse rules stack
        return: [logic!]  ; TRUE: continue parsing, FALSE: exit
    ]

    Events list:
    - push    : once a rule or block has been pushed on stack
    - pop     : before a rule or block is popped from stack
    - fetch   : before a new rule is fetched
    - match   : after a value matching occurs
    - iterate : after a new iteration pass begins (ANY, SOME, ...)
    - paren   : after a paren expression was evaluated
    - end     : after end of input has been reached
This API will be extended in the future to get more fine-grained events. This API could be used to provide Parse with tracing, stats, debugging, ... Let's see what Red users will come up with! ;-)

A default callback has been implemented for tracing purposes. It can be accessed using the handy parse-trace function wrapper:
    parse-trace <input> <rules>

You can try it with simple parsing rules to see the resulting output.


What about DSL support?

Parse is a powerful tool for implementing DSL parsers (both embedded and externals), thanks to its ability to inline Red expressions directly into the rules, allowing to easily link the DSL syntax with its corresponding semantics. To illustrate that, here is a simple Brainfuck interpreter written using Parse:
    bf: function [prog [string!]][
        size: 30000
        cells: make string! size
        append/dup cells null size

        parse prog [
            some [
                  ">" (cells: next cells)
                | "<" (cells: back cells)
                | "+" (cells/1: cells/1 + 1)
                | "-" (cells/1: cells/1 - 1)
                | "." (prin cells/1)
                | "," (cells/1: first input "")
                | "[" [if (cells/1 = null) thru "]" | none]
                | "]" [
                   pos: if (cells/1 <> null)
                   (pos: find/reverse pos #"[") :pos
                   | none
                  ]
                | skip
            ]
        ]
    ]

    ; This code will print a Hello World! message
    bf {
        ++++++++++[>+++++++>++++++++++>+++>+<<<<-]>++.>+.+++++++..+++.
        >++.<<+++++++++++++++.>.+++.------.--------.>+.>.
    }

    ; This one will print a famous quote
    bf {
        ++++++++[>+>++>+++>++++>+++++>++++++>+++++++>++++++++>
        +++++++++>++++++++++>+++++++++++>++++++++++++>++++++++
        +++++>++++++++++++++>+++++++++++++++>++++++++++++++++<
        <<<<<<<<<<<<<<<-]>>>>>>>>>>>----.++++<<<<<<<<<<<>>>>>>>
        >>>>>>.<<<<<<<<<<<<<>>>>>>>>>>>>>---.+++<<<<<<<<<<<<<>>
        >>>>>>>>>>>>++.--<<<<<<<<<<<<<<>>>>>>>>>>>>>---.+++<<<<
        <<<<<<<<<>>>>.<<<<>>>>>>>>>>>>>+.-<<<<<<<<<<<<<>>>>>>>>
        >>>>>>+++.---<<<<<<<<<<<<<<>>>>.<<<<>>>>>>>>>>>>>>--.++
        <<<<<<<<<<<<<<>>>>>>>>>>>>>>-.+<<<<<<<<<<<<<<>>>>.<<<<>
        >>>>>>>>>>>>>+++.---<<<<<<<<<<<<<<>>>>>>>>>>>>>>.<<<<<<
        <<<<<<<<>>>>>>>>>>>>>>-.+<<<<<<<<<<<<<<>>>>>>>>>>>>>>-.
        +<<<<<<<<<<<<<<>>>>>>>>>>>>>>--.++<<<<<<<<<<<<<<>>>>+.-
        <<<<.
    }
Note: this implementation is limited to one-level of "[...]" nesting to keep it as short as possible. A complete, but a bit more longer and complex implementation using Parse only, is availaible here.

So, such approach works incredibly well for small DSLs. For more sophisticated ones, Parse still works fine, but it might be less helpful as the DSL semantics get more complex. Implementing an interpreter or compiler for a more advanced DSL is not a trivial task for most users. Red will address that in the future by providing a meta-DSL wrapper on top of Parse, exposing a higher-level API to build more sophisticated DSL by abstracting away core parts of an interpreter or compiler. A DSL for building other DSLs is not a crazy idea, it already exists in a very powerful form as the Rascal language for example. What Red will provide, will be just one step in that direction, but nothing as sophisticated (and complex) as Rascal.

Other changes in this release
Just to mention other changes in this release, now that we got rid of the 800 pound gorilla in the room. This release brings a significant number of bug fixes, both for Red and Red/System. Also, thanks to qtxie, the ELF file emitter is now on par with the other ones when producing shared libraries.

Thanks to all the people involved in helping for this big release, including design fixes and improvements, testing, bug reporting and test writing.

Enjoy! :-)

September 27, 2013

0.4.0: Red goes binary!

What's that?!

As we are getting closer to the end of the alpha period, we are now moving to a more convenient way to use and distribute the Red toolchain. So far, you needed to download a Rebol interpreter and the sources from Github separately, and run it using, a bit verbose, command-lines. This is fine for developping Red  with contributors that are interested in the inner workings of the toolchain, but for the end users, the goal has always been to provide a simpler and much more convenient way, like Rebol teached us in the past.

So, from now, you can get Red as a single binary (< 1 MB) from the new Download page. Yes, all of Red toolchain and runtime is packed in that small binary, even the REPL is built-in!

The Red repo landing page has been reworked to show Red usage in binary form, all previous command-line options are present, a new one (-c) has been introduced. Here is a short overview of the main options:

Launch the REPL:
    $ red 
Run a Red script directly (interpreted):
    $ red <script>
Compile a script as executable with default name in working path:
    $ red -c <script>
Compile a script as shared library with default name in working path:
    $ red -dlib <script>
Compile a script as executable with alternative name:
    $ red -o <new> <script>
Compile a script as executable with default name in other folder:
    $ red -o <path/> <script>
Compile a script as executable with new name in other folder:
    $ red  -o <path/new> <script>
Cross-compile to another platform:
    $ red -t Windows <script>
Display a description of all possible options:
    $ red -h
Notice that -c option is implied when -o or -t are used. It is designed to keep command-lines as simple and short as possible.

Moreover, for standalone Red/System programs, the Red binary will be able to compile them directly, no special option needed, they will be recognized automatically.

Thanks very much to Tamás Herman for helping with setting up the build farm and providing the Mac OSX machine, and thanks to the HackJam hackerspace group from Hong-Kong for the hosting!

Other changes

  • In addition to that new binary form, 17 issues have been fixed since the 0.3.3 release about a month ago (not counting regression tickets).
  • The work on objects support is half-done, objects are working fine with advanced semantics on the interpreter (see object branch), now the focus will be to support them at the Red compiler level.

What's next?

As we are moving closer to the beta state, version numbers will increase faster, e.g., once objects will be done, the release will be the 0.5.0, while 0.6.0 will bring full I/O support. Between these major versions, smaller versions should be still possible, this means that the release cycle should accelerate with at least one release each month from now on. So, what you should expect in the next ones?

0.4.x
  • Simple I/O support: (just read, write and exists? on files)
  • PARSE support
  • Pre-compiled runtime (much faster compilation times)
0.5.0
  • Full object support
0.5.x
  • VID-like cross-platform dialect binding to native widgets.
  • Mezzanine functions additions
  • Redbin (accurate Red values serialization in binary format)
  • Full errors management
  • Red-level exceptions handling

Enjoy!


August 11, 2013

0.3.3: Shared libraries and Android!

This new release took a while, first because a lot of work was done since the last release, about 390 new commits were added and 44 issues were fixed over 80 new tickets opened since the last release. But it was also delayed because, a month ago, I was invited at Recode 2013 developer conference to give a talk and present the latest advancements on Red.




I would like to thank again people that donated to cover the costs of my trip, Brian "HostileFork" Dickens and Coginov in the first place, and all the others that have contributed. I have spent a great time in Montréal and met with lot of Red and Rebol coders, it was great to meet you IRL and spend a few days with you guys!

What's new in 0.3.3?

Java bridge

Since this version, Red is now able to interface with a 32-bit Java Virtual Machine through JNI, using a dedicated bridge written in Red, including a very small part in Java. This bridge allows Red programs to instantiate Java objects and call their methods from Red directly at run-time.

Instructions for using this bridge and compiling the hello example are given on this page.

Here are some screenshots of the resulting AWT hello app done with the Red/Java bridge:

Windows / x86

Linux / ARM

Android support

Red/System runs on Android since more than a year, but we were unable to access the Android API....until now. Thanks to the Red/Java bridge, an Android specific bridge has been made, in order to allow Red to call the whole Android API in Java. This is still alpha stuff, but it works really well, allowing you to build true native Android app in Red, without having to touch any Java code!

This is the hello Red/Android app screenshot, called eval, which simply shows a multiline input field where you can type any Red valid code and run it with the [Do] button. The last expression value gets printed below the button.



You can download the APK file (116KB) with the QR code on the right. 

This is a proof of concept. Now that we know that this way works fine, we will continue to improve the bridge and add new layers of abstraction, in order to keep the low-level Java objects hidden as much as possible for the Red coders and replace them with DSLs and Red ports.

In order to build the eval demo app from the sources, just run the build script from a Rebol2 console.


Shared lib generation

Since a year, we were working on bringing shared library generation, now it is available in the main branch. New features were added to support library generation like a way to declare the exported symbols and special callback functions when loading and freeing the library. Here is a simple example of a Red/System library:

    Red/System [
        File: %testlib.reds
    ]

    inc: func [n [integer!] return: [integer!]][n + 1] 

    #export [inc]


You compile such shared library using the new -dlib command-line option:

   >> do/args %rsc.r "-dlib testlib.reds"
The output binary name will have a platform-specifc suffix (.dll, .so or .dylib).

You can then load this shared library from Red or any other programming language having a good enough FFI. For example, from Red/System, it could be done as simply as:

    Red/System [
        File: %loadlib.reds
    ]

    #import [
        "testlib.dll" stdcall [
            inc: "inc" [n [integer!] return: [integer!]]
        ]
    ]

    print inc 123

This will print 124 when run.


Other Red language changes
  • new action: insert
  • new native: bind
  • new mezzanines: system, any-series?, replace, zero?
  • finished interpreter by adding exit and return support.
  • Function keyword now collects counter words from: repeat, foreach, forall.
  • new #call compilation directive to enable calling Red functions from Red/System.
  • paths now support get-words fully
Red/System got also several improvements:
  • PIC support for both x86 and ARM backends
  • Kernel driver compilation support (Windows only for now)
  • improved function! pointer support, now they can be used as struct members.
  • added new function attribute for dynamic calls: custom
  • new compiler directive for exporting symbols: #export
  • a way to manually align the native stack before making system calls: system/stack/align


What's next?

Android is becoming the dominant OS, so we need to have Red support it as best and as soon as possible. This remains a priority, so improved Android bridge and new sample apps will be available in the next releases. At the same time, we still miss some core features in Red, so these are the new features that you can expect in the next releases of Red:
  • object support
  • simple I/O support
  • whole toolchain released as a single binary (encapped)
  • PARSE support
  • VID-like cross-platform dialect binding to native widgets.
  • mezzanines function additions

We are now heading towards the end of the alpha tunnel, if you look well enough, you'll see the light already. ;-)

Enjoy!