Subpixel GUI

Maybe you didn't notice, but Red/View, our GUI engine, has subpixel precision from the beginning! Unfortunately, that level of precision was not directly accessible to end users, until now. 

Actually, it would be more accurate to say that we had subpixel resolution only so far. The guilty part is the pair! datatype being limited to integer components only, while subpixel precison requires decimal numbers. So we have recently introduced new datatypes to cope with that.

What urged us to make those changes now was a very peculiar visual glitch caused by that dissonance. That glitch happens during face dragging operations. Here is an example using our View test script:

As you can see, on some positions, the face starts shaking while the mouse cursor remains still. This affects any type of face. The shaking is about ±2 pixels. It is caused by the difference in precision between the /offset facet expressed in integer numbers and the backend API, which only deals with floating point numbers. The accumulated error when converting integer->float->integer gives a 2 pixels difference. Such error happens on displays where the scaling factor is different from 100%. With the rise of 2K, 3K and 4K displays, a scaling factor > 100% has become the norm, making this glitch more frequent. You might think that this is not a big issue until you start building custom scrollbars and see your entire scrolled content shaking massively...

New point datatypes

In order to provide decimal positions and sizes for View faces, extending the existing pair! datatype was considered, though, the pair syntax can hardly scale up for such needs:

    2343.122x54239.44
    2343.122x54239.44x6309.332
    2343.122x54239.44x6309.332x442.3321
    2.33487e9x54239.44
    2.33487e9x54239.44x9.83242e17
    2.33487e9x54239.44x9.83242e17x5223.112
    1.#infx1.#infx1.#inf

As you can notice there, it quickly becomes difficult to read and identify the individual components. So we opted for adding a new literal form (hence a new datatype) that matches how coordinates for two or more dimensions are commonly represented:

    (2343.122, 54239.44)
    (2343.122, 54239.44, 6309.332)
    (2343.122, 54239.44, 6309.332, 442.3321)
    (2.33487e9, 54239.44)
    (2.33487e9, 54239.44, 9.83242e17)
    (2.33487e9, 54239.44, 9.83242e17, 5223.112)
    (1.#inf, 1.#inf, 1.#inf)

Such literal forms requires the comma character to be a delimiter, so that it cannot be used anymore as a decimal separator. That was, unfortunately, a necessary decision in order to unlock such literal forms. The gains should be bigger than the loss.

So, two new datatypes have been added:

• point2D!: a two-dimensional coordinate or size.
• point3D!: a three-dimensional coordinate or size. 

Their canonical lexical forms are:

    (<x>, <y>)
    (<x>, <y>, <z>)

    where <x>, <y> and <z> are integer or float numbers.

Optional spaces are allowed anywhere inside the point literals on input, they will be removed on loading.

    >> (1,2)
    == (1, 2)
    >> (  1.35 ,  2.4  )
    == (1.35, 2.4)

Both for 2D and 3D points, their components are internally stored as 32-bit floating point numbers, so that their precision is limited to 7 digits. This should be far enough for their use-cases though.

When one of the components has a fractional part equal to zero, it is displayed without the .0 part for easier reading. Similarly, integers are accepted as input for any component and are internally converted to a 32-bit float.

    >> (0.3, 0.5) + (0.7, 0.5)
    == (1, 1)
    >> (2.0, 3.0)
    == (2, 3) 

Creation

Besides literal points, it is possible to create them dynamically, the same way as pairs, using make, to or one of the as-* native functions:

    >> make point2D! [2 4.5]
    == (2, 4.5)
    >> to-point2D 1x2
    == (1, 2)
    >> as-point3D 1 (3 / 2) 7 * 0.5
    == (1, 1.5, 3.5)

Accessors

Point components can be individually accessed using ordinal numbers or component names using action accessors or path syntax:

    >> pick (2, 4.5) 1
    == 2.0
    >> pick (2, 4.5) 'y
    == 4.5
    >> p: (2, 4.5)
    == (2, 4.5)
    >> p/x
    == 2.0
    >> p/y: 3.14159
    == 3.14159
    >> p
    == (2, 3.14159)

Math operations

Basic math operations are supported as well:

    >> (1, 1) + (2, 3.5)
    == (3, 4.5)
    >> (1, 1) - (2, 3.5)
    == (-1, -2.5)
    >> (2, 3) * (10, 3.5)
    == (20, 10.5)
    >> (20, 30) / (10, 3)
    == (2, 10)

Notice that mixing pairs with point2D in math expressions is allowed. The pair value will be promoted to a point2D in such case (as integers with floats):

    >> 1x1 + (2, 3.5)
    == (3, 4.5)

Other actions/natives

    >> round (2.78, 3.34)
    == (3, 3)
    >> round/down (2.78, 3.34)
    == (2, 3)
    >> random (100, 100)
    == (53, 81)
    >> zero? (0, 0)
    == true
    >> min (10, 100) (24, 35)
    == (10, 35)
    >> max 10x100 (24, 35)
    == (24, 100)    

Notice that pairs will be promoted to point2D in mixed use cases with min/max.

 

View and VID adjustments

The main changes are in face! object:

• /offset: now only accepts point2D! values.
• /size: accepts both pair! and point2D! values.

In VID, both pair and point2D values can be used to denote positions and sizes, so that VID is backward compatible. All previous VID code should work without any change. VID will convert all positions to point2D values. Sizes by default in VID, keep using pairs, unless a point2D is provided by the user.

All Draw commands that were accepting pairs now also accept point2D values for higher precision.

The related documentation will get updated soon to reflect those changes.

In order to illustrate the difference in using pairs and point2D positions, here is a (not so) simple animation comparison showing the subpixel positioning difference (correctness of animation in this case is privileged over simplicity of code):

    view/no-wait [
        size 800x200 space 0x0
        b1: box 2x40 red return
        b2: box 2x40 blue
    ]
    x: b1/offset/x
    until [
        do-events/no-wait                   ; processes GUI events in queue
        wait 0.1                            ; slows down the animation
        do-no-sync [                        ; switches to manual faces redrawing
            b1/offset/x: b1/offset/x + 0.1
            b2/offset/x: to-integer x: x + 0.1
            if all [b1/state b2/state][show [b1 b2]] ; redraws both faces
        ]
        any [b1/offset/x > 700 none? b1/state none? b2/state]
    ]

Here's the zoomed capture of the result (on a display with 200% scaling factor):

The red bar uses the newly enabled subpixel precision, while the blue bar simulates the old pair positioning precision (so only allowing integer positions). What you can see is that the red bar makes two smaller steps while the blue bar makes a single one, looking more "jumpy".

This means that now animations on displays with a scaling factor > 100% can be smoother as they benefit from more accurate positioning.

Note: the animation code above is far from being simple or elegant, we'll be working on improving that. The animation code could have been quite simpler by using a rate option in VID and putting the animation code in a on-time handler. Though, timer events firing (especially on Windows) are not very reliable, so unrolling a custom event loop lowers that risk when the timing is critical (like for fast game loops).

As a conclusion, here is an old-school style starfield code demo using 2D and 3D points (moving mouse left/right changes the stars speed):

Let us hear your feedback about those changes on our Gitter (now Matrix) channel.

Enjoy!

Dynamic Refinements and Function Application

It's Time to Apply Yourself to Red

Red has never had an apply function, though we knew it would come someday. In the meantime, some of us rolled our own. Gregg's was simple, neither flexible nor efficient, and just a couple lines of code. Boris made a much more capable version, but it could still only be so fast as a mezzanine. R2 had a mezz version, which suffered the same problem. All that changes now. Apply is dead! Long live Apply! It required deep work, and a lot of design effort, but we think you'll like the results, whether you're a high-level Reducer, or anxious to see how much leverage you can, um, apply, from a functional perspective. Everybody wins.

If you don't know what apply is, in terms of functional languages, take a moment and read up. If you can get through the introduction there without getting dizzy, great. If your head is spinning, feel free to stop after the first section of this article and ignore the deep dive. You still get 90% of the value for most high level use cases. Gregg got so dizzy that he fell down, but was still able to help with this article.

Function application is largely about composition. How you can combine functions in a concise way for maximum leverage and minimum code. The problem with its design in many languages is that it makes things harder to understand. Rather than concrete functions names, there is indirection and abstraction. It can be tricky to get right, especially in a flexible language like Red, while also maintaining as much safety as possible. You can drive fast, but still wear your seat belt.

Dynamic Refinements

This subtle feature is likely to see wide use, because it will reduce code and let people build more flexible functions. It's also easy to explain. Here's an example. First, how you would write it today:

repend: func [
    {Appends a reduced value to a series and returns the series head} 
    series [series!] 
    value 
    /only "Appends a block value as a block"
][
    either only [
        append/only series reduce :value
    ][
        append series reduce :value
    ]
]

With dynamic refinements, it can be done like this.

repend: func [
    {Appends a reduced value to a series and returns the series head} 
    series [series!] 
    value 
    /only "Appends a block value as a block"
][
    append/:only series reduce :value
]

In case you missed the subtlety, it's :only being a get-word in the path. That's right, it's evaluated, rather than being treated literally, just like you use in selector paths. The value for the dynamic refinement is taken from its context, which can be a refinement in the function, or a local value. It can be any truthy value to use the refinement, and it is only retrieved not evaluated. That means you can't use a computed value directly, which makes it safer. Other than that, paths work just as they always have. If a refinement is used, fixed (literal) or dynamic, which can be mixed however you want, any arguments it requires will be fetched and used. Otherwise they are silently ignored, so you don't have to clutter your code or worry about what a dynamic path expression will consume.

repend: func [
    {Appends a reduced value to a series and returns the series head} 
    series [series!] 
    value 
    /only "Appends a block value as a block"
    /dup  "Duplicate the value"
        count [integer!]
][
    append/:only/:dup series reduce :value count
]

>> only: false  dup: false  repend/:only/:dup [] [1] 3
== [1]
>> only: true  dup: false  repend/:only/:dup [] [1] 3
== [[1]]
>> only: false  dup: true  repend/:only/:dup [] [1] 3
== [1 1 1]
>> only: true  dup: true  repend/:only/:dup [] [1] 3
== [[1] [1] [1]]

This is an incredibly exciting and powerful feature. It's a shame there isn't more to explain. ;^)

Function Application

Functional Programming has never yet become mainstream, though it has periodic rises in popularity and a devoted following in many language camps. Even Red. Yes, Red is a functional language. It's not a pure functional language, because functions can have side effects, but functions are first class values and can be passed around like any other. It lets you do things like this:

>> do-math-op: func [
    fn    [any-function!]
    arg-1 [number!]
    arg-2 [number!]
][
    fn arg-1 arg-2
]

== func [fn [any-function!] arg-1 [number!] arg-2 [number!]][fn arg-1 arg-2]
>> do-math-op :add 1 2
== 3
>> do-math-op :subtract 1 2
== -1

That's called a Higher Order Function, or HOF. It also means you can return a function as a result.

>> make-multiplier: func [
    arg-1 [number!]
][
    ;; We're making a function and returning it here.
    func [n [number!]] compose [(arg-1) * n]
]

== func [arg-1 [number!]][func [n [number!]] compose [(arg-1) * n]]
>> fn-m: make-multiplier 4
== func [n [number!]][4 * n]
>> fn-m 3
== 12

That's all well and good, but what if you want to call different functions that take a different number or type of arguments? Now it gets tricky, and inefficient. Because Red uses free-ranging evaluation (function args are not contained in a paren or marked as ending in any way at the call site), how do you handle different arities (number of arguments)? Here's a very simple apply mezzanine:

apply: func [
    "Apply a function to a block of arguments." 
    fn [any-function!] "Function to apply" 
    args [block!] "Arguments for function" 
    /only "Use arg values as-is, do not reduce the block"
][
    args: either only [copy args] [reduce args] 
    do head insert args :fn
]

So easy! The power of Red. But there is a cost. It's a mezzanine function, so it's slower than a native function, the args are copied or reduced, and then do is used to evaluate it. We can live with this kind of overhead for a great deal of Red code, but apply is a building block, and may be used in deep code where performance is important. You may also have noticed that the fn argument is any-function!, that means two things: 1) If you want to know the name of the function, the word that refers to it, too bad. You'd have to pass another arg for that. 2) Refinements. You can't use them with this model. And that limitation is a killer. For example, you could pass :append but not :append/:only. And there's no way you could have an /only refinement in your function and just pass that along. Until now. 

The Real Apply

Here's the new apply native that is now available in Red:

USAGE:
    APPLY func args

DESCRIPTION: 
    Apply a function to a reduced block of arguments. 
    APPLY is a native! value.

ARGUMENTS:
    func    [word! path! any-function!] "Function to apply, with eventual refinements."
    args    [block!] "Block of args, reduced first."

REFINEMENTS:
    /all    => Provides a continuous list of arguments, tail-completed with false/none.
    /safer  => Forces single refinement arguments, skip them when inactive instead of evaluating.

Notice that the func arg can now be a word! or path!, so you can use the name, or a path including refinements. That's right, the Dynamic Refinements feature explained above works with apply too. And having access to the name being used to call the function is enormously valuable when it comes to tracing and debugging. It's a huge win. 

One big difference is that all the arguments for apply are in a single block. Another is that you MUST include the on/off values for each dynamic refinement in the arg block, they DO NOT come from the environment (context). Compare this version to the one in the Dynamic Refinements section. Really, paste them into an editor and look at them side by side. 

>> only: false  dup: false  apply 'append/:only/:dup [[] [1] only dup 3] 
== [1]
>> only: true  dup: false  apply 'append/:only/:dup [[] [1] only dup 3] 
== [[1]]
>> only: false  dup: true  apply 'append/:only/:dup [[] [1] only dup 3] 
== [1 1 1]
>> only: true  dup: true  apply 'append/:only/:dup [[] [1] only dup 3] 
== [[1] [1] [1]]

; Refinement names in the arg block don't have to match the spec.
; You can use other names, or literal values. For example:

apply 'append/:only/:dup [[] [1] false false 3] 

a: b: false  apply 'append/:only/:dup [[] [1] a b 3]

It means you have to be more careful in lining things up with the function spec, in a different way than you're used to, but here's where you could use a computed refinement value, which may be useful for generative scenarios like testing. You can also see that both refinements are dynamic, so both need an associated on/off value in the args block.

>> only: does [random true]
== func [][random true]
>> blk: copy [] dup: no  loop 10 [apply 'append/:only/:dup [blk [1] only dup none]]
== [1 [1] 1 [1] [1] 1 [1] [1] 1 1]

But if you use a fixed refinement, it does not need the extra on/off value. In this example, /dup is always used, so there is no on/off value for it in the args, block, but its associated count arg has to be there, and is type-checked normally.

>> only: does [random true]
== func [][random true]
>> blk: copy [] dup: no  loop 10 [apply 'append/:only/dup [blk [1] only 2]]
== [[1] [1] 1 1 1 1 [1] [1] [1] [1] 1 1 [1] [1] [1] [1] 1 1 [1] [1]]
>> blk: copy [] dup: no  loop 10 [apply 'append/:only/dup [blk [1] only none]]
*** Script Error: append does not allow none! for its count argument
*** Where: append
*** Near : apply 'append/:only/dup [blk [1] only none]
*** Stack: 

But those aren't your only options. During the design of apply there was a lot of discussion about the interface(s) to it, and different use cases benefit from different models. For example, programmatically constructed calls means you need to build the path, and keep the args in sync. It may be easier to build a single block with everything in it, which you can do.

>> only: does [random true]
== func [][random true]
>> blk: copy []  loop 5 [apply :append [blk [1] /only only /dup no none]]
== [1 1 [1] [1] 1]
>> blk: copy []  loop 5 [apply :append [blk [1] /only only /dup yes 2]]
== [[1] [1] 1 1 1 1 [1] [1] 1 1]
>> blk: copy []  loop 5 [apply 'append [blk [1] /only only /dup no none]]
== [1 1 [1] [1] 1]
>> blk: copy []  loop 5 [apply 'append [blk [1] /only only /dup yes 2]]
== [[1] [1] [1] [1] 1 1 1 1 [1] [1]]

This interface is used if the first argument to apply is a function or lit-word, and /all is not used.

Apply/all

This is the most direct model, and what the others map to internally. In those models, you get friendly refinements, which may be optional, and those may have their own optional args. It's great for humans, and one of the best things about Redbol languages. But look at it from the view of a function spec.

>> print mold spec-of :append
[
    {Inserts value(s) at series tail; returns series head} 
    series [series! bitset! port!] 
    value [any-type!] 
    /part "Limit the number of values inserted" 
    length [number! series!] 
    /only {Insert block types as single values (overrides /part)} 
    /dup "Duplicate the inserted values" 
    count [integer!] 
    return: [series! port! bitset!]
]

The doc string isn't used when calling functions, so we can ignore that for this discussion. We can also ignore return: here. What's left is all the parameters (we often interchange arg and parameter, but there's a technical difference. Parameters are the named slots in a function spec, and arguments are the actual values passed in those slots). There are seven of those. Some are required args, some are refinements, and some are optional args. But there are seven slots, and when a function is invoked it expects there to be seven values on the stack that it could use if needed, or ignored if not.

When you use /all, you're telling apply that you are going to provide all those values in a single block, in the order of the function spec, like a stack frame (don't worry about the terminology too much if it's unfamiliar). Apply/all calls look like this:

1.
>> blk: copy []  loop 5 [apply/all 'append [blk [1] false none false true 2]]
== [1 1 1 1 1 1 1 1 1 1]

2.
>> blk: copy []  loop 5 [apply/all 'append [blk [1] false none true true 2]]
== [[1] [1] [1] [1] [1] [1] [1] [1] [1] [1]]

3.
>> blk: copy []  loop 5 [apply/all 'append [blk [1] false none true false 2]]
== [[1] [1] [1] [1] [1]]

4.
>> blk: copy []  loop 5 [apply/all 'append [blk [1] false none only false none]]
== [1 [1] 1 1 [1]]

5.
>> blk: copy []  loop 5 [apply/all 'append [blk [1] false none 1 false none]]
*** Script Error: append does not allow integer! for its only argument
*** Where: append
*** Near : apply/all 'append [blk [1] false none 1 ]
*** Stack:  

6.
>> blk: copy []  loop 5 [apply/all 'append [blk [1] false none true true none]]
*** Script Error: append does not allow none! for its count argument
*** Where: append
*** Near : apply/all 'append [blk [1] false none true ]
*** Stack:  

7.
>> blk: copy []  loop 5 [apply/all 'append [blk [1]]]
== [1 1 1 1 1]

You can see that the refinement slots are now anonymous logic values in examples 1-3, but 4 uses only, our func from earlier examples, which randomly returns true or false. You can use anything that evaluates to logic! for a refinement slot. 5 shows that it has to be logic!, not just truthy, because types are checked (and logic refinement values then stand out against none argument values). And 6 shows that if you use /dup (second from the last arg), the count arg is also type checked, where 4 didn't complain because /dup was false. Confused yet? Look at 7. How can that work? I thought we had to fill all the slots! Yes and No. Apply "tail completes" the block with false/none values for you, if you don't provide enough arguments. Think of find, which has 16 slots. You only have to include enough args up to the last one you need. That may help, but if you need to use /match, the last refinement in find, you will have to provide all 16 args. Before you think this is unacceptable, consider our first repend example:

repend: func [
    {Appends a reduced value to a series and returns the series head} 
    series [series!] 
    value 
    /only "Appends a block value as a block"
][
    append/:only series reduce :value
]

It would look like this:

repend: func [
    {Appends a reduced value to a series and returns the series head} 
    series [series!] 
    value 
    /as-one "Appends a block value as a block"
][
    apply/all 'append [series reduce :value false none :as-one]
]

The point here is not to show that apply/all is longer, but that we can use a different name, where the first example must use :only in the path (make your own version to try it). Not all refinements will propagate using the same name. With /all it cares only about the logic value in the slot.

Apply/safer

/Safer is a form of short-circuit logic. Its purpose is to avoid the evaluation of unused args. Without it, everything in the args block is evaluated, but may be discarded if an associated refinement isn't active. The easiest way to explain this is with an example. 

; This is the function we're going to apply
applied: func [i [integer!] b /ref c1 /ref2 /ref3 c3 c4][
    reduce ['i i 'b b 'ref ref 'c1 c1 'ref2 ref2 'ref3 ref3 'c3 c3 'c4 c4]
]
; And some passive and active arg values
c: 0
bar40: does [4 * 2]
baz40: does [c: c + 1 456]

; Refinement args are evaluated
apply       'applied [10 "hi" /ref on bar40  /ref3 on  baz40            "ok"]

; No /safer difference because all refinement args are single values.
apply       'applied [10 "hi" /ref no bar40  /ref3 no  baz40            "ok"]
apply       'applied [10 "hi" /ref no bar40  /ref3 no  (c: c + 1 4 * 2) "ok"]
apply/safer 'applied [10 "hi" /ref no bar40  /ref3 no  (c: c + 1 4 * 2) "ok"]


apply       'applied [10 "hi" /ref no 4 * 2  /ref3 no  baz40            "ok"]
apply       'applied [10 "hi" /ref no bar40  /ref3 no  c: c + 1 4 * 2   "ok"]
apply/safer 'applied [10 "hi" /ref no 4 * 2  /ref3 no  baz40            "ok"]
apply/safer 'applied [10 "hi" /ref no bar40  /ref3 no  c: c + 1 4 * 2   "ok"]

In Real Life

Here are some examples of these new features being applied in the Red code base. The parse-trace example is jaw-dropping, not because it turns 9 lines into 1 (though, wow!), but because it makes the intent so much clearer and eliminates so much redundant code and the errors they can lead to. Not only that, it adds a capability! Before now you couldn't use both refinements together, i.e. parse-trace/case/part, but now you can.

Things we left out

The design of apply took many turns, with long and vigorous discussion and analysis. Many views and preferences were expressed, and which ultimately led to dynamic refinements as what we called straight sugar. That is, syntactic sugar at its sweetest. We knew /all had to be there, as that's what the others build on, but it was originally the default. We all eventually agreed that it shouldn't be, as it's the lowest level and likely the least directly used, though still vital for some use cases. Our problem was striking a balance between what would be most useful, with minimal overlap in use cases, and making the rules too complex to remember and get right. So a couple models didn't make the cut.

If Dynamic Refinements are straight sugar, the candy-wrapper version might be something like this, where you can still use a path, but only the args are in the block.

apply-args: function [
    "Make straight sugar call from semi-sweet model."
    fn [word! path!] "Get-word refinements come from context."
    args [block!]	 "Args only, no refinement values."
][
    ; Use temp result so we don't return any extra args
    do compose [res: (fn) (args)]
    res
]

>> apply-args 'append [[] [a]]
== [a]
>> apply-args 'append/only [[] [a]]
== [[a]]
>> only: false
== false
>> apply-arg 'append/:only [[] [a]]
== [a]
>> only: true
== true
>> apply-args 'append/:only [[] [a]]
== [[a]]

It's easy, but has quite a bit of overhead, because of compose and do. Remember, this amount of overhead only matters in loops running thousands of times at the very least, or in a real-time interactive interface. When in doubt, clock it. Write for humans to understand, and only optimize as needed (and after you know what's slow)

Another model is name-value args. That is, you provide a structure of arg names and their values, which is applied. It can make some code much clearer, but you also have to make sure the names match, so any refactoring of names will break code, which doesn't happen if args are positional. This is a bit involved, but it shows the power of Red. We'll use objects in our example, for a particular reason. That reason is values-of. The idea being that apply/all wants all the args, in order, and every slot filled. If your object matches the function spec, it's a perfect match. But making objects manually that way is error prone. So we'll use reflection for an object tailor-made for a given function.

Step 1: Find all the words in the spec. Remember it could have doc strings and arg types as well.

func-spec-words: function [
    "Get all the word-type values from a func spec."
    fn [any-function!]
    /as type [datatype!] "Cast results to this type."
][
    arg-types: make typeset! [word! lit-word! get-word! refinement!]
    parse spec-of :fn [
        ; If we want an apply-specific version of objects, we could
        ; denote refinements with a sigil for added clarity.
        collect [
            any [set w arg-types keep (either type [to type w][w]) | skip]
        ]
    ]
]

Step 2:  Make an object from that

func-spec-to-obj-proto: function [
    "Returns an object whose words match a function's spec."
    fn [any-function!]
    ; The idea here is that you can both preset values that are in the spec,
    ; and also extend the object with extra words, which will be ignored.
    /with args [block!] "APPLY/ALL ignores extra values."
][
    obj: construct/with any [args []]
    construct append func-spec-words/as :fn set-word! none
    ; Refinement values in APPLY/ALL calls MUST be logic!, not none!.
    foreach word func-spec-words :fn [
        if refinement? word [set in obj to word! word false]
    ]
    obj
]

Step Aside: Here's another approach, which combines steps 1 and 2, and lets you use a path for the function arg.

; Alt approach to func-spec-to-obj-proto, does NOT allow extending the spec.
make-apply-obj-proto: function [
    "Returns an object whose words match a function's spec."
    fn [any-function! word! path!]
    /with args [block!] "TBD: If APPLY doesn't ignore extra values, keys must be in spec."
][
    if path? :fn [refs: next fn   fn: first fn]        ; split path
    if word? :fn [
        name: fn                                       ; hold on to this for error report
        set/any 'fn get/any fn                         ; get func value
        if not any-function? :fn [
            do make error! rejoin ["FN argument (" name ") does not refer to a function."]
        ]
    ]                                                  ; get func value
    obj: construct append func-spec-words/as :fn set-word! none	; make object
    ; Refinement values in apply calls MUST be logic!, not none!.
    foreach word func-spec-words :fn [
        if refinement? word [set in obj to word! word false]
    ]
    if refs [foreach ref refs [obj/:ref: true]]        ; set refinements
    ; can't use obj/:key: if key is a set-word!
    if args [foreach [key val] args [set in obj key :val]]  ; set arg values
    obj
]

o-1: make-apply-obj-proto/with 'find/case/part/tail/skip/with [wild: "*?+" length: 10 size: 2]

Step 3. Using the object with APPLY

; We finally get here, and it's anticlimactic.
apply-object: func [
    "Call APPLY using an object's values as args."
    fn  [any-function!]
    obj [object!]
][
    apply/all :fn values-of obj
]

And an example call:

>> o-fctm: make-apply-obj-proto/with 'find/case/tail/match [series: [a b c] value: 'a]
== make object! [
    series: [a b c]
    value: 'a
    part: false
    length: none
    only: false
    case: true
    same: fal...
>> apply-object :find o-fctm
== [b c]

But you may see that this is verbose and inefficient, making a whole object just for a call like this. And you'd be right. It's just an example.

You don't want to recreate objects like this, especially in a loop. But you don't have to. You can reuse the object and just change the important values in it. This blog is getting long already, so we'll leave that as an exercise for the reader, or a question in community chat. And if you reuse the same object, the overhead is minimal.

We even talked about an idea whose time has not come, and is not guaranteed to work in the future. Here's the idea:

dyna-ref: func [p [path!]][
    res: make path! collect [
        keep p/1
        foreach val next p [
            case [
                get-word? val [if get :val [keep to word! val]]
                all [paren? val get-word? val/1] [if do next val [keep to word! val/1]]
                paren? val [do make error! "Sorry, has to be (get-word expr) for use with dyna-path."]
                'else [keep val]
            ]
        ]
    ]
    res
]

c: true
print mold dyna-ref 'a/b/:c/(:d true)
print mold dyna-ref 'a/b/:c/(:d false)
c: false
print mold dyna-ref 'a/b/:c/(:d true)
print mold dyna-ref 'a/b/:c/(:d false)

That's right, it's a dialected path! that builds a dynamic path. Crazy, right? You may know that while paths can currently contain parens, for Rebol compatibility, that feature may go away. It has deep design implications, but is also very handy at times. And while this isn't part of Red, it's another example of how Red lets us think off the beaten path.

Interpreter improvements

Dynamic refinements and function application are supported at interpreter level, for maximum efficiency and code reuse (mostly arguments fetching and type-checking). In addition to that, long standing issues and needed simplifications have been made in the interpreter code. Here is the changelog:

Function arguments cache entirely reimplemented:

• Massively reduces the amount of code needed to manage the caches.
• Simpler and faster cache design, O(1) lookup time for refinements in paths.
• Adds a context! to native!, action! and routine!, to speed up word lookups.
• Fixes long standing issue #4854 (CRASH & CORRUPTION in "dynamic" function calls with refinements)

Simpler reimplementation of op! datatype and its evaluation:

• red-op! structure redesigned: it is now a shallow copy of the underlying function (nodes are not copied), the sub-type is stored in the op! header.
• is infix function has been deprecated and replaced by a prefix version: relate.
• Massive code reduction and simplification compared to previous version. Now op! maximally reuses the interpreter code for evaluating other functions.

Those changes lead to a general interpreter speed-up of 3-5% and up to 20% in some cases.

Additional language changes:

• The in native now accepts any-function! as its first argument and refinements as the second argument. Refinements, if matched, will be converted to word values. This makes in a fast way to check if a symbol is part of an object or function spec and return a word bound to that context.

Conclusion

The most important thing you should do now is try it. It's a new design, and we want to hear what people like, see what they try, and where it falls short.

Happy Reducing!

-Red Team

New Red binaries

Since many years, we are offering pre-built binaries for the Red toolchain, as a more convenient way to use Red, even if it is not strictly needed, as Red can be run from its sources, the toolchain being run by a Rebol2 interpreter. As the Red REPL and toolchain are not run by the same engine, the console (REPL) used to be compiled on first run of the `red` executable (when no arguments was provided or a Red script was passed). This resulted in a significant delay on the first use of the console (both for the GUI and CLI versions). 

We have now decided to change that by providing separate pre-built binaries for the consoles and toolchain. This is a temporary split until Red gets self-hosted, at which point we can recombine everything into a single binary.

Another change is the temporary dropping of the semantic versioning until version 1.0 and related "stable" releases, as it seems to be too confusing to some users (Red being still in alpha stage). This also will remove a tendency from some users to care more about version increments than feature availability and work being done overall. We will now be proposing only pre-built binaries for latest commit, though older binaries will still be available if that can be of any help to anyone.

So the pre-built binaries now are:

• Red GUI : Red interpreter + View + GUI console
• Red CLI : Red interpreter + CLI console
• Red Toolchain : Encapper for Red + Red/System compiler

We are also considering ways to merge the GUI and CLI consoles into a single binary which can work even if no GUI API is available, falling back on CLI mode. We will also have the console(s) act as a front-end for the toolchain, even downloading it for you in the background when needed. Though for that we need a proper asynchronous `call` function implementation. More news about this soon.

In the meantime, enjoy running Red consoles almost instantly from just a click on the Download page!

The Road To 1.0

You cannot have missed that in the last months (and even last years), our overall progress has slowed down drastically. One of the main reasons is that we have spread our limited resources chasing different objectives while making little progress on the core language. That is not satisfying at all and would bring us most likely to a dead-end as we exhaust our funding. We have spent the last weeks discussing about how to change that. This is our updated action plan.

From now on, our only focus will be to finish the core language and bring it to the much-awaited version 1.0. We need to reach that point in order to kickstart a broader adoption and provide us and our users a stable and robust foundation upon which we can build commercial products and services necessary for sustainability.

Given the complexities involved in completing the language and bringing an implementation that can run on modern 64-bit platforms, we have devised a two-stage plan.

Upgrade the current 32-bit Red implementation

👉 Language specification

It is now time to do so in order to clean-up some semantic rules and address all possible edge cases which will help fulfill our goals of implementation robustness and stability. The process of writing down the complete language specs will result in dropping some features that we currently have that end up being problematic or inconsistent. OTOH, we might add some new features that will need to be implemented for 1.0.

👉 Modules

We need a proper module system in order to be scalable. We also need to have a proper package management system which will be tied to a central repo where we can gather third-party libraries. That would also enable modular/incremental compilation (or encapping) which will be most probably supported in the self-hosted toolchain.

👉 Concurrency

We need a proper model for concurrent execution in order to leverage multicore architectures. We will define one and make a prototype implementation in the 32-bit version.

👉 Toolchain

Before starting to work on the new toolchain, we will make some changes to the existing version in order to prepare for the transition. The biggest change is the dropping of the Red compiler, which will only act as a (smart) encapper. Routines and #system directives will still be supported, but probably with some restrictions. The Red preprocessor might also see some changes. This change means that Red will only have one execution model instead of the two it has currently. The Red compiler has become more of a burden than a help. The speed gains are not that significant in real code (even if they can be in some micro-benchmarks), but the impossibility for the compiler to support the exact same semantics as the interpreter is a bigger problem. This move not only will bring more stability by eliminating some edge case issues but also will reduce the toolchain by almost 25% in size, which will help reduce the number of features to support in the new toolchain.

👉 Runtime library

Some improvements are long overdue in the Red runtime library. Among them:

• unified Red evaluation stack.
• unified node! management.
• improved processing of path calls with refinements.
• improved object! semantics.

All those changes are meant to simplify, reduce the runtime library code and address some systemic issues (e.g. stack management issues and GC node leaks).

👉 Documentation

We need proper, exhaustive, user-oriented documentation for the Red core language. This is one of the mandatory tasks that needs to be completed and done well for wider adoption.

Self-hosted Red for 64-bit version

👉 Toolchain

In order to go 64-bit, we have to drop entirely our current toolchain code based on Rebol2 and rewrite it with a newer architecture in Red itself. The current toolchain code was disposable anyway, it was not meant to live this long, so this was a move we had to do for 1.0 anyway.

So the new toolchain will feature:

• a new compilation pipeline with a plugin model.
• an IR layer.
• one or more optimizing layers.
• modular/incremental compilation support.
• x64, AArch64 and WASM backends.
• linker support for 64-bit executable file formats for the big 3 OS.
• support for linking third-party static libraries.

32-bit backends will not be supported in 1.0, though, they could be added back in the future.

👉 Runtime library

The current Red runtime library written in R/S will be kept and some adjustments will be needed in order to be fully compatible with a 64-bit environment (like updating all imported OS API to their 64-bit versions). 

View engine will not be part of that upgrade for 1.0, but will be done in a 1.1 version, priority is given to Red/Core for the 1.0.

Roadmap

Here are the main milestones:

• v0.7   : Full I/O with async support.
• v1.0b : (beta) completed self-hosted Red with 64-bit support.
• v1.0r  : (release) first official stable and complete Red/Core language release.
• v1.1   : View 64-bit release.
• v1.2   : Android backend and toolchain release.
• v1.3   : Red/C3 release.
• v1.4   : Web backend for View release.
• v2.0   : Red JIT-compiler release.
• v3.0   : Red/...

The 0.7 should be the last version for the 32-bit Red version and current toolchain and we will be working on that first.

For reaching the 1.0-beta milestone, we target 12 months of intensive work, so that will bring us to Q3 2023. That's an ambitious goal but necessary to reach for the sake of Red's future.

The currently planned beta period for 1.0 is 2-3 months. We want a polished, rock-solid, production-ready 1.0 release.

For the 1.1, we will probably make some (needed) improvements to View engine architecture and backends.

For Red/C3, as the Ethereum network is transitioning to 2.0 and a new EVM, we need the WASM backend in order to support it.

Version 1.4 will bring a proper web runtime environment to the WASM backend, including GUI support.

The 2.0 will be focused on bringing a proper JIT-compiler to Red runtime, that should radically improve code execution of critical parts without having to drop to R/S.

Version 3.0 is already planned, but I will announce that once 1.0 will be released. ;-)

One major platform is missing from the above plan, that is iOS. Given how closed that platform is, we will need to come up with a specific plan on how to support it, as it won't be able to cross-compile for it (you would need a Mac computer), nor probably generate iOS apps without relying on Xcode at some point (not even mentioning dynamic code restrictions on the AppStore), which are layers of complexity that Red is trying to fight against in the first place... So for now, that platform is not among our priorities.

To finish, let me borrow some words from someone who succeeded more than anyone else in our industry:

Expect me to say "no" even more so from now on, as we get laser-focused on our primary goal.

Cheers and let's go!

2021 Winding Down

Another quarter, another blog post. Seems almost rushed after the previous drought. 

To set the stage, I'll start with a bit of a rant about complexity. If you just want the meat of what's happening in the Red world, feel free to skip the introduction. 

Complexity Considerations: Part 1

I liked what the InfoWorld article, Complexity is Killing Software Developers said, which we all know, about difficult domains (voice and image recognition, etc.) being available as APIs. This lets us tackle things we couldn't in some cases. Though I imagine @dockimbel or others also used Dragon Dictate's libraries back in the 90s. What we have now is massive data to train systems like that. Those work well, allowing us to add features we otherwise couldn't with a small team.

The problem I see is that the trend has become for everything to be outsourced, including simple features like logging, and those libraries have exploded. There must be graphs available to show the change. Moderately complex domains, UIs for example, have risen in number and lead to what @hiiamboris says about Brownian Movement. It's a random collection of things, not designed to work together, without a coherent vision. A quote from the above article says it this way:

"Complexity is less the issue than inconsistency in an environment."

It used to be that you could take a FORTRAN, COBOL, Lisp, VB, Pascal/Delphi, Access/PowerBuilder, dBase/Clipper/Paradox, or even a Java developer, drop them into a project, and they could work from a solid core, learning the team's custom bits and any commercial tools as they went. With JS leading the way, but not alone in this, a programmer can only rely on a much smaller core, relative to how many libraries are used.

Because those libraries, and the choices to use a particular combination of them were not designed to work together, there is no guarantee (or perhaps hope) of consistency to leverage. It's worse if you came from a history of other tools that were based on different principles or priorities, because you have to unlearn, breaking the patterns in your mind. Or you convince people to use what you did before, even if there is overlap with tools already in use.

Things are changing now, and will even more. New service-based companies are coming, and a drive to APIs rather than libraries. So we not only have risks like LeftPad, but also companies going out of business under you. The modern trend means it's no longer dependent on an author or team committed to a project long term, but to what investors want, and what changes are made to gain adoption at all costs. As a service-based company you can't hold dearly to design principles if the investors tell you to pivot. Because it's no longer about your vision, but their return. If it is a solo FOSS author or small team, what is their incentive to maintain a project for free, while others profit from it? Success can be your worst enemy, and we need a more equitable solution than what we have now. The software business model has changed dramatically, and will likely continue to do so.

Here is what I personally see as the crux of the problem: the goal of scaling. FOSS projects and companies are only considered successful if they have millions (or, indirectly, billions) of users. Companies that want to be sustainable, providing long term, moderate profits don't make headlines, but they make the world go 'round. They are not the next big social media disruption where end users are the product, to be bought and sold. It is a popular business model and profit is the goal. It's nothing personal.

This has led us to the thinking that every project needs to be designed for millions of users at the very least. Sub-second telemetry for all the data collected, another explosion, giving rise to data analytics for everyone; not just Business Intelligence (BI) for large companies. I won't argue against having data. I love data and learning from it. But I do believe there is a point of diminishing returns which is often ignored. Rather, in this case, there is a cost of entry that small projects wouldn't otherwise need to pay.

What do you do, as an "architect" (see the previous blog post about my thoughts on software architecture) or developer on a team? Your small team (we all know small teams are best, plenty of research and history there) simply can't design and build every piece to support these scaling demands, while the sword of Damocles hangs over you in the form of potential pivots (dramatic changes in goals).

As an industry, we are being inexorably forced to make these choices. Either you're a leader and make your own Faustian bargain, or you're in the general mass of developers being whipped and driven to the gates of Hell.

Only you, dear reader, can decide the turns this tragic story will take, and what you forgive in this telling perchance I should exaggerate.

Complexity Considerations: Part 2

Complexity doesn't come only in the form McCabe is famous for, the decision points in a piece of code, but in how many pieces there are and how often they change either by choice or necessity. Temporal Complexity if you will. This concept is unrelated to algorithmic time complexity. Rebol2 for any faults we can point out, still works to this day (except in cases where the world changed out from under it, e.g., in protocols). It was self-contained, and relied only on what the OS (Operating System) provided. As long as OSs don't break a core set of functionality that tools rely on, things keep working. R2 had a full GUI system (non-native, which insulated it from changes there), and I can only smile when I run code that is 20 years old and it works flawlessly. If that sounds silly, remember that technology, in most cases, is not the goal. It is a means to an end. A lot of very old code is still in production, keeping businesses running.

We talk about needing to keep up with changes, but some things don't change very much, if at all. Other things change rapidly, but for no good reason, and without being an improvement. If a change is just a lateral move there is no value in it, unless it is to align us on a different, and better, path in the future. I started programming with QuickBASIC, but also used other tools as I quickly learned my tool of choice came with a stigma attached, and I wanted to be a serious, "real" programmer. What became clear was that QB was a great tool, with a few companies providing terrific ASM libraries, and had a wonderful IDE to boot. It was simpler, not only as a language, but because every 12-18 months (the release cycle way back when) my new C compiler would break something in my code. But QB, and later BASIC/PDS and then VB very rarely broke working code. Temporal complexity.

Even then there were more complex options. The cool kids used Zortech C++ and there were various cross-platform GUI toolkits. But those advanced tools were often misapplied to simple projects. We still do that today. Much of that is human nature, and the nature of programmers. If it's easy we are no longer special. We may not mean to, but we make things harder than they need to be. Some of us are even elitist about what we do, to our own detriment. If you don't need to be cross platform, why do you have multiple machines or VMs each with a different compiler setup? If you need a GUI, why are you using a language that was not designed with them in mind? If you need easy deployment, which is simpler: a single EXE with no dependencies, or a containerization approach with all that entails? How many technologies do you need in your web stack? Are you the victim of peer pressure, where you feel your site has to be shiny and "responsive", or use the latest framework?

A big argument for using other's work is performance. They've taken time, and may be experts, to optimize Thing X far beyond what you could ever do. That JIT compiler, an incredible virtual DOM, such clever CSS tricks, the key-value DB with no limits, and yet...and yet our software is slower and more bloated than ever. How can that be? Is it possible we're overbuilding? Is software sprawl just something we accept now?

Earlier I mentioned that a hodge-podge assembly of parts that have no standards, norms, or even aesthetic sense applied does not make our lives easier. Lego blocks, the originals anyway, are limited, but consistent in how they can be used. We misapply that analogy, because the things we build are far from consistent or designed to interact. Even in the realm of UX and A/B testing on subsets of users that companies apply today. I love the idea of data-driven HCI to guide us to a more evidence-oriented approach. This includes languages. But when a site or service moves fast and changes their interface based on their own A/B testing, they don't account for the others doing the same. Temporal complexity.

As a user, every app or site I access may change out from under me in the flash of refresh or automatic update I didn't ask for. Maybe it's better, an actual improvement, if you only use that one site. But if all your tools constantly change out from under you, it's like someone sneaking into your office and rearranging it every night while you sleep. Maybe this is the developer's revenge, for the pain we inflict on ourselves by constantly changing our own tools. If we suffer, why shouldn't our users? For those who truly have empathy for their users and don't want to drive them mad, or away, perhaps the lesson is to have empathy for ourselves, for our own tribe. I don't want to see my friends and colleagues burn out, when it was probably the enjoyment and passion that solving problems with software can bring which led them here to begin with.

Every moving part in your system is a potential point of failure. Reduce the moving parts and reliability increases. Whether it's the OS you run on (we now have more of those than ever, between Linux distros and mobile platforms always trying to outdo each other), extra packages or commercial tools, FOSS libraries, environments, [?]aaS, or platform components like containers and cluster management, every single piece is a point of failure. And if any of them break your code, or your system, even in the name of improvements or bug fixes, you may find yourself running just to stay in the same place. Many of those pieces are touted as the solution to reliability problems, but a lot of them just push problems around, or target problems you don't have. Don't solve problems you don't have. That adds complexity, and now you really have a problem.

Less Philosophy, More Red

Interpreter Events

Having a debugger in Red has been a request of many users for a long time, even since the Rebol era. We have tackled this feature from a larger perspective, considering general instrumentation of the interpreter (note: not the compiler), extending it with an event system and user-provided event handlers, similar to how parse and lexer tracing operate today. This approach allows us to build more than just a debugger, though it was a lot of work to design and we expect it will be refined once people start using it in earnest. It's a brave new world, with a lot of tooling possibilities.

It's important to note that this is not magic. Because it operates as the interpreter evaluates values and expressions, including functions, it can't see into the future. In order to get a complete trace, you have to evaluate everything. That means we'll see tools which silently collect data, like a profiler does, which can later be viewed and analyzed, perhaps up to the point where an error occurred. This is an important aspect, and plays once again into the power of Red as data. Your event handlers can easily collect data into any structure or model you like. And because event handlers can filter events, you can tailor them for specific needs. It should even be possible to build interpreter level DTrace-like tools in the future. We also hope to build higher level observability and monitoring tools, based on eventing systems, in the future, but those are long term projects.

Event generation is not active by default, it is enabled using do/trace and by providing an event handler function. For example, here's a simple logging function:

  logger: function [
      event  [word!]                      ;-- Event name
      code   [any-block! none!]           ;-- Currently evaluated block
      offset [integer!]                   ;-- Offset in evaluated block
      value  [any-type!]                  ;-- Value currently processed
      ref    [any-type!]                  ;-- Reference of current call
      frame  [pair!]                      ;-- Stack frame start/top positions
  ][
      print [
          pad uppercase form event 8
          mold/part/flat either any-function? :value [:ref][:value] 20
      ]
  ]

Given this code:

  do/trace [print 1 + 2] :logger

It will output:

  INIT    none                    ;-- Initializing tracing mode
  ENTER   none                    ;-- Entering block to evaluate
  FETCH   print                   ;-- Fetching and evaluating `print` value
  OPEN    print                   ;-- Results in opening a new call stack frame
  FETCH   +                       ;-- Fetching and evaluating `+` infix operator
  OPEN    +                       ;-- Results in opening a new call stack frame
  FETCH   1                       ;-- Fetching left operand `1`
  PUSH    1                       ;-- Pushing integer! value `1` on stack
  FETCH   2                       ;-- Fetching and evaluating right operand
  PUSH    2                       ;-- Pushing integer! value `2`
  CALL    +                       ;-- Calling `+` operator
  RETURN  3                       ;-- Returning the resulting value
  CALL    print                   ;-- Calling `print`
  3                               ;-- Outputting 3
  RETURN  unset                   ;-- Returning the resulting value
  EXIT    none                    ;-- Exiting evaluated block
  END     none                    ;-- Ending tracing mode

Several tools are now provided in the Red runtime library, built on top of this event system:

• An interactive debugger console, with many capabilities (step by step evaluation, a flexible breakpoint system, and call stack visualisation).
• A simple profiler that we will improve over time (especially on the accuracy aspects).
• A simple tracer. The current evaluation steps are quite low-level, but @hiiamboris has already built an extended version, operating at the expression level that will soon be integrated into the master branch.

Full docs are here.

Format

Boy, I really thought this was was going to be easy, or at least not too hard. I couldn't have been more wrong. When I did my format experiments, I imagined at least some of the code would be useful, requiring polish and more work of course, providing a foundation to work from. It turns out that I missed a key aspect, and my approach was just one of many possible. @hiiamboris and @giesse both weighed in, and we chatted about specific parts. Then it sat idle for a while, and I asked Boris to take it over to get it into production. He identified the key missing piece, which would have limited its usefulness until we eventually had to address it. Better now than later. He also made a strong case for a different approach to the core masked-number and I told him to run with it. That led to a lot of design chat about one aspect, which is as yet undecided. It's not a fight to the death, but there has definitely been some sparring. :^)

The missing piece I've alluded to is Localization (L10N). As an American who has never had to develop software requiring Internationalization (I18N), I've been blissfully ignorant of all the aspects that come into play when Globalization (G11N) becomes part of the process. We have talked about how to implement L10N in Red, and have system/locale for a months, weekdays, and currency codes. The first two we inherited from Rebol's design, the latter was added when @9214 designed the currency! datatype. Thinking of locale data in a system catalog of some kind is easy enough, but how to actually apply it (and not apply it when necessary) is a different story entirely. And I mean entirely. Format forced us to start down this path, and is a guinea pig feature that will guide future plans for all future L10N work. But keep my complexity rant in mind. While we want to make it as easy as possible for Reducers to write globally aware apps, if you don't need it, don't do it. We don't yet know if we can make it so magical that you can write your app ignoring that for the most part, and then flip a switch, or simply include local data, and have it work. Don't get your hopes up. There's a lot that can go wrong with that approach.

We agreed to start with masked numbers but, in order to do that, L10N R&D had to be done. This led to broad and deep dives into unicode.org and other resources. While they cover far more than we need, and is overly complex in many cases (or just doesn't match our aesthetic sense for Red), the data they have there is enormously valuable, and we deeply appreciate it being available. We just draw the line around a smaller scope than they do, and no committees are involved where people fight to get their own bits included. Well, we do that too, to some extent. What Boris managed to do was identify the key elements needed for our work, and then wrote tools (using Red of course) to extract and reformat the data for use in Red. I can't stress how much work this was. Truly a heroic and mostly thankless effort most people will never know about.

In order to test masked number formatting, and give others an easy way to play, Boris created a Playground App and I can't tell you how important that was. You see, a particular piece of behavior came up while I was playing with it and got unexpected results. Unexpected to me, but Boris confirmed it was by design. I will just say here that it's about a significant digits mode, and let you play with the app from there. Named formats will be available, but everything will likely boil down to wrappers around masks, which should cover almost any need.

Next up is date formatting. This time I knew locales would play a role because some IETF RFCs specify that date elements be in English. So you may have localized dates for some things, but if you use RFC2822 dates or HTTP cookie dates, they must not be affected by any locale settings. Dates will use masks at the core, like numbers, because masks are an easy to understand WYSIWYG format. Well, easy if the masks make sense. If you look at printf and some other mask syntax, it can be quite obscure. By trying to cram things into a limited syntax, people end up using whatever low ASCII letters might be left over for some elements. We hope to avoid that. 

Our main choices are what Boris termed the stuttering format. e.g. MMDDYYYY/HHMMSS. Think in terms of "progressing in a hesitant or irregular way." rather than stuttering in terms of human speech. I prefer to call this a symbolic format, where the letters map to date elements. This, of course, isn't perfect. e.g. is MM month or minute? Context is required. We don't want to be case sensitive, or use other letters randomly to avoid that conflict. So there's an alternate approach; a literal mask. e.g. 1-Jan-2022. We're not the first to consider it, and it is in use elsewhere, but it's not a perfect solution either. Do masks have to be written in English terms, or can they use any locale? How do you disambiguate numbers (does 01-01 mean MM-DD or DD-MM, and how do you write that without the separator to get MMDD?) Does it make code more or less readable, because Red already has a literal date form, and it would add what look like literal dates as strings in code.

Play with the app, give us feedback, and stay tuned. We think this will be a crucial feature for a lot of users, and we want to make it the best it can be.

Split

Like format, split seems a relatively simple subject at a glance. And if you limit it to basic functionality, it is. That's what other languages do, though some add a few extra features. See this table for examples. Wolfram appears quite broad in scope, because there are multiple variants for each named function. Something else common to all other languages is that they split only strings and sometimes byte arrays. In Red we have blocks, and while `parse` is great for string parsing, where it really shines is when applied to blocks to build dialects. We knew split should be block aware for more leverage. I (Gregg) helped design the version in R3, and used DiaGrammar to design a new dialected interface that aimed to extend the functionality. Wanting to do more evidence based language design, I also prototyped a small practice/playground app, thinking we'd put it out and see what kind of feedback we could get. 

Toomas stepped up and suggested an alternative, refinement-based, interface. He did a number of versions of that, and then we had to decide what to do next. There was a great deal of design discussion, still going on, about behavior details. Once you start adding options, it's easy for things to become confusing for the user. We need to strike a balance between ease of use and flexibility. Split is meant to handle the most common cases, and those with the most leverage, not every case. And while a refinement-based interface seems natural for Reducers, we also know how readable parse, draw, and VID dialects are. There are pros and cons to each, but we don't want dual interfaces, which will be confusing. If a function is dialected, any refinements should work in support of that dialect. So the test app was reimagined by @GalenIvanov to compare the two approaches.

Here's a screenshot of the test app, which we'll release to the community in January.

We learned by doing this that it's hard to compare them side by side, without having the user write full calls directly. That defeats some of the purpose, and the DRY principle, so we'll put this one out, then revise it based on feedback.

Markup Codec

Who knew that parsing HTML and XML would be the easy part? Well, many Reducers would. What they, and we on the team, might not have guessed, is just how hard it is to decide on a data format for the output. Red gives us many options, and XML gives us many headaches. The two formats, while closely related, also have some critical differences. Fortunately, once @rebolek set things up so we could play, and made the emitter modular, we could look at real examples and dive even deeper. What we discovered is that there is no perfect solution. No elegant model to fit all uses and cases. Key to many insights was @dander's input, as he works with XML a lot. Turns out, an infinitely extensible format is infinitely challenging to nail down.

Should we emphasize path access? Being data driven, people probably shouldn't hardcode their field names, but working with known data makes it a clear access model. Should attributes come before or after the text/content for a tag? As we learned, attributes aren't always small, so the locality argument isn't won either way there. Is it better to provide an interface to the structure and tell people to always use that, or to create a bland and obvious data structure that is possible to access in many ways? Will these things all complicate HOF access, which we know we want to leverage? How much do we need to care about efficiency? We don't want to be wasteful without purpose, but if we're too miserly, users may pay the price because it's harder to use. If we make more things implicit, do we paint ourselves into a corner somehow?

What we settled on was a modular approach, so there will be more than one standard emitter. What is yet undecided is how other emitters might be supported. They will likely be quite custom, as the standard versions will cover most needs. But is it worth making the system extensible? Once you have a result, it's easy to post-process into your preferred format. For now that's our recommended approach.

CLI Module

If you don't follow our channels on Gitter, you may not know about Boris' CLI module. It's very slick, very Reddish, and will become a standard part of Red in the near future. You won't believe how easy it is to create rich command line interfaces for your Red apps with this feature. Huge thanks to @hiiamboris for all his innovation and work on it.

IPv6 Datatype

It hasn't been merged to the mainline yet, but it's fully operational. You can see the code here, and some lexer tests here. You may be impressed that it's only a couple hundred lines of code, not counting the lexer changes, and think it was easy. It wasn't. As usual, there was a lot of design chat and compromise involved. For example, the name is not 100% finalized because, technically, the datatype itself is more generally applicable, being simply a vector of numbers internally. You can think of it like a tuple! on steroids. Less slots (8 vs 12), but each slot can hold a larger value (tuple! slots are limited to byte values).

Just as tuple! is a general name, used both for IPv4 addresses and colors, but also useful for other things, IPv6! could be used for things like GUIDs or extended time values. But the lexical form for GUID/UUID values is quite different, even ignoring the shortcut forms in the IPv6 specification. As you probably know, lexical space is tight in Red, and the colon is an important character in other places, and URL lexical forms were impacted, so this is a deep change and commitment, in that regard. Why do it then?

Because IPv6 networking support was already in place in Red, and IPv6 is the future. How often people will write literal URLs like http://[FEDC:BA98:7654:3210:FEDC:BA98:7654:3210]:80/index.html we can't say. But we do know that addresses often end up in config files as data and that modern, dynamic systems generate addresses dynamically. They will appear in log files, messages, and more. As with the value of other lexical forms in Red, it's an important one that is part of our modern networking vocabulary.

Getting Near

@dockimbel created a new branch here, which will interest almost every Reducer. It's not ready yet, but expect it to be available in January. For those who used R2, you may recall that errors gave you a Near field, to hint at where the error occurred. Red will get this feature when the new branch is merged. e.g., in Red today you get this:

    >> 1 / 0

    *** Math Error: attempt to divide by zero

    *** Where: /

    *** Stack:  

Where in R2 you got this:

    >> 1 / 0

    ** Math Error: Attempt to divide by zero

    ** Near: 1 / 0

A little extra information goes a long way. We're anxious to see all the virtual smiles this features brings.

The Daily Grind

We closed roughly 120 tickets in 2021, that's 10 per month. We also merged almost 50 PRs. These numbers don't sound large, but when you consider how much time and effort may go into the deep ones, along with all the other work done, it's steady progress. We'd love for both tickets and pending PRs to be at zero, but that's not practical for a project like Red. The deep core team must have uninterrupted time for design and bigger, more complex tasks.

Roadmap

Q4 2021 (retrospective)

• We hoped to have `format` and `split` deployed, but they will push back to Jan-2022.
• `CLI` module approved, needs to be merged, then refined as necessary.
• `Markup Codec` took longer than expected due to extensive design chat on formats.
• Interpreter instrumentation, with PoC debugger and profiler. Took longer than expected, but are out now.
• Async I/O, out but some extra bits didn't make it in. One unplanned addition was `IPv6!` as a datatype. It's experimental, and subject to change.
• @galenivanov did some great work on his animation dialect, but @toomasv's `diagram` dialect took a back seat and will move to Q1 2022.
• Audio has 3 working back ends and a basic port implementation. Next up is higher level design, device and format enumeration, and device control. A `port!` may not be the way to go for all this, but it was step one.
• Animation has more great examples all the time. Like this and this. @GalenIvanov is doing great work, and we are planning to make his dialect a standard addition to Red.

2022

I'm not going to list items in any particular order, because our plans often change. This way you have things to look forward to, but still with an element of surprise.

• `Table` module, `node!` datatype and other REP reviews
• Full HTTP/S protocol and basic web server framework
• New DiaGrammar release
• Animation dialect
• New release process
• New web sites updated and live
• Red/C3 (Including ETH 2.0 client protocol)
• Red Language Specification (Principles, Core Language, Evaluation Rules, Datatype Specs (including literal forms), Action/Native specs, Modules spec.
• 64-bit support (LLVM was a possibility, but we learned from Zig that LLVM breaking changes can be quite painful for small teams to keep up with. We may be better off continuing to roll our own, though it's a big task.)
• Android update
• Red Spaces cross-platform GUI
• Module and package system design
• RAPIDE (Rapid API Development Environment)

RAPIDE, from Redlake Technologies

If you've used Postman or Insomnia, you know what the most popular tools in the API IDE space look like today. If you haven't used them, but use APIs, they're worth a look. For all that those tools do, and there are a few other players in the space, there is a lot they don't do. We think we can add a lot of value in the API arena, thanks to Red's superpowers and how important data-centric thinking is. For example, testing a group or series of APIs together seems like it could be greatly improved. Also, how APIs are found, and collaboration possibilities.

While we haven't set a release date, the plan is to start work on RAPIDE in Q2 2022, after we wrap up some infrastructure pieces it will rely on. 

In conclusion

Happy New Year to all, and may 2022 see us all healthy, happy, and writing more Red. :^)