July 14, 2022

The Road To 1.0

You cannot have missed that in the last months (and even last years), our overall progress has slowed down drastically. One of the main reasons is that we have spread our limited resources chasing different objectives while making little progress on the core language. That is not satisfying at all and would bring us most likely to a dead-end as we exhaust our funding. We have spent the last weeks discussing about how to change that. This is our updated action plan.

From now on, our only focus will be to finish the core language and bring it to the much-awaited version 1.0. We need to reach that point in order to kickstart a broader adoption and provide us and our users a stable and robust foundation upon which we can build commercial products and services necessary for sustainability.

Given the complexities involved in completing the language and bringing an implementation that can run on modern 64-bit platforms, we have devised a two-stage plan.


Upgrade the current 32-bit Red implementation


👉 Language specification

It is now time to do so in order to clean-up some semantic rules and address all possible edge cases which will help fulfill our goals of implementation robustness and stability. The process of writing down the complete language specs will result in dropping some features that we currently have that end up being problematic or inconsistent. OTOH, we might add some new features that will need to be implemented for 1.0.

👉 Modules

We need a proper module system in order to be scalable. We also need to have a proper package management system which will be tied to a central repo where we can gather third-party libraries. That would also enable modular/incremental compilation (or encapping) which will be most probably supported in the self-hosted toolchain.

👉 Concurrency

We need a proper model for concurrent execution in order to leverage multicore architectures. We will define one and make a prototype implementation in the 32-bit version.

👉 Toolchain

Before starting to work on the new toolchain, we will make some changes to the existing version in order to prepare for the transition. The biggest change is the dropping of the Red compiler, which will only act as a (smart) encapper. Routines and #system directives will still be supported, but probably with some restrictions. The Red preprocessor might also see some changes. This change means that Red will only have one execution model instead of the two it has currently. The Red compiler has become more of a burden than a help. The speed gains are not that significant in real code (even if they can be in some micro-benchmarks), but the impossibility for the compiler to support the exact same semantics as the interpreter is a bigger problem. This move not only will bring more stability by eliminating some edge case issues but also will reduce the toolchain by almost 25% in size, which will help reduce the number of features to support in the new toolchain.

👉 Runtime library

Some improvements are long overdue in the Red runtime library. Among them:

  • unified Red evaluation stack.
  • unified node! management.
  • improved processing of path calls with refinements.
  • improved object! semantics.

All those changes are meant to simplify, reduce the runtime library code and address some systemic issues (e.g. stack management issues and GC node leaks).

👉 Documentation

We need proper, exhaustive, user-oriented documentation for the Red core language. This is one of the mandatory tasks that needs to be completed and done well for wider adoption.


Self-hosted Red for 64-bit version


👉 Toolchain

In order to go 64-bit, we have to drop entirely our current toolchain code based on Rebol2 and rewrite it with a newer architecture in Red itself. The current toolchain code was disposable anyway, it was not meant to live this long, so this was a move we had to do for 1.0 anyway.

So the new toolchain will feature:

  • a new compilation pipeline with a plugin model.
  • an IR layer.
  • one or more optimizing layers.
  • modular/incremental compilation support.
  • x64, AArch64 and WASM backends.
  • linker support for 64-bit executable file formats for the big 3 OS.
  • support for linking third-party static libraries.
32-bit backends will not be supported in 1.0, though, they could be added back in the future.

👉 Runtime library

The current Red runtime library written in R/S will be kept and some adjustments will be needed in order to be fully compatible with a 64-bit environment (like updating all imported OS API to their 64-bit versions). 

View engine will not be part of that upgrade for 1.0, but will be done in a 1.1 version, priority is given to Red/Core for the 1.0.


Roadmap

Here are the main milestones:

  • v0.7   : Full I/O with async support.
  • v1.0b : (beta) completed self-hosted Red with 64-bit support.
  • v1.0r  : (release) first official stable and complete Red/Core language release.
  • v1.1   : View 64-bit release.
  • v1.2   : Android backend and toolchain release.
  • v1.3   : Red/C3 release.
  • v1.4   : Web backend for View release.
  • v2.0   : Red JIT-compiler release.
  • v3.0   : Red/...

The 0.7 should be the last version for the 32-bit Red version and current toolchain and we will be working on that first.

For reaching the 1.0-beta milestone, we target 12 months of intensive work, so that will bring us to Q3 2023. That's an ambitious goal but necessary to reach for the sake of Red's future.

The currently planned beta period for 1.0 is 2-3 months. We want a polished, rock-solid, production-ready 1.0 release.

For the 1.1, we will probably make some (needed) improvements to View engine architecture and backends.

For Red/C3, as the Ethereum network is transitioning to 2.0 and a new EVM, we need the WASM backend in order to support it.

Version 1.4 will bring a proper web runtime environment to the WASM backend, including GUI support.

The 2.0 will be focused on bringing a proper JIT-compiler to Red runtime, that should radically improve code execution of critical parts without having to drop to R/S.

Version 3.0 is already planned, but I will announce that once 1.0 will be released. ;-)

One major platform is missing from the above plan, that is iOS. Given how closed that platform is, we will need to come up with a specific plan on how to support it, as it won't be able to cross-compile for it (you would need a Mac computer), nor probably generate iOS apps without relying on Xcode at some point (not even mentioning dynamic code restrictions on the AppStore), which are layers of complexity that Red is trying to fight against in the first place... So for now, that platform is not among our priorities.


To finish, let me borrow some words from someone who succeeded more than anyone else in our industry:




Expect me to say "no" even more so from now on, as we get laser-focused on our primary goal.

Cheers and let's go!

32 comments:

  1. When is sound coming to Red?

    ReplyDelete
    Replies
    1. I think should be a part of ports

      Delete
    2. There are some contributed audio libs in https://github.com/red/code/tree/master/Library. Unless it is contributed don't expect sound support in the Red runtime library before 1.0.

      Delete
  2. Following since 2014, finally some good news (priorities {core language}), hope you make it that in12 month timeline.

    ReplyDelete
  3. Finally a concrete and focused plan, right to the point! Really thanks for that. I really like the plan I am seeing and keeping my fingers crossed.

    ReplyDelete
  4. "View engine will not be part of that upgrade for 1.0, but will be done in a 1.1 version, priority is given to Red/Core for the 1.0." Does that mean that one cannot write GUI applications in Red 1.0? Rudolf Meijer (meijeru)

    ReplyDelete
    Replies
    1. Correct, but View will be back in 1.1. OTOH, you can still play with the 32-bit version in the meantime.

      Delete
  5. That's awesome news!

    ReplyDelete
  6. At last, a v1.0 version!!

    ReplyDelete
  7. Are you in the languages+tools industry or the cult leader industry? (Steve Jobs can only be considered a success in the latter--not the former.) I think a Nietzsche quote is more relevant here: "Many are stubborn in pursuit of the path they have chosen. Few in pursuit of the goal." But...if you someday experience a change of heart and find yourself actually interested in programming language design and being relevant, Ren-C is still here and enjoying success in many areas you are not--across the board in design, as well as on the web and in WASI containers. Let me be clear: this success *could* belong to all of us, IF you cared more about actually solving problems than maintaining your odd cult. But seems you'd rather fail completely than admit you need my help and need the designs that Ren-C has painstakingly solved. In fact...if you paint yourself into a corner by insisting on doing things another way simply out of spite...you will be tying your hands, *badly*. Look at UPARSE, look at the Wasm stuff, learn about definitional errors and isotopes... and ask yourself if you really insist on going it on your own vs. coming and seeing what we could build together. That's really all on you--and always has been. I'm the one who cares enough to outreach, even now. If Red/System were compiling in the browser and integrating with Ren-C that would be impressive and we could say we did something cool. Or...hey...just do it your way--we've seen how well that's gone. Dungeon's on the web now, BTW: https://forum.rebol.info/t/enter-the-dungeon-if-you-dare/1865

    ReplyDelete
    Replies
    1. I guarantee this is hostilef**k... I don't even have to look. Still the same donkeyhat you were 7 years ago.

      Delete
  8. Thanks for all your works.

    ReplyDelete
    Replies
    1. Thanks for your kind support!

      Delete
  9. At last, you guys finally woke up to reality. We already knew that although we are not part of the project team. People have been asking for years when is 1.0 coming out, and the answer always was Red project team does not publish timelines. This was already a red flag that the team had lost direction. I think it is already too late now. Red has made the same deadly mistake that Rebol did i.e missed its main window of opportunity which came only once in the life of a product. Good Luck. Hopefully you still make it after all...

    ReplyDelete
    Replies
    1. That main window of opportunity is still in front of us, we have not played our trump cards yet. ;-)

      Delete
  10. Finally! I was about to give up on Red...

    ReplyDelete
    Replies
    1. I gave up on Red long ago when I saw they went into blochain "business". Otoh, I'm sad they were led away from original purposes and wasted so much time on practically not much. However, I'll check in Q3 2023 what is going on.

      Delete
  11. Finally - was about to give up on Red…

    ReplyDelete
  12. At last! It has been a very long wait since I first discovered Rebol and later Red.

    Both projects seemed to stall or sidetrack forever or, in the case of Rebol, that even was abandoned more or less.

    From the Rebol "children" I find Red the most attractive, but I gave up on it some years ago because of never reaching a solid 1.0 and how much I appreciate that real steps are taken now (or are promised at least) I very much regret that the GUI won't be a part of 1.0. How long will it take between 1.0 and 1.1?

    BUT (a big but) I am *VERY* grateful for all the folks that have worked on / work on Red and especially to Nenad Rakocevic who has shown relentless perseverance. Thank you all!!!

    As of now I'll start with fresh energy to be part of the Red family again.

    Thank you very much!

    Best regards,
    Arie van Wingerden

    ReplyDelete
  13. Oh thank God, a firm hand at last.

    ReplyDelete
  14. You must target Android as a platform ASAP or else risk suffering the same fate as REBOL for missing out on the browser as a platform. I still remember that discussion from around 2003 or so...I sooo wanted REBOL to succeed but Carl completely missed the boat.

    ReplyDelete
  15. Based on this post, you are planning to do code generation yourself *again*, now adding to it by making your own IL. Oh, and multiple optimisation passes and so on. Doing codegen yourself was maybe defensible in 2011. It isn't in 2022. LLVM has solved all of those problems. You want to do all that in a year, yourself? The key to finishing projects is (besides overall competence) restricting scope where possible. Yet you've *added* scope that amounts to one of the largest open source projects in the compiler space. It's not ambition, it's silliness, and it will end in tears.

    ReplyDelete
  16. Good luck, i might try it once in a while from now on.

    ReplyDelete
  17. Wow, it's refreshing to see the team's dedication to refocusing their efforts on completing the core language and reaching version 1.0. The decision to upgrade the current 32-bit Red implementation and make necessary changes to the toolchain shows their commitment to stability and scalability. The roadmap for future releases, including the plans for a self-hosted 64-bit version and a JIT-compiler, is ambitious but exciting. I'm looking forward to seeing Red's progress and the polished 1.0 release they aim to achieve.

    ReplyDelete

Fork me on GitHub