September 15, 2019

The Latest: Red could help AI be more precise; community stars; one CSV codec to rule them all?

Hello to all the great makers, doers and creative people who are using Red, helping the Red Language grow and improve! As always, there's a standing invitation for you to join us on Gitter, Telegram or Github (if you haven't already) to ask questions and tell us about your Red-powered projects.

Here are some recent highlights we’d like to share with you:

1. Tickets Get Priority

In the last month, our core team has closed a large number of tickets.We’d like to thank community members rgchris, giesse, and dumblob who are just a few of the passionate contributors putting Red through its paces and providing feedback as fixes and changes occur. @WArP ran the numbers for us, showing a cyclical growth pattern linking bursts of closed issues and some serious Red progress, and September’s not even done yet!...:


2. CSV Codec Available

Our newly updated CSV codec has been merged in the master branch and is now a part of the nightly (or automatic) build here. It is in an experimental phase, and we want your feedback.

Should the standard codec only support block results, so it’s as simple as possible? Or do people want and need record and column formats as well (using the load-csv/to-csv helper funcs, rather than load/as)? Including those features as standard means they’re always available, rather than moving them to an extended CSV module; but the downside is added size to the base Red binary.

Applause goes to @rebolek’s excellent organization and his wiki on the codec, which explains the various ways in which Red can represent data matrices. He writes, “Choosing the right format depends on a lot of circumstances, for example, memory usage - column store is more efficient if you have more rows than columns. The bigger the difference, the more efficient.”

You can judge their efficiency here, where @rebolek has laid out the compile time, size and speed of each version, including encapping and lite. Be sure to get the latest build, and chat with everyone on Gitter to tell us what you think.

3. Red has reached 4K stars on GitHub!

We’re truly grateful for all the interest and support, and we are proud of the way our growth has been powered by this community.

4. AI + Red Lang Stack: Precision Tuning With Local OR Web-Based Datasets

In conversation with @ameridroid:
“Presently, it seems like most AI systems available today either allow building an AI from scratch using low level code (difficult and time-consuming), *OR* using a pre-built AI system that doesn't allow any fine-tuning or low-level programming...with the advent of NPUs (Neural Processing Units) akin to CPUs and GPUs, an AI toolkit would allow specifying what type of AI we want to perform (facial, image or speech recognition, generic neural net for specialized AI functions, etc.), the training data (images, audio, etc.) and then allow us to send it the input data stream and receive an output data stream…[using Red] would also allow us to integrate with the AI system at a low level if we have specific needs not addressed by the higher-level funcs. Red dialects would be a good way to define the AI functionality that's desired (a lot like VID does for graphics), but also allow the AI components, like the learning dataset or output data stream sanitization routines, to be fine-tuned via functions. Red can already work on web-based data using 'read or 'load, or work on local data in the same way; the learning data for a particular AI task could be located on the web or on the local machine. That's not easily possible with a lot of the AI solutions available today.”

Check back in the next few days for an update from @dockimbel!

Ideas, contributions, feedback? Leave a comment here, or c’mon over and join our conversation on TelegramGitter, or Github.

July 18, 2019

GPIO port for Raspberry Pi

As part of the R&D work on port! datatype in port-type branch, we have implemented a GPIO driver for Raspberry Pi boards, as a gpio:// scheme (no third-party library needed). This work helped define the low-level API for ports written in purely Red/System or a mix of Red and Red/System.

The Raspberry Pi is a very popular board with millions of units sold, so this is a market where Red could be potentially helpful to developers. We could run Red on such boards for years but did not have proper GPIO support, so this is now fixed!

The current features supported by the GPIO port are:
  • auto-detecting the Raspberry Pi board type
  • uses /dev/mem or/dev/gpiomem for direct and fast access.
  • reading a GPIO pin state.
  • writing to a GPIO pin state.
  • hardware PWM output support (on capable pins).
  • a simple DSL for sending commands.

Planned (but not scheduled) future features include:
  • generating events when a pin state changes.
  • software PWM on all GPIO pins.
  • a higher-level reactive object layer for a API-less interface.
  • various drivers for common peripherals.

The source code for the gpio:// scheme can be found there.

In the short video below, you can find a little pet project meant for testing some of the features. It implements a simple joypad with 4 directions support, a red LED indicating when the pad is active and a green one for notifying when a level is completed. The game is the Red port contributed by Huang Yongzhao of Rebox!, my old clone of BoxWorld written in Rebol. In the video below, it is running on a Raspberry Pi 3 using our work-in-progress red/GTK backend for Red (contributed by Rcqls), locally merged with the red/port-type development branch (EDIT: that branch has been merged into master now).




You can find the breadboard layout below made using Fritzing. Sorry for the messy wiring, it is my first try with such kind of tool. If you have the skills to improve it, here is the sketch file. The buttons rely on the internal pull-down resistors. Note that in the video, the board is reversed.



The source code of Redbox has been modified to add GPIO support, you can find the modified code here. The GPIO-related code is enclosed in a context:

    joypad: context [
        mapping: [
            16 down  20 up
            17 left  21 right
        ]

        row: [(id) state: #[false] direction: (direction)]
        table: collect [
            foreach [id direction] mapping [keep compose row]
        ]

        pins: extract table length? row
        port: none

        acquire: has [pin][
            port: open gpio://
            foreach pin pins [
                insert port [
                    set-mode  pin in
                    pull-down pin
                ]
            ]

            insert port [
                set-mode 18 out
                set-mode  4 out
                set 4 on
            ]
        ]

        pressed?: function [][
            foreach pin pins [
                entry: find table pin
                old: entry/state

                insert port [get pin]
                entry/state: make logic! port/data

                ;-- detect 0-to-1 transitions only, to avoid auto-firing
                if all [not old entry/state][return entry/direction]
            ]
            none
        ]

        show-win: does [insert port [set 18 on]]

        release: does [
            insert port [
                set  4 off
                set 18 off
            ]
            close port
        ]
    ]


The GPIO port has a simple API:

  • opening: port: open gpio://
  • sending commands: insert port [...commands...]
  • closing: close port

The sent commands form a small DSL:
Set the working mode for a given pin:
 
    set-mode <pin> <mode>

    <pin> : pin number (integer!)
    <mode>: in, out, pwm

Write a value on a pin: 
 
    set <pin> <value>
  
    <pin>  : pin number (integer!)
    <value>: true, false, on, off, yes, no, 0, 1

Read a value from a pin (the returned value is in port/data):

    get <pin>

    <pin>: pin number (integer!)    

Manage pull-up/down resistors:

   pull-off <pin>    ;-- disable any pull previously set
   pull-down <pin>   ;-- activate pull-down on the given pin
   pull-up <pin>     ;-- activate pull-up on the given pin   

Write a PWM value on a pin:

   set-pwm <pin> <value>
 
   <pin>  : pin number (integer!)
   <value>: an integer between 0 and 1024, or a percentage for duty cycle.

Fade in/out values on a PWM pin:

   fade <pin> from <start> to <end> <delay>   

   <pin>  : pin number (integer!)
   <start>: starting value (0-1024)
   <end>  : ending value (0-1024)
   <delay>: duration of the whole fading (time!)

Wait for a given duration:

   pause <delay>

   <delay>: integer => pause in miliseconds, float => pause in seconds.

Important notes
  • Only BCM numbering for GPIO pins is supported.
  • A word, or path or paren expression containing regular Red code can be used in place of any numeric value.
  • A single command block can contain an arbitrary number of commands.
  • The get command can be used multiple times, a block of corresponding results will then be returned in port/data.
  • Use sudo when running your GPIO code if it involves PWM!
  • An example of using fade and other commands is available here.
  • The PWM range is currently preset to 1024, though, the DSL can be trivially extended to allow a user-provided range value.

Happy hardware hacking! ;-)

July 9, 2019

RED Wallet 0.4.0

We are releasing today the version 0.4.0 of the RED Wallet with several major new features. As a reminder, the RED Wallet aims to be a simple and very secure wallet for the major cryptocurrencies (BTC, ETH, and ERC-20 tokens). Safety is enforced through the mandatory use of a hardware key (Ledger or Trezor keys), which protects against any failure of the wallet app (being it a bug or an attacker). The RED Wallet code is fully written in Red (using the Red/System DSL for USB drivers implementation) and open source.

BTC support

The RED Wallet now supports the bitcoin network. The main features are:
  • retrieval of address balances.
  • sending transactions (with follow-up on an online block explorer).
  • support both SegWit and Legacy addresses.
  • support both mainnet and testnet.




The balance retrieval can be quite slow on the bitcoin network, so be patient.


ERC-20 tokens support

ERC-20 tokens are supported since version 0.3.0. Though, Ledger removed the ERC-20 tokens list from the embedded Ethereum app since version 1.2.6, leaving it to the wallet app to manage the tokens list. This release now incorporates that list in the RED Wallet binary directly, resulting in a significantly bigger binary (about 100 KB bigger than 0.3.0). The Nano keys will still check the validity of the tokens addresses internally for extra safety. Another consequence of Ledger's recent firmware updates is noticeably slower addresses retrieval when plugging the key. That's an internal change in the Ledger key, so we cannot do anything to bring back the fast speed we could achieve with older firmware.

As a reminder, BIP44 address derivation support is present in the wallet since 0.3.0. If you are using a key initialized in an older Ledger Live version, the derivation path would be different and in order to make the wallet recognize your key, you need to click on the "(Legacy)" label (we will improve that UI in further versions).

Batch transactions were introduced in 0.2.0, they now work for any ERC-20 tokens in addition to ETH. They can be accessed from a contextual menu using a right click on an ETH account.



TREZOR Key

The RED Wallet has updated the firmware support:

  • Wallet v0.3.0: Compatibility with firmware 1.7.1+
  • Wallet v0.4.0: Support for the latest firmware of the Trezor key. 1.8

TREZOR model T is fully supported on macOS. On Windows 7, if the key is not recognized, you need to install the TREZOR bridge first.


Bug fixes

  • FIX: Ledger key does not work on Windows in some cases.
  • FIX: crashes in some cases when using Ledger key on Windows.
  • FIX: only ledger key is usable when plugging both ledger key and trezor key.
  • FIX: trezor-driver context name is wrong.


Download the RED Wallet 0.4.0




Just click on the executable to run it (extract the .app file first on macOS), no installation or setup process required. Plug in your Ledger or TREZOR key and enjoy a smooth and simple experience!

If you want to check if your RED Wallet binary has been tampered with in any way, you can simply drag'n drop the wallet executable on our binary checking service. If it's legit, the screen will turn green. If it turns red with a warning message, please notify us on Gitter or Twitter at once.

Only download the RED Wallet app from this page, do not trust any other website for that.

Enjoy!

June 30, 2019

June 2019 Update

If you are an active member of the Red community, you will have likely been keeping track of Red's progress through our Gitter rooms and Github repositories, but we wanted to officially bring everyone up to speed with a shiny new blog post. The team has been busy since January on many fronts, multiplexing its efforts on many different branches of development. Here is an overview of what we have been cooking and what is coming.

RED Wallet

We are preparing to release v0.4.0 of the RED Wallet which will feature:
  • Bitcoin support with both Legacy and SegWit addresses!
  • Updated support for ERC-20 tokens for Ledger Nano keys (after Ledger externalized such token management to the wallet software earlier this year).
  • Support for the latest firmware of the Trezor key.
  • Some bug fixes.
The new wallet is currently under heavy testing, we plan to release it publicly in about a week.

C3

The first public alpha of the C3/System compiler is coming this summer. It is still under heavy development though. C3/High has made little progress as it needs real-world use-cases from the smart contract ecosystem for modelling the DSL(s). That ecosystem has not spread around much, probably due to the so-called "crypto-winter". As new big players are now entering the space, we are looking forward to the possibilities for providing new backends for the C3/System, like for the Libra network.

We will be soon releasing our ETH connector for Red. The final part missing is the integration of hardware and software key support for signing transactions on real online networks (we use private networks during development). The code for hardware key support is already there in our wallet, so we just need to integrate it inside the connector. For software keys, we have developed specific support for them, but it relies on third-party libraries, which are currently not a good match for our single binary distribution model.

GTK support

Red should offer soon a pretty good GUI backend for Linux, thanks to the community effort led by R cqls. The last PR for red/GTK branch weighs in at about 400 commits and covers most of View features.

The main remaining features to cover are:
  • Camera widget support (already available in an experimental branch)
  • Rich-text and Draw improvements
  • Shape dialect support
  • Some stability fixes

Here are some examples of scripts and apps using the GTK backend:


GTK theming is also supported:


Red/System

The ARM backend got many improvements:
  • Largely improved ARMhf (hard-float) support.
  • Fixes on regressions for ARMsf (soft-float).
  • Various libraries loading fixes, now all unit tests are passing again.
  • New ARMhf target added to our Travis CI backend.

A few low-level features were added, that would make it easier to write device drivers or operating systems directly in Red/System:

  • Ability to save/reload all CPU registers on stack using intrinsics.
  • Ability to read/write CPU I/O ports using new intrinsics.
  • Ability to inline machine-code pretty much anywhere in R/S source code.
  • Replacing hard native support for modulo operation on floats by the soft fmod() import.

VSCode plugin

The Red plugin for VSCode got a major refresh with a full reimplementation using LSP model improving the efficiency of existing features and adding some new ones:
  • Smarter code completion
  • Realtime syntax checking
  • Goto definition support



Other WIP features

JSON support

JSON format is now supported by Red's codec system. So it can be used in load/as and save/as to convert JSON data back and forth to Red values. More info about its usage in the Red 0.6.5 release notes.

Shadows in Draw

Some simple shadow effects in Draw have been implemented. They have the capabilities of the box-shadow model of CSS.



GPIO library for Raspberry Pi

Red has supported Raspberry Pi as a compilation target for many years, but we were still lacking a proper GPIO library, which is now under development. It will be implemented as a gpio:// port for now in the port-type branch. A dedicated library or dialect should be added also as a higher abstraction layer in order to simplify GPIO programming even further. Hobbyists and seasoned programmers will be able soon to build IoT projects on RPi easily using Red with GTK for the GUI support.


Next Red releases

0.6.5: Splitting the console and compiler

This is an important change that will be the new way to use Red and, hopefully, provide an even better experience, especially for newcomers. Not much coding is involved but we still need a prototype before giving the final go-ahead then proceeding with the code and infrastructure changes. We plan to finish those tasks during this summer (no precise ETA for now).

0.7.0: Full async I/O

Work started on this release many months ago but with a low-priority, as we have many other features to release first. The current state on the main features is:
  • port! type is almost completed and already usable (e.g. the eth:// port) 
  • the native ports TCP, UDP, DNS should be implemented this summer.
  • the async API still needs design work before being implemented.

Tokens retro-distribution

We are setting up the last steps for the RED tokens retro-distribution (past contributions) and the first monthly distribution for new contributors. We will publish the first article in July that will provide all the rules that we have established for that process. Then shortly after, we will publish the list of the contributors who were selected according to those rules, with the reward amounts and proceed with the distribution (using the batch transactions feature of the RED Wallet).

For example, as part of our tokens rewarding program for contributors, R cqls should receive a good amount of them in the next distribution,


The road ahead...

The most notable next planned milestones are:
  • RED Wallet 0.4.0 in July.
  • C3/System alpha release in August!
  • Red/Android beta release in September!
  • Red/Pro for New year!
  • Red 64-bit will be split into two phases:
    • cross-compiled from 32-bit platform: end of the year
    • toolchain ported to 64-bit: 2020

We plan to give more information about each of those big new additions to the Red family this summer.

As several development tracks are coming to fruition in the next couple of months, new articles should come up more frequently.

Go Red Go!

February 14, 2019

January 2019 Update



Stepping-up


2019 started with continued advances in the main development branch, implementation of features, and triage of pending tickets. For example, smart merging of a style's actors with the actors of a particular face instance was introduced in the VID dialect. Example:

view [
    style my-button: base
        on-down [face/color: face/color / 2 do-actor face event 'click]
        on-up   [face/color: face/color * 2]

    my-button "Say hi!" red on-click [print "hi"]
]

But this is just a tiny drop in the January ocean, and nothing special in the grand scheme of things. More than 50 commits to the mainline, lots of doc updates, and many improvements to the Red Wallet, which we'll talk about in another blog post. On the other hand, Red's progress is rarely trivial and silky smooth, and this write-up gives a deeper insight into our daily routine. Well, what did you expect, Pumpkin Juice? ;-)

Alas, as much as we love coding, organizational heavy-lifting needs to be done too — one of our short-term priorities this year is relocation of core team to Europe, and (ongoing) activity on this front took a lot of our time and resources in January. That is to say, after little overture, the core team's main activity shifted behind the scenes, giving the stage to our outstanding community's efforts. 1

Community highlights

GTK backend


Red strives to be a truly cross-platform language, but the project's roadmap always implies a certain order of priorities, where some milestones come later than others. One such missing cross-platform feature is the View engine's backend for Linux — its development requires dedicated resources, which the core team cannot fully afford, probably until near the 1.0 release.

With that in mind, it is much to our delight that development of GTK backend is being carried out as a dedicated community effort. We'd like to take an opportunity and thank all the people participating in it — their proactive contributions make Red a truly open-source project, and help us to advance it towards our collective vision. Big thanks to @rcqls for taking the lead on this.

Last but not least, you can track development's progress in a recently created GTK room. Don't forget to visit it and give our contributors all the praise they deserve! Or even better — become one of them and get a share of fame and glory! ;-)


View and Rich-text tweaking


Not only do missing parts of the View engine get developed, but existing ones receive thorough maintenance too. Features and fixes.

Among fixes are base and text rendering, Red's codebase received a contribution for testing of image-related features on Windows platforms, accompanied by more than 300 tests. Coupled with our "null" testing backend (covered in some detail in here) this casts a wide net for catching regressions in View.

After the 0.6.4 release announcement, many users checked out rich-text support and gave Red's Rich-text dialect a proper spin, with numerous bug fixes and tweaks being submitted in the process.


Editing, with a capital E


As you can see from the above, native GUI support is one of our strongest aspects, which piques the interest of many community members.

Another special feature that catches everyone's fancy is Red's "code as data" nature. But, while giving Red enormous expressive power (metaprogramming and the creation of domain-specific languages, DSLs for short), it makes some things trickier — with the boundary between code and data being blurred, and multiple dialects (our term for embedded DSLs) at one's disposal, it is extremely challenging to assign any particular meaning to a given piece of code (data!); both for humans (esp. newr Red programmers) and automatic tools, such as syntax highlighters, static code analyzers, debuggers and linters.

Historically, this was the main argument when it came to Rebol's tooling. Sprinkling your code with print and probe is still the most common debugging method, and error reporting boils down to stack traces rather than source text info; text editing support can accommodate only cosmetic niceties like "keywords" highlighting and indentation, covering only general subset of the language.

Being a direct descendant of Rebol, we carried over this bag of concerns, aiming to resolve them at some point in development. A quite natural answer to points raised would be an editor written in pure Red and its dialects, integrated in our self-contained toolchain.

And while writing a basic version of such editor is rather trivial, such a naive approach leaves much to be desired, and keeps questions about proper tooling surrounded with an air of uncertainty, begging for a detailed investigation. Thankfully, this is where community projects and experiments come to the rescue!
  • @rebolek's Values, announced at last year's RedCon, is a sophisticated projectional editor with Vim's flavor and an ingenious schtick — rather than working with textual form, it loads a program and directly manipulates its internal data structure (a block! of values), reflecting all changes back in the text area.

    Idiomatically written, and offering sheer power in a few keystrokes, it struck us in awe back at RedCon, and stimulated a lot of technical discussions. With our new GUI console's plug-in API on its way for 0.6.5 release, we are eager to see how it will get integrated in day-to-day user experience, and what fresh insights it can bring to all of us.
  • @toomasv is well-known in our community as an avid Draw and View tinkerer, and his recent experiment only firms this reputation.

    Datatype-aware syntax highlighting, contextual help, auto-completion, step-by-step evaluation and bunch of other niceties — all in ~1K lines of code! Magic, you say? No, Reducing at its finest. ;-)


Projects like this (and all others that we didn't get a chance to mention) aren't just eye-candy for the public — they help us, core developers, to evaluate all options and to revisit old ideas in a new light. Besides, it's extremely rewarding to see Red (in its alpha stage) being put to real use.

Once again, our cheers go to all the people who dedicate their time and efforts to support us. No matter how small your contribution, it makes Red stronger and readier for the prime-time: day by day, bit by bit.

C3 project


Since the core team relocation is our ongoing No.1 priority, development of the C3 project got sidetracked a bit. There are still a few final touches that need to be done (e.g. finishing AST parser, hex! and bigint! datatypes support) but, aside from that, the first alpha version of compiler is almost ready for a public release. The first 90% of a task takes 90% of the time, the last 10% takes the other 90%.

In the meantime, we have one appetizer to serve: RedCon talk about C3! Due to technical issues and time constraints, it became available only recently, and was captured on a nearby web-camera. Because of that, video quality and recording angle are so-so, but the good news is that you can gain a deeper insight into the blockchain aspect of Red... and witness the talking belly. :-)




As evident from the talk2, C3's design proved to be much more challenging than we initially anticipated, muddled by pesky EVM (Ethereum Virtual Machine) constraints (like gas), lack of good reference documentation, complex tooling and blockchain hype haze.

At this point, we're convinced that at least two layers are required for C3 to be a sound smart-contract development solution:
  1. The low-level one, named C3/System, is a language with simple semantics (close to Solidity, but without foot-shooting), serving as a base for building high-level dialects on top of it. We also want to implement a couple of experimental features, like off-chain Cron-like scheduler of Smart Contract execution and embedded FSM (finite-state machine) sub-dialect.
  2. Possibly multiple high-level layers, each covering different blockchain aspect and tailored to specific solutions: creation of crypto tokens, gambling, in-game collectibles, legal contracts and business logic, bids and auctions, asset tracking... who-knows-what people may come up with.
Despite all the media craze surrounding blockchains, the technology itself is still in a nascent state, constantly searching for a real-world use-case. Because of that, navigating this new and constantly shifting space (not to mention building a development platform like C3) is like sailing in stormy weather.

In light of the above, we owe much to @9214 for his help with C3 preliminaries and conducting in-depth research on Smart Contract languages. His constant background presence affirmed many of our assumptions, and guided us past many reefs and pitfalls.

Back in the day, our "Red goes blockchain" announcement generated much controversy, and prompted many people (both community members and outsiders) to cast their doubts on this decision. Some go as far as to dismiss the Red project entirely, based on our new roadmap.

Yet, very few realize that, in some sense, we're picking up the game that Rebol left — with Carl's conception of X-Internet3 replaced by decentralized consensus-driven P2P networks, and EVM instead of IOS. Indeed, Smart Contracts are just very limited...slow Reblets/services in disguise ;-) And behind blockchain fatigue lies the brave new world, where Red can be the missing puzzle piece and stepping stone towards simple, human-friendly tooling.

All in all, we hold to our beliefs, and going to play this card, aiming to hit the jackpot. But we're also not putting all our eggs in one basket. Bear with us as we tread forward, and stay tuned for future updates!

Wrapping-up


January, with its holidays and family reunions, is nothing but a calendar sheet. The work on Red never ceases, spans through weekends, occupies all time and space (both physical and mental) and now reaches across the globe. For some, it became the animating principle of their lives.

With our hard work at stake, and with 1.0 release in near sight, encounters with the outside world get more frequent and progressively challenging — criticism and cross-checks with reality are important, but it's equally important to apply a fair yardstick, and judge the project on its own merits.

That's why, when it comes to media coverage and public discussions, we expect outsiders to stay open-minded and not to cast aspersions lightly. We want Red to succeed, and welcome technical debate and comparisons with other projects (like, e.g., Red vs. Racket; again, kudos to @9214 for writing such a well-articulated response). But, more often than not, discussion threads turn into exercises at damage control, with constructive points being hamstrung and torn to pieces in a cross-fire.

Our mission to fight software complexity is challenging, and not for the faint-hearted. Remember — the best way to prove one's case is not in word, but in deed... and the strongest argument is a tech-savvy pull-request. ;-)

Come to think of it, I can't help remembering Pierre Rabhi's tale:
Once upon a time, the scorching fire swallowed the forest. All the animals, terrified and powerless, watched it destroying their homes from afar. Only little Hummingbird kept flying back and worth between the fire and nearby pond, carrying tiny drops of water in its beak and throwing them in the flames.  
Irritated by bird's constant flickering, grouchy Armadillo yelled: "Hummingbird! Don't you see that your pathetic attempts won't save the forest?"
"I know.", answered Hummingbird, "but I'm doing my part."

Hectic 2018 barely passed by, but we're already steering wheels on our new roadmap. There is more news to announce, once the dust has settled. We'll be in touch. Until then, Happy Reducing!

-- Post by 9214, edits and amendments by Gregg


Footnotes

  1. Remember that you can track progress of Red projects (both community-driven and official ones) at https://progress.red-lang.org/
  2. Reminder: C3 presentation slides in PDF format are available here.
  3. "X" here stands for eXecutable and eXtendable.

January 6, 2019

Full steam ahead!

The past year

It has been a tough year 2018, filled with roller-coasters and instability. The constant dropping of the crypto markets has forced us to change our plans pretty much every three months, in order to cope with that market uncertainty and our diminishing means. Though we have taken measures to ensure the proper funding for Red project for the next few years, so no worries on that side. All that has slowed down work on Red's core and even our new C3 language. That said we still have managed to make some great progress in the past year, e.g.:

  • RED Wallet released: a good showcase of what can be achieved in a small codebase using a truly full-stack programming language like Red, from USB and hardware keys drivers to blockchain network interfacing and cross-platform UI!
  • Garbage Collector: a simple mark & sweep, compacting GC for Red! That is a big step towards a completed production-ready Red version 1.0.
  • Red console fully re-implemented in Red itself, allowing new features like syntactic coloring and user plugins.
  • New rich-text display widget and related RTD dialect.
  • DPI-independent support for Red/View.
  • Preliminary port! and bigint! datatypes implementations.
  • eth:// port for interacting with the Ethereum blockchain (to be released publicly very soon).
  • 350 tickets closed, among which 136 are bugfixes.
  • RedCon1 held in Prague in November!
  • Extended Red team with now about 12 people in total.

We are also working on structuring the Red Foundation and have a new recruit working on tokenomics. So the tokens retro-distribution to kick-start tokenomics should happen very soon.

The plan for 2019

We are starting the year with big targets as we need some last big pieces of the Red project to be implemented to reach both our technical and business goals. Here are the main tasks we will handle during this new year:

  • C3 toolchain release: the C3/System language spec was defined last October and work on the compiler and related toolchain is on-going. This is our main short-term priority as we want to test our business options in the smart contract market while it is still in its early days.

  • Red full async I/O support (0.7.0 milestone): we already have a PoC for async tcp/udp support, with Rebol-like event handling API. There is still a need for more design work (on-going right now) in order to provide the best possible API for Red users. For now, we are trying to avoid going full-force into the deep territory of defining a general concurrency support (it's scheduled for 0.9.0 milestone). I'm not yet sure if we can escape stepping into it anyway if we want to go beyond the Rebol async model.

  • Full Android support for Red: we have a preliminary implementation that has proven to be very promising! With the integration of async I/O and ports, we will be able to complete the wrapping of the Android API using decent human-friendly abstractions. We need to advance Android support to at least beta level by the end of this year. This is one of the major goals we need to achieve.

  • Red/Pro: it is coming this year! It will be our first commercial product and set of online services targetting both individual developers and enterprises. It will include a new backend for Red's toolchain, providing a state-of-the-art optimizing layer and support for many new platforms, including 64-bit ones (though full 64-bit Red support will probably be for next year). That new backend will provide an alternative to our current code emitter and linker, the rest of the toolchain remaining the same. The current Red toolchain will be known as the "community" version, and continue being the main development branch.

In addition to all that, a 0.6.5 milestone for Red is in the plans, aiming at making the Red console the main focus for Red users by providing:
  • prebuilt versions of the Red console.
  • ability to invoke the Red toolchain from the Red console.
  • a plugin API to extend the console in many ways.

In order to better focus on these tasks and better interact with our new team members in Europe, we will be relocating our core team from different parts of the world to Europe (Red Foundation is located in Paris). We also plan to make a RedCon2 developer conference in Europe later this year.

Many thanks to all the people who have contributed or helped in any way with advancing Red and managing the community. How do we climb the tallest mountain? One step at a time! And the top is in sight now, we're almost there. ;-)

Happy New Year to all, and keep up the great work you are all doing!

December 7, 2018

0.6.4 Simple GC and Pure Red GUI Console

It's been a long time coming.

While work on Red has progressed every day, work on the blockchain aspects of the project, and their associated logistics, came at a time when that workload temporarily prevented us from making new, formal/stable releases. There was a key feature that needed to be in place, and it wasn't a small one. Thanks for bearing with us!

Fixes and features kept rolling out on a regular basis, so most active community members were not affected by the lack of a fresh new "official" release. It did, however, become an issue for people new to Red, because as the stable download for 0.6.3 became more and more out of date, users would report bugs that had already been fixed! So now 0.6.4 should see us on the same page.

We'd like to promise we'll never go that long again between official releases, and that is our goal, but we hope you'll roll with the punches, as we do the same. The best way to keep up to date is to use the automated builds, so you're always current. Some people will want to stay with the stable version for various reasons, and that's fine too.

This new Red release contains 1'116 commits, closes 167 issues tagged for this milestone, and adds a lot of  small features. Looking at the list of bugs fixed, and features added, is a great reminder of all Red has to offer.


1. Simple GC


The main feature for 0.6.4 is what we call the Simple GC (Garbage Collector). A more advanced GC is planned for the future, but this release will make it easier to create long running processes, without having to be aware of all your allocations, or manually reusing series to conserve memory. Things just got a whole lot easier!

There are 2 functions related to this, that you can use:

1. You can use stats to see the amount of memory in use, and its /info refinement provides more details to aid deeper debugging if needed.

2. The second function is called recycle, which triggers a manual garbage collection pass, rather than waiting for internal heuristics and timing events to do so. Recycle has 2 refinements: /off and /on. They allow you to disable the garbage collector, and re-enable it. This gives you control in scenarios where you want to manage things more directly yourself.

It's also important to note that the GC operates at the level of Red values, and does not affect Red/System code. Red/System is a C level language, where you are responsible for allocating and freeing memory, and where you may need precise manual control over those actions. (It's possible to add an optional GC for Red/System in the future, but it's not currently on our roadmap.)


2. Simple Isn't Always Easy


Our main goal is to fight software complexity, but nothing can eliminate it completely.

Our small size belies the number of details that have to be considered, interactions between the large number of datatypes, and how to make each most useful with the standard library of functions. And then there's the reactive GUI system, with its draw dialect, both very flexible and powerful, which leads to more combinations that need to be handled, and edge cases to uncover. We accept the effort needed, because we are our own toolchain. A great deal of work went into the new GUI console, with a lot more planned.

Apple's talk of eliminating all 32-bit support could impact us. We are currently 32-bit only, and need to decide the best path for adding 64-bit support, should we elect to do so. libRed is also kept up-to-date in this release as well, for integration from other tools and languages. Additional improvements were made in the ELF format emitter.

We still face being flagged by a few big AV products, and these false positives hurt our image. It's frustrating to be marked clean one day, then have their heuristics change and flag you the next. Keep reporting the problem to them, and check VirusTotal to see how they compare. It's just one more challenge, because we're small, and use our own toolchain, so they don't know what to do with us.

Not all changes are big, and a lot of work also goes into documentation and tests on a regular basis. The code is only part of the project. A lot of our time this past year has been spent on logistics related to business, and making the Red project sustainable long term.


2.1: Operating System Support


Providing the maximum number of features across platforms is not easy, and we will need to freeze support for older OSs, like Windows XP, so we can focus efforts on the systems used by more people today.

Keeping up with mainline changes on all the operating systems we support is a challenge itself. We've kept ARM soft float working, and FreeBSD support has remained largely intact. We got a report about it breaking recently, and a fix went out the next day. FreeBSD isn't critical for us, but that fast response shows how a smaller and simpler code base can help.


2.2: A REPL In Time


Having an instant-on REPL is enormously helpful to newcomers and experts alike, and we'll build on this new foundation to add more features.

One change we expect to make for 0.6.5 is to change the download binary, so REPL use is even easier. Right now, when you download Red, you get the toolchain. The first time you run it, it builds the REPL console on your system locally. That takes some time, and isn't a great first user experience. So as we move forward, we're considering making the REPL the download, and then when you go to compile, if Red sees you haven't installed the toolchain, it will download it at that time. We'll keep you updated as we try our alternatives.


3. Rich Text


There's a new style in fashion this season: rich-text. 

Full documentation is forthcoming, but you can see the source code for the RTD (Rich Text Dialect) in %red/modules/view/RTD.red (only ~150 lines), and reference information in the wiki.

Eight attributes are currently supported. More can be added, but these gives you a great deal of control, and are by far the most commonly used: bold, italic, underline, strikethrough, text color, background color, font-name, and font-size. Border is planned but not yet implemented, as are some advanced options for the above attributes (e.g. wavy underline as is often used for spell checking markup). Attributes can be nested, of course.

@toomasv put together a small editor, which is about 400 lines of code, but since the rich-text style doesn't have built in keyboard handling yet, he had to roll his own, which is the majority of the code. 



Using the rich-text style directly in code is easy too. The data facet just needs to contain RTD commands and, internally rtd-layout is called to render those commands into the text and data facets of the face. You can also create a rich-text face, and later call rtd-layout/with to update commands and render them.

While Red prefers plain words for markup elements, RTD also supports HTML style tags, for convenience. (Note, RTD is not designed to be HTML compatible, and it's not a goal on our roadmap.) 

You can see different approaches to formatting, below: the first three examples all produce the same output, and show how attributes can be applied to blocks of commands, which is more idiomatic in Red. That is, when you have a formatting command followed by a block, everything in that block has the formatting command applied. It makes viewing the formatting structure much easier, and also eliminates attribute ordering issues. 

The last example below shows how a group of attributes can be expressed as a single path! value, applying all of them to the block that follows.

    view [
        rich-text data [
            <i> <b> "Hello" </b> <font> 24 red " Red " </font> blue "World!" </i>
        ]
        rich-text data [i b "Hello" /b font 24 red " Red " /font blue "World!" /i]
        rich-text data [i [b ["Hello"] red font 24 [" Red "] blue "World!"]]
        rich-text data [i/b/u/red ["Hello" font 32 " Red " /font blue "World!"]]
    ]

This is only the most basic use. See @toomasv's example gist for more uses, and consult the wiki doc to see how to query information like line height and caret position. Beyond that rich-text is already hardware optimized and supports multiple rendering areas in a single face, which makes it suitable for quite complex layouts of richly formatted text. You can also combine draw commands in a rich-text face, for even more power.


4. GUI Console: More Than Meets the Eye


While the GC took the spotlight in this release, the console also saw huge changes and improvements. The biggest change isn't visible to the naked eye: the new GUI console is now implemented in pure Red, using the rich-text face style. This sets the foundation for building more complete Red editors, tools, and IDEs. Basic features in place right now:

  • History and Undo/Redo
  • Copy/Paste support
  • Hot-keys and shortcuts
  • Syntax Coloring
  • Auto-completion


4.1: Console Hot Keys


  • Ctrl + Backspace  delete previous word
  • Ctrl + Del        delete next word
  • Ctrl + Up/Down    scroll up/down
  • Ctrl + C          copy
  • Ctrl + V          paste
  • Shift + Insert    paste
  • Ctrl + X          cut
  • Ctrl + Z          undo
  • Ctrl + Y          redo
  • Ctrl + L          clear screen
  • Ctrl + K          delete the whole line


4.2: Syntax Coloring


You can turn syntax highlighting on with the following command:

    gui-console-ctx/terminal/color?: yes

It is not dynamic, so you may find the colors don't work well with the colors you've selected in the Settings dialog for the console. There is no support for customizing themes yet. If you want to experiment with changing colors, you have to manually set values in gui-console-ctx/terminal/theme.


4.3: Auto-Completion


This feature can support completion for 3 types of values: words, files, and paths. 

For words, if you type a word and hit the Tab key, one of two things can happen. Either there's a single match for what you've typed so far, which will then be auto-completed for you, or all matches will be shown.

Try typing p <tab>. You'll get a list of all words in the system that start with the letter P.

Now, type pa <tab>. The help function works in a similar way, but with different output.

Filename completion works the same way. If you type a file value, starting with %, the letters you type are then prefix matched against the current directory in the file system. It works very much like a system shell's autocomplete feature, but built directly into the Red console. Snazzy!

Path completion is a little different:
If you type a path, e.g., system/, and then hit the Tab key, the console will display the list of words available for that path value.
If the value is an object, the words of that object will be displayed.
If it's a function, the possible refinements will be listed.

For objects, it makes it easy to drill down into nested structures. For functions, it offers basic help, but doesn't do anything advanced yet. For example, if you use more than one refinement, auto-completion will still list every refinement available, rather than removing those you've already used from the list.


5. Other Highlights & Changes in 0.6.4


Here are just a few more details:

5.1: View and VID

  • DPI support added, which is great, but also raises new OS system scaling questions, currently being reviewed
  • rich-text style added
  • Password mode added to field style (not supported in VID yet); include 'password in face flags
  • Make-face helper function added
  • RTD (Rich Text Dialect) support (MacOS & Windows)
  • Ctrl+A hot-key works in area now
  • Text selection support improved for area and field

5.2: Core Language

  • Added /deep and /case refinements to replace
  • Simple I/O now supports all HTTP verbs
  • Added decompress native, with support for gzip, zlib and deflate formats
  • Adler32 support added to checksum
  • Maps now support binary! values as keys
  • Latin-1 strings can now be decoded
  • Trim now supports binary! values. By default, only null values are removed. If you use the /all refinement, whitespace chars are trimmed as if it were a string
  • Basic query action added for files, which returns their timestamp
  • Added offset-to-char function. Given a face and a coordinate, it returns the corresponding character position
  • Added os-info func. It returns detailed operating system version information, which can be helpful when debugging and submitting tickets (used by about/debug)
  • Very simple alert function added; more are available as community experiments
  • Enbase/Debase now support base 58
  • Sum and average functions added


6. Looking Ahead


  • Full I/O is our next major milestone, and some other key features, like async I/O and a networking port model, will come with that.
  • Android: We've changed up the roadmap a bit, so Android will come after Full I/O, because it will benefit from having it, and save rework if done the other way around. This is a very important -- dare we say, critical? -- milestone, and one we will put all our efforts behind in the coming months.
  • Blockchain Aspects: The RED Wallet is expected to get some new features, and possibly a new look. We need a dedicated product manager for it, and another team member who can focus on it. The C3 project is going full steam, and we'll be organizing resources to keep that moving as we redirect key players to Red's core early next year.
  • We have an experimental money! datatype, based on Douglas Crockford's Dec64 design. It will have a limited value range initially, due to our 32-bit foundation, but it's a first test, and can be used to see how it works in action. We believe we're the first language to implement it.
  • JSON and CSV codecs. These are works in progress, and their design had led to discussion of how best to do certain things outside them -- for example, how to represent JSON null values in Red maps, which isn't yet possible.

We're enormously proud of how Red's growth and community support have accelerated. For a project like ours -- ambitious, idealistic, open-source -- to have garnered interest in the way we have, has been greatly rewarding. We look forward to a 2019 rich with REDucing software complexity, yet celebrating the increasing power and impact of the language and the movement.

Fork me on GitHub