Showing posts with label map. Show all posts
Showing posts with label map. Show all posts

February 11, 2024

Important Change! Switching map and construction syntax.

Sometimes deep changes take a huge amount of code. Sometimes they take a lot of detailed explanation and consideration, leading to long discussions and people taking sides. Rarely does an important syntactic change to a language happen quickly, with universal agreement, simple implementation, and tools to help update scripts in the wild. Today is one of those rare days.

Admittedly, this idea has been discussed for a long time. It would surface, people nodded their virtual heads, and it would submerge again. Today it's ready to deploy. Not only that, but Rebol3 is making the same change, so the two languages will still be compatible in this regard.

What is the change?

It's easy to describe. Today, map! values use this syntax: #(...) and construction syntax (sometimes called serialized form or loadable form) looks like this: #[...]. Going forward, those syntactic forms will be swapped. Why? The answer is easy. In Redbol langs, blocks do not evaluate by default, you have to do or reduce them. Parens, on the other hand, do evaluate by default. Today, maps use paren-like syntax, but they do not evaluate, while construction syntax uses block-like syntax, but does evaluate. This is a carryover from Rebol, so the major concession here is that Red and Rebol3 will no longer be compatible with Rebol2's construction syntax.

If you've never heard of construction syntax, there's a nice explanation of it here. Red only supports a few values via construction syntax today, all datatype literals, true, false, none, and unset; but eventually it will support much more. If you look at the help for mold, you'll see that /all is TBD (very partially implemented for now), and that's how you create loadable, serialized, data that can safely and easily contain any value (like redbin but readable by humans). It helps avoid cases where none or true/false may load as words. This is also why construct evaluates those specific words (including also on/off/yes/no), but not others. When loading untrusted data, we have to strike a balance between ease of use and safety.

What do I have to do?

Not much. There are two tools available, which will convert your scripts automatically. The first is small and simple, showing just how powerful Red is, and leveraging its lexer instrumentation. You can find it here (once merged, that branch may go away and the tool will be in the main branch). The second is a more advanced and standalone tool written by @hiiamboris. You can find that here.

For the simple script it's necessary that you run it under a current version of Red's lexer. Once the change is in place, running it under the new lexer will make the exact opposite change. Of course, you can compile it into a standalone EXE, or use Boris' app, which is already available.

To run the simple script from a Red console, you can just:

  do https://raw.githubusercontent.com/red/red/master/utils/migration/map-conv.red

then you can use help map-conv to see all the available options. By default it runs in preview mode, making no changes, and showing you all the instances it found, which will be changed if you use the /save refinement. A copy of each changed file is created with a .saved extension in the same folder. If you don't want them, you can use the /no-copy refinement.

A word of warning, if you run the conversion tools a second time on the same files, they will convert the data back, because they can't know what you're thinking. On the bright side, this is an effective "undo" feature. Still, it's wise to back up your data before running any tools against them.

Thanks to the power of Red, and these tools, there are already PRs pending for updates to docs and community scripts. But the other thing you can do to help is to let us know when you find things that need to be updated for this change, and especially if you run into any issues when converting your own code.

Conclusion

We know changes like this can be hard, but better now than when there is even more code in the wild that would be affected. If we had been any other language, this long-view improvement might not have happened. Only because Red (and Redbol langs in general) can consume their own code as data and have powerful parse and lexing features, was this change so easy and safe. It's still a code-porting process, and if you run multiple versions of Red you may need to maintain separate versions for a while. The other hard part is retraining your hands and eyes to the new syntax.

Happy Reducing!

June 12, 2015

0.5.4: New datatypes, exceptions and set operations

This new version turned out to be a major release, given the vast number of new features which made their way into it. Hope it was worth the waiting. ;-)

In preparation for the GUI support and the DSL that will come with it, a new range of datatypes has been implemented.

Pair! datatype

A pair is a couple of integer! values used to represent mainly dimensions and coordinates. Its literal representation separates the two integers by an `x` character.
    1x2
    -56x1234
    -3x-8
Pair! is a scalar datatype that supports math and bitwise operations.
    1x2 + 2x3
    == 3x5

    3x4 * 2x3
    == 6x12

    10x10 * 2
    == 20x20

    60x10 % 4
    == 0x2
To access pair elements, path syntax provides a handy way. The left value is referenced by `x` or just 1, the right value is referenced by `y` or just 2.
    a: 5x6
    a/x
    == 5
    a/y
    == 6

    b: a + 10x1
    b/1
    == 15
    b/2
    == 7
It also works for modifying pair elements.
    a: 5x6
    a/x: 0
    a/2: -1
    a
    == 0x-1
A pair! can be constructed using different approaches, depending on how pair elements are provided (as separate values or as a block).
    as-pair 3 4
    == 3x4

    make pair! [4 5]
    == 4x5

    make pair! 10
    == 10x10
The following notable actions are supported on pairs: random, absolute, add, divide, multiply, negate, remainder, subtract, and, or, xor, pick, reverse.

Pair! will be extended in the future to handle float values, in addition to integers.

Percent! datatype

Percent! provides an elegant way to represent numbers as fractions of 100 using a natural syntax:
    20%
    -45.285%
    0.1%
    100%
As it is implemented using a 64-bit float internally, it supports all the same math actions and operations as float! values, namely: random, absolute, add, divide, multiply, negate, power, remainder, round, subtract.
    20 * 80%
    == 16
    33,33% * 250
    == 83.325
Tuple! datatype

The tuple! datatype is a finite list of 3 up to 12 bytes. It offers a versatile way to represent different kind of values like:

Version numbers:
    0.5.4
    1.0.0
    2.10.120
RGB colors:
    red
    == 255.0.0
    blue
    == 0.0.255
    orange
    == 255.150.10
or IPv4 addresses:
    127.0.0.1
    192.168.1.1
    255.255.0.0
Once a tuple! value is created, its size cannot be changed anymore (it is not a series!), but its elements can still be modified, using, for example, path syntax.
    ip: 192.168.0.0
    ip/1
    == 192
    ip/4: 1
    ip
    == 192.168.0.1
The following actions are supported by tuple! values: random, add, divide, multiply, remainder, subtract, and, or, xor, length?, pick, poke, reverse.
    1.2.3 + 100.10.1
    == 101.12.4

    255.0.0 or 0.128.0
    == 255.128.0

    192.168.2.1 and 255.255.0.0
    == 192.168.0.0

    green + blue = cyan
    == true

    red + green = yellow
    == true
Math operations are allowed with some other scalar datatypes, like integer!, float! and percent!.
    1.2.3 * 2
    == 2.4.6

    red * 33%
    == 84.0.0

    blue / 2
    == 0.0.127
Tip: in order to get the list of all predefined color names from console, just use:
    red>> help tuple!

We are considering for a future release, the option of extending the tuples with 3 or 4 elements to 16-bit values, as the internal storage space allow us to do so. This would make possible to support color values with up to 16-bit per component and version numbers with high build values.

Map! datatype

This datatype provides a dictionary-like data structure, to make it easy to store key/value pairs while providing very fast lookups. Internally, keys are stored in an hashtable, using our existing MurmurHash3 implementation.

The map! datatype has its own literal format:
    #(a: 3 b: "hello" c: 10)
    == #(
        a: 3
        b: "hello"
        c: 10
    )
Keys can be any value among the following datatypes: any-word!, any-string!, integer!, float!, char!.

Reading and setting keys and values in maps can be done using the familiar path syntax.
    m: #(a: 3 b: "hello" c: 10)
    m/a
    == 3
    m/b: 14
    m
    == #(
        a: 3
        b: 14
        c: 10
    )
In addition to that, some actions can be used when keys are not words, or when paths are not the best fit.
    select m 'a
    == 3
    put m 'a 42
    put m "Monday" "pizza"
    m
    == #(
        a: 42
        b: 14
        c: 10
        "Monday" "pizza"
    )
The new put action works as a counterpart of select, changing the value associated with a key. It will be extended to act on all series in the future.

Read more about map! in the specification document.

Exception handling

Red now provides a more complete exception system, allowing to throw and catch exceptions in a convenient way.

Syntax
    throw <value>
    throw/name <value> <name>

    catch [...]
    catch/name [...] <names>

    <value> : any value
    <name>  : a word for naming the exception
    <names> : a name or block of names
A throw will interrupt the code flow and go back up through the call stack, until a catch is reached, then resume execution just after it. By default, exceptions are anonymous, but for finer-grained control, they can be named through the /name refinement, targeting one or several specific catch statements.
    catch  [print "Hello" throw 1 print "John"]
    "Hello"
    == 1
Catch will catch all thrown exceptions and return the thrown value. If /name is used, only the matching named exceptions will be caught, all the others will pass through it.

If an exception is not caught by a catch call, it will result in a runtime error.
    red>> throw 1
    *** Throw error: no catch for throw: 1
    *** Where: throw
In order to improve the ability to handle both errors and thrown exceptions at the same time, try has been extended with a new /all refinement, allowing it to catch all possible forms of exceptions, including return, exit, break and continue misuses. It is an ultimate barrier in your code, to handle runtime issues.

Set operations

Set operations are now fully supported:
  • union: returns the union of two data sets.
  • exclude: returns the first data set less the second data set.
  • intersect: returns the intersection of two data sets.
  • difference: returns all the values which differ from two data sets.

Those operations can be applied on following datatypes: block!, string!, bitset!, typeset!. (Hash! datatype support will be added in the next release)

In order to produce sets out of block! and string! values, a unique operation is provided:
  • unique: returns the data set with duplicates removed.
    union [a b c] [d a hello 123]
    == [a b c d hello 123]

    intersect [2 6 3 4 2] [5 6 9 4 3]
    == [6 3 4]

    unique "hello red world"
    == "helo rdw"
A /case refinement is provided for performing case-sensitive set operations (default is case-insensitive).

The /skip refinement allows to process series arguments in a record-oriented way.

Note: all these operations are implemented using hashtables internally, in order to ensure the best performances.

New natives

as-pair

Returns a pair! value made out of two integer arguments.
    as-pair <x> <y>

    <x> : integer x value
    <y> : integer y value
Example:
    as-pair 2 3
    == 2x3
break

Exits a loop and resume evaluation after it. Can optionally return a value from the loop.
    break
    break/return <value>

    <value> : returned value, any type.
Example:
    loop 3 [print "hi!" break print "hidden"]
    hi!
continue

Interrupts a loop evaluation flow and resume at next loop iteration.
    continue
Example:
    loop 3 [print "hi!" continue print "hidden"]
    hi!hi!hi!
extend

Adds pairs of keys and values to an aggregate value. If the key is already defined, it will replace its value. Additions are done in a case-preserving way. Default lookups are case-insensitive, a /case refinement can optionally enforce case-sensitivity.
    extend <aggregate> <spec>
    extend/case <aggredate> <spec>

    <aggregate> : an object! or a map! value
    <spec>      : a block of key and values pairs
Note: support for object! is not implemented yet.

Example:
    m: #(a: 3)
    extend m [b: 4 "c" 123]
    m
    == #(
        a: 3
        b: 4
        "c" 123
    )
New action

A new put action has been added in order to provide a modifying counterpart to the select action. Select offers a way to access series values as if they were associative data structures. Put and select are natural accessor actions for objects and maps. Default lookups are case-insensitive.
    put <aggregate> key value
    put/case <aggregate> key value
Note: PUT support is only implemented for map! values for now, series and objects support will come in a future release.

Example:
    m: #(a: 1)
    put m 'a 2
    m
    == #(
        a: 2
    )
New function

cause-error

Causes an immediate error throw, with the provided information. Use it to generate standard or custom errors.
    cause-error <type> <id> <arguments>

    <type>      : word representing an error class
    <id>        : word representing an error definition
    <arguments> : a block of optional arguments (can be empty)
Error classes and definitions can be inspected using:
    help system/catalog/errors
Example:
    red>> cause-error 'math 'zero-divide []
    *** Math error: attempt to divide by zero
    *** Where: do
Red/System additions

New natives were added in order to better support new Red features:


Other changes

The global symbol table is now hashed, so lookups are now very fast and the startup time of Red runtime greatly improved. On fast modern hardware, there is no noticeable difference, but on low-end or old hardware, it can be very significant.

Set and get natives have been extended to allow path arguments, to access a word within an associative structure (an object or a map). Moreover, they also feature a /case refinement now, allowing to distinguish target words by case (useful for maps).

Collation tables that were visible in system/locale are temporarily hidden now, until we figure out a better way to expose that feature to users, without allowing nasty errors caused by race conditions (loading a script that changes the tables unexpectedly, concurrency issues,...)

More changes:
  • exit/return are now defined as natives instead of volatile keywords.
  • do can accept error! values.
  • parse and load are now more stable when errors are raised from parsing rules.
  • load errors handling greatly improved (no console exit on syntax errors anymore).
  • value? now supports any type, except unset! as argument.
  • fixed bugs and little improvement of help output.
  • minor Redbin speed and generated payload size improvement.
  • prin output in console fixed.
  • fixed Red/System's #get directive not working in some cases.
  • system/words now defined as an object!.
  • compiler now supports system/words/ prefix to access global context words.
  • many fixes and improvements on vector! datatype, especially on math operations.
  • color definitions are now available.
  • vector! unit tests significantly extended.
  • an op! used without arguments in the interpreter now reports an error.
  • pick and poke now accept a logic! value as index.
  • added missing comparison operators for vector!.
  • paths evaluation errors in interpreter are now more accurate.
  • first memory frame allocation increased from 512KB to 1MB.
  • fixed memory corruptions caused by function with refinements in interpreter.
  • division by zero now properly caught for floats.
  • last but not least, 44 bugs reported on Github's tracker fixed in this release!

Also, thanks to PeterWAWood for the huge work on bringing a big number of new unit tests (especially for vector! datatype).

Pre-compiled runtime

This was originally the main feature planned for the release: Compile the runtime library once and cache it on disk to speed-up compilation. Unfortunately, it has been more challenging than we expected, because the Red/System compiler constructs many word values that cannot be properly serialized by Rebol's mold action, so cannot be load-ed back. Writing ad-hoc serializers and loaders was taking too much time, so this feature was postponed for a future release. We will probably implement shadow objects first, in the Red/System compiler, and reduce the use of non-serializable words before supporting the pre-compiled runtime. It is still a high priority, because it is the first step towards modular compilation.

Moving to Gitter

We moved our chat rooms to Gitter. It's not perfect, but nicley combines the features of Stackoverflow chat and AltME, where the Rebol and Red community meet. So far, so good. Gitter is still young, but very promising. You just need a Github account, which is a lower entrance barrier than Stackoverflow chat and its required 20 reputation points. Come there, say Hello, and ask questions about Red. :-)

What's next?

This release marks the beginning of the re-integration of the android branch in master. That branch has grown up a lot in the last 12 months, to the point where merging back changes from master has become a long, complex and error-prone process. The pair! and tuple! datatype implementations were pulled from the android branch, and more code will be pulled in the next release in order to close the gap between the two branches.

Finally, the time has come to have official GUI support, and that is what the next release (0.6.0) will bring. Trying to build the GUI engine, the GUI DSL, and the Android back-end at the same time was not the best plan. The development cycle on Android is slow and debugging options limited. So Windows will be the first GUI target, where we can quickly complete the engine and DSL. Then we will merge the Android GUI back-end and toolchain in a 0.6.1 release. Those two releases should come quickly, so don't go on holiday for too long this summer. ;-)

In the meantime, enjoy this new release! :-)

The Red Team.

Fork me on GitHub