May 14, 2011

Red/System compiler overview

Source Navigation

As requested by several users, I am giving a little more insights on the Red/System compiler inner workings and a map for navigating in the source code.

Current Red/System source tree:
 red-system/
%compiler.r ; Main compiler code, loads everything else
%emitter.r ; Target code emitter abstract layer
%linker.r ; Format files loader
%rsc.r ; Compiler's front-end for standalone usage

formats/ ; Contains all supported executable formats
%PE.r ; Windows PE/COFF file format emitter
%ELF.r ; UNIX ELF file format emitter

library/ ; Third-party libraries

runtime/ ; Contains all runtime definitions
%common.reds ; Cross-platform definitions
%win32.r ; Windows-specific bindings
%linux.r ; Linux-specific bindings

targets/ ; Contains target execution unit code emitters
%target-class.r ; Base utility class for emitters
%IA32.r ; Intel IA32 code emitter

tests/ ; Unit tests

Once the compiler code is loaded in memory, the objects hierarchy looks like:
 system/words/          ; global REBOL context

system-dialect/ ; main object
loader/ ; preprocessor object
process ; preprocessor entry point function
compiler/ ; compiler object
compile ; compiler entry point function

emitter/ ; code emitter object
compiler/ ; short-cut reference to compiler object
target/ ; reference the target code emitter object
compiler/ ; short-cut reference to compiler object

linker/ ; executable file emitter
PE/ ; Windows PE/COFF format emitter object
ELF/ ; UNIX ELF format emitter object


Note: the linker file formats are currently statically loaded, this will be probably changed to a dynamic loading model.

Compilation Phases

The compilation is a process that goes through several phases to transform a textual source code to an executable file. Here is an overview of the process:


1) Source loading

This is a preparatory phase that would convert the text source code to its memory representation (close to an AST). This is achieved in 3 steps:
  1. source is preprocessed in its text form to make the syntax REBOL-compatible
  2. source is converted to a tree of nested blocks using the REBOL's LOAD function
  3. source is postprocessed to interpret some compiler directives (like #include and #define)
2) Compilation

The compiler walks through the source tree using a recursive descent parser. It attempts to match keywords and values with its internal rules and emits:
  • the corresponding machine code in the code buffer
  • the global variables and complex data (c-string! and struct! literals) in the data buffer
The internal entry point function for the compilation is compiler/comp-dialect. All the compiler/comp-* functions are used to recursively analyze the source code and each one matches a specific semantic rule (or set of rules) from the Red/System language specification.

The production of native code is direct, there is no intermediary representation, machine code is generated as soon as a language statement or expression is matched. This is the simplest way to do it, but code cannot be efficiently optimised without a proper Intermediate Representation (IR). When Red/System will be rewritten in Red, a simple IR will be introduced to enable the full range of possible code optimisations.

As you know, a Red/System program entry point is at the beginning of the source code. During the compilation, the source code in the global context is compiled first and all functions are collected and compiled after all global code. So the generated code is always organised the same way:
  • global code (including proper program finalization)
  • function 1
  • function 2
  • ...
The results of the compilation process are:
  • a global symbol table
  • a machine code buffer
  • a global data buffer
  • a list of functions from external libraries (usually these are OS API mappings)
The compiler is able to process one or several source files this way before entering the linking phase.

3) Linking

The linking process goal is to create an executable file that could be loaded by the target platform. So the executable file needs to conform to the target ABI for that, like PE for Windows or ELF for Linux.

During the linking, the global symbol table is used to "patch" the code and data buffer (see linker/resolve-symbol-refs), inserting the final memory address for the pointed resources (variable, function, global data). The different parts to assemble are grouped into so-called "sections", that can be themselves be grouped into "segments" (as, e.g., in ELF).

Finally, some headers describing the file and its sections/segments are inserted to complete the file. The file is then written down on disk, marking the end of the whole process.

Static linking of external libraries (*.lib, *.a,...) will be added at some point in the future (when the need for such feature will appear).


I hope this short description gives you a better picture on how Red/System compiler works, even if it is probably obvious for the most experienced readers. Feel free to ask for more in the comments, or better, on the Google Groups mailing-list.

5 comments:

  1. Again, OS X and BSD left out of the picture...
    Should targets like Android and iPhone be considered at this point in order to clarify a GUI roadmap to cover all the bases?

    It needn't be part of the current development, as Red is currently still on the runway, but surely you should be considering where you are going and accepting as many passengers as possible! Otherwise it's just another Rebol...

    ReplyDelete
  2. You seems a bit misinformed:
    - OS X support is under work, but not ready yet.
    - BSD: which *BSD flavor are you referring to? AFAICT, the current ELF emitter should work for FreeBSD too (don't know about other *BSD)? Maybe you could help with the port of Red/System to your personal favorite BSD?

    Porting Red to Android and iOS platforms is a clearly distinct task from adding GUI support. Raw ports on these platforms will most probably be achieved before any work on GUI will be done.

    Clearly working on GUI support now is premature, Red implementation has not yet started and Red/System is not yet 100% finished. If you have propositions for GUI support, please post them on the mailing-list for discussion and future reference.

    Thank for your advice, but I don't feel lost on my way and my roadmap is clearly explained in my initial blog post: http://www.red-lang.org/2011/02/journey-begins.html. I will publish soon a more detailed one.

    As you might have noticed, and contrary to REBOL, Red is a fully OSS project. I am glad you are so concern by Red not being "another Rebol", so I guess this means that you will invest your time in helping porting Red & Red/System on the various platforms you've mentioned?

    ReplyDelete
  3. I translate this article to chinese: https://github.com/red/red/wiki/Red-System-compiler-overview-chinese-translation

    ReplyDelete
  4. I wish there was a simple compiler tutorial.

    ReplyDelete
  5. This comment has been removed by the author.

    ReplyDelete

Fork me on GitHub