ic/rodfiles

Source   Edit  

Low level binary format used by the compiler to store and load various AST and related data.

NB: this is incredibly low level and if you're interested in how the compiler works and less a storage format, you're probably looking for the ic or packed_ast modules to understand the logical format.

Overview

RodFile represents a Rod File (versioned binary format), and the associated data for common interactions such as IO and error tracking (RodFileError). The file format broken up into sections (RodSection) and preceded by a header (see: cookie). The precise layout, section ordering and data following the section are determined by the user. See ic.loadRodFile.

A basic but "wrong" example of the lifecycle:

  1. create or open - create a new one or open an existing
  2. storeHeader - header info
  3. storePrim or storeSeq - save your stuff
  4. close - and we're done

Now read the bits below to understand what's missing.

Issues with the Example

Missing Sections: This is a low level API, so headers and sections need to be stored and loaded by the user, see storeHeader & loadHeader and storeSection & loadSection, respectively.

No Error Handling: The API is centered around IO and prone to error, each operation checks or sets the RodFile.err field. A user of this API needs to handle these appropriately.

API Notes

Valid inputs for Rod files

ASTs, hopes, dreams, and anything as long as it and any children it may have support copyMem. This means anything that is not a pointer and that does not contain a pointer. At a glance these are:

  • string
  • objects & tuples (fields are recursed)
  • sequences AKA seq[T]

Note on error handling style

A flag based approach is used where operations no-op in case of a preexisting error and set the flag if they encounter one.

Misc

  • 'Prim' is short for 'primitive', as in a non-sequence type

Types

RodFile = object
  f*: File
  currentSection*: RodSection
  err*: RodFileError
Source   Edit  
RodFileError = enum
  ok, tooBig, cannotOpen, ioFailure, wrongHeader, wrongSection, configMismatch,
  includeFileChanged
Source   Edit  
RodSection = enum
  versionSection, configSection, stringsSection, checkSumsSection, depsSection,
  numbersSection, exportsSection, hiddenSection, reexportsSection,
  compilerProcsSection, trmacrosSection, convertersSection, methodsSection,
  pureEnumsSection, toReplaySection, topLevelSection, bodiesSection,
  symsSection, typesSection, typeInstCacheSection, procInstCacheSection,
  attachedOpsSection, methodsPerTypeSection, enumToStringProcsSection,
  typeInfoSection, backendFlagsSection, aliveSymsSection
Source   Edit  

Procs

proc close(f: var RodFile) {....raises: [], tags: [], forbids: [].}
Source   Edit  
proc create(filename: string): RodFile {....raises: [], tags: [], forbids: [].}
create the file and open it for writing Source   Edit  
proc loadHeader(f: var RodFile) {....raises: [IOError], tags: [ReadIOEffect],
                                  forbids: [].}
Loads the header which is described by cookie. Source   Edit  
proc loadPrim(f: var RodFile; s: var string) {....raises: [IOError],
    tags: [ReadIOEffect], forbids: [].}
Read a string, the length was stored as a prefix Source   Edit  
proc loadPrim[T](f: var RodFile; x: var T)
Load a non-sequence/string T. Source   Edit  
proc loadSection(f: var RodFile; expected: RodSection) {....raises: [IOError],
    tags: [ReadIOEffect], forbids: [].}
read the bytes value of s, sets and error if the section is incorrect. Source   Edit  
proc loadSeq[T](f: var RodFile; s: var seq[T])
T must be compatible with copyMem, see loadPrim Source   Edit  
proc open(filename: string): RodFile {....raises: [], tags: [], forbids: [].}
open the file for reading Source   Edit  
proc storeHeader(f: var RodFile) {....raises: [IOError], tags: [WriteIOEffect],
                                   forbids: [].}
stores the header which is described by cookie. Source   Edit  
proc storePrim(f: var RodFile; s: string) {....raises: [IOError],
    tags: [WriteIOEffect], forbids: [].}
Stores a string. The len is prefixed to allow for later retreival. Source   Edit  
proc storePrim[T](f: var RodFile; x: T)
Stores a non-sequence/string T. If T doesn't support copyMem and is an object or tuple then the fields are written -- the user from context will need to know which T to load. Source   Edit  
proc storeSection(f: var RodFile; s: RodSection) {....raises: [IOError],
    tags: [WriteIOEffect], forbids: [].}
update currentSection and writes the bytes value of s. Source   Edit  
proc storeSeq[T](f: var RodFile; s: seq[T])
Stores a sequence of Ts, with the len as a prefix for later retrieval. Source   Edit