macroutils

Search:
Group by:

This module is meant to supplement the macros module in the standard library. It adds bits and pieces that I've personally missed while writing macros over the years.

Creating and accessing the fields of NimNodes

One of the major things is the ability to create and access the members of nodes more easily. With this module imported you can create all of the useful nodes simply by dropping nnk from their name. So instead of doing something like this:

newStmtList(
  nnkCommand.newTree(
    newIdentNode("echo"),
    newLit("Hello world")))

You can do something like this:

StmtList(
  Command(
    Ident "echo",
    Lit "Hello world"))

This just removes a lot of noise from creating trees like these. But the procedures here are also smarter than the regular newTree, and literals are automatically converted, so the above can also be written as:

StmtList(Command("echo", "Hello world"))

The Command procedure here is aware that the first argument is often supposed to be an nnkIdent, so it will automatically convert that string into one. And the literal automatically gets turned into a nnkStrLit.

Another neat feature are the setters and getters for the properties of nodes. You might have come across code like this (if you haven't, consider yourself lucky):

procImpls[0][6][2][1][1].add(
  nnkElse.newTree(
    nnkStmtList.newTree(nnkDiscardStmt.newTree(newEmptyNode()))))

This example is taken from my protobuf module and shows how the flat structure of NimNodes with just a list of children nodes can be really confusing. If you look at the generator procedures we used above you can use the same names of the arguments there to access the nodes in those nodes. So the above can be written as:

procImpls[0].body[2].body[1].branches.add(
  Else(StmtList(DiscardStmt(Empty()))))

As you can see it is now more obvious that we're accessing the branches of the second statement in the body which is the third child of the body of the first statement in procImpls. This is still not very clear, but at least it gives us _some context to what we're doing, which makes it easier to read and debug. Thanks to the Slice type defined in this module it is also possible to add, insert, index, and assign to positions in lists of children that are offset in the node.

This alone is a useful feature when working with macros. But this module also has some more convenience things.

Traversing the tree

Often times when writing macros you want to manipulate only certain nodes within the AST. Either to parse a DSL, or to modify passed in code. For this purpose I've implemented various tree traversal procedures here. But before they're explained I want to include the normal disclaimer that traversing the tree isn't foolproof as it can contain template or macro calls that won't be expanded. It's often times better to solve replacement a different way. That being said, let's dive in.

First off we have forNode, it accepts a NimNode tree; a NimNodeKind, or set[NimNodeKind]; an action procedure; and a max depth. There are also templates that offer various variants of these arguments. It will traverse the entire tree, and for each node that matches the kind it will replace that node with the result of applying action to it. Note that it goes down the tree first, then applies the action on the way up. An example that replaces all string literals with the word "goodbye" would look like this:

ourTree.forNode(nnkStrLit, (x) => Lit"goodbye")

A version of forNode named forNodePos also exists. It takes an action with two arguments, the node that matched and a sequence of indices into the tree to get to that node. This is useful if you need to know the context of the node to change it.

As a simple helper to forNode there is also replaceAll which takes either a kind, a set of kinds, or a node along with a node to be inserted and replaces every node in the tree that has the same kind, is in the set of kinds, or is the same as the node with that node.

Verifying DSL trees

When writing DSLs it's also interesting to check if your tree is the same as the structure you wanted. This can be done by a lot of asserts and if and for statements. But with this module you can also use the sameTree procedure that compares trees. It also accepts a list of node kinds to ignore, if you need a placeholder for any kind, and you can specify a max depth as well. Combine this with forNode and you can pretty much check any passed in tree fairly easily. An example of what a sameTree check would look like:

ourTree.sameTree(quote do:
  echo "A string"
  if something:
    echo 100
)

This would return true iff ourTree was a tree that contained one call that took a string, and an if statement on a single ident, with a similar call that took an int. Note that it only verifies node kinds, so it wouldn't have to be a call to echo, merely a call to any ident. If you wanted to verify that the two echo statements where actually the same you could use forNode or forNodePos to implement that.

Building trees

One of the most welcome additions to the macros module has been the quote macro. It is able to take a tree and interpolate symbols from your surrounding code into it. Much like string interpolation works, just for the AST. But it has certain limits, the most annoying of which is that it only works for simple symbols. This module includes a superQuote macro that allows you to put anything in the quotes, and rewrites it to a normal quote statement that declares these as let statements. With this you can do things like:

proc testSuperQuote(input: untyped): untyped =
  let x = [newLit(100), newLit(200)]
  result = superQuote do:
    echo `$input[0].name`
    if `x[0]` == 300:
      echo "test"
    elif `x[1]` == 200:
      echo "hello world"

testSuperQuote:
  proc someproc()

Extracting nodes from a tree

Creating trees is all well and good, and with forNode and the accessors it's easy to get things from the tree. But to take things one step further this module also implements what is essentially a reverse superQuote statement. Since NimNode object can have a variable amount of children you can also postfix your arguments with * to collect them into a sequence of nodes. If the identifier exists it will assign or add to it, otherwise it will simply create them. With this you can do something like:

macro testExtract(input: untyped): untyped =
  var arguments = newSeq[NimNode](1) # Create space for body
  input.extract do:
    import `packages*`
    proc `procname`(`arguments*`): `retval` =
      `arguments[0]`
    let x: seq[`gen`]
  assert packages == @[Ident "one", Ident "two", Ident "three"]

testExtract:
  import one, two, three
  proc someproc(arg: int, test: string): string =
    echo "Hello world"
    echo "Hello"
  let x: seq[int]

Types

Slice = object
  offset, length: int
  node: NimNode
Object that wraps a certain slice of a NimNodes children.

Consts

AllNodeKinds = {nnkNone..nnkTupleConstr}
ContainerNodeKinds = {nnkIdent..nnkType, nnkFloat128Lit,
                    nnkComesFrom..nnkTupleConstr}

Procs

proc Ident(name: string): NimNode {...}{.raises: [], tags: [].}
proc Sym(name: static[string]): NimNode
proc RStrLit(argument: string): NimNode {...}{.raises: [], tags: [].}
proc CommentStmt(argument: string): NimNode {...}{.raises: [], tags: [].}
proc BlockStmt(body: NimNode): NimNode {...}{.raises: [], tags: [].}
proc ContinueStmt(): NimNode {...}{.raises: [], tags: [].}
proc Empty(): NimNode {...}{.raises: [], tags: [].}
proc AsmStmt(body: string | NimNode): NimNode
proc RaiseStmt(): NimNode {...}{.raises: [], tags: [].}
proc Command(name: string | NimNode; arguments: varargs[NimNode]): NimNode
proc Call(name: string | NimNode; arguments: varargs[NimNode]): NimNode
proc Infix(name: string | NimNode; left: NimNode; right: NimNode): NimNode
proc Prefix(name: string | NimNode; argument: NimNode): NimNode
proc Postfix(name: string | NimNode; argument: NimNode): NimNode
proc ExprEqExpr(left: NimNode; right: NimNode): NimNode {...}{.raises: [], tags: [].}
proc ExprColonExpr(left: NimNode; right: NimNode): NimNode {...}{.raises: [], tags: [].}
proc CallStrLit(name: string | NimNode; argument: string | NimNode): NimNode
proc DerefExpr(node: NimNode): NimNode {...}{.raises: [], tags: [].}
proc Addr(node: NimNode): NimNode {...}{.raises: [], tags: [].}
proc Cast(bracket: NimNode; node: NimNode): NimNode {...}{.raises: [], tags: [].}
proc DotExpr(left: NimNode; right: NimNode): NimNode {...}{.raises: [], tags: [].}
proc BracketExpr(node: NimNode; bracket: NimNode): NimNode {...}{.raises: [], tags: [].}
proc Par(arguments: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc Curly(arguments: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc Bracket(arguments: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc TableConstr(arguments: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc IfExpr(branches: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc IfStmt(branches: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc WhenStmt(branches: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc ElifExpr(cond: NimNode; body: NimNode): NimNode {...}{.raises: [], tags: [].}
proc ElifBranch(cond: NimNode; body: NimNode): NimNode {...}{.raises: [], tags: [].}
proc ElseExpr(body: NimNode): NimNode {...}{.raises: [], tags: [].}
proc Else(body: NimNode): NimNode {...}{.raises: [], tags: [].}
proc Pragma(arguments: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc Asgn(left: NimNode; right: NimNode): NimNode {...}{.raises: [], tags: [].}
proc StmtList(arguments: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc CaseStmt(cond: NimNode; branches: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc OfBranch(arguments: openArray[NimNode]; body: NimNode): NimNode {...}{.raises: [],
    tags: [].}
proc WhileStmt(cond: NimNode; body: NimNode): NimNode {...}{.raises: [], tags: [].}
proc ForStmt(arguments: openArray[NimNode]; iter: NimNode; body: NimNode): NimNode {...}{.
    raises: [], tags: [].}
proc TryStmt(body: NimNode; branches: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc ExceptBranch(arguments: openArray[NimNode]; body: NimNode): NimNode {...}{.raises: [],
    tags: [].}
proc Finally(body: NimNode): NimNode {...}{.raises: [], tags: [].}
proc ReturnStmt(argument: NimNode): NimNode {...}{.raises: [], tags: [].}
proc YieldStmt(argument: NimNode): NimNode {...}{.raises: [], tags: [].}
proc DiscardStmt(argument: NimNode): NimNode {...}{.raises: [], tags: [].}
proc BreakStmt(argument: NimNode): NimNode {...}{.raises: [], tags: [].}
proc BlockStmt(name: string | NimNode; body: NimNode): NimNode
proc AsmStmt(pragmas: NimNode; body: string | NimNode): NimNode
proc ImportStmt(arguments: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc ImportExceptStmt(left: NimNode; right: NimNode): NimNode {...}{.raises: [], tags: [].}
proc FromStmt(left: NimNode; right: NimNode): NimNode {...}{.raises: [], tags: [].}
proc ExportStmt(argument: NimNode): NimNode {...}{.raises: [], tags: [].}
proc ExportExceptStmt(left: NimNode; right: NimNode): NimNode {...}{.raises: [], tags: [].}
proc IncludeStmt(arguments: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc VarSection(definitions: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc LetSection(definitions: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc ConstSection(definitions: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc IdentDefs(name: string | NimNode; typ: NimNode; body: NimNode): NimNode
proc ConstDef(name: NimNode; typ: NimNode; body: NimNode): NimNode {...}{.raises: [], tags: [].}
proc Range(left: NimNode; right: NimNode): NimNode {...}{.raises: [], tags: [].}
proc AccQuoted(arguments: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc BindStmt(argument: NimNode): NimNode {...}{.raises: [], tags: [].}
proc GenericParams(definitions: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc FormalParams(retval: NimNode; arguments: varargs[NimNode]): NimNode {...}{.raises: [],
    tags: [].}
proc OfInherit(argument: NimNode): NimNode {...}{.raises: [], tags: [].}
proc VarTy(argument: NimNode): NimNode {...}{.raises: [], tags: [].}
proc ProcDef(name: NimNode; generics: NimNode; params: NimNode; pragmas: NimNode;
            body: NimNode): NimNode {...}{.raises: [], tags: [].}
proc FuncDef(name: NimNode; generics: NimNode; params: NimNode; pragmas: NimNode;
            body: NimNode): NimNode {...}{.raises: [], tags: [].}
proc IteratorDef(name: NimNode; generics: NimNode; params: NimNode; pragmas: NimNode;
                body: NimNode): NimNode {...}{.raises: [], tags: [].}
proc ConverterDef(name: NimNode; generics: NimNode; params: NimNode; pragmas: NimNode;
                 body: NimNode): NimNode {...}{.raises: [], tags: [].}
proc MethodDef(name: NimNode; generics: NimNode; params: NimNode; pragmas: NimNode;
              body: NimNode): NimNode {...}{.raises: [], tags: [].}
proc TemplateDef(name: NimNode; terms: NimNode; generics: NimNode; params: NimNode;
                pragmas: NimNode; body: NimNode): NimNode {...}{.raises: [], tags: [].}
proc MacroDef(name: NimNode; terms: NimNode; generics: NimNode; params: NimNode;
             pragmas: NimNode; body: NimNode): NimNode {...}{.raises: [], tags: [].}
proc TypeSection(definitions: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc TypeDef(name: string | NimNode; generics: NimNode; typ: NimNode): NimNode
proc Defer(body: NimNode): NimNode {...}{.raises: [], tags: [].}
proc RaiseStmt(node: NimNode): NimNode {...}{.raises: [], tags: [].}
proc MixinStmt(node: NimNode): NimNode {...}{.raises: [], tags: [].}
proc TupleTy(node: NimNode): NimNode {...}{.raises: [], tags: [].}
proc PtrTy(node: NimNode): NimNode {...}{.raises: [], tags: [].}
proc RefTy(node: NimNode): NimNode {...}{.raises: [], tags: [].}
proc DistinctTy(node: NimNode): NimNode {...}{.raises: [], tags: [].}
proc EnumTy(definitions: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc TypeClassTy(arglist: NimNode; body: NimNode): NimNode {...}{.raises: [], tags: [].}
proc Arglist(arguments: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc ProcTy(name: NimNode; generics: NimNode; params: NimNode; pragmas: NimNode;
           body: NimNode): NimNode {...}{.raises: [], tags: [].}
proc IteratorTy(name: NimNode; generics: NimNode; params: NimNode; pragmas: NimNode;
               body: NimNode): NimNode {...}{.raises: [], tags: [].}
proc StaticTy(node: NimNode): NimNode {...}{.raises: [], tags: [].}
proc ConstTy(node: NimNode): NimNode {...}{.raises: [], tags: [].}
proc MutableTy(node: NimNode): NimNode {...}{.raises: [], tags: [].}
proc SharedTy(node: NimNode): NimNode {...}{.raises: [], tags: [].}
proc StaticStmt(body: NimNode): NimNode {...}{.raises: [], tags: [].}
proc UsingStmt(definitions: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc TypeOfExpr(node: NimNode): NimNode {...}{.raises: [], tags: [].}
proc ObjectTy(pragmas: NimNode; inherits: NimNode; body: NimNode): NimNode {...}{.raises: [],
    tags: [].}
proc RecList(arguments: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc RecCase(cond: NimNode; branches: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc RecWhen(branches: varargs[NimNode]): NimNode {...}{.raises: [], tags: [].}
proc EnumFieldDef(left: NimNode; right: NimNode): NimNode {...}{.raises: [], tags: [].}
proc ObjConstr(name: string | NimNode; definitions: varargs[NimNode]): NimNode
proc forNodePos(node: NimNode; kind: NimNodeKind or NimNodeKinds;
               action: proc (x: NimNode; y: seq[int]): NimNode; depth, maxDepth: int;
               expr: seq[int] = @[]): NimNode {...}{.discardable.}
Takes a NimNode and a NimNodeKind (or a set of NimNodeKind) and applies the procedure passed in as action to every node that matches the kind throughout the tree. The x passed to action is not the node itself, but rather it's position in the tree as nested BrakcketExpr s. NOTE: This modifies the original node tree and only returns for easy chaining. 
proc forNode(node: NimNode; kind: NimNodeKind or NimNodeKinds;
            action: proc (x: NimNode): NimNode; depth, maxDepth: int): NimNode {...}{.
    discardable.}
Takes a NimNode and a NimNodeKind (or a set of NimNodeKind) and applies the procedure passed in as action to every node that matches the kind throughout the tree. The x passed to action is the node in the tree. NOTE: This modifies the original node tree and only returns for easy chaining. 
proc sameTree(node: NimNode; ignored: NimNodeKind or NimNodeKinds; comp: NimNode;
             depth = 0; maxDepth = int.high): bool
Compares two NimNode trees and verifies that the structure and all kinds are the same.

Iterators

iterator items(s: Slice): NimNode {...}{.raises: [], tags: [].}
Iterate over the nodes in a NimNode slice

Converters

converter Lit(x: char): NimNode {...}{.raises: [], tags: [].}
converter Lit(x: int): NimNode {...}{.raises: [], tags: [].}
converter Lit(x: int8): NimNode {...}{.raises: [], tags: [].}
converter Lit(x: int16): NimNode {...}{.raises: [], tags: [].}
converter Lit(x: int32): NimNode {...}{.raises: [], tags: [].}
converter Lit(x: int64): NimNode {...}{.raises: [], tags: [].}
converter Lit(x: uint): NimNode {...}{.raises: [], tags: [].}
converter Lit(x: uint8): NimNode {...}{.raises: [], tags: [].}
converter Lit(x: uint16): NimNode {...}{.raises: [], tags: [].}
converter Lit(x: uint32): NimNode {...}{.raises: [], tags: [].}
converter Lit(x: uint64): NimNode {...}{.raises: [], tags: [].}
converter Lit(x: bool): NimNode {...}{.raises: [], tags: [].}
converter Lit(x: string): NimNode {...}{.raises: [], tags: [].}
converter Lit(x: float32): NimNode {...}{.raises: [], tags: [].}
converter Lit(x: float64): NimNode {...}{.raises: [], tags: [].}

Macros

macro superQuote(x: untyped): untyped
Converts the input to a quote statement, but lifts out the content of the quoted sections as variables which allows you to do things like:
macro someMacro(input: untyped): untyped =
  let x = [newLit(100), newLit(200)]
  result = superQuote do:
    echo `$input[0].name` # If input is a procedure, this is the name of the procedure
    if `x[0]` == 300: # Grabs the first literal
      echo "test"
    elif `x[1]` == 200: # Grabs the second literal
      echo "hello world"

Note that the content of a quoted node is not parsed by Nim, so the content of the quoted node goes through parseExpr which might introduce some weird behaviour for complex statements.

macro extract(ast, pattern: untyped): untyped
A reverse superQuote macro, takes an AST and a pattern with the same tree structure but where nodes can be replaced by quoted statements. The macro will then assign the node in the tree to this statement. If the statement ends with * it will be a sequence of NimNode that gets added to.

Templates

template `=`(s: Slice; values: openArray[NimNode]): untyped
template `[]`(s: Slice; idx: int): untyped
Access child in a NimNode slice 
template `[]=`(s: Slice; idx: int; val: untyped): untyped
Assign to child in a NimNode slice 
template add(s: Slice; val: untyped): untyped
Add a child into a NimNode slice 
template insert(s: Slice; pos: untyped; val: untyped): untyped
Insert a child into a NimNode slice 
template len(s: Slice): untyped
Get length of NimNode slice 
template definitions(x: NimNode): untyped
template definitions=(x: NimNode; definitions: untyped): untyped
template terms(x: NimNode): untyped
template terms=(x: NimNode; terms: untyped): untyped
template pragmas(x: NimNode): untyped
template pragmas=(x: NimNode; pragmas: untyped): untyped
template iter(x: NimNode): untyped
template iter=(x: NimNode; iter: untyped): untyped
template left(x: NimNode): untyped
template left=(x: NimNode; left: untyped): untyped
template typ(x: NimNode): untyped
template typ=(x: NimNode; typ: untyped): untyped
template arguments(x: NimNode): untyped
template arguments=(x: NimNode; arguments: untyped): untyped
template argument(x: NimNode): untyped
template argument=(x: NimNode; argument: untyped): untyped
template params(x: NimNode): untyped
template params=(x: NimNode; params: untyped): untyped
template arglist(x: NimNode): untyped
template arglist=(x: NimNode; arglist: untyped): untyped
template body(x: NimNode): untyped
template body=(x: NimNode; body: untyped): untyped
template generics(x: NimNode): untyped
template generics=(x: NimNode; generics: untyped): untyped
template name(x: NimNode): untyped
template name=(x: NimNode; name: untyped): untyped
template inherits(x: NimNode): untyped
template inherits=(x: NimNode; inherits: untyped): untyped
template cond(x: NimNode): untyped
template cond=(x: NimNode; cond: untyped): untyped
template node(x: NimNode): untyped
template node=(x: NimNode; node: untyped): untyped
template bracket(x: NimNode): untyped
template bracket=(x: NimNode; bracket: untyped): untyped
template branches(x: NimNode): untyped
template branches=(x: NimNode; branches: untyped): untyped
template retval(x: NimNode): untyped
template retval=(x: NimNode; retval: untyped): untyped
template right(x: NimNode): untyped
template right=(x: NimNode; right: untyped): untyped
template forNodePos(node: NimNode; kind: NimNodeKind or NimNodeKinds; maxdepth: int;
                   action: proc (x: NimNode; y: seq[int]): NimNode): NimNode
template forNodePos(node: NimNode; kind: NimNodeKind or NimNodeKinds;
                   action: proc (x: NimNode; y: seq[int]): NimNode): NimNode
template forNode(node: NimNode; kind: NimNodeKind or NimNodeKinds; maxdepth: int;
                action: proc (x: NimNode): NimNode): NimNode
template forNode(node: NimNode; kind: NimNodeKind or NimNodeKinds;
                action: proc (x: NimNode): NimNode): NimNode
template replaceAll(node: NimNode; kind: NimNodeKind or NimNodeKinds; replace: NimNode;
                   maxdepth = int.high): NimNode
Replaces all nodes of kind in the tree given by node with replace 
template replaceAll(node: NimNode; find: NimNode; replace: NimNode; maxdepth = int.high): NimNode
Replaces all nodes that are equal to find in the tree given by node with replace 
template sameTree(node: NimNode; comp: NimNode): bool
template sameTree(node: NimNode; maxdepth: int; comp: NimNode): bool

Made with Nim. Generated: 2020-03-16 14:42:13 UTC