Roslyn - How to create a custom debuggable scripting language?

Home

A screenshot comes from Visual Studio 2015

Sometime ago I decided to play a little bit with Cakebuild. It's a build automation tool/system that allows you to write build scripts using C# domain specific language. What's more it is possible to debug these scripts in Visual Studio. It is interesting because Cake scripts are neither "normal" C# files nor are added to projects (csproj). I was curious how it was achieved and it is result of my analysis. I'll tell you how to create a simple debuggable scripting language. By debuggable I mean that it'll be possible to debug scripts in our language in Visual Studio almost as any "normal" program in C#. Cakebuild uses Roslyn i.e. a compiler as a service from Microsft and we'll do the same.

Language

Our sample language will support write command. It's trivial and it'll simply write something to a console. Additionally it'll be possible to use any C# expression. Here is a sample script:

var i = 0;
write a
write 1
i++;
write 2
i++;
System.Console.WriteLine(i);
write b

Transalation

We'd like to use Roslyn to compile our scripts. It means that firstly we need to translate our language into C#. Here is a code that will do so. It simply reads line by line. If a line starts with write then it is replaced with Console.WriteLine. Otherwise it's assumed that a line contains C# code so no action is needed.

var lines = File.ReadAllLines(path);
var sb = new StringBuilder();
foreach (var l in lines)
{
   if (l.StartsWith("write"))
   {
      var res = l.Substring(l.IndexOf(" ", StringComparison.Ordinal)).Trim();
      sb.AppendLine($"System.Console.WriteLine(\"{res}\");");
   }
   else
      sb.AppendLine(l);
}

Compilation

Now we need to compile our script. The following snipped shows how to do that (it is based on DebugXPlatScriptSession class from Cakebuild project). It is worth noting that we used OptimizationLevel.Debug which disables all optimizations what improves debugging experience. The result of compilations are two streams: assemblyStream contains actual compiled code and symbolStream contains symbols.

var options = Microsoft.CodeAnalysis.Scripting.ScriptOptions.Default;
var roslynScript = CSharpScript.Create(script, options);
var compilation = roslynScript.GetCompilation();

compilation = compilation.WithOptions(compilation.Options
   .WithOptimizationLevel(OptimizationLevel.Debug)
   .WithOutputKind(OutputKind.DynamicallyLinkedLibrary));

using (var assemblyStream = new MemoryStream())
{
   using (var symbolStream = new MemoryStream())
   {
      var emitOptions = new EmitOptions(false, DebugInformationFormat.PortablePdb);
      var result = compilation.Emit(assemblyStream, symbolStream, options: emitOptions);
      if (!result.Success)
      {
         var errors = string.Join(Environment.NewLine, result.Diagnostics.Select(x => x));
         Console.WriteLine(errors);
         return;
      }

       //Execute the script
   }
}

Execution

Now we are ready to execute our script. To to so firstly we need to load an assembly and symbols that were generated by Roslyn Then we need to find Submission#0 class and <Factory> method in order to call it. These strange names are generated by Roslyn and execution of <Factory> method is equal to executing our script.

var assembly = Assembly.Load(assemblyStream.ToArray(), symbolStream.ToArray());
var type = assembly.GetType("Submission#0");
var method = type.GetMethod("<Factory>", BindingFlags.Static | BindingFlags.Public);

method.Invoke(null, new object[] {new object[2]});

The last step

I deliberately didn't write one thing. If we run the code know, the script will be executed correctly. However, the debugging will not work i.e. if we create a breakpoint in the script it won't hit. Why? Because a debugger doesn't know anything about a file with a script.

In other words there is no link between symbols and this file. In order to fix this problem we need to use line directive as follows. It tells a debugger that a debuggable code starts in a given line and that it should use a given file while debugging. Here is a modified version of the snippet that performs translation:

...
var sb = new StringBuilder();
sb.AppendLine($"#line 1 \"{path}\"");
foreach (var l in lines)
...

Summary

You will find the working example on my Github in the Roslyn repository. After download, open the solution, go to the RoslynScriptDebugging project, select and open the script file and put a breakpoint in any line. Finally press F5 and vualá!