腳本文檔_驗證腳本文檔– EditorTests的樂趣

時間 2021-07-14 標籤 java python web 單元測試 gwt

腳本文檔

When I encounter an API in Unity that I am unfamiliar with the first thing I (and most of us) do is go to the Unity Scripting API Manual to see how it works via one of the examples. If that example will not compile when I try it, I assume that I must be doing something wrong. The example couldn’t possibly be broken, could it…?

當我在Unity中遇到一個我不熟悉的第一件事(以及我們大多數人)時，請去Unity Scripting API手冊，通過其中一個示例瞭解它的工作方式。如果在嘗試時無法編譯該示例，則認爲我必須做錯了什麼。這個例子不可能被打破，是吧？

This is how I discovered that we do indeed have examples in our scripting docs that do not compile, as a result of API changes over time and the odd case of an the example never compiling to start with. At Unity we have a lot of freedom in how we work; if we see a problem we can report it to the relevant team or fix it ourselves. At one of our recent Sustained Engineering team weeks we decided to do our own hackweek and picked several issues we wanted to tackle. Some of us chose to look into a solution for there being broken examples in the scripting docs.

這就是我發現腳本編寫文檔中確實存在無法編譯的示例的原因，這是由於API隨時間而變化以及該示例從未編譯開始的奇怪情況。在Unity，我們在工作方式上有很多自由。如果我們發現問題，可以向相關團隊報告或自行解決。在我們最近的「可持續工程團隊」中，有一個星期我們決定進行自己的「黑客周」活動，並挑選了一些我們想解決的問題。我們中有些人選擇研究解決方案，以解決腳本文檔中的破例。

There are about 15,000 scripting docs pages. Not all of them contain examples (a different problem which we are working to improve); however a large portion do. Going through each example and testing them manually would be unachievable in a week. It would not solve the problem of API changes or broken examples being written in the future either either.

大約有15,000個腳本文檔頁面。並非所有示例都包含示例(我們正在努力改善的另一個問題)；但是很大一部分。在一週之內無法完成每個示例並對其進行手動測試。它也不會解決API更改或將來編寫的示例不完整的問題。

Last year as part of the Unity 5.3 release we included a new feature called the Editor Test Runner. This is a unit test framework that can be run from within Unity. We have been using the Editor Test Runner internally for our own automated tests since its introduction. I decided to tackle the problem using an editor test. All our scripting docs are stored in XML files which we edit through an internal Unity project.

去年，作爲Unity 5.3版本的一部分，我們加入了一項稱爲「編輯器測試運行器」的新功能。這是一個可以在Unity內部運行的單元測試框架。自推出以來，我們一直在內部將Editor Test Runner用於我們自己的自動化測試。我決定使用編輯器測試解決問題。我們所有的腳本文檔都存儲在XML文件中，我們可以通過內部Unity項目進行編輯。

The code to parse all these files is already available in this project so it made sense to add the editor test into the same project so we could reuse it.

解析所有這些文件的代碼已經在該項目中可用，因此將編輯器測試添加到同一項目中是有意義的，以便我們可以重用它。

In our editor test framework (which is using NUnit) there is an attribute that can be applied to a test called TestCaseSource. This lets a test be run multiple times with different source data. In this case the source data would be our list of script examples.

在我們的編輯器測試框架(使用NUnit)中，有一個屬性可以應用於名爲TestCaseSource的測試。這樣就可以使用不同的源數據多次運行測試。在這種情況下，源數據將是我們的腳本示例列表。

public class ScriptVerification

{

public static IEnumerable TestFiles

{

get

{

// Get all the xml files

var files = Directory.GetFiles("OurDocsApiPath/*.mem.xml", SearchOption.AllDirectories);

// Each file is a separate test.

foreach (var file in files)

{

string testName = Path.GetFileName(file).Replace(k_FileExtension, "");

yield return new TestCaseData(file).SetName(testName);

}

[Test]

[TestCaseSource("TestFiles")]

public void TestDocumentationExampleScripts(string docXmlFile)

{

// Do the test

}

public class ScriptVerification

{

public static IEnumerable TestFiles

{

get

{

// Get all the xml files

var files = Directory . GetFiles ( "OurDocsApiPath/*.mem.xml" , SearchOption . AllDirectories ) ;

// Each file is a separate test.

foreach ( var file in files )

{

string testName = Path . GetFileName ( file ) . Replace ( k_FileExtension , "" ) ;

yield return new TestCaseData ( file ) . SetName ( testName ) ;

}

[ Test ]

[ TestCaseSource ( "TestFiles" ) ]

public void TestDocumentationExampleScripts ( string docXmlFile )

{

// Do the test

}

Using this method now shows a list of all the tests that will be run in the test runner. Each test can be run individually or they can all be run using the Run All option.

現在，使用此方法將顯示將在測試運行器中運行的所有測試的列表。每個測試可以單獨運行，也可以使用「全部運行」選項全部運行。

To compile the examples we use CodeDomProvider. It allows us to pass in one or more strings that represent a script, and it will compile and return information on errors and warnings.

爲了編譯示例，我們使用CodeDomProvider 。它允許我們傳入一個或多個表示腳本的字符串，它將編譯並返回有關錯誤和警告的信息。

This is a cutdown version (XML parsing removed) of the first iteration of the test:

這是測試的第一次迭代的簡化版本(刪除了XML解析)：

using UnityEngine;
using NUnit.Framework;
using System.CodeDom.Compiler;
using System.Collections;
using System.Reflection;
using System.Xml;
using System.IO;
using UnityEditor;

public class ScriptVerification
{
    public static IEnumerable TestFiles
    {
        get
        {
            // Get all the xml files
            var files = Directory.GetFiles("OurDocsApiPath/*.mem.xml", SearchOption.AllDirectories);

// Each file is a seperate test
            foreach (var file in files)
            {
                string testName = Path.GetFileName(file).Replace(k_FileExtension, "");
                yield return new TestCaseData(file).SetName(testName);
            }
        }
    }

CodeDomProvider m_DomProvider;
    CompilerParameters m_CompilerParams;

[SetUp]
    public void InitScriptCompiler()
    {
        m_DomProvider = CodeDomProvider.CreateProvider("CSharp");
        m_CompilerParams = new CompilerParameters
        {
           GenerateExecutable = false,
           GenerateInMemory = false,
           TreatWarningsAsErrors = false,
        };
        Assembly unityEngineAssembly = Assembly.GetAssembly(typeof(MonoBehaviour));
        Assembly unityEditorAssembly = Assembly.GetAssembly(typeof(Editor));
        m_CompilerParams.ReferencedAssemblies.Add(unityEngineAssembly.Location);
        m_CompilerParams.ReferencedAssemblies.Add(unityEditorAssembly.Location);
    }

[Test]
    [TestCaseSource("TestFiles")]
    public void TestDocumentationExampleScripts(string docXmlFile)
    {
        // Parse the xml and extract the scripts
        // foreach script example in our doc call TestCsharpScript
    }

void TestCsharpScript(string scriptText)
    {
        // Check for errors
        CompilerResults compilerResults = m_DomProvider.CompileAssemblyFromSource(m_CompilerParams, scriptText);
 
        string errors = "";
        if (compilerResults.Errors.HasErrors)
        {
            foreach (CompilerError compilerError in compilerResults.Errors)
            {
                errors += compilerError.ToString() + "\n";
            } 
        } 
        Assert.IsFalse(compilerResults.Errors.HasErrors, errors);
    }
}

using UnityEngine;
using NUnit.Framework;
using System.CodeDom.Compiler;
using System.Collections;
using System.Reflection;
using System.Xml;
using System.IO;
using UnityEditor;

CodeDomProvider m_DomProvider;
    CompilerParameters m_CompilerParams;

using UnityEngine;

using NUnit.Framework;

using System.CodeDom.Compiler;

using System.Collections;

using System.Reflection;

using System.Xml;

using System.IO;

using UnityEditor;

public class ScriptVerification

{

public static IEnumerable TestFiles

{

get

{

// Get all the xml files

var files = Directory.GetFiles("OurDocsApiPath/*.mem.xml", SearchOption.AllDirectories);

// Each file is a seperate test

foreach (var file in files)

{

string testName = Path.GetFileName(file).Replace(k_FileExtension, "");

yield return new TestCaseData(file).SetName(testName);

}

CodeDomProvider m_DomProvider;

CompilerParameters m_CompilerParams;

[SetUp]

public void InitScriptCompiler()

{

m_DomProvider = CodeDomProvider.CreateProvider("CSharp");

m_CompilerParams = new CompilerParameters

{

GenerateExecutable = false,

GenerateInMemory = false,

TreatWarningsAsErrors = false,

};

Assembly unityEngineAssembly = Assembly.GetAssembly(typeof(MonoBehaviour));

Assembly unityEditorAssembly = Assembly.GetAssembly(typeof(Editor));

m_CompilerParams.ReferencedAssemblies.Add(unityEngineAssembly.Location);

m_CompilerParams.ReferencedAssemblies.Add(unityEditorAssembly.Location);

}

[Test]

[TestCaseSource("TestFiles")]

public void TestDocumentationExampleScripts(string docXmlFile)

{

// Parse the xml and extract the scripts

// foreach script example in our doc call TestCsharpScript

}

void TestCsharpScript(string scriptText)

{

// Check for errors

CompilerResults compilerResults = m_DomProvider.CompileAssemblyFromSource(m_CompilerParams, scriptText);

string errors = "";

if (compilerResults.Errors.HasErrors)

{

foreach (CompilerError compilerError in compilerResults.Errors)

{

errors += compilerError.ToString() + "\n";

}

Assert.IsFalse(compilerResults.Errors.HasErrors, errors);

}

using UnityEngine ;

using NUnit . Framework ;

using System . CodeDom . Compiler ;

using System . Collections ;

using System . Reflection ;

using System . Xml ;

using System . IO ;

using UnityEditor ;

public class ScriptVerification

{

public static IEnumerable TestFiles

{

get

{

// Get all the xml files

var files = Directory . GetFiles ( "OurDocsApiPath/*.mem.xml" , SearchOption . AllDirectories ) ;

// Each file is a seperate test

foreach ( var file in files )

{

string testName = Path . GetFileName ( file ) . Replace ( k_FileExtension , "" ) ;

yield return new TestCaseData ( file ) . SetName ( testName ) ;

}

CodeDomProvider m_DomProvider ;

CompilerParameters m_CompilerParams ;

[ SetUp ]

public void InitScriptCompiler ( )

{

m_DomProvider = CodeDomProvider . CreateProvider ( "CSharp" ) ;

m_CompilerParams = new CompilerParameters

{

GenerateExecutable = false ,

GenerateInMemory = false ,

TreatWarningsAsErrors = false ,

} ;

Assembly unityEngineAssembly = Assembly . GetAssembly ( typeof ( MonoBehaviour ) ) ;

Assembly unityEditorAssembly = Assembly . GetAssembly ( typeof ( Editor ) ) ;

m_CompilerParams . ReferencedAssemblies . Add ( unityEngineAssembly . Location ) ;

m_CompilerParams . ReferencedAssemblies . Add ( unityEditorAssembly . Location ) ;

}

[ Test ]

[ TestCaseSource ( "TestFiles" ) ]

public void TestDocumentationExampleScripts ( string docXmlFile )

{

// Parse the xml and extract the scripts

// foreach script example in our doc call TestCsharpScript

}

void TestCsharpScript ( string scriptText )

{

// Check for errors

CompilerResults compilerResults = m_DomProvider . CompileAssemblyFromSource ( m_CompilerParams , scriptText ) ;

string errors = "" ;

if ( compilerResults . Errors . HasErrors )

{

foreach ( CompilerError compilerError in compilerResults . Errors )

{

errors += compilerError . ToString ( ) + "\n" ;

}

Assert . IsFalse ( compilerResults . Errors . HasErrors , errors ) ;

}

And it worked! We needed to make some small changes in how we compile the examples, though, as some scripts are designed to go together as a larger example. To check for this we compiled them separately; if we found an error, we then compiled them again combined to see if that worked.

而且有效！但是，我們需要對示例的編譯方式進行一些小的更改，因爲某些腳本被設計爲可以作爲一個更大的示例一起使用。爲了對此進行檢查，我們分別編譯了它們。如果發現錯誤，則將它們重新組合起來以查看是否可行。

Some examples are written as single lines of code which are not wrapped in a class or function. We could fix this by wrapping them in our test, but we have a rule that all examples should compile standalone (i.e. if a user copies and pastes it into a new file it should compile and work), so we count those examples as test failures.

一些示例是用單行代碼編寫的，這些代碼沒有包裝在類或函數中。我們可以通過將它們包裝在測試中來解決此問題，但是我們有一個規則，即所有示例都應獨立編譯(即，如果用戶將其複製並粘貼到新文件中，則可以編譯並正常工作)，因此我們將這些示例視爲測試失敗。

The test was now in a state where it could be run as part of our build verification on the path to trunk. However there was one small problem: the test took 30 minutes to run. This is far too long for a test running in build verification, considering we run around 7000 builds a day.

現在，測試處於可以在通往中繼路徑的構建驗證中運行的狀態。但是，有一個小問題：測試耗時30分鐘。考慮到我們每天運行大約7000個構建，對於構建驗證中運行的測試來說，這太長了。

The test was running sequentially, one script after another, but there was no reason we could not run them in parallel as the tests were independent of each other and did not need to make any calls to the Unity API;and we are only testing that they compile, not the behaviour. Introducing ThreadPool, a .NET API that can be used to execute tasks in parallel. We push the tests as individual tasks into the ThreadPool and they will be executed as soon as a thread becomes available. This needs to be driven from a single function, meaning that we can’t have individual NUnit test cases for testing specific examples from the docs. As a result we lose the ability to run any one of the tests individually, but we gain the ability to run them all quickly.

測試按順序運行，一個腳本接一個腳本運行，但是沒有理由我們不能並行運行它們，因爲測試彼此獨立並且不需要對Unity API進行任何調用；而我們只是在測試那個它們會編譯，而不是行爲。引入ThreadPool ，這是一個.NET API，可用於並行執行任務。我們將測試作爲單獨的任務推送到ThreadPool中，並在線程可用時立即執行它們。這需要從單個函數驅動，這意味着我們無法使用單獨的NUnit測試用例來測試文檔中的特定示例。結果，我們失去了單獨運行任何一個測試的能力，但獲得了快速運行所有測試的能力。

[Test]
public void ScriptVerificationCSharp()
{
    // Setup. Start all tests running on multiple threads.
    s_ThreadEvents = new ManualResetEvent[s_DocInfo.Count];

for (int i = 0; i < s_DocInfo.Count; ++i)
    {
        // Queue this example up for testing
        s_ThreadEvents[i] = new ManualResetEvent(false);
        ThreadPool.QueueUserWorkItem(TestDocumentationExampleScriptsThreaded, i);
    }
    // Check for errors and build the error output if required.
    bool testFailed = false;
    StringBuilder results = new StringBuilder();
    for (int i = 0; i < s_ThreadEvents.Length; ++i)
    {
        // Wait for the test to finish.
        s_ThreadEvents[i].WaitOne();
     
        if (s_DocInfo[i].status == TestStatus.Failed)
        {
            testFailed = true;
            GenerateFailureMessage(results, s_DocInfo[i]);
        }
    }

// If a single item has failed then the test is considered a failure.
    Assert.IsFalse(testFailed, results.ToString());
}

public static void TestDocumentationExampleScriptsThreaded(object o)
{
    var infoIdx = (int)o;
    var info = s_DocInfo[infoIdx];
    try
    {
        TestScriptsCompile(info);
    }
    catch (Exception e)
    {
        info.status = TestStatus.Failed;
        info.testRunnerFailure = e.ToString();
    }
    finally
    {
        s_ThreadEvents[infoIdx].Set();
    }
}

[Test]
public void ScriptVerificationCSharp()
{
    // Setup. Start all tests running on multiple threads.
    s_ThreadEvents = new ManualResetEvent[s_DocInfo.Count];

// If a single item has failed then the test is considered a failure.
    Assert.IsFalse(testFailed, results.ToString());
}

[Test]

public void ScriptVerificationCSharp()

{

// Setup. Start all tests running on multiple threads.

s_ThreadEvents = new ManualResetEvent[s_DocInfo.Count];

for (int i = 0; i < s_DocInfo.Count; ++i)

{

// Queue this example up for testing

s_ThreadEvents[i] = new ManualResetEvent(false);

ThreadPool.QueueUserWorkItem(TestDocumentationExampleScriptsThreaded, i);

}

// Check for errors and build the error output if required.

bool testFailed = false;

StringBuilder results = new StringBuilder();

for (int i = 0; i < s_ThreadEvents.Length; ++i)

{

// Wait for the test to finish.

s_ThreadEvents[i].WaitOne();

if (s_DocInfo[i].status == TestStatus.Failed)

{

testFailed = true;

GenerateFailureMessage(results, s_DocInfo[i]);

}

// If a single item has failed then the test is considered a failure.

Assert.IsFalse(testFailed, results.ToString());

}

public static void TestDocumentationExampleScriptsThreaded(object o)

{

var infoIdx = (int)o;

var info = s_DocInfo[infoIdx];

try

{

TestScriptsCompile(info);

}

catch (Exception e)

{

info.status = TestStatus.Failed;

info.testRunnerFailure = e.ToString();

}

finally

{

s_ThreadEvents[infoIdx].Set();

}

[ Test ]

public void ScriptVerificationCSharp ( )

{

// Setup. Start all tests running on multiple threads.

s_ThreadEvents = new ManualResetEvent [ s_DocInfo . Count ] ;

for ( int i = 0 ; i < s_DocInfo . Count ; ++ i )

{

// Queue this example up for testing

s_ThreadEvents [ i ] = new ManualResetEvent ( false ) ;

ThreadPool . QueueUserWorkItem ( TestDocumentationExampleScriptsThreaded , i ) ;

}

// Check for errors and build the error output if required.

bool testFailed = false ;

StringBuilder results = new StringBuilder ( ) ;

for ( int i = 0 ; i < s_ThreadEvents . Length ; ++ i )

{

// Wait for the test to finish.

s_ThreadEvents [ i ] . WaitOne ( ) ;

if ( s_DocInfo [ i ] . status == TestStatus . Failed )

{

testFailed = true ;

GenerateFailureMessage ( results , s_DocInfo [ i ] ) ;

}

// If a single item has failed then the test is considered a failure.

Assert . IsFalse ( testFailed , results . ToString ( ) ) ;

}

public static void TestDocumentationExampleScriptsThreaded ( object o )

{

var infoIdx = ( int ) o ;

var info = s_DocInfo [ infoIdx ] ;

try

{

TestScriptsCompile ( info ) ;

}

catch ( Exception e )

{

info . status = TestStatus . Failed ;

info . testRunnerFailure = e . ToString ( ) ;

}

finally

{

s_ThreadEvents [ infoIdx ] . Set ( ) ;

}

This took the test time from 30 minutes to 2, which is fine for running as part of our build verification.

測試時間從30分鐘縮短到2分鐘，這對於作爲我們的構建驗證的一部分運行是很好的。

Since we couldn’t test individual examples with NUnit any more, we added a button to the scripting doc editor to allow developers to test the examples as they write them. The script with an error is now colored red when the test is run and error messages are displayed beneath.

由於我們無法再使用NUnit測試單個示例，因此在腳本文檔編輯器中添加了一個按鈕，以允許開發人員在編寫示例時對其進行測試。在運行測試時，帶有錯誤的腳本現在顯示爲紅色，並且錯誤消息顯示在下方。

When the test was first run we had 326 failures which I whitelisted (so they could be fixed at a later date). We now have that down to 32, of which most are failures in the test runner mainly due to not having access to some specific assemblies. There have been no new issues introduced and we can rest assured that when we deprecate parts of the API the test will fail and we can then update the example to use the new API.

首次運行測試時，我將326個失敗列入了白名單(因此可以在以後進行修復)。現在，我們將其減少到32個，其中大多數是測試運行程序中的失敗，主要是由於無法訪問某些特定的程序集。沒有引入新的問題，我們可以放心，當我們棄用API的某些部分時，測試將失敗，然後可以更新示例以使用新的API。

Overall I thought this was an interesting use of the Editor Test Runner. It does have some limitations: We only test C# examples, and I have not managed to get JS compilation working, although that won’t be an issue in the future.

總的來說，我認爲這是Editor Test Runner的有趣用法。它確實有一些侷限性：我們僅測試C＃示例，儘管將來這不會成爲問題，但我沒有設法使JS編譯正常工作。

Here is the full test.

這是完整的測試。

using System;
using System.CodeDom.Compiler;
using UnityEngine;
using NUnit.Framework;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Reflection;
using System.Text;
using System.Threading;
using System.Xml;
using Microsoft.CSharp;
using UnderlyingModel;
using UnityEditor;

public class ScriptVerification
{
    const string k_PathToApiDocs = @"../../../../Documentation/ApiDocs/";
    const string k_FileExtension = ".mem.xml";
    const string k_WhiteList = "Assets/Editor/ScriptVerificationWhiteList.txt";

public enum TestStatus
    {
        Unknown,     // Nothing has been done to this test yet.
        Ignored,     // Tests are ignored if they contain no example code
        Failed,      // The test failed to compile one or more of the examples.
        Passed,      // All examples were compiled successfully.
        Whitelisted, // Test was ignored as the member is in the white list file.
    }

public class ExampleScript
    {
        public string code;
        public CompilerResults compileResults;
    }

public class ScriptingDocMember
    {
        public TestStatus status = TestStatus.Unknown;

// Information on the test and the xml file it can be found in.
        public string path;
        public string parent;
        public string name;
        public string nspace;
        public bool editor;

public List<ExampleScript> csharpExamples = new List<ExampleScript>();

// If we fail to compile multiple examples we also attempt to compile them as a single example.
        public CompilerResults combinedResults;

// Error message if something caused the test runner to fail.
        public string testRunnerFailure;
    }

static List<ScriptingDocMember> s_DocInfo;
    static ManualResetEvent[] s_ThreadEvents;

[SetUp]
    public void SetupScriptVerification()
    {
        // Parse the scripting doc files and prepare the test data.
        string path = k_PathToApiDocs;
        if (!path.Contains(":"))
        {
            path = Application.dataPath + "/" + k_PathToApiDocs;
        }
        var files = Directory.GetFiles(path, "*" + k_FileExtension, SearchOption.AllDirectories);

s_DocInfo = new List<ScriptingDocMember>();
        var whiteList = GetWhiteList();

for (int i = 0; i < files.Length; ++i)
        {
            var xml = new XmlDocument();
            xml.Load(files[i]);
            XmlNode xmlheader = xml.FirstChild;
            XmlNode docsheader = xmlheader.NextSibling;
            XmlNode namespaceTag = docsheader.FirstChild;
            ParseMemberNode(namespaceTag, files[i], "", s_DocInfo, whiteList);
        }
    }

[Test]
    public void ScriptVerificationCSharp()
    {
        // Setup. Start all tests running on multiple threads.
        // This gets the test time down from 30 minutes to around 2 minutes.
        s_ThreadEvents = new ManualResetEvent[s_DocInfo.Count];
        for (int i = 0; i < s_DocInfo.Count; ++i)
        {
            if (s_DocInfo[i].csharpExamples.Count == 0)
            {
                // Ignore items with no examples
                s_ThreadEvents[i] = new ManualResetEvent(true);
                s_DocInfo[i].status = TestStatus.Ignored;
            }
            else if (s_DocInfo[i].status == TestStatus.Whitelisted)
            {
                // Skip white listed items
                s_ThreadEvents[i] = new ManualResetEvent(true);
            }
            else
            {
                // Queue this example up for testing
                s_ThreadEvents[i] = new ManualResetEvent(false);
                ThreadPool.QueueUserWorkItem(TestDocumentationExampleScriptsThreaded, i);
            }
        }

// Check for errors and build the error output if required.
        bool testFailed = false;
        StringBuilder results = new StringBuilder();
        for (int i = 0; i < s_ThreadEvents.Length; ++i)
        {
            s_ThreadEvents[i].WaitOne();

if (s_DocInfo[i].status == TestStatus.Failed)
            {
                testFailed = true;
                GenerateFailureMessage(results, s_DocInfo[i]);
            }
        }

// If a single item has failed then the test is considered a failure.
        Assert.IsFalse(testFailed, results.ToString());
    }

static void GenerateFailureMessage(StringBuilder output, ScriptingDocMember info)
    {
        output.AppendLine(new string('-', 100));
        output.AppendLine("Name: " + info.name);
        output.AppendLine("Path: " + info.path + "\n");

// Print out the example scripts along with their errors.
        for (int i = 0; i < info.csharpExamples.Count; ++i)
        {
            var example = info.csharpExamples[i];
            if (example.compileResults != null && example.compileResults.Errors.HasErrors)
            {
                output.AppendLine("Example Script " + i + ":\n");

// Add line numbers
                var lines = example.code.SplitLines();
                int lineNumber = 0;
                int startLine = 0;

// Find the first line of code so the line numbers align correctly.
                // The compiler will ignore any empty lines at the start.
                for (; startLine < lines.Length; ++startLine)
                {
                    if (string.IsNullOrEmpty(lines[startLine]))
                        startLine++;
                    else
                        break;
                }

for (; startLine < lines.Length; ++startLine)
                {
                    // Does this line contain an error?
                    string lineMarker = " ";
                    foreach (CompilerError compileResultsError in example.compileResults.Errors)
                    {
                        // Add a mark to indicate this line has a reported error.
                        if (compileResultsError.Line == lineNumber)
                        {
                            lineMarker = "-";
                            break;
                        }
                    }

output.AppendFormat("{0}{1:000} | {2}\n", lineMarker, lineNumber++, lines[startLine]);
                }

output.Append("\n\n");
                output.AppendLine(ErrorMessagesToString(example.compileResults));
            }
        }

if (info.combinedResults != null)
        {
            output.AppendLine("Combined Example Scripts:\n");
            output.AppendLine(ErrorMessagesToString(info.combinedResults));
        }

if (!string.IsNullOrEmpty(info.testRunnerFailure))
        {
            output.AppendLine("Test Runner Failure: " + info.testRunnerFailure);
        }
    }

// Concatenates all the errors into a formated list.
    public static string ErrorMessagesToString(CompilerResults cr)
    {
        string errorMessages = "";
        foreach (CompilerError compilerError in cr.Errors)
        {
            errorMessages += string.Format("{0}({1},{2}): {3}\n", compilerError.ErrorNumber, compilerError.Line, compilerError.Column, compilerError.ErrorText);
        }
        return errorMessages;
    }

public static void TestDocumentationExampleScriptsThreaded(object o)
    {
        var infoIdx = (int)o;
        var info = s_DocInfo[infoIdx];
        try
        {
            TestScriptsCompile(info);
        }
        catch (Exception e)
        {
            info.status = TestStatus.Failed;
            info.testRunnerFailure = e.ToString();
        }
        finally
        {
            s_ThreadEvents[infoIdx].Set();
        }
    }

// Tests all scripts compile for the selected scripting member.
    // First attempts to compile all scripts separately, if this fails then compiles them combined as a single example.
    public static void TestScriptsCompile(ScriptingDocMember info)
    {
        var scripts = info.csharpExamples;
        if (scripts.Count == 0)
        {
            info.status = TestStatus.Ignored;
            return;
        }

// Setup compiler
        var providerOptions = new Dictionary<string, string>();
        providerOptions.Add("CompilerVersion", "v3.5");
        var domProvider = new CSharpCodeProvider(providerOptions);
        var compilerParams = new CompilerParameters
        {
            GenerateExecutable = false,
            GenerateInMemory = false,
            TreatWarningsAsErrors = false,
        };
        Assembly[] assemblies = AppDomain.CurrentDomain.GetAssemblies();
        foreach (var assembly in assemblies)
        {
            compilerParams.ReferencedAssemblies.Add(assembly.Location);
        }

// Attempt to compile the scripts separately.
        bool error = false;
        for (int i = 0; i < scripts.Count; i++)
        {
            scripts[i].compileResults = domProvider.CompileAssemblyFromSource(compilerParams, scripts[i].code);
            if (scripts[i].compileResults.Errors.HasErrors)
                error = true;
        }

if (error)
        {
            // Its possible that the scripts are all part of one example so we should compile them together and see if that works instead.
            info.combinedResults = domProvider.CompileAssemblyFromSource(compilerParams, scripts.Select(s => s.code).ToArray());
            if (!info.combinedResults.Errors.HasErrors)
                error = false;
        }

info.status = error ? TestStatus.Failed : TestStatus.Passed;
    }

static HashSet<string> GetWhiteList()
    {
        var textAsset = AssetDatabase.LoadAssetAtPath(k_WhiteList, typeof(TextAsset)) as TextAsset;
        var whiteList = new HashSet<string>();
        if (textAsset)
        {
            foreach (var line in textAsset.text.Split('\n'))
            {
                whiteList.Add(line.Replace("\r", "").TrimEnd(' '));
            }
        }
        return whiteList;
    }

// Parses the scripting docs and generates our test data.
    static void ParseMemberNode(XmlNode node, string file, string parent, List<ScriptingDocMember> infoList, HashSet<string> whiteList)
    {
        ScriptingDocMember info = new ScriptingDocMember();
        info.path = file;
        infoList.Add(info);
        info.parent = parent;
        foreach (XmlAttribute attr in node.Attributes)
        {
            // potential tag attributes: name, namespace, type
            var attrLowercase = attr.Name.ToLower();
            if (attrLowercase == "name") info.name = attr.Value;
            else if (attrLowercase == "namespace") info.nspace = attr.Value;
        }

if (whiteList.Contains(info.name))
            info.status = TestStatus.Whitelisted;

if (!string.IsNullOrEmpty(info.nspace))
        {
            // trim down the namespace to remove UnityEngine and UnityEditor
            if (info.nspace.StartsWith("UnityEngine"))
            {
                info.editor = false;
                info.nspace = info.nspace.Remove(0, "UnityEngine".Length);
            }
            if (info.nspace.StartsWith("UnityEditor"))
            {
                info.editor = true;
                info.nspace = info.nspace.Remove(0, "UnityEditor".Length);
            }
            if (info.nspace.StartsWith("."))
                info.nspace = info.nspace.Remove(0, 1);
        }

foreach (XmlNode child in node.ChildNodes)
        {
            var childNameLowercase = child.Name.ToLower();
            if (childNameLowercase == "section")
            {
                // see if this section is undoc
                for (int i = 0; i < child.Attributes.Count; i++)
                {
                    if (child.Attributes[i].Name == "undoc" && child.Attributes[i].Value == "true")
                    {
                        infoList.Remove(info);
                        break;
                    }
                }

foreach (XmlNode grandChild in child.ChildNodes)
                {
                    var codeLangNode = GetExample(grandChild);
                    if (codeLangNode != null)
                    {
                        var scriptInfo = new ExampleScript();
                        scriptInfo.code = codeLangNode.InnerXml.Replace("<![CDATA[", "").Replace("]]>", "");
                        info.csharpExamples.Add(scriptInfo);
                    }
                }
            }
            else if (childNameLowercase == "member")
            {
                ParseMemberNode(child, file, info.name, infoList, whiteList);
            }
        }
    }

// Extract the cs example code.
    static XmlNode GetExample(XmlNode node)
    {
        if (node.Name.ToLower() == "example")
        {
            for (int i = 0; i < node.Attributes.Count; ++i)
            {
                if (node.Attributes[i].Name == "nocheck" && node.Attributes[i].Value == "true")
                    return null;
            }
            return node.SelectSingleNode("code[@lang='cs']");
        }
        return null;
    }
}