Automated Unit Test Execution

Source code:

Problem

Unit tests are great. When you follow best practices and implement them when a feature has been implemented, they are a lifesaver when they fail because something has changed.

The problem is that right now using just the platform, as far as I can tell, executing unit tests is a manual process, which typically involves going to the "Apex Test Execution" page or deploying to a production org. Using apex code and some scheduled jobs, we'll automate the execution of our unit tests, so that we can know much sooner if our changes have broken something.

Solution

Automating unit test execution is conceptually simple. The overall algorithm is

  1. Create a scheduled job that queues the test classes containing the desired unit tests to be executed by inserting "ApexTestQueueItem" objects.
  2. Store the Apex Job Id somewhere so that we can process the test results later. This is necessary because the unit tests are executed asynchronously.
  3. Create one or more other scheduled jobs that periodically check the status of the unit tests, and when complete, email us the results.

First, create a custom object whose object name is "AutomatedTestingQueue". Next, create a "Text, Required, Length 40" custom field whose api name is "AsyncId". This object stores the aysnchronous job id while the unit tests are executed so we can periodically check to see if they're done executing and then process them.

Next, save the following classes

[code apex] global with sharing class AutomatedTestJobQueuer implements schedulable { global void execute(SchedulableContext SC) { doExecute(); } @future (callout=true) public static void doExecute(){ enqueueUnitTests(); } public static void createDaily4AMScheduledJob(){ AutomatedTestJobQueuer atj = new AutomatedTestJobQueuer(); string sch = '0 0 4 * * ?'; system.schedule('Enqueue Unit Tests 4 AM',sch,atj); } /* Allows us to externally enqueue our unit tests. For example, whenever we check our code into source control, we could run our unit tests. */ webservice static void enqueueUnitTests(){ enqueueTests(); } // Enqueue all classes beginning with "Test". public static void enqueueTests() { /* The first thing you need to do is query the classes that contain the unit tests you want executed. In our org, our test classes are named "Test" so that all the test classes are grouped together in Eclipse. Change the where clause as necessary to query the desired classes. */ ApexClass[] testClasses = [SELECT Id, Name FROM ApexClass WHERE Name LIKE 'Test%']; Integer testClassCnt = testClasses != null ? testClasses.size() : 0; system.debug(' enqueueTests::testClassCnt ' + testClassCnt); if (testClassCnt > 0) { /* Insertion of the ApexTestQueueItem causes the unit tests to be executed. Since they're asynchronous, the apex async job id needs to be stored somewhere so we can process the test results when the job is complete. */ ApexTestQueueItem[] queueItems = new List(); for (ApexClass testClass : testClasses) { system.debug(' enqueueTests::testClass ' + testClass); queueItems.add(new ApexTestQueueItem(ApexClassId=testClass.Id)); } insert queueItems; // Get the job ID of the first queue item returned. ApexTestQueueItem item = [SELECT ParentJobId FROM ApexTestQueueItem WHERE Id=:queueItems[0].Id LIMIT 1]; AutomatedTestingQueue__c atq = new AutomatedTestingQueue__c( AsyncId__c = item.parentjobid ); insert atq; } } } [/code] [code apex] global with sharing class AutomatedTestingJob implements Schedulable { global void execute(SchedulableContext SC) { doExecute(); } // Have to use a future method so the email will be sent out. @future (callout=true) public static void doExecute(){ processAsyncResults(); } /* Schedule String Format: Seconds Minutes Hours Day_of_month Month Day_of_week optional_year */ public static void createEvery15MinuteScheduledJobs(){ AutomatedTestingJob atj = new AutomatedTestingJob(); string sch = '0 0 * * * ?'; system.schedule('Process Queued Unit Tests Every Top Of The Hour',sch,atj); sch = '0 15 * * * ?'; system.schedule('Process Queued Unit Tests At Each Quarter After',sch,atj); sch = '0 30 * * * ?'; system.schedule('Process Queued Unit Tests At Each Bottom Of The Hour',sch,atj); sch = '0 45 * * * ?'; system.schedule('Process Queued Unit Tests At Each Quarter To The Hour',sch,atj); } public static void processAsyncResults(){ List queuedTests = [select id, name, AsyncId__c from AutomatedTestingQueue__c limit 5]; if (queuedTests != null && queuedTests.size() > 0){ Set AsyncIds = new Set(); for (AutomatedTestingQueue__c queuedJob : queuedTests){ AsyncIds.add(queuedJob.AsyncId__c); } List queuedItems = checkClassStatus(AsyncIds); Map> groupedTestsByJob = new Map>(); for (ApexTestQueueItem atqi : queuedItems){ if (groupedTestsByJob.containsKey(atqi.ParentJobId) == true){ List groupedTests = groupedTestsByJob.get(atqi.ParentJobId); groupedTests.add(atqi); } else{ List groupedTests = new List(); groupedTests.add(atqi); groupedTestsByJob.put(atqi.ParentJobId, groupedTests); } } Set completedAsyncIds = getCompletedAsyncJobsIds(groupedTestsByJob); if (completedAsyncIds != null && completedAsyncIds.size() > 0){ List testResults = checkMethodStatus(completedAsyncIds); Map> groupedTestResultsByJob = new Map>(); for (ApexTestResult testResult : testResults){ if (groupedTestResultsByJob.containsKey(testResult.AsyncApexJobId)){ List groupedTestsResults = groupedTestResultsByJob.get(testResult.AsyncApexJobId); groupedTestsResults.add(testResult); } else{ List groupedTestsResults = new List(); groupedTestsResults.add(testResult); groupedTestResultsByJob.put(testResult.AsyncApexJobId, groupedTestsResults ); } } List queuedTestsToDelete = new List(); for (List jobTestResults : groupedTestResultsByJob.values()){ sendTestResultEmail(jobTestResults); } for (AutomatedTestingQueue__c queuedTest : queuedTests){ for (Id completedAsyncId : completedAsyncIds){ if (queuedTest.AsyncId__c == completedAsyncId){ queuedTestsToDelete.add(queuedTest); break; } } if (groupedTestsByJob.containsKey(queuedTest.asyncId__c) == false){ queuedTestsToDelete.add(queuedTest); } } if (queuedTestsToDelete.size() > 0){ delete queuedTestsToDelete; } } } } public static Set getCompletedAsyncJobsIds(Map> groupedTestsByJob){ Set completedAsyncJobIds = new Set(); for (List jobTests : groupedTestsByJob.values()){ if (jobTests == null || jobTests.size() == 0){ continue; } Boolean allCompleted = true; for (ApexTestQueueItem queuedTest : jobTests){ if (queuedTest.Status != 'Completed' && queuedTest.Status != 'Failed' && queuedTest.Status != 'Aborted'){ allCompleted = false; break; } } if (allCompleted == true){ completedAsyncJobIds.add(jobTests[0].ParentJobId); } } return completedAsyncJobIds; } private static void sendTestResultEmail(List jobTestResults){ system.debug(' In sendTestResultEmail'); Messaging.SingleEmailMessage mail = new Messaging.SingleEmailMessage(); String emailAddress = 'Your email address here'; String[] toAddresses = new String[] { emailAddress }; mail.setToAddresses(toAddresses); String emailSubject = 'Dev Unit Test Results ' + String.valueOf(Date.today()); mail.setSubject(emailSubject); String testResultEmailbody = getTestResultHtmlEmailBody(jobTestResults); mail.setHtmlBody(testResultEmailbody); Messaging.sendEmail(new Messaging.Email[] { mail }); system.debug(' sent test results email'); } private static String getTestResultHtmlEmailBody(List jobTestResults){ system.debug(' In getTestResultHtmlEmailBody'); List successTests = new List(); List failedTests = new List(); for (ApexTestResult jobTestResult : jobTestResults){ if (jobTestResult.Outcome == 'Pass'){ successTests.add(jobTestResult); } else{ failedTests.add(jobTestResult); } } Integer numTestsRun = successTests.size() + failedTests.size(); Integer numFailures = failedTests.size(); Integer successNum = numTestsRun - numFailures; if (successNum < 0){ successNum = 0; } String testResultBody = ''; // Unfortunately, css has to be inlined because many email service providers now exclude external CSS // because it can pose a security risk. testResultBody += ''; testResultBody += ''; testResultBody += ''; testResultBody += '
Tests Run:' + numTestsRun + '
Failure Count:' + numFailures + '
Success Count:' + successNum + '
'; if (numFailures > 0){ testResultBody += '
Test Failures
'; testResultBody += ''; testResultBody += ''; testResultBody += ''; testResultBody += ''; testResultBody += ''; testResultBody += ''; testResultBody += ''; testResultBody += ''; for (ApexTestResult testFailure : failedTests){ testResultBody += ''; testResultBody += ''; testResultBody += ''; testResultBody += ''; testResultBody += ''; testResultBody += ''; //testResultBody += ''; testResultBody += ''; } testResultBody += '
Test ClassUnit TestMessageStack TraceTime (Ms)
' + testFailure.ApexClass.Name +'' + testFailure.MethodName +'' + testFailure.message +'' + testFailure.stackTrace +'' + testFailure.ApexLog.DurationMilliseconds +'' + testFailure.type_x +'
'; } return testResultBody; } // Get the status and pass rate for each class // whose tests were run by the job. // that correspond to the specified job ID. public static List checkClassStatus(Set jobIds) { ApexTestQueueItem[] items = [SELECT ApexClass.Name, Status, ExtendedStatus, ParentJobId FROM ApexTestQueueItem WHERE ParentJobId in :jobIds]; for (ApexTestQueueItem item : items) { String extStatus = item.extendedstatus == null ? '' : item.extendedStatus; System.debug(item.ApexClass.Name + ': ' + item.Status + extStatus); } return items; } // Get the result for each test method that was executed. public static List checkMethodStatus(Set jobIds) { ApexTestResult[] results = [SELECT Outcome, MethodName, Message, StackTrace, AsyncApexJobId, ApexClass.Name, ApexClass.Body, ApexClass.LengthWithoutComments, ApexClass.NamespacePrefix, ApexClass.Status, ApexLogId, ApexLog.DurationMilliseconds, ApexLog.Operation, ApexLog.Request, ApexLog.Status, ApexLog.Location, ApexLog.Application FROM ApexTestResult WHERE AsyncApexJobId in :jobIds]; for (ApexTestResult atr : results) { System.debug(atr.ApexClass.Name + '.' + atr.MethodName + ': ' + atr.Outcome); if (atr.message != null) { System.debug(atr.Message + '\n at ' + atr.StackTrace); } } return results; } } [/code]

Now that you've the custom object to store the Async Job Ids and the schedulable classes, it's a simple matter of scheduling the jobs. Note: The defaults used in the classes require 5 scheduled jobs and salesforce imposes a limit of 10, so please make sure that you have enough or you'll have to refactor the code or change the scheduling to get it to work.

Use the following to create a scheduled job that will execute your unit tests daily at 4 AM.

[code apex] AutomatedTestJobQueuer.createDaily4AMScheduledJob(); [/code]

Use the following to create four scheduled jobs that effectively will process the test results and email them every 15 minutes, if needed.

[code apex] AutomatedTestingJob.createEvery15MinuteScheduledJobs(); [/code]

There's also a webservice method "enqueueUnitTests" that allows you to queue them on demand and have the results emailed to you. I envisioned this to be useful for continuous integration purposes.

For example, if you're using source control, which I really hope you are, and you check in code, this method would be called to enqueue and run your unit tests. Essentially, whenever code is checked in, the unit tests will run in semi real-time and you'll know fairly quickly whether or not you broke something. Since finding and fixing bugs is cheaper and faster when you're already coding somewhere, this'll save you from many future headaches.

I hope this helps and happy coding.

Luke