Problem

Unit tests are great. When you follow best practices and implement them when a feature has been implemented, they are a lifesaver when they fail because something has changed.

The problem is that right now using just the platform, as far as I can tell, executing unit tests is a manual process, which typically involves going to the "Apex Test Execution" page or deploying to a production org. Using apex code and some scheduled jobs, we'll automate the execution of our unit tests, so that we can know much sooner if our changes have broken something.

Solution

Automating unit test execution is conceptually simple. The overall algorithm is

1. Create a scheduled job that queues the test classes containing the desired unit tests to be executed by inserting "ApexTestQueueItem" objects.
2. Store the Apex Job Id somewhere so that we can process the test results later. This is necessary because the unit tests are executed asynchronously.
3. Create one or more other scheduled jobs that periodically check the status of the unit tests, and when complete, email us the results.

First, create a custom object whose object name is "AutomatedTestingQueue". Next, create a "Text, Required, Length 40" custom field whose api name is "AsyncId". This object stores the aysnchronous job id while the unit tests are executed so we can periodically check to see if they're done executing and then process them.

Next, save the following classes

global with sharing class AutomatedTestJobQueuer implements schedulable {
    
    global void execute(SchedulableContext SC) {
        doExecute();
    }
    
    @future (callout=true)
    public static void doExecute(){
        enqueueUnitTests();
    }
    
    public static void createDaily4AMScheduledJob(){
        AutomatedTestJobQueuer atj = new AutomatedTestJobQueuer();  
        string sch = '0 0 4 * * ?';  
        system.schedule('Enqueue Unit Tests 4 AM',sch,atj);
    }

    /* Allows us to externally enqueue our unit tests. For example,
       whenever we check our code into source control, we could
       run our unit tests.
    */
    webservice static void enqueueUnitTests(){      
        enqueueTests();
    }


    // Enqueue all classes beginning with "Test".  
    
    public static void enqueueTests() {
       /* The first thing you need to do is query the classes that contain
         the unit tests you want executed.

         In our org, our test classes are named "Test<Class_Name_Here>"
         so that all the test classes are grouped together in Eclipse.
         Change the where clause as necessary to query the desired classes.
       */
            
       ApexClass[] testClasses = 
         [SELECT Id,
                 Name
            FROM ApexClass 
           WHERE Name LIKE 'Test%'];
              
       Integer testClassCnt = testClasses != null ? testClasses.size() : 0;
        
       system.debug('   enqueueTests::testClassCnt ' + testClassCnt);
            
       if (testClassCnt > 0) {
          /*
             Insertion of the ApexTestQueueItem causes the unit tests to be 
             executed. Since they're asynchronous, the apex async job id
             needs to be stored somewhere so we can process the test results
             when the job is complete.
          */
          ApexTestQueueItem[] queueItems = new List<ApexTestQueueItem>();
            
          for (ApexClass testClass : testClasses) {
              system.debug('   enqueueTests::testClass ' + testClass);
                
              queueItems.add(new ApexTestQueueItem(ApexClassId=testClass.Id));
          }

          insert queueItems;

          // Get the job ID of the first queue item returned. 
    
          ApexTestQueueItem item = 
            [SELECT ParentJobId
               FROM ApexTestQueueItem 
              WHERE Id=:queueItems[0].Id
              LIMIT 1];
            
          AutomatedTestingQueue__c atq = new AutomatedTestingQueue__c(
              AsyncId__c = item.parentjobid
          );

          insert atq;
       }
    }
}

global with sharing class AutomatedTestingJob implements Schedulable {
    
    global void execute(SchedulableContext SC) {
        doExecute();
    }

    // Have to use a future method so the email will be sent out.
    @future (callout=true)
    public static void doExecute(){
        processAsyncResults();
    }
    
    /*
        Schedule String Format: Seconds Minutes Hours Day_of_month Month Day_of_week optional_year
    */
    
    public static void createEvery15MinuteScheduledJobs(){
        AutomatedTestingJob atj = new AutomatedTestingJob();  
        string sch = '0 0 * * * ?';  
        system.schedule('Process Queued Unit Tests Every Top Of The Hour',sch,atj);
        
        sch = '0 15 * * * ?';  
        system.schedule('Process Queued Unit Tests At Each Quarter After',sch,atj);
        
        sch = '0 30 * * * ?';  
        system.schedule('Process Queued Unit Tests At Each Bottom Of The Hour',sch,atj);
        
        sch = '0 45 * * * ?';  
        system.schedule('Process Queued Unit Tests At Each Quarter To The Hour',sch,atj);
    }
    
    public static void processAsyncResults(){
        List<AutomatedTestingQueue__c> queuedTests = 
           [select id,
                   name,
                   AsyncId__c
              from AutomatedTestingQueue__c
             limit 5];
        
        if (queuedTests != null && queuedTests.size() > 0){
            Set<Id> AsyncIds = new Set<Id>();

            for (AutomatedTestingQueue__c queuedJob : queuedTests){
                 AsyncIds.add(queuedJob.AsyncId__c);
            }
            
            List<ApexTestQueueItem> queuedItems = checkClassStatus(AsyncIds);
            
            Map<Id, List<ApexTestQueueItem>> groupedTestsByJob = new Map<Id, List<ApexTestQueueItem>>();

            for (ApexTestQueueItem atqi : queuedItems){
                 if (groupedTestsByJob.containsKey(atqi.ParentJobId) == true){
                     List<ApexTestQueueItem> groupedTests = groupedTestsByJob.get(atqi.ParentJobId);
                     groupedTests.add(atqi);
                 }
                 else{
                     List<ApexTestQueueItem> groupedTests = new List<ApexTestQueueItem>();
                     groupedTests.add(atqi);
                     groupedTestsByJob.put(atqi.ParentJobId, groupedTests);
                 }
            }
            
            Set<Id> completedAsyncIds = getCompletedAsyncJobsIds(groupedTestsByJob);
            
            if (completedAsyncIds != null && completedAsyncIds.size() > 0){
                
                List<ApexTestResult> testResults = checkMethodStatus(completedAsyncIds);
                
                Map<Id, List<ApexTestResult>> groupedTestResultsByJob = new Map<Id, List<ApexTestResult>>();
                
                
                for (ApexTestResult testResult : testResults){
                    if (groupedTestResultsByJob.containsKey(testResult.AsyncApexJobId)){
                        List<ApexTestResult> groupedTestsResults = groupedTestResultsByJob.get(testResult.AsyncApexJobId);
                        groupedTestsResults.add(testResult);
                    }
                    else{
                        List<ApexTestResult> groupedTestsResults = new List<ApexTestResult>();
                        groupedTestsResults.add(testResult);
                        
                        groupedTestResultsByJob.put(testResult.AsyncApexJobId, groupedTestsResults );
                    }
                }

                List<AutomatedTestingQueue__c> queuedTestsToDelete = new List<AutomatedTestingQueue__c>(); 
                
                for (List<ApexTestResult> jobTestResults : groupedTestResultsByJob.values()){
                    sendTestResultEmail(jobTestResults);
                }
                
                for (AutomatedTestingQueue__c queuedTest : queuedTests){
                    for (Id completedAsyncId : completedAsyncIds){
                        if (queuedTest.AsyncId__c == completedAsyncId){
                            queuedTestsToDelete.add(queuedTest);
                            break;
                        }
                    }
                    if (groupedTestsByJob.containsKey(queuedTest.asyncId__c) == false){
                        queuedTestsToDelete.add(queuedTest);
                    }
                }
                
                if (queuedTestsToDelete.size() > 0){
                    delete queuedTestsToDelete;
                }
            }
        }
    }
    
    public static Set<Id> getCompletedAsyncJobsIds(Map<Id, List<ApexTestQueueItem>> groupedTestsByJob){
        Set<Id> completedAsyncJobIds = new Set<Id>();
        
        for (List<ApexTestQueueItem> jobTests : groupedTestsByJob.values()){
            if (jobTests == null || jobTests.size() == 0){
                continue;
            }
            
            Boolean allCompleted = true;
            
            for (ApexTestQueueItem queuedTest : jobTests){
                if (queuedTest.Status != 'Completed' && queuedTest.Status != 'Failed' && queuedTest.Status != 'Aborted'){
                    allCompleted = false;
                    break;
                }
            }
            
            if (allCompleted == true){
                completedAsyncJobIds.add(jobTests[0].ParentJobId);
            }
        }
        
        return completedAsyncJobIds;
    }
    
    private static void sendTestResultEmail(List<ApexTestResult> jobTestResults){
        system.debug(' In sendTestResultEmail');
            
        Messaging.SingleEmailMessage mail = new Messaging.SingleEmailMessage();
        
        String emailAddress = 'Your email address here';
        
        String[] toAddresses = new String[] { emailAddress };
    
        mail.setToAddresses(toAddresses);
        
        String emailSubject = 'Dev Unit Test Results ' + String.valueOf(Date.today()); 
    
        mail.setSubject(emailSubject);

        String testResultEmailbody = getTestResultHtmlEmailBody(jobTestResults);

        mail.setHtmlBody(testResultEmailbody);
        Messaging.sendEmail(new Messaging.Email[] { mail });
        
        system.debug(' sent test results email');
    }
    
    private static String getTestResultHtmlEmailBody(List<ApexTestResult> jobTestResults){
        system.debug(' In getTestResultHtmlEmailBody');
        
        List<ApexTestResult> successTests = new List<ApexTestResult>();
        List<ApexTestResult> failedTests = new List<ApexTestResult>();
        
        for (ApexTestResult jobTestResult : jobTestResults){
            if (jobTestResult.Outcome == 'Pass'){
                successTests.add(jobTestResult);
            }
            else{
                failedTests.add(jobTestResult);
            }
        }
        
        Integer numTestsRun = successTests.size() + failedTests.size();
        Integer numFailures = failedTests.size();
        Integer successNum = numTestsRun - numFailures;
        
        if (successNum < 0){
            successNum = 0;
        }
        
        String testResultBody = '';
        
        // Unfortunately, css has to be inlined because many email service providers now exclude external CSS
        // because it can pose a security risk.
        testResultBody += '<td style="text-align: right;">' + numTestsRun + '';
        testResultBody += '<td style="text-align: right;">' + numFailures + '';
        testResultBody += '<td style="text-align: right;">' + successNum + '';
        
        testResultBody += '
Tests Run:
Failure Count:
Success Count:
';
                
        if (numFailures > 0){
            
            testResultBody += '<div style="margin: 5px 0px; font-weight: bold;">Test Failures</div>';
            
            testResultBody += '';
            
            testResultBody += '';
            testResultBody += '<th style="text-align: left; padding-left: 5px;">Test Class</th>';
            testResultBody += '<th style="text-align: left; padding-left: 5px;">Unit Test</th>';
            testResultBody += '<th style="text-align: left; padding-left: 5px;">Message</th>';
            testResultBody += '<th style="text-align: left; padding-left: 5px;">Stack Trace</th>';
            testResultBody += '<th style="text-align: left; padding-left: 5px;">Time (Ms)</th>';
            testResultBody += '';
            
            for (ApexTestResult testFailure : failedTests){
                testResultBody += '';
                
                testResultBody += '<td style="padding: 5px; vertical-align: top;">' + testFailure.ApexClass.Name +'';
                testResultBody += '<td style="padding: 5px; vertical-align: top;">' + testFailure.MethodName +'';
                testResultBody += '<td style="padding: 5px; vertical-align: top;">' + testFailure.message +'';
                testResultBody += '<td style="padding: 5px; vertical-align: top;">' + testFailure.stackTrace +'';
                testResultBody += '<td style="padding: 5px; vertical-align: top;">' + testFailure.ApexLog.DurationMilliseconds +'';
                //testResultBody += '<td style="vertical-align: top;">' + testFailure.type_x +'';
                
                testResultBody += '';
            }
            
            testResultBody += '';
        }       
        
        return testResultBody;
    }


    // Get the status and pass rate for each class 
    
    // whose tests were run by the job. 
    
    // that correspond to the specified job ID. 
    
    public static List<ApexTestQueueItem> checkClassStatus(Set<ID> jobIds) {
        ApexTestQueueItem[] items = 
           [SELECT ApexClass.Name,
                   Status,
                   ExtendedStatus,
                   ParentJobId
            FROM ApexTestQueueItem 
            WHERE ParentJobId in :jobIds];
            
        for (ApexTestQueueItem item : items) {
            String extStatus = item.extendedstatus == null ? '' : item.extendedStatus;
            System.debug(item.ApexClass.Name + ': ' + item.Status + extStatus);
        }
        
        return items;
    }

    // Get the result for each test method that was executed. 
    
    public static List<ApexTestResult> checkMethodStatus(Set<ID> jobIds) {
        ApexTestResult[] results = 
           [SELECT Outcome,
                   MethodName,
                   Message,
                   StackTrace,
                   AsyncApexJobId, 
                   ApexClass.Name,
                   ApexClass.Body,
                   ApexClass.LengthWithoutComments,
                   ApexClass.NamespacePrefix,
                   ApexClass.Status,
                   ApexLogId,
                   ApexLog.DurationMilliseconds,
                   ApexLog.Operation,
                   ApexLog.Request,
                   ApexLog.Status,
                   ApexLog.Location,
                   ApexLog.Application
            FROM ApexTestResult 
            WHERE AsyncApexJobId in :jobIds];
            
        for (ApexTestResult atr : results) {
            System.debug(atr.ApexClass.Name + '.' + atr.MethodName + ': ' + atr.Outcome);
            if (atr.message != null) {
                System.debug(atr.Message + '\n at ' + atr.StackTrace);
            }
        }
        
        return results;
    }
}

Now that you've the custom object to store the Async Job Ids and the schedulable classes, it's a simple matter of scheduling the jobs. Note: The defaults used in the classes require 5 scheduled jobs and salesforce imposes a limit of 10, so please make sure that you have enough or you'll have to refactor the code or change the scheduling to get it to work.

Use the following to create a scheduled job that will execute your unit tests daily at 4 AM.

AutomatedTestJobQueuer.createDaily4AMScheduledJob();

Use the following to create four scheduled jobs that effectively will process the test results and email them every 15 minutes, if needed.

AutomatedTestingJob.createEvery15MinuteScheduledJobs();

There's also a webservice method "enqueueUnitTests" that allows you to queue them on demand and have the results emailed to you. I envisioned this to be useful for continuous integration purposes.

For example, if you're using source control, which I really hope you are, and you check in code, this method would be called to enqueue and run your unit tests. Essentially, whenever code is checked in, the unit tests will run in semi real-time and you'll know fairly quickly whether or not you broke something. Since finding and fixing bugs is cheaper and faster when you're already coding somewhere, this'll save you from many future headaches.

I hope this helps and happy coding.