What is Order of operation with LTRIM/RTRIM/ISNULL in SQL server?

SQL server rtrim 

In SQL Server RTRIM is a Transact-SQL function that removes all space characters from the right-hand side of a string. SQL Server will evaluate functions inside to out. Your first example is equivalent to: In SQL Server RTRIM is a Transact-SQL function that removes all space characters from the right-hand side of a string. SQL Server will evaluate functions inside to out. Your first example is equivalent to:

DECLARE @Test1 varchar(16) = '    '
SET @Test1 = ISNULL(@Test1,'')
SET @Test1 = RTRIM(@Test1)
SET @Test1 = LTRIM(@Test1)
IF @Test1 = ''
BEGIN
    SET @Test1 = NULL
END
SELECT @Test1

Since ISNULL just returns the first non-NULL value, the function just returns @Test1. It then goes on to trim whitespace. If you reverse the order of these it's going to act the exact same way because ISNULL will always evaluate to FALSE. For your described purpose, the order is largely irrelevant. The ISNULL function is only needed for situations where you don't care about trimming whitespace, and vice versa (as LTRIM/RTRIM do nothing to NULL values).