How to Write Small,Useful and Efficient Programs

So far I have written over 40 articles on The Code Project, most of them are about tools and software components I developed. I am not good at explaining and teaching complicated technical stuff, that is why my articles stay at the level of "here is some cool code I wrote and look at how easy it is to use ...". I admire a lot of good writers here at The Code Project such as Marc Clifton, Matt Gullet, and Michael Dunn (those are just the ones whose names start with M.

Today I am going to try something different, that is, write an article that aims at teaching others how to program. But what can I teach? I find most books and lectures about computing technology boring and avoid them as much as I can. As a result, I don't have solid detailed knowledge of the new and popular technologies. I don't even have a basic understanding of some popular terms and catchy phrases. Sometimes I think those terms were invented to confuse ordinary people in order to sell more products and services. Here is a funny exchange between the Detective and Dr. Calvin in the movie iRobot.

Detective: So, what do you do around here?
Dr. Calvin: My general fields are advanced robotics and psychiatry although I specialize in hardware-to-wetware interfaces in an effort to advance U.S.R.'s anthropomorphization program.
Detective: So, what exactly do you do around here?
Dr. Calvin: I make the robots seem more human.
Detective: Now wasn't that easier to say?
Dr. Calvin: Not really. No.

I am not saying that popular terms and catchy phrases are useless, they are just not that useful in solving real world programming problems. I think I do know how to write useful programs. At my work place, I have a reputation of being able to produce long-lasting code. We are constantly fixing/upgrading/rewriting our applications, but a lot of code I wrote (some are from 5 years ago) are still being used without any significant change. Of course, for people who don't like my code, they can always say my code is so bad that nobody wants to touch it.

Like many people have said, programming is not an exact science, it is more like an art. To be successful in programming, you need a combination of talent, experience, persistence, and most of all, a lot of common sense. Speaking of common sense, I like the following Chinese idioms:

• Believing everything in books is worse than not reading any book.
  
• A teacher can only get a student started to learn, it is up to the student to finish the learning.
  
• Black cat or white cat, the one that catches mice is the good cat.

The above are just some common sense we should have. Many programmers and managers are bad because they don't use much common sense in their work. Solving programming problems is no different than solving other problems in life, you use whatever is available to you and there is no point in restricting yourself to some artificial patterns or popular technologies just to prove to others that you know them. I would prefer to attack the problem directly instead of spending a lot of time and energy on abstractions/frameworks in order to fit the final solution into some well-known design pattern. One thing I hate when reading some textbook-like code is that almost all the methods call some other methods, it is very hard to find a method that actually does the real work. Don't get me wrong, abstraction has its use, that doesn't mean you must use it everywhere. In his Kong Fu movie, Bruce Lee teaches his students to hit the target directly instead of dancing around. Great minds think alike, don't you agree? :-)

I like "service oriented architecture". I think it is just like the old times when you write a function to solve a problem, except the function is now a "service" and it can potentially be used by more programs. I may be completely wrong about what exactly "service oriented architecture" is, who cares.

In this article, I am going to use two Java classes as examples. Why Java? You may ask (come on, don't tell me you like C# syntax better than Java syntax). I wrote similar code first in C++ and VB then in VB.NET and C#. I haven't used Java for a while, it is a good opportunity for me to enhance what I did before and to pick up things I have already forgotten.

Before I go into specifics, I would like to state that the programs I am talking about here are not large frameworks or libraries that are intended for the users to expand (such as .NET or Java Framework). My interest is in developing small reusable components or utilities. All the methods in the sample classes of this article are "final" which means that users are supposed to use them without overriding (but you can change whatever you want in the code). Building large frameworks and libraries require different approaches, of course.

Disclaimer: It goes without saying that my ideas and methods may be incorrect, misleading, or even offensive. I know it is already too late for the disclaimer.
A Class to Write Log Files

The first of my two Java classes is XYTrace, it is for writing application information into log files. It is intended to be used for both development and production environments. Before I begin the next sentence, some well-informed readers will tell me that there already exist a lot of commercial and free software tools for doing the same thing, why would I bother to reinvent the wheel? Let me give you my reasons, if I cannot convince you, that is too bad.

The existing software tools and packages for logging may be more flexible and powerful. For example, they may allow you to write log messages into different kinds of media, such as Windows Event Viewer or your mother's cell phone or your sister's iPod, besides simple text files. However, are these extra features really useful? For myself, I have never seen the need to write trace information into anything else other than simple text files. Tools with too many features are usually harder to learn and use.

I would like a small self-contained class for this purpose. Users should be able to use it without much learning and it should not depend on some external libraries (except the Java run-time environment, of course). I would like to include enough useful features and at the same time keep the design simple. Remember, it is never hard to produce a complicated design, the challenge is to come up with a design that makes complicated stuff look simple.
The Class Interface

After some "extensive" research and analysis, I decided that the basic interface of my XYTrace class should be the following two public static methods:


java code

final public static void SetTraceOptions
    (String sTraceFolderPath, String sAppName, int nAppTraceLevel, int nCleanupDays);
final public static void WriteTrace(int nTraceLevel, String sMessage);

What? Two static methods? That's it? Yes, that's good enough for me. There are no exceptions or errors for these methods. They should work, period. I prefer static methods because I think in this case it is unnatural to create an instance of the class then use it: why should we create anything if we don't need to? Let me first list the features I want my class to have.

1. Users can call the WriteTrace method directly to write any string message into a log file in the current working directory, no initialization or configuration is required. The message header in the log file will include a time stamp, a number indicating the seriousness of the message (trace level), and a number identifying the current thread.
   
2. There will be one log file per process per day with some exceptions explained later. The log file name will contain a date-time stamp so that you know exactly when it was created. The size of the log file won't get too big. Old log files will be deleted automatically.
   
3. You can dynamically determine where to create the log file and how much information is written to the log file by calling the SetTraceOptions method.
   
4. Thread-safety and reasonable performance.

Application Trace Level

This is a private static integer member of the class. It determines how much trace information is written to the log file at run-time. I only use five different values: 0 for no tracing, 10 for error messages only; 20 for error messages plus warning messages; 30 for error, warning, and some additional information; 40 for the most detailed tracing which includes debugging messages and all previously mentioned messages.

If you prefer, you can use other positive integers as the value for application trace level. The default application trace level is 40. The application trace level interacts with the first parameter in the WriteTrace method. The only way to modify this value is calling the SetTraceOptions method. For example, you can have the following statements in your code:


java code

XYTrace.WriteTrace(10, "This is an error message");
XYTrace.WriteTrace(20, "This is a warning message");
XYTrace.WriteTrace(30, "This is just a normal message");
XYTrace.WriteTrace(40, "This is a debugging message");

If the application trace level is 0, then none of the above statements will generate a message in the log file. If the application trace level is 10, then only the first statement will generate a message in the log file. If the application trace level is 40, then all of the above statements will generate messages in the log file.
Descriptions of the Methods

Now I am going to give more detailed descriptions of these two static methods and their parameters.

The SetTraceOptions method has four parameters:

• sTraceFolderPath
  
• sAppName
  
• nAppTraceLevel
  
• nCleanupDays

The value of sTraceFolderPath determines where the log files should be created. In case this method is not called or the specified folder does not exist, log files will be created in the current working folder of the application process.

sAppName is a string that is going to be part of the log file name (in addition to the time stamp) so that if we store log files of different application processes in the same folder, it will be clear which log file belongs to which application.

nAppTraceLevel is the application trace level described above, the default value of application trace level is 40.

nCleanupDays is the number of days before old log files get deleted. If this value is 0, then old log files won't be deleted. The default value for this property is 3, if you don't call SetTraceOptions method.

Log files are just simple text files that can be viewed in Notepad or other simple text viewers/editors. Typically, this class creates one log file per process per day. If the application is restarted, a new log file will be created. Calling the SetTraceOptions method while the application is running will also close the old log file and create a new one. I recommend that this method be called only once at the beginning of the application. If you don't call this method at all, then default values will be used for all the options. Another situation where a new log file will be created is when the size of the current log file exceeds 10 mega bytes.

The WriteTrace method has two parameters:

• nTraceLevel
  
• sMessage

The first one is the intended trace level which interacts with the application trace level to determine whether or not the message is actually written to the log file. Error messages should have a lower trace level (say 10) and detailed debug messages should have a higher trace level (say 40).

The second parameter is the message string. Messages written by the above statements look like the following:

java code

072800000_1_10: This is an error message 
072801000_1_20: This is a warning message  
072802000_1_30: This is just a normal message  
072803000_1_40: This is a debugging message

Please note that the first part (9 digit number) of the message is a time stamp in the "hour minute second millisecond" format. The next number is the thread id, and then the intended trace level. So we know the first of the above messages was written at 07:28:00, the second at 07:28:01, ..., and all of the above messages were written by thread 1.
Implementation, Enhancement, and Performance

In order to implement this class, I was forced to add two more public methods, run and accept.

The run method is for the Java Runnable Interface. With .NET, you need to use a ThreadStart delegate to pass a method to be invoked by a new thread. Java works differently by passing an object implementing the Runnable interface to a new thread, the new thread will invoke the run method of the object.

The accept method is for the Java FilenameFilter Interface. In Java, the File class can represent either a file or a folder. I use a File object in XYTrace internally to represent the trace folder. If an instance of the File class is a folder, the listFiles method will return a list of all files and subfolders of this folder. If you pass an object that implements the FilenameFilter Interface to the listFiles method, then the accept method of the object will be invoked to filter the files and subfolders so that the list returned by listFiles will only have the ones you want.

In XYTrace the run method is invoked by a background thread that deletes old log files periodically, the background thread will be started at most once in the life of the process. The accept method is also invoked (indirectly) by the background thread to make sure that only log files generated by this class will be deleted (no other files will be deleted). Although users are never supposed to call these two methods explicitly, Java dictates that they must be public.

I hate any method with more than five parameters. I think it reflects poor design in many cases. If the information you need to use in the code can be obtained elsewhere, why bother the users? To make your program more useful, it is better that you make the obvious decisions for the users instead of letting them get confused. It is also a good practice to provide default values if you do need to have more parameters and options. I had a rule for myself to use as few parameters as possible for methods in my class. However, if adding one or two parameters does provide convenience to the user, then the rule should be broken. That's why I added other WriteTrace methods with more or less parameters. Here is the list of the additional methods:


java code

// we need these because Java does not allow default values for parameters 
final public static void WriteTrace(int nTraceLevel, Object oMsg); 
final public static void WriteTrace(int nTraceLevel, Object oMsg0, Object oMsg1); 
final public static void WriteTrace
    (int nTraceLevel, Object oMsg0, Object oMsg1, Object oMsg2); 
final public static void WriteTrace
    (int nTraceLevel, Object oMsg0, Object oMsg1, Object oMsg2, 
    Object oMsg3); 
final public static void WriteTrace
    (int nTraceLevel, Object oMsg0, Object oMsg1, Object oMsg2, 
    Object oMsg3, Object oMsg4); 
final public static void WriteTrace
    (int nTraceLevel, Object oMsg0, Object oMsg1, Object oMsg2, 
    Object oMsg3, Object oMsg4, Object oMsg5); 
final public static void WriteTrace(int nTraceLevel, Object[] pMessages);
final public static void WriteTrace(int nTraceLevel, Exception oBug); 
final public static void WriteTrace(Exception oBug);

All of the above methods are implemented using the original WriteTrace method. They allow users to write code below:

java code

try 
{ 
    ... 
    int nUserID = 4; 
    String sUserName = "James"; 
    // compose a single message from several objects 
    XYTrace.WriteTrace(40, "ID = ", nUserID, ", Name = ", sUserName");
    .... 
    return true;
} 
catch(Exception oBug)
{ 
    // Exception object as the only parameter
    XYTrace.WriteTrace(oBug);
    return false;
}

The two calls to different versions of the WriteTrace method in the above code will produce messages like the following:

java code

073200000_1_40: ID = 4, Name = James
... 
073201000_1_10: java.sql.SQLException: [Microsoft][ODBC Driver Manager] 
Data source name not found and no default driver specified
    sun.jdbc.odbc.JdbcOdbc.createSQLException(JdbcOdbc.java:6958)
    sun.jdbc.odbc.JdbcOdbc.standardError(JdbcOdbc.java:7115)
    sun.jdbc.odbc.JdbcOdbc.SQLDriverConnect(JdbcOdbc.java:3074)
    sun.jdbc.odbc.JdbcOdbcConnection.initialize(JdbcOdbcConnection.java:323)
    sun.jdbc.odbc.JdbcOdbcDriver.connect(JdbcOdbcDriver.java:174)
    java.sql.DriverManager.getConnection(DriverManager.java:525)
    java.sql.DriverManager.getConnection(DriverManager.java:193)
    XYDB.OpenDB(XYDB.java:95)

Please note that the second call takes an Exception object as the only parameter and the resulting log message includes the stack trace of the exception object.

As for performance, we can test the code of the WriteTrace method in a simple "for loop" to see how much time on average it takes to write a log message, but the result is not meaningful because real applications never write log messages this way. However, we can at least do the test to make sure that executing the code in WriteTrace will not have any significant impact on the performance of most real applications. Here is the result I got, it takes 29 seconds on my computer to write 1 million log messages. For those who wonder, I also tested similar VB.NET code on the same machine, the result is 14 seconds. I think we can pretty much rule out the possibility that the WriteTrace statements in your code will drag down performance.
Error Handling

In this case, there is not much you can or need to do. Think about it, if an error occurs while logging an error message, where and how would you log it? If we let WriteTrace fail silently, the worst that can happen is you don't get log messages written and there could be (not likely) a slight performance hit. In general, what we need is sensible error handling which could mean no error handling at all. We will come back to the topic of error handling later.
A Brief Summary
To summarize, here are the guidelines and rules I use to design and implement small useful programs.

1. Attack problems directly. Don't use unnatural abstractions.
   
2. Keep design and interface simple. Excessive flexibility and power can hurt usability.
   
3. Reduce the number of parameters in your class methods.
   
4. Make decisions for the users instead of offering options that are not exactly meaningful or useful to them.
   
5. Sensible error handling.
   
6. Break the above rules when necessary.
   
7. Common sense, common sense, common sense ...

Imagine that an OOP zealot is designing the XYTrace class. First, you will have to create an instance of XYTrace using some kind of factory class. The parameter to the WriteTrace method will be an instance of an abstract TraceMessage class from which 3 or more subclasses are derived; Each of the derived classes will have 2 to 5 constructors taking from 3 to 8 parameters; And there will be at least 2 custom exception classes to "help" users to handle error conditions. OOP is supposed to be flexible and easy to use, in reality it can be just the opposite if we go too far.
A Class to Access Database

The second class is XYDB, it is for accessing relational databases via JDBC. This class is designed and implemented according to the same guidelines and rules explained previously. Again, I want a self-contained class and it should satisfy all basic database needs which are listed below.

1. Connect to all databases for which JDBC or ODBC driver is available
   
2. Use dynamic SQL statement to query or update data
   
3. Call stored procedures to query or update data
   
4. Return data in XML format
   
5. Perform simple transactions
   
6. Implement connection pooling
   
7. Simple error handling

The XYDB class is built on top of the classes in the java.sql package which is part of the standard Java SKD. It does not support all possible data types. Users need only to know how to write SQL statements and how to use XML, they don't need to know details of JDBC or ODBC. The reason you can use almost any ODBC data source with this class is that a special JDBC driver called jdbc-odbc bridge is included with the standard Java environment.
The Class Interface

Let's discuss the class interface. Here is the list of all public methods in this class:


java code

final public static boolean LoadDriver(String sDriverClass);
final public static XYDB GetDB(String sDBURL);
final public void CloseDB(); 
final public void ReleaseDB(); 
final public String ExecuteSQL(String sSQL); 
final public String ExecuteSQL(String[] pSQL); 
final public String ExecuteSQL(String sSQL, int[] pParamType, Object[] pParamValue);
final public String ExecuteSQLTransaction(String[] pSQL);
final public String CallProc
    (String sSQL, String[] pParamName, int[] pParamType, Object[] pParamValue);
final public boolean BeginTrans();
final public boolean CommitTrans(); 
final public boolean RollbackTrans(); 
final public Exception GetLastError();

Again, I want this class to be small and useful. I put a lot of thought into the design of the interface. Let's look at the ExecuteSQL method. It is supposed to be used for both query and update actions.

Why does it return a string? In Java, you can use a ResultSet object to hold a collection of database records returned from a query (a ResultSet is similar to a RecordSet in ADO). However, using the ResultSet class as return type means the user has to know several other classes, for example the Statement class in the java.sql package, in order to manipulate the returned data. It also means the internal Statement object used in the implementation of the ExecuteSQL method has to be kept open (cannot be closed immediately after the query). Things get even more complicated if we want the ExecuteSQL method to be able to return more than one result set.

So I decided to use string as the return type. The return value is null if the query or update action fails. If the call is successful, the return value will be an XML string. The format of the string looks like the following:


xml code

<DataSet>
    <Table0>
        <Column00>...</Column00>
        <Column01>...</Column01>
        <Column02>...</Column02>
        ...
    </Table0>
    <Table0>
        <Column00>...</Column00>
        <Column01>...</Column01>
        <Column02>...</Column02>
        ...
    </Table0>
    ...
    <Table1>
        <Column10>...</Column10>
        <Column11>...</Column11>
        <Column12>...</Column12>
        ...
    </Table1>
    <Table1>
        <Column10>...</Column10>
        <Column11>...</Column11>
        <Column12>...</Column12>
        ...
    </Table1>
    ...
</DataSet>

Table0 and Table1 represent records from two different select queries in the SQL statement. ColumnXXs are actual field names in the corresponding select query. As you can see, the ExecuteSQL method may return more than one result set depending on the input SQL statement and the underlying database. What if the SQL statement is for update only? In that case, the return string will be <DataSet></DateSet>. Remember, for any failure the return value is null.

If you are familiar with the DataSet class in .NET, you will see that the above XML string can be used to populate a DataSet object. So you can write your .NET application that calls another program written in Java which uses this class to query the database. Once you get the return string, use it to populate a DataSet object and pass the DataSet object to other methods in your application.

Typically an SQL statement will only return one result set. With SQL Server 2000 and may be some other databases, you can include two or more select statements in one string and execute the batch to get more result sets (more than one table in the above XML output). Even if the database you are using does not support this, you can call a different version of the Execute-SQL method which takes an array of SQL statements as input, the result will be as if your database does support multiple selects in one SQL statement. For example, if you know an employee's id, you can pass two SQL select statements in an array to ExecuteSQL, one queries the employee's phone numbers and address (which is stored in the Employee table) and the other queries the same employee's past salary history (which is stored in the Salary table).

Another question you may ask is, why does the Execute-SQL method take only the SQL statement as parameter? This reflects my personal preference in writing code to access databases. If I want to do parametrized query, I will construct the SQL statement using code similar to the following:


java code

String GetData(XYDB oDB, String sParam, int nParam)
{
    String sSQL = "select * from MyTable where Col_1 = 
            '"+sParam.replaceAll("'","''")+"' and Col_2="+nParam;
    return oDB.ExecuteSQL(sSQL);
}

However, there is another version of the ExecuteSQL method and the CallProc method in this class that can be used to pass an array of parameters to your SQL statement.
Database Connection and Connection Pool

Before connecting to the database, you need to load the corresponding JDBC driver. The following statements load Sun's jdbc-odbc bridge and Microsoft's JDBC Driver for SQL Server 2000. The static method LoadDriver will return true if successful and false if the driver class cannot be found.


java code

if(XYDB.LoadDriver("sun.jdbc.odbc.JdbcOdbcDriver")==false) 
    throw new Exception("Failed to load jdbc-odbc bridge");
if(XYDB.LoadDriver("com.microsoft.jdbc.sqlserver.SQLServerDriver")==false) 
    throw new Exception("Failed to load JDBC Driver for SQL Server 2000");

Each XYDB object is a physical connection to a database. The static method GetDB will return an XYDB object for a database connection represented by the sDBURL string parameter, in case of failure, it will return null.

The format of the sDBURL parameter depends on the corresponding JDBC driver. sDBURL is typically made up of protocol name, company name, server name, port number, database or data source name, user, and password. To learn the database URL format and JDBC in general, I recommend a little book "JDBC Pocket Reference" by Donald Bales , the author should definitely write a "How to write small and useful books" article on The Code Project.

Here is an example of getting a database connection. If connection to the database cannot be established, GetDB will return null.


java code

XYDB oDB = XYDB.GetDB("jdbc:odbc:MyODBCDataSource;user=MyUser;password=MyPassword"); 
if(oDB==null) throw new Exception("Failed to connect to database via jdbc-odbc bridge");

After getting the XYDB object, you can call its ExecuteSQL method to do query or update. If you no longer need the database connection, you can call the CloseDB method or the ReleaseDB method. CloseDB will close the database connection. ReleaseDB will return the connection to an internal pool so that it can be reused later. That is, GetDB only returns a brand new XYDB object when there is no unused connection with the same sDBURL parameter in the internal pool. If you decide to take advantage of the connection pool, you should always call ReleaseDB instead of CloseDB.

We are only talking about connection pooling within the same application process here. If you want to share and reuse database connections among different applications, then you need to build a multithreaded server using this class. Other applications can send database requests to your server via HTTP or soap or TCP protocols. Your server will query and update the database and send the results back to the other applications. This way you centralized database access in your server and made it possible for other applications to share the same pool of connections.

Another feature I need to mention is that whenever a database error occurs, XYDB will close the underlying connection. And it will automatically reconnect to the database the next time the same object is used. This feature does have an implication when you want to do explicit database transaction in your code. I will explain it below.
Simple Database Transactions

If you want to execute a batch of SQL statements in a transaction, you can do one of the following:

1. Include the SQL statements in a stored procedure and use the ExecuteSQL method to invoke the stored procedure.
   
2. Use the ExecuteSQLTransaction method, the return data format is the same as that of ExecuteSQL (null is returned for failure).
   
3. Call BeginTrans, ExecuteSQL (one or more times), CommitTrans or RollbackTrans manually in your code.

Please note that it is very important to check for failure (indicated by false or null return values) of the BeginTrans method and the ExecuteSQL method if you are choosing approach #3 above. This is because if a database error occurs, the connection will be closed and the transaction will be lost. If you keep calling ExecuteSQL regardless of the failure and without starting from BeginTrans, the database connection will be re-established and this time the SQL statement will be executed outside of any transaction!

The BeginTrans, CommitTrans, and RollbackTrans methods return true or false to indicate success or failure. They do not support nested transactions.
Error Handling

The XYDB class uses the XYTrace class described in the first part of the article to log error messages. You need to call the XYTrace.SetTraceOptions method if you don't want log files to be written to the current working directory of the application process. By default, log files will be kept for 3 days before they are deleted automatically. Besides basic error information (exception message and stack trace), all JDBC database URL strings and all SQL statements will be written to the log file to help in identifying problems.

You may have noticed that all methods of the XYDB class do not throw any exception. The same is true for the XYTrace class. Users do need to check the return value to learn whether a method call has failed or not. They also need to look at the log file if they want to know more about specific database errors. I think this is better than throwing exceptions, especially custom exceptions derived from the Java Exception class. When you throw a custom exception, you put a big burden on the users. Typically, there is not much the user can do besides log the error, fail the current operation or ignore the error and keep trying. They can do the same without catching exceptions. Just in case some users want to handle exceptions in their code, the GetLastError method is there to return the last exception that happened within the current connection.
Final Note

The code included in this article is written with JDK 1.5.0. No sample code is provided to use these two classes. If you cannot figure out how to use it, then the article or the code is not well-written. Please suggest improvement ideas. Thanks