Be careful of large log files using TeamCity

Previous week I had very interesting experience with TeamCity continuous integration server. From some moment it became unable to perform any actions, just hanging and them crashing. After analysis of the server logs and running processes we identified that it uses 100% CPU time and then fail with errors related to memory limitations. The first level of the issue was that it runs garbage collector very often and without success (GC couldn't release memory in any generation). But why? To understand such behavior we used one of the very well done open source profilers - VisualVM. We found that one of the threads created a very large list of strings (more than 100000) and continued to add new strings. The root of the issue was very large log file from one of our builds (this build always works some hours and gathers all actions in log). TeamCity saved this log and on each start it tried to read it all in memory line by line to allow users view build history logs. We have already configured TeamCity to perform clean up procedure for all historical artifacts and logs one per week, but it didn't save us from crash. To fix the issue we cleaned manually all historical data from .BuildServer/system/messages. Also we configured problematic build to output all log messages in log file and now we use this log file only as artifact of the build. So, logging is well, but be aware of large log files especially when somebody read them. Develop with pleasure!

Risk of inheritance

Inheritance is one of the basics in any object oriented language but you always should think a lot before use it in your code. Of course inheritance is very useful when you design your code for it, use abstract classes, template method design pattern, create different implementations overriding well defined protected methods and so on. But in some cases inheritance may make damage to your application in future very silently.

Imagine situation when you want to add some behavior to class and decided to override its public methods in the child class. The following hidden issues are waiting for you:

* You don't know how this method is used and how it will be used in future in code of the parent class. Your changes may cause unpredictable behavior.
* You want to control some access points of the parent class, but in future some more points can be added and your code will fail in unpredictable time interval.
* Overridden methods will be removed in future, deprecated or change their access level.
* Some other issues depending of concrete scenario.

To avoid most of these issues use composition instead of inheritance. Create new class and declare needed behavior by implementing the same set of interfaces such a basic class. May be you need only some of them, so you can localize your needs and have easier implementation saving time for support. Then implement some methods and delegate all other to the instance of basic class.

Composition has its own disadvantages when basic class is too large or creation of the instances is too complicated, then you need to duplicate a lot of code or logic. IDE or other automation tools can help you with generation of such kind of code.

But what to do if basic class doesn't implement any interfaces and declared abstract? In this case you have to protect yourself from as many issues as possible. Create unit tests to control basic class methods and validate if all of them are known by you. These tests will notify you if basic class was changed so you need to look at new implementation and may be make changes in code or tests. To access information about declared methods use reflection API or some open source libraries that simplify access to this information. But think before if you really need inheritance in this case (if class was not designed for inheritance) and try to avoid it if possible. Develop with pleasure!

Crisis in action

As far as you know in many countries economical crisis came and many "experienced" people was fired in IT market. But crisis makes a good job for some companies. I want to share my experience in interviewing one of such "experienced" guys. This person has a lot of practical experience with Java language at all levels from core to web. After some general questions I proposed following interface declaration:

public interface I {
   int b = 5;
   void a();
}

The question was if this declaration is correct. A received the following answer: "No. You need to change it to be correct." Here is what he wrote:

public interface I {
   int b = 5;
   private void a();
}

The explanation was great: "Instead this method will have visibility only in the package". The next question was about difference between abstract class and interface. The answer makes me smile: "The advantage of abstract class is that I can declare method in it and don't think about it anymore". Imagine how hard is to think about method in all other cases. :) "And why do you ask all these stupid questions? All this is so simple and doesn't matter" - he continued. After that phrase interview was finished.

So, do you want your code to be developed by such guys? If no be aware of crisis. And develop with pleasure...

Generics commons collections

Everybody of you who have large experience with Java thanks Apache for their commons project. This project includes a lot of libraries to extend or simplify work with JDK. One of these libraries is commons-collections.

It was very helpful when you needed to work with specific kinds of collections like multi map, lazy collections, bidirectional map, etc. Also it had a great number of iterators, comparators and utilities to avoid writing unnecessary code in your application. It allowed us to transform one collection to another, filter collection elements and many other things very easy. It introduced completely new types of collections like buffer (collection that allows objects to be removed in some well-defined order), bag (collection that counts the number of times an object appears in the collection), bidirectional map (map that allows bidirectional lookup between key and values) and created a great number of custom implementations of existing collections types.

But why I always use past sentences? Because from the beginning of Java 5 usage this library is not more so helpful. One of the main features introduced in Java 5 was generics. And of course the most useful place for generics to be applied is collections framework. I hoped that it takes not more than year to refactor existing library and add support of generics by commons-collections developers. But it didn't happen. :( But don't worry! There are some solutions of this issue. First one is to use generics version of commons-collections developed in parallel with main Apache version. At this moment they have functionality synchronized with version 3.1. Another option is to use google-collections instead. This project was started as free time development of one developer in Google for internal usage and now it contains a lot of features in comparison to commons-collections. It is driven by community, so only really needed things are included. To start work with this library or to get your own opinion take a look at the video presentation. Develop with pleasure!

Acceptance Test Driven Development training in Kiev

In august I'm going to conduct a training on the Acceptance Test Driven Development in Kiev. Date and agenda of the training was announced with help of Agile Ukraine. Most of the presented tools for automated acceptance testing will be related to Java language (have implementation for Java or native Java support). You are welcome!

Advanced EasyMock techniques

EasyMock is one of many mock libraries available for Java testing. Now it archive version 2.4 and is very powerful. For those of you who are not familiar with mock concepts start with Martin Fowler article. Detailed guide about EasyMock you can read here. I'm going to show some advanced techniques of using EasyMock in your tests.

Imagine that you have following document processing code:

public class DocumentService {
    private DocumentProcessor processor;
    
    public Document getDocument(long id) {        
        Document doc = //code to retrive document
        processor.process(doc);
        return doc;
    }

    public void setProcessor(DocumentProcessor processor) {
        this.processor = processor;
    }
}

public interface DocumentProcessor {
    void process(Document document);
}

public class Document {
    private String text;
    private long wordsCount;

    public Document(String text) {
        this.text = text;
    }

    public String getText() {
        return text;
    }

    public void setText(String text) {
        this.text = text;
    }

    public long getWordsCount() {
        return wordsCount;
    }

    public void setWordsCount(long wordsCount) {
        this.wordsCount = wordsCount;
    }
}

In your tests you want to check if changes made by processor really change returned document. To check this you want to set 'wordsCount' variable value to 5. You may create your own processor to do this:

public class DocumentServiceTest {
    private DocumentService documentService = new DocumentService();
    private DocumentProcessor processor = new DocumentProcessor() {
        public void process(Document document) {
            document.setWordsCount(5);
        }
    };

    @Test
    public void testGetDocument() {              
        documentService.setProcessor(processor);
        Document document = documentService.getDocument(3);
        Assert.assertEquals(5, document.getWordsCount());
    }
}

But also you can use EasyMock (in my example I use Unitils to easy manage mocks):

@RunWith(UnitilsJUnit4TestClassRunner.class)
public class DocumentServiceTest {
    private DocumentService documentService = new DocumentService();
    
    @RegularMock
    private DocumentProcessor processor;

    @Test
    public void testGetDocument() {
        expectDocumentProcessed();
        
        documentService.setProcessor(processor);
        Document document = documentService.getDocument(3);
        Assert.assertEquals(5, document.getWordsCount());
    }

    private void expectDocumentProcessed() {
        processor.process(isA(Document.class));
        expectLastCall().andAnswer(new IAnswer<Object>() {
            public Object answer() throws Throwable {
                ((Document) getCurrentArguments()[0]).setWordsCount(5);
                return null;
            }
        });
        replay(processor);
    }
}

Why this is more powerful solution? Because you can change behavior of the processor in runtime and reuse the same mock in different tests with different behavior. Some times your dependencies are not so simple as document processor in my example and it is not so easy to extend or implements their methods (for example interface has 10 methods and you need only one of them). There is one more side effect of this usage - you have access to the parameters passed to the methods. You can save the parameter instance to use it in asserts or for other dependencies behavior mocking.

In 2.4 version of EasyMock 'capture' option was introduced to simplify work with method parameters. Document class from previous example doesn't have custom 'equals' method, so you can't use 'eq' method of EasyMock to check parameter of 'process' method. You can implement your own argument matcher (see IArgumentMatcher in the EasyMock guide) or use Unitils reflection comparison 'refEq', but another good option for you is to use new 'capture' option:

Capture<Document> capture = new Capture<Document>();
processor.process(and(isA(Document.class), capture(capture)));
replay(processor);

Document captured = capture.getValue();

These two techniques must be in your arsenal to use full power of EasyMock library and make your tests better. Develop with pleasure!

Design for scalability

If you develop large J2EE or WEB application using Java you should always think about scalability of your application (of course if you expect your application to be popular). Nobody tell you that you should scale your application initially, but take in mind that some architecture solutions may make it really hard to scale in the future. You have two common approaches: vertical and horizontal scalability. Both approaches has their requirements for code base to be applied effectively. I have found very interesting article on the www.theserverside.com about J2EE scalability written by Wang Yu from of Sun Microsystems. Article contains two parts: vertical and horizontal scalability. Take this information in mind when make architectural decisions. Develop with pleasure!