Kill your application with Hibernate subselect!

If you want to kill your application silently the best way is to use Hibernate 'subselect' fetching strategy. Lets discuss the issue in details. Imagine that you have entities 'Post' and 'Comment' in the domain model with corresponding tables in the database:

public class Post {
    private Long id;
    private String title;
    private String body;
    private Set<Comment> comments;
    ...
}

public class Comment {
    private Long id;
    private String text;
    ...
}

Of course you use 'one to many' relationship to map relations between posts and comments (using 'post_id' column from comments table). It doesn't matter what mapping type you prefer (annotations of XML based). All seems good at this moment and you decide to choose 'subselect' fetching strategy for comments collection. You also create a number of unit tests to make sure that you mapping is well done and works fine. Now you want to create method to find last 10 posts for your home page. You decide to use Criteria API for this purpose:

List posts = session.createCriteria(Post.class)
     .addOrder(Order.desc("id"))
     .setMaxResults(10)
     .list();

You again create some unit tests to make sure that method works as expected. But of course you don't review SQL queries in Hibernate logs to check if all is as you expect to be. Hibernate generates following SQL queries for you (example from PostgreSQL database):

select post0_.id as id0_, 
post0_.title as title0_, 
post0_.body as body0_ 
from Post post0_ 
order by
post0_.id desc
limit 10;

select
comment0_.post_id as post5_1_,
comment0_.id as id1_0_,
comment0_.text as text1_0_
from Comment comment0_ 
where comment0_.post_id in (
   select post0_.id 
   from Post post0_
); 

Pay attention to the second query, especially to its subquery part. Limitation is not included in the subquery! What is going on here? It seems that Hibernate is going to load full comments table in memory and then select from them related to top posts selected before. Crazy behavior! Initially, when database is empty or there is low amount of comments, your application will work well. But every time somebody opens your home page with top blogs shown query loads all comments in memory, so you have performance penalty. This penalty is proportional to the number of comments in the database. Once (may be after some monthes of usage in production) you will find that all memory allocated to your application is filled and garbage collector eats 100% of CPU time. Thanks god for profilers, especially for VisualVM. It is hard to believe that such small issue in Hibernate may cause such dramatic effects. There is an opened issue in Hibernate bug tracker, but it has minor priority. We need to keep living with it, so another approach should be applied. The best way to avoid these issues is to use 'batch select' fetching strategy with lazy loading (or without it depending on application needs). Be careful and develop with pleasure!

Defining batch size for batch fetching in Hibernate

One of the Hibernate performance tuning ways when you need to work with parent/children relationships is to use lazy loading collections with batch fetching. This way allows you to perform far less than n+1 SQL queries to initialize your entities. To enable this mode you just need to specify batch-size attribute in XML mapping for collection or mark it with @BatchSize annotation in Java code. But how to define appropriate batch size? You need to understand how it works internally to do it well.

From examples you may see that if there are 25 objects in the database and batch size is set to 10 then Hibernate will perform 3 SQL queries: 10, 10, 5 items. But it is not so simple in the case of larger batch size. Hibernate internally creates an array of batch sizes using following strategy: if batch size <= 10 (for example 5) then it fills array with numbers from 1 to batch size ([1,2,3,4,5] for 5), else (for example 50) it fills array with numbers from 1 to 10 and integer parts of division batch size by powers of 2 ([1,2,3,4,5,6,7,8,9,10,12,25,50] for 50). So, now lets see how many SQL queries will Hibernate perform in case when batch size is 50 and there are 38 records in the database. The answer is 3: 25, 12, 1. Are you surprised? Hibernate doesn't create too many JDBC prepared statements for batch fetching so it performs querying using array of batch sizes. So, when you know the truth how to define the best batch size for your application? The answer is simple and relies on the mathematical theory: you should use powers of 2 multiplied by 10 (for example 10, 20, 40, 80, etc.) because each positive integer may be represented as a sum of powers of 2 (for example 13 = 8 + 4 + 1). If you select 40 instead of 50 in the previous example you will see benefit for example when number of records is 23: 2 SQL queries (20, 3) instead of 3 SQL queries (12, 10, 1). Of course if you know that a number of records in the database will always be small enough then use smallest batch size from recommended - 10. If you don't know how many records will be in the database then switch on logs for SQL queries and analyze how many of them are performed to define batch size from the recommended formula. I hope this will help you make Hibernate more productive. Develop with pleasure!

Make your tests excellent with Unitils

I am going to share my experience in using Unitils - framework that makes life easier. I have started to work with this framework from early release candidates and can't imagine my tests without it now. If you need to test application that use Hibernate, Spring, other ways of database access, but you don't want to write much test code and create some homegrown frameworks, then Unitils is your choice. If you already use some specific testing frameworks like EasyMock or DbUnit, but your tests are hard to create and maintain then Unitils will remove this pain for you.

Unitils started as a discussion group on unit testing. The result of these discussions is the list of guidelines that will be useful without Unitils itself. Then some people decide to build framework that will contains all in one, but be flexible enough to use it in different applications. So Unitils was born. To help developers with mapping guidelines to the framework there are tutorial and cookbook both published on the Unitils web site.

Unitils supports JUnit3, JUnit4 and TestNG and contains following modules:

- Spring module (for loading application contexts and retrieving and injecting Spring beans)
- Hibernate module (Hibernate configuration support and automatic database mapping checking)
- DbUnit module (test data management using DbUnit)
- Database module (unit-test database maintenance and connection pooling)
- EasyMock module (for creating mocks and lenient reflection argument matching)
- Inject module (for injecting (mock) objects into other objects)

Unitils configuration is very flexible and very small (if you are satisfied with default behavior). You can select any modules for your application and don't use others. Unitils work is based on annotations, so you will not need additional configuration files like except general settings file. Also Unitils provide to developers very useful assertion utilities, that based on reflection and support different modes for asserts.

Unitils has modular architecture so you can easy write your own modules to extend existing functionality. It is open source project so your modules may be included in future releases. General Maven repository contains last version of the framework so those of you who use Maven will not need manually install it. One important part is missed in Unitils - mocks for Java 'native' technologies like Servlets, JSP, JNDI and etc. I use a part of Spring framework (spring-test) for this purpose and it makes job very well.

From June 25, 2008 Unitils 1.1 became available. This release includes support of JPA, Spring transactions and complex database scripts structure. Also many integration bugs was fixed for database module.

I don't want to include examples for Unitils usage, you can find them in the tutorial or cookbook with detailed descriptions. Also there is useful video presentation with concrete samples. Make your tests better and your application code will be better. Develop with pleasure!

Select implementation of Spring bean in runtime

Some of you who uses Spring in your application as dependency injection framework or as development platform may be need change implementation of one particular dependency in runtime. Some examples of this need: integration testing with fake external subsystem, local and remote versions of services, mock implementation to speed up development.

I have found very useful to have such option in my applications and implemented special factory bean for this purpose:

public class SelectImplementationFactoryBean implements FactoryBean {
  private String key;
  private Map<String, Object> implementations;

  public Object getObject() {
      Object impl = implementations.get(key);
      if (impl == null) {
        throw new IllegalStateException("There is no implementation for key = " 
                                        + key);
      }
      return impl;
  }

  public Class getObjectType() {
      return getObject().getClass();
  }

  public boolean isSingleton() {
      return false;
  }

  @Required
  public void setKey(String key) {
      this.key = key;
  }

  @Required
  public void setImplementations(Map<String, Object> implementations) {
      this.implementations = implementations;
  }
}

Simple enough but very powerful. I will demonstrate how to use it to control database properties for integration with Hibernate:

<bean id="hibernateProperties" class="com.example.SelectImplementationFactoryBean">
  <property name="key" value="${database.dialect}"/>
  <property name="implementations">
    <map>
      <entry key="postgre">
          <util:map map-class="java.util.Properties">
            <entry key="hibernate.hbm2ddl.auto" value="validate"/>
            <entry key="hibernate.show_sql" value="true"/>
            <entry key="hibernate.dialect"
                      value="org.hibernate.dialect.PostgreSQLDialect"/>
            <entry key="hibernate.default_schema" value="test"/>
            <entry key="hibernate.cache.use_second_level_cache" value="false"/>
            <entry key="hibernate.cache.provider_class" 
                      value="org.hibernate.cache.NoCacheProvider"/>
          </util:map>
      </entry>
      <entry key="hsqldb">
          <util:map map-class="java.util.Properties">
            <entry key="hibernate.hbm2ddl.auto" value="validate"/>
            <entry key="hibernate.show_sql" value="true"/>
            <entry key="hibernate.dialect" 
                      value="org.hibernate.dialect.HSQLDialect"/>
            <entry key="hibernate.cache.use_second_level_cache" value="false"/>
            <entry key="hibernate.cache.provider_class" 
                      value="org.hibernate.cache.NoCacheProvider"/>
          </util:map>
      </entry>
    </map>
  </property>
</bean>

Now to switch your tests to another database you should change value of the property database.dialect between postgre and hsqldb (it can be configuration properties file or system property). I hope it will save your time. Develop with pleasure!