Be careful of large log files using TeamCity

Previous week I had very interesting experience with TeamCity continuous integration server. From some moment it became unable to perform any actions, just hanging and them crashing. After analysis of the server logs and running processes we identified that it uses 100% CPU time and then fail with errors related to memory limitations. The first level of the issue was that it runs garbage collector very often and without success (GC couldn't release memory in any generation). But why? To understand such behavior we used one of the very well done open source profilers - VisualVM. We found that one of the threads created a very large list of strings (more than 100000) and continued to add new strings. The root of the issue was very large log file from one of our builds (this build always works some hours and gathers all actions in log). TeamCity saved this log and on each start it tried to read it all in memory line by line to allow users view build history logs. We have already configured TeamCity to perform clean up procedure for all historical artifacts and logs one per week, but it didn't save us from crash. To fix the issue we cleaned manually all historical data from .BuildServer/system/messages. Also we configured problematic build to output all log messages in log file and now we use this log file only as artifact of the build. So, logging is well, but be aware of large log files especially when somebody read them. Develop with pleasure!