Problem: Given a system which is not performing very well, what steps can be taken to increase its performance
System performance depends on a lot of factors.
Below are some of these factors and the corrective measures to increase performance:
Check file IO and try to minimize it.
Look for database locks.
Run some analytics on DB to find long running queries.
Normalize columns where group-by is running on strings
Denormalize tables having frequent joins.
One-to-one mapping tables must be denormalized.
Use SOLR/Lucene for running text-search queries.
Normal databases do not do this very well.
Create indexes wherever missing.
Use hadoop if too many requests are coming per second.
Try to use map-reduce to avoid loading of one server.
Use caching to improve performance.
Turn on Hibernate second-level caching if there are too many surrogate primary key lookups.
Try using distributed caching.
Check that no library is running in debug mode.
Log message printing should not be too excessive to cause slow down.
Check if logger is becoming a bottleneck by creating too much synchronization.
Threads and synchronization:
Take a thread dump to see if some threads are deadlocked or livelocked.
Grep through the code and look for synchronized keyword. See if the blocks are correctly synchronized in code.
Use thread-pool executors.
Use ThreadLocal variables wherever possible to reduce synchronization.
Use AtomicLong, AtomicInteger etc
Use ConcurrentHashMap in place of synchronizedMap(map) to make use of CAS
Do not use Hashtable or Vector if synchronization is not required
Check garbage collector usage. If too much concurrency is there, use concurrent garbage collector or G1 garbage collector.
Check is OS is 32-bit or 64-bit
Check size of RAM
Check the runtime flag with which the binary was launched. Does it restrict memory usage or network bandwidth etc.
Third party libraries and useful commands:
If some third party library is causing bottleneck, try decompiling their libraries.
Use ldd command in linux to see dynamically loaded libraries.
In linux, use nm command to see the symbols of a binary and check if some wrong version of library is being loaded due to wrong PATH variable.
Trace all system calls with the appropriate tool (strace/trace/truss, depending on the OS)
Strace: It intercepts and records the system calls which are called by a process and the signals which are received by a process.
The name of each system call, its arguments and its return value are printed on standard error or to the file specified with the -o option.
Truss: is a debugging utility in Solaris and FreeBSD to monitor the system calls used.
Got a thought to share or found a bug in the code? We'd love to hear from you: