Possible performance bottleneck

I want to share with you my own experiences of load testing and especially what are the possible bottlenecks you can encounter.

I summarize into this diagram.


It represents a common architecture of load test plateforme :

• a injector in its own network
• a cloud of network injection has to pass through
• a reverse proxy in a public DMZ
• an Apache Server in a private DMZ
• a loadbalancer, two application server and a database in a private network

Red note are the points you have to check first. It is low level consideration but they must be check first to avoir spending time on details.
Orange note are about configuration of software in use on the machine. Check those points in second time.
At last, yellow note are application problem or miss configuration you have to check at last.

Here some other points which can make you mad and misunderstand the load test results :

• is the target application simply available ?
• is there any application concurrency on some servers ?
• is the date you use correct ? bad data can cause incoherence
• do you access data sequentially ? this can cause Oracle or database contention
• is the tcp access layer well configured ? what about hashcode ? timeout ? ....
• the application is not responding ? you can not restart it ? Check diskspace ! Perhaps diskspace for logs is full and application wont start if it can not write some logs.
  

And to finish some tips :

• ever add %T or %D parameters on the apache log. They provide apache service time. You can use this time to get cloud network latency (response time minus service time).
• always choose the lowest level of log in production mode