Load Test First Numbers

[Nate Klingenstein]

So, after a brief 2 day detour because of _eventId_proceed and a few other small mistakes I made in the environment, I've got everything set up and I'm able to celebrate the 4th of July by blowing up VM's since fireworks are banned due to drought here.

Specs:

2 IdP nodes, easily expanded to however many
4 The Grinder nodes, easily expanded to however many
1 Big Gulp LDAP server with 511111 users because 500000 wasn't enough over 9000.  Not easily expanded because RedHat 389 replication agreements are indecipherable to me, but since it's dedicated to the IdP's, this should never be the bottleneck in any realistic scenario.  I'll watch it.

Changed all Grinder nodes to m3.mediums, the LDAP to c3.xlarge, and the IdP's to c3.large.  I was able to do this without connecting to any of the nodes, yay.

It's definitely pegging the CPU on the IdP's.  I have a hard number of about 45 logins per node per second on c3.large hardware with no tuning(beyond running from jetty-base and whatever was done to the AMI distro).  Adding a second node doubled the TPS so I'm sure that's the bottleneck.  It appears to be threading properly.

And c3.large isn't actually that large...

Model	vCPU	Mem (GiB)	SSD Storage  (GB)
c3.large	2	3.75	2 x 16

1 of the IdP nodes:

top - 02:32:46 up 15 min,  1 user,  load average: 9.16, 5.66, 3.07
Tasks:  69 total,   1 running,  68 sleeping,   0 stopped,   0 zombie
Cpu0  : 92.7%us,  6.0%sy,  0.0%ni,  1.0%id,  0.0%wa,  0.3%hi,  0.0%si,  0.0%st
Cpu1  : 89.7%us,  7.3%sy,  0.0%ni,  0.7%id,  0.0%wa,  0.7%hi,  1.7%si,  0.0%st
Mem:   3842900k total,   893964k used,  2948936k free,    10928k buffers
Swap:        0k total,        0k used,        0k free,   191196k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                  
11696 root      20   0 2466m 586m  15m S 195.3 15.6   7:23.98 java                                                                     

LDAP not breaking a sweat after I gave it a ton of hardware:

Tasks: 127 total,   2 running, 125 sleeping,   0 stopped,   0 zombie
Cpu0  :  9.7%us,  4.1%sy,  0.0%ni, 76.2%id,  8.3%wa,  0.0%hi,  1.7%si,  0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   7513652k total,   663680k used,  6849972k free,    17324k buffers
Swap:        0k total,        0k used,        0k free,   456316k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                  
 1732 ldap      20   0 2611m  50m  16m R 16.3  0.7   0:38.94 ns-slapd                                                                 

35 random errors happened in about 30000 tests.  These errors were roughly evenly distributed both across nodes and throughout time.  The error rate corresponds to the TPS rate and is about 1 in 1000.

Interestingly, it took a -long- time accelerating.  It went from 20 to 45 logins per node per second over a period of several minutes.

So... maybe about 25-30% slower than the v2 IdP at the end of the day.  All nodes were freshly started and conformed to the spec's in the first mail.

I then took the IdP's up to c3.4xlarge, which is 16 cores per node.  And I found I will need vastly larger Grinders for that.  So, I just stopped the whole thing and I'll pick it up again shortly.