How Task Queue dispatches each Task to multiple JVMs

I've been wondering how Task Queue of App Engine dispatches each Task to multiple JVMs (App Servers). So I just tested with my application hosted on App Engine, which uses Task Queue to brakedown a large query into tens of small queries.

I've added a code to write log of "Runtime.getRuntime().hashCode()" at the end of each task code. Here's the log I got:

[19/Oct/2009:12:10:50 +0900] [4040305]
[19/Oct/2009:12:10:52 +0900] [9749991]
[19/Oct/2009:12:10:53 +0900] [5956071]
[19/Oct/2009:12:10:53 +0900] [23518822]
[19/Oct/2009:12:10:54 +0900] [1512816]
[19/Oct/2009:12:10:54 +0900] [9241323]
[19/Oct/2009:12:10:54 +0900] [32885885]
[19/Oct/2009:12:10:54 +0900] [14891863]
[19/Oct/2009:12:10:54 +0900] [2168094]
[19/Oct/2009:12:10:55 +0900] [23469803]
[19/Oct/2009:12:10:55 +0900] [9460493]
[19/Oct/2009:12:10:55 +0900] [11819261]
[19/Oct/2009:12:10:55 +0900] [16723022]
[19/Oct/2009:12:10:55 +0900] [26422362]
[19/Oct/2009:12:10:56 +0900] [32885885]
[19/Oct/2009:12:10:57 +0900] [26422362]
[19/Oct/2009:12:10:58 +0900] [23469803]
[19/Oct/2009:12:10:58 +0900] [5956071]
[19/Oct/2009:12:10:58 +0900] [9749991]
[19/Oct/2009:12:10:59 +0900] [1512816]
[19/Oct/2009:12:11:00 +0900] [25885348]
[19/Oct/2009:12:11:00 +0900] [25281697]
[19/Oct/2009:12:11:01 +0900] [9241323]
[19/Oct/2009:12:11:05 +0900] [4040305]
[19/Oct/2009:12:11:06 +0900] [1512816]
[19/Oct/2009:12:11:06 +0900] [14891863]
[19/Oct/2009:12:11:07 +0900] [25885348]
[19/Oct/2009:12:11:07 +0900] [11819261]
[19/Oct/2009:12:11:10 +0900] [4040305]
[19/Oct/2009:12:11:11 +0900] [25281697]
[19/Oct/2009:12:11:12 +0900] [9241323]
[19/Oct/2009:12:11:15 +0900] [25885348]
[19/Oct/2009:12:11:17 +0900] [9749991]
[19/Oct/2009:12:11:17 +0900] [23469803]
[19/Oct/2009:12:11:20 +0900] [11819261]
[19/Oct/2009:12:11:23 +0900] [26422362]
[19/Oct/2009:12:11:25 +0900] [11819261]
[19/Oct/2009:12:11:30 +0900] [25281697]
[19/Oct/2009:12:11:30 +0900] [23469803]
[19/Oct/2009:12:11:32 +0900] [11819261]
[19/Oct/2009:12:11:33 +0900] [25885348]
[19/Oct/2009:12:11:36 +0900] [9460493]
[19/Oct/2009:12:11:37 +0900] [9241323]
[19/Oct/2009:12:11:40 +0900] [11819261]
[19/Oct/2009:12:11:42 +0900] [25281697]
[19/Oct/2009:12:11:46 +0900] [4040305]
[19/Oct/2009:12:11:49 +0900] [26422362]
[19/Oct/2009:12:11:51 +0900] [32885885]
[19/Oct/2009:12:11:54 +0900] [4040305]
[19/Oct/2009:12:11:56 +0900] [25885348]
[19/Oct/2009:12:12:00 +0900] [11819261]
[19/Oct/2009:12:12:02 +0900] [32885885]
[19/Oct/2009:12:12:06 +0900] [9460493]
[19/Oct/2009:12:12:08 +0900] [4040305]
[19/Oct/2009:12:12:12 +0900] [9241323]
[19/Oct/2009:12:12:13 +0900] [26422362]
[19/Oct/2009:12:12:19 +0900] [9460493]
[19/Oct/2009:12:12:24 +0900] [4040305]
[19/Oct/2009:12:12:24 +0900] [26422362]
[19/Oct/2009:12:12:28 +0900] [14891863]
[19/Oct/2009:12:12:30 +0900] [9749991]
[19/Oct/2009:12:12:33 +0900] [9241323]
[19/Oct/2009:12:12:35 +0900] [9460493]
[19/Oct/2009:12:12:37 +0900] [14891863]
[19/Oct/2009:12:12:39 +0900] [26422362]
[19/Oct/2009:12:12:44 +0900] [4040305]
[19/Oct/2009:12:12:51 +0900] [9241323]
[19/Oct/2009:12:12:56 +0900] [23469803]
[19/Oct/2009:12:12:56 +0900] [32885885]

And here's the summary of the counts for each JVM:

[11819261] 7
[14891863] 4
[1512816] 3
[16723022] 1
[2168094] 1
[23469803] 5
[23518822] 1
[25281697] 4
[25885348] 5
[26422362] 7
[32885885] 5
[4040305] 8
[5956071] 2
[9241323] 7
[9460493] 5
[9749991] 4
total 69 tasks

So, we can state that:

• There're 69 tasks dispatched to 16 JVMs without any concentration on a certain JVM.
• The tasks are processed at about 5 tasks/sec maximum (the setting is 9/s).

FYI, the tested environment has been used for my development work and it is possible that those JVMs had been launched already for the past workloads.

And I also noticed that the source IP addresses of each HTTP request for tasks are something like "0.1.0.1" or "0.1.0.2". It seems the request is originated from the same App Server node. I suppose this would be working like this: "A remote Task Queue service" -> "An agent-like component on the App Server" -> "the Web container".