time per token goes down up to a certain batch size and then flattens - however the times vary a lot so we don't want to get too close to the maximum of 50s.