Have you actually separated the sections of the control valve yet? Or are you only extracting the spools and springs for cleaning?
There is a fine mesh strainer (Toyota part # 32804-23330-71) inside the control valve that may have a broken section of mesh which is allowing metal grit from the pump to enter the shuttles and bores of the valve. The control valve sections will have to be separated to get at the filter. This will also permit you to blow out all the very small holes in the separator plate inside the valve.
There may be some metal flakes stuck in some of the tiny holes.
If the shuttles/spools are sticking, grit has entered the valve body.
That is what it is beginning to look like from here.
What was the reason for the new oil pump?