Chad, the hose that leads to the plastic baffle box is just a vent that allows the secondary chamber diaphragm to move inward whenever the throttle is opened allowing intake manifold vacuum to act against that secondary diaphragm.
When the intake vacuum pulls against the diaphragm, the diaphragm presses on the secondary valve lever which permits fuel to be drawn to the engine through the large rubber fuel hose to the carb.
The fact that you can notice a difference in how the engine runs when you remove the venting hose from the picture probably indicates there might be a defect in the secondary diaphragm.
Let me make sure which hose you are actually covering with your finger........you are referring to the hose that connectes to a 90 degree nipple on the bottom of the 4 screw secondary chamber cover, is this correct?
And when you cover something with your finger.....are you covering the rubber hose end or are you covering the 90 degree nipple opening?