Why did the Soviet union get a pass on invading Poland?

If the Soviet Union was at least an equal partner with Germany in the invasion of Poland, as well as Soviets conquering the Baltic States on their own, why did allies not declare war on them as well? I understand the enemy of my enemy is my friend but that didn't come into play until 1941when Germany then invaded them.