Yes, it does matter... to an extent. If speaker wire permits it... move the boxes around and find out! move them around to you find the optimal position and you are happy.