The contraction of tensor networks is a central task in the application of tensor network methods to the study of quantum and classical many-body systems. In this talk, I will discuss the impact of gauge degrees of freedom in the virtual indices of the tensor network on the contraction process, specifically focusing on boundary matrix product state methods for contracting two-dimensional tensor networks. We show that the gauge transformation can affect the entanglement structures of the eigenstates of the transfer matrix and change how the physical information is encoded in the eigenstates, which can influence the accuracy of the numerical simulation. We illustrate this effect through a systematic analysis of local gauge transformations. Additionally, we go beyond the scope of local gauge transformations and analyze an example that incorporates non-local gauge transformations.