Sensitive, high resolution observations of Galactic neutral hydrogen (HI) reveal an intricate network of slender linear features. Across the high Galactic latitude sky, this HI is aligned with the magnetic field as traced by both starlight polarization (Clark et al. 2014) and Planck 353 GHz polarized dust emission (Clark et al. 2015). The structure of the neutral interstellar medium is more tightly coupled to the magnetic field than previously known. At high Galactic latitudes, where the Planck data are noise-dominated, the HI data provide an independent constraint on the Galactic magnetic field orientation, and hence the local dust polarization angle. The HI data thus provide a new tool in the search for inflationary gravitational wave B-mode polarization in the cosmic microwave background, which is currently limited by dust foreground contamination. By using HI orientation as a Bayesian prior on the dust polarization angle, we can better constrain the properties of the polarized CMB foreground. This gives us a new mechanism for testing models of ISM-magnetic field interactions.