Resonant flow instability (RFI) and Kelvin-Helmholtz instability (KHI) are investigated as possible wave generating mechanisms in the mantle-like boundary layer of the Earth's magnetotail where all equilibrium quantities transition continuously from magnetosheath values to values more characteristic of the tail lobe. It is shown that as in the case of a sharp interface the KHI requiring high flow speeds in the magnetosheath is unlikely to be operative under typical conditions. RFI which is physically distinct from KHI may appear at lower flow speeds due to the inhomogeneity of the mantle-like boundary layer. It is shown that RFI can be important when the variation length-scale of the flow velocity is smaller than the variation length-scales of other equilibrium quantities such as density and magnetic field strength. Interpretation in terms of the wave energy flux is presented and the applicability to the magnetotail is discussed. The obtained results could explain the observed low power of ULF waves in the tail lobes compared with other parts of the magnetosphere.