Here’s a passage from Ray Allen Billington’s The Far Western Frontier: 1830–1860:
American support for annexation [of Texas] cannot be simply explained. To state that the South favored and the North opposed is to rely on a generality that does little justice to the complexity of human motivation. That a majority of Southerners wanted Texas is incontestably clear; that they wanted to increase the slave territory within the United States does not necessarily follow. Those who expressed themselves were more concerned with fear of England than with the expansion of slavery, while a sizable segment opposed adding Texas to the Union. Some among these felt that Texans would enjoy a “bright destiny” if they could remain free of meddling northern abolitionists and high-tariff advocates; others sought to avoid the sectional conflicts that loomed if more slave states were acquired, or disliked taking on new lands that would drain population westward. Similarly, a significant portion of Northerners violated the sectional pattern to favor annexation. These included speculators who had invested in Texan securities or lands and stood to gain fortunes when the United States assumed the obligations of the poorly managed Republic. Jay Cooke, a leading financier, believed that the “selfish exertion in their own interests” of this small but powerful group was responsible for the passage of the joint resolution. Other Northerners welcomed Texas as a new area for commercial exploitation, or because the spirit of manifest destiny so dictated, or as a means of glorifying the national honor. Whatever the reasons, the American people had decided. The new spirit of expansionism had borne its first fruit.
Slavery was one issue, but it was far from the only issue; powerful financial interests exerting a disproportionately large influence is not only a modern problem. Belief in the superiority of “Anglo-Saxon” ways was obviously a part of the national conversation at the time, but this alone is not sufficient (or even necessary) to explain enthusiasm for expansion.
As Robin Hanson might say, nuance is near, caricature is far.