Only a little more than a decade later, a whirlwind of unforeseen events blew in. In the mid-1800s, new fortune-seeking settlers poured in from the eastern U.S. and elsewhere because of the California Gold Rush and its prospects of statehood. Upon leaving the gold fields (after having struck it rich, though more commonly not), or even bypassing them entirely, new Milpitas townfolk settled on the western edge of the Rancho along what was referred to as The Mission Road (from El Pueblo San José Guadalupe to Mission San José); others put down roots in the more open eastern foothill rangelands. They typically grew private gardens of squash, corn, beans and other edible crops along or near the banks of the nearby Penitencia and Coyote Creeks. Penitencia Creek was the western boundary of Rancho Milpitas; Coyote Creek was the western boundary of Rincon de los Esteros.
Briefly Milpitas township was known as Penitencia by its locals until the first postmaster objected that the name was too reminiscent of “penitentiary,” which is quite ironic because in this era Milpitas is the site of Elwood Correctional Facility. During the 1870s immigrant John O’Toole owned a 584-acre property in southern Milpitas with Elmwood trees lining the lane leading from the Mission Road (aka the Old Oakland Highway) to his residence; it was later purchased by Santa Clara County to be used initially as a almshouse (a home for the poor), but it gradually transitioned into housing for low-security inmates presaging its current use.