Herein, it is aimed to correlate the crystallographic structure, crack propagation, and toughness scatter with the prior austenite grain size (PAGS) in a low-carbon low-alloy (LCLA) steel from the perspective of crystallography. Two average austenite grain sizes, 16.1 ± 13.4 μm and 30.1 ± 24.9 μm, are obtained by reheating the experimental steel to 950 °C (S950) and 1100 °C (S1100), respectively. Overall, the weaker variant selection and higher density of high-angle grain boundaries (HAGBs) are observed in the as-quenched microstructure of S1100. However, coarser blocks and Bain groups (BPs) also appear in S1100 due to some oversized PAGs, which cause a more uneven microstructure in S1100 than that in S950. The ductile-to-brittle transition temperature (DBTT) is estimated to be −60 and −88 °C for S1100 and S950, respectively. In addition, a larger scatter of impact toughness occurs in S1100 in the DBTT range from −35 to −80 °C. The inhomogeneous microstructure inherited from oversized PAGs explains higher DBTT and large scatter in impact toughness of S1100. Therefore, to improve low-temperature toughness, it is more important to balance the blocks and BPs’ size refinement and uniformity rather than only consider the fraction of HAGBs in quenched LALC steel.