Remarkable differences in the shape of the nematic-smectic-B interface in a quasi-two-dimensional geometry have been experimentally observed in three liquid crystals of very similar molecular structure, i.e., neighboring members of a homologous series. In the thermal equilibrium of the two mesophases a faceted rectanglelike shape was observed with considerably different shape anisotropies for the three homologs. Various morphologies such as dendritic, dendriticlike, and faceted shapes of the rapidly growing smectic-B germ were also observed for the three substances. Experimental results were compared with computer simulations based on the phase field model. The pattern forming behavior of a binary mixture of two homologs was also studied.