On the nanoscale, CoO can have different polymorph crystal structures, zinc blende and wurtzite, apart from rock salt, which is the stable one in bulk. However, the magnetic structures of the zinc blende and wurtzite phases remain virtually unexplored. Here we discuss some of the main parameters controlling the growth of the CoO wurtzite and zinc blende polymorphs by thermal decomposition of cobalt (II) acetylacetonate. In addition, we present a detailed neutron diffraction study of oxygen deficient CoO (CoO) nanoparticles with zinc blende (∼15 nm) and wurtzite (∼30 nm) crystal structures to unravel their magnetic order and its temperature evolution. The magnetic order of the zinc blende nanoparticles is antiferromagnetic and appears at the Néel temperature T ∼ 203 K. It corresponds to the 3rd type of magnetic ordering in a face-centered cubic lattice with magnetic moments aligned along a cube edge. The magnetic structure in the wurtzite nanoparticles turned out to be rather complex with two perpendicular components. One component is incommensurate, of the longitudinal spin wave type, with the magnetic moments confined in the ab-plane. In the perpendicular direction, this magnetic order is uncorrelated, forming quasi-two-dimensional magnetic layers. The component of the magnetic moment, aligned along the hexagonal axis, is commensurate and corresponds to the antiferromagnetic order known as the 2nd type in a wurtzite structure. The Néel temperature of wurtzite phase is estimated to be ∼109 K. The temperature dependence of the magnetic reflections confirms the reduced dimensionality of the incommensurate magnetic order. Incommensurate magnetic structures in nanoparticles are an unusual phenomenon and in the case of wurtzite CoO it is probably caused by structural defects (e.g., vacancies, strains and stacking faults).