We prove that $\mathrm{SU}(n)$ ($n \ge 3$) and $\mathrm{Sp}(n)U(1)$ ($n \ge 2$) are the only connected Lie groups acting transitively and effectively on some sphere which can be weak holonomy groups of a Riemannian manifold without having to contain its holonomy group. In both cases the manifold is Kähler.