Objects floating at a liquid interface, such as breakfast cereals floating in a bowl of milk or bubbles at the surface of a soft drink, clump together as a result of capillary attraction. This attraction arises from deformation of the liquid interface due to gravitational forces; these deformations cause excess surface area that can be reduced if the particles move closer together. For micrometer-sized colloids, however, the gravitational force is too small to produce significant interfacial deformations, so capillary forces between spherical colloids at a flat interface are negligible. Here, we show that this is different when the confining liquid interface has a finite curvature that is also anisotropic. In that case, the condition of constant contact angle along the three-phase contact line can only be satisfied when the interface is deformed. We present experiments and numerical calculations that demonstrate how this leads to quadrupolar capillary interactions between the particles, giving rise to organization into regular square lattices. We demonstrate that the strength of the governing anisotropic interactions can be rescaled with the deviatoric curvature alone, irrespective of the exact shape of the liquid interface. Our results suggest that anisotropic interactions can easily be induced between isotropic colloids through tailoring of the interfacial curvature.