Edible insects are expected to become an important nutrient source for animals and humans in the Western world in the near future. However, before insects can be put on the market, the safety of their use for feed and food is warranted. This literature study was prepared to provide an overview of the actual knowledge of possible food safety hazards, including chemical, microbiological, and allergenic agents and prions, to human and animal health upon the use of insects for food and feed, and to highlight data gaps and suggest the way forward. From the data available, heavy metals of concern are cadmium in black soldier fly and arsenic in yellow mealworm larvae. Investigated mycotoxins do not seem to accumulate. Residues of pesticides, veterinary drugs, and hormones, as well as dioxins and PCBs, are sometimes found in insects. Contamination of insects with pathogens to human health is a consequence of a combination of the substrates used and the farming and processing steps applied. Insects harbor a wide variety of microorganisms, and some human pathogenic bacteria may be present. In addition, insects may harbor and transmit parasites. There is no evidence so far insects may harbor pathogenic viruses or prions, but they may act as vectors. Insects and insect-derived products may have allergenic potential. In this review, evidence on some safety aspects is displayed, and data gaps are identified. Recommendations are given for future research to fill the most relevant data gaps.