Probably, yes. Without worms, beetles and other animal detrivores it would be difficult, if not impossible to maintain soil condition to grow crops. In addition, hand pollination is laborious and unfeasible on large scales. Wild forests and grasslands would die because they are adapted to rely on animal decomposers as well as pollinators and seed dispersers. This would cause abrupt loss of rainfall, atmospheric change and climate change. Widespread starvation combined with lack of decomposition would cause rampant disease. States would collapse into warring tribes with no capacity to develop the technology to counter these issues.
That is, assuming the extinction is sudden. If it happened slowly enough, and people were aware that it was happening and prepared for it, we could perhaps manage to preserve some pockets of humanity. But, given our apathy about the fact that we're already causing a mass extinction that's expected to wipe out 30-50% of species in the next century, I doubt we would even try.
More questions on Quora:
- Extinction: Why should humans care about other animals becoming extinct?
- Nature: What is poaching and what can be done about it?
- Evolutionary Biology: How does evolution impact biodiversity?