A hole in the drywall is usually considered cosmetic. Some inspectors will make a note about it for FYI but it does not affect the structural integrity of a home. The exception to that is if it is allowing moisture or insects to cause further damage to the interior or exterior.
But it can also be an indication of how well a home is maintaned. A random hole that has been untouched can be suspect for further typical preventive measures not performed by the homeowner and can certainly be a negative for a potential homebuyer.
If it is in a lower area (below your knees) in a room adjoining a bathroom, it can also be a treatment “trap” hole left by a Pest Control Company. It is used to inject chemicals into an area needing treatment. Usually it is covered by a vent, not left uncovered.
When in doubt, better to be repaired than left as is. It shows pride in ownership to a potential buyer when considering your house for purchase.