How big is "larger"? I'm preparing to make some ~3 inch holes..
About 4" is where I start paying attention. It also depends on what your are cutting, and the speed of the drill. At 3" 400rpm is about the max I would go, if your drill is a 600RPM unit, you may find it has a tendency to grab and try to spin out of your hands.
The other factor is the inertia of the drills motor and gearbox. Larger drills, or those with high RPM motors, (even with low speed gearbox attached) can have considerable energy stored up during operation. If the drill and saw combo takes more than a second to spin down when you let off the trigger, you need to pay attention, or attach a handle.
If you must go large without a handle, hold the drill so that the rotation is braced against your arm/forearm (non primary arm), or leg/body (not near the face). Do hold it loosely with your trigger hand, so if it catches, it will spin out of your hand, instead of yanking your wrist. I have on occasion held the drill "upside down" and used my small finger on the trigger. You have much less grip this way, so the drill will readily spin out of your hand, and not load your wrist if it catches. (Or just drill at a very slow speed!)
I have done some 3" and 4" holes with my dual speed lightweight cordless (no handle) and been okay. It has a compact motor, and ~350rpm slow speed which limits the kick when it catches.
I have a 1/2" chuck gear reduction hammer drill (right angle) that takes almost 3 seconds to spin down. Even with the large handle attached, it can seriously injure if the bit catches and you are positioned badly.