Your specification needs to include guidance on the interfaces to the rest of the system. Sometimes the “envelop” is wide open, other times it is very confined — such as when you are replacing an existing drive. I actually prefer the latter situation in that the fewer options you have the easier it is to focus on a solution.
Something as simple as a coupling selection can save you from designing yourself into a corner. Shaft stresses and keyway capacities are calculated using the same formulas — regardless of whether they are on the prime mover, gearbox, or driven equipment. Material grade and hardness can influence the connection size, but not as significantly as you might think.
We once were asked to replace an existing gearbox on a flattener drive because it was failing every three to six months. It did not take long to see why the drive was failing, as the flattener input shaft was SEVEN inches in diameter and the gearbox output shaft was only FOUR inches. The difference was so large that the mill had to special-order small-bore coupling halves to avoid delays when swapping out that damn gearbox.
Surprisingly, we were the first to point out that the gearbox was so obviously undersized.
Another instance that comes to mind had belt-driven input shafts where the diameter and orientation of the sheaves could not have been worse for bearing life. A simple change in gearbox ratio allowed the belt ratio to be lowered, and the existing product envelop maintained. A higher input speed more than compensated for the increased gearbox ratio.
Know that you have to be prepared to ask questions on the interfaces; e.g. — are they open to negotiation or carved in stone? Would a different motor speed reduce the number of stages needed in the gearbox? Would a different style of coupling ease assembly and alignment?
Sometimes designing the gears is the easy part of the project — except for the fact that those gears need to “fit” into a complete system. And your job as a designer requires you to venture “outside the box.”