First things first: MAKE SURE YOU HAVE ALL REQUIREMENTS FOR WHAT YOU ARE DESIGNING FIRST.
Designing without a clear scope of what you are going to do will lead to over engineered design, paralysis and sleeplessness.
If you are designing from scratch, start with requirements for a small prototype, don’t try to design a giant system from the start.
If you already have a working prototype from the previous iteration, keep refining the design, add on for new iteration. Keep iterating until a big system is completed. Again don’t design for a complex system.
Now that all previous points are addressed, I’ve found that it’s great to follow Newton’s approach to solving problems: break them into smaller problems and tackle them one by one.
break big problems to smaller ones, top to bottom, and tackle each one problems at a time, use multiple UML diagram for each problem.
Once you have a list of diagrams and see the interactions between classes, try to reduce couplings where possible, so that the number of dependencies is as small as possible.
If designing for extensibility, create interfaces to separate implementations so components can be swapped out. Be careful not to over design if you don’t need to, it’s better to design to meet requirements first, then add extensibility later on. Let the design evolve instead of trying to over design for future.
Keep iterating so the design evolves to meet all requirements.