To use your arms effectively, it helps to position your shoulder blades relative to your ribcage and it helps to give them a stable foundation once positioning has been achieved. It's akin to using a camera tripod. You loosen the head so that you can adjust positioning, then you tighten the head of the tripod once that position has been found.
In terms of positioning, that can mean moving your shoulder blades (and collar bones) relative to your ribcage. It can also involve shaping the ribcage. Once positioning is achieved, then it helps to stabilize your ribcage and your shoulder blades (and collarbones) so that your shoulder and arm muscles have an anchored end point (stability) from which to work.
You could also create stability from the hands, wrists and elbows. Then your arms can act as the anchor.
When looking at the shoulders and arms with a view to making them stronger and/or more flexible, a good place to start is with the ribcage. Why is the ribcage so important for shoulder and arm strength? To find out why it can help to understand why turtles have shells. That and more in Ribcage
For a series of youtube videos on feeling and controlling your ribcage, spine, and shoulders, and using your arms poses like chaturanga dandasana and dolphin pose check out the Upper body awareness and arm control exercise videos page
One thing that teach fairly regularly in class is shoulder blade awareness. I teach it because a lot of people don't have it. Learn some basic shoulder blade awareness in scapular awareness.
You can then work at applying scapular awareness while your arms are bearing weight as in these shoulder strengthening exercises for yoga.
The serratus anterior is one of a group of muscles called the scapular stabilizers. I'd say that a better name for the group is scapular controllers. These muscles are responsible for moving the scapulae, or shoulder blades, relative to the ribcage as well as stabilizing them. As mentioned, the serratus anterior is one of those muscles. And I've actually written two articles about it. The other is called serratus anterior muscle.
For arm over the head positions, as well as for lifting the shoulder blades, the trapezius muscle can work in concert with the serratus to optimally position the shoulder blades. Note that a problem with lifting the arms is that if you don't position your shoulder blades optimally, your arm bones can squeeze against the accromion process causing shoulder impingement.
So since the upper portion of the trapezius attaches near the accromion process, a good tip to prevent impingement is to pull upwards and inwards on the accromion process. Note that the lower traps can create a downards pull on the inner border of the shoulder blades, and this can be another aid to avoiding shoulder impingement.
The pectoralis minor is another muscle that helps to control the shoulder blade. if you have winged scapula the pec minor may be one possible culprit.
Read more about the pec minor and how you can use it when reaching the arms behind the back in pectoralis minor.
The effective use of our shoulder anatomy means looking at the shoulder blades, collarbones, ribcage and arms, and how they all interact. You can get a taste of that from the shoulder anatomy article. There's also a part 2: shoulder anatomy 2
For an overview of different ways to use your shoulder and stretch them, check out shoulder yoga.
The shoulder rotators include more than just the rotator cuff muscles. Rotator cuff muscles are muscles that are located close to the shoulder socket. There are other muscles that can help to rotate the arms. That being said, for a look at the rotator cuff, and how you can exercise it check out rotator cuff exercises.
For more exercises for the shoulder rotators check out shoulder rotation exercises.
These exercises can be used to exercise the shoulder rotators in general!
For a look at the muscles that make up the rotator cuff, plus other muscles that can rotate the upper arm bone relative to the shoulder blade, check out rotator cuff anatomy
Something important to realize with respect to the arm rotators is that because the arms have mass, the shoulder rotators can use the mass of the arm as a foundation and so act on the shoulder blade.
We generally consider the elbow as something that bends and straightens. While the elbow joint does bend and straighten, another movement that occurs quite close to it is rotation of the forearm. This is actually a movement of the radius relative to the ulna. Some of the muscles that cross the elbow joint can be used to rotate the forearm one way or another. So when talking about elbow joint stability, as with the knee, we can stabilize the elbow against bending and straightening. We can also stabilize it, or rather the forearm, against rotation.
For more on elbow joint anatomy and the mechanism of forearm rotation read elbow joint anatomy for yoga teachers.
For even more (there may be some duplication) check out arm stability.