We tend to think of the knee as a simple hinge joint. As a result it's easy to suggest that all the knee does is bend and straighten. But when the knee is bent it also allows the shin to rotate relative to the thigh. And so when stabilizing the knee it helps to think of stability both in terms of stabilizing or resisting changes in knee bend as well as changes in shin rotation.
The main muscles that act to straighten the knee are the vastus muscles. These are the vastus medials, vastus intermedius and vastus lateralis. These muscles make up most of the bulk at the front of the thigh. In addition to straightening the knee these muscles also resist the knee being bent.
Because these muscles are all relatively massive, when they are active they tend to press out against overlying muscles.
And one other possible function of these muscles is that they help to remove slack from overlying muscles like the IT Band muscles (the tensor fascia latae and gluteus maximus) as well as the sartorius and gracilis both of which attach to tibia as part of the pes anserinus.
Muscles that can be used to bend the knee or resist is being straightened are the hamstring muscles. This includes the biceps femoris (both long head and short head) which attaches to the fibula, and the semimembranous and semitendinosus, both of which attach to the inside of the tibia.
The semitendinosus is the third muscle that forms the pes anserinus.
Another muscle that can help bend the knee or resist it being straightened is the gastrocnemius, the larger of the two calf muscles. The two upper tendons of this muscle pass between the tendons of the inner and outer hamstrings before attaching the the bottom of the femur.
To stabilize the knee against hinging the vastus muscles can be used against the calf muscles, particularly while standing. With weight pressing through the feet, the calf muscles are anchored at the heel. The quadriceps can then work against the gastrocnemius to stabilize the knee joint.
If the pelvis is stable then the hamstrings have a stable foundation and it may be possible to activate the hamstrings against the quadriceps to stabilize the knee joint against bending or straightening.
Either of these muscle pairs (vastus against gastrocnemius or vastus against hamstrings) could be used with the knee straight, or bent in various positions.
In terms of shin rotation, the IT band passes over the vastus lateralis to attach to the top of the tibia in front of the fibula.
Along the inside of the thigh the sartorius can act to rotate the shin outwards relative to the thigh when the knee is straight.
If the knee is bent then the sartorius can act, along with the gracilis and semitendinosus to rotate the shin inwards relative to the thigh.
If the knee is bent, the pes anserinus muscles can work against the IT band muscles to stabilize the shin against rotation.
If the knee is straight then the tensor fascia latae can work against the gluteus maximus to prevent rotation.
Another possibility with the knee straight is that the sartorius could work against either the gracilis or the semitendinosus to prevent shin rotation.
Part of controlling the knee can involve stabilizing either the femur or the lower leg bones.
To stabilize the femur against rotation two important muscles are the adductor magnus long head and gluteus minimus. Both can cause internal rotation or at the very least resist external rotation of the thigh relative to the pelvis.
If the femur is stabilized against rotation at the hip joint, the hip bone can then act as an anchor for the shin rotation muscles to effectively control the lower leg bones.
By the same token, if the shin is stabilized against rotation via muscles of the foot and ankle, then the shin rotation muscles have an anchor from which to work on the hip bone. Muscles that attach between the lower leg and femur also have an anchor from which to act on the femur.
Since the shin rotations attach to the hip bone, another way to make the knee joints more controllable (and/or more stable) is to make anchor the hip bones via the abs. For example, creating an upward pull on the ASICs via the external obliques can give sartorius, rectus femoris and tensor fascia latae an anchor point from which to effectively control the lower leg bones.
Exercises to develop muscle control for all of the above muscles (except the gastrocnemius) are included in Sensational Leg Anatomy. It includes videos of simple exercises for learning to feel and control the muscles of your legs from your hip bones down to your toes.