Gluteus maximus is a major extensor of the hip. It moves the leg backwards relative to the pelvis.
If the foot (or feet) are on the floor then it moves the pelvis forwards relative to the ground.
You could think of this as "back bending" the hip.
It can be helpful in backbending yoga poses as follows:
I'd suggest that this muscle be considered a multi-joint muscle because it has fibers that cross the SI Joint, attaching the Sacrum to the thigh, and it has fibers that attach from the pelvis to the side of the knee via the Iliotibial band.
As a yoga teacher and practitioner there is often a large influence to "relax" the gluteus maximus in poses like those mentioned above. Why? One suggested reason is because the gluteus maximus is an "external rotator" of the hip as well as an extensor, and this external rotation function can cause damage to the SI Joint.
Based on what I've learned from Michael Boyles "Advances in Functional Training" and also based on my own experience the gluteus maximus can be used in these poses along with the single joint extensors of the hip (gluteus medius, quadratus femoris, posterior fibers of adductor magnus). Not only that, properly training and using your gluteus maximus may help relieve low back pain and if you are an athlete then being able to use your gluteus maximus can help give you more leg power.
Rather than relaxing this muscle, you should be learning to activate it (and relax it) at will.
If you didn't already know, gluteus maximus is the technical name for the muscle that makes up the bulk of your butt cheeks. It has two main group of fibers.
The Iliotibial Band is the belt of connective tissue that runs down the side of the pelvis from the iliac crest down to the top of the lower leg bones (the tibia and fibula.)
The fibers of glute max that attach to the back of the thigh bone can be used to pull the thigh back relative to the pelvis (or push the pelvis forwards or up depending on whether you are standing, laying on your back or kneeling.)
The fibers that attach to the iliotibial band are the ones that can be used to externally rotate the thigh.
However, these same fibers are "balanced" by the fibers of the tensor fascae latae muscle while can be used to rotate the thigh internally (as well as pull the thigh forwards, "flexing" the hip.)
While fibers of the gluteus max reach forward and down to attach to the IT Band the fibers of the TFL (Tensor Fascae Latae) reach back and down from the front of the pelvis to attach to the IT band. You could thus think of the TFL and the Glute Max as forming a Y shape with the IT Band forming the "trunk" of the y when viewed from the side.
Along with the Gluteus medius (and perhaps even gluteus minimus) these muscles can be used to help stabilize the side-to-side (lateral) tilt of the pelvis when standing on one leg.
Where gluteus medius and minimus work directly on the hip joint, gluteus maximus and tensor fascae latae use the lower leg bones as an anchor to pull down on the top of the pelvis.
One trick you can use to help stabilize your hip joint when standing on one leg is to focus on "squeezing" your hip joint. One of my Tai Ji teachers was having trouble with his hip one day. I suggested this action to him and he told me it helped alot.
These two functions of the gluteus maximus can be separated so that if you choose you can activate it in such a way as to pull your thigh back or press your pelvis forwards without causing your thighs to externally rotate. (This is assuming the feet are on the floor!)
All you have to do is focus on keeping your thighs pointing straight ahead.
If you have difficulty doing this then you can practice rotating your thighs so that you get a feel for when they are parallel with knees straight ahead and when they are not.
While laying down on your back with knees straight, you can roll your thighs in and out while feeling your thighs. Then practice positioning your thighs and feeling when they are parallel with hips lifted (and feet flat on the floor). Next, as you push up into the pose, keep your thighs parallel and work at using gluteus maximus to push your pelvis upwards.
It's when I am laying on my back and using my glutes to push my pelvis up that I feel like I can isolate the hip extensor fibers of gluteus maximus. Or perhaps it is equally accurate to say that I can feel the external rotation causing fibers and choose not to activate them.
Both gluteus maximus and tensor fascia latae attach to the iliotibial band and together the two of them can be used to help stabilize the pelvis when balancing on one leg. In this instance the external rotation function of the gluteus maximus can be used but it is then balanced by the internally rotating action of the tensor fascae latae.
Another important point about gluteus maximus is that it oppose the psoas and so can be used to help stretch it.
A tight psoas can be released and stretched by using the gluteus maximus as in this hip flexor stretch.
Optionally, lift your free arm.
You could use the same action in poses like upward facing dog and upright pigeon.
As a yoga teacher, I'm constantly exploring new exercises, new ways of doing yoga poses.
There is no single "right way" of doing a yoga pose. Instead, there are options. And the better you are at "feeling" your body, the better you can get at choosing the right option for your body as it is now.
For any technique, the point of practice is to learn feel it and to control it, so that it can be used without thinking about how to use it.
And that is more or less the approach taken in all of my ebooks and videos. They help you to feel your body and control it so that you can work towards using it effectively in anything that you do.