Ok so here's what will happen.
If you lose while battling the legendary Pokemon, you will be kicked out of the den. The person standing outside (the one who you talk to if you want to start a Dynamax Adventure) will ask you if you want to take note of the legendary Pokemon. If you do, you can rebattle it at any time until you catch it, though you have to complete the Dynamax Adventure again.
If you don't encounter the legendary Pokemon during in the Max Lair before you lose, you'll have to start at the beginning and you can't take note of the legendary (since it wasn't revealed).
Hope this helps! :)