A few thing to keep in mind when starting out would be to work within the Nuke B-Line meaning to keep the main flow of the comp with the B side of the merge operation. A quick and dirty test you can do for this is if you select all your nodes in your comp and disable them it should read back to the source (plate) and not some random roto node.
Bounding boxes (bbox) should also be kept to the smallest amount required. Be sure to set the "set bbox to" operation in your merge node to whichever results in the smallest bbox required to output. Having ridiculously big bounding boxes will make the computer calculate unnecessary pixels void of information.
Although mask and stencil operations are great for 2D compositing sometimes they don't give you the result you want when working with multichannel source plates. To get the equivalent result in your multichannels you can use the in and out operations respectively. You would also have to switch the A and B inputs which breaks the first rule I said but you gotta know the rules first to break 'em! Don't forget to check your bounding boxes!
Good luck with your VFX journey!