When you choose to turn the Xbox off by holding the button on your controller, or when you press the button on the console, you're really just putting it into sleep mode. In sleep mode, it'll continue downloads (I can't remember if it does by default, you might need to search for it in settings)
The only way to actually turn the box off is to unplug it, or to hold the button on the console for a few seconds, and you'll actually hear it power completely down.