Click Settings on your home screen.
Once in settings, click General.
In General, you will see a tab for Siri. Click that tab.
Once in Siri, you will see a toggle for Allow "Hey Siri". Swipe the toggle button to off.
Once the toggle button is no longer green, Siri will no longer be activated by your voice.