Nevermind, I was changing the Products Search Results CMS settings instead of the Full Site Search settings so, turns out that's why I wasn't seeing changes when I added blocks
What's the difference between the Products Search Results and the Full Site Search anyways?