Eep, you probably are not using a neutral rock--if you have them in with your betta I would remove them. Over time, they will raise the pH of that tank as well. Most rocks used in the aquarium are neutral and won't raise pH. Quartz, obsidian, jasper, jade, slate, granite, lava rock, petrified wood and most crystalline rocks are safe for the aquarium. Other rocks like pyrite (fool's gold), limestone, fossils, marble, sandstone, dolomite, corals and sea shells are not safe for the aquarium because they will raise your pH. Fluctuating pH is not healthy for your fish--and though the boiling almost certainly sped up the process of hardening your water, eventually without boiling, the rock is likely to leech minerals into the water.
I would remove it from the tank and get a neutral substrate from a store.