Ha! I figured it out. Now I know the difference between wiring a circuit in parallel and wiring it in series (after spending a while online and doing many Google searches). Hopefully someone will read this some day and learn from my foibles. Now I have to tear everything apart and rewire the battery pack to the rest of the circuit. This time I will connect a positive and negative wire from each pack, and the other positive and negative will feed the rest of the circuit.

I'll post pictures of what I imagine will be a much brighter lightsaber after it gets the right amount of juice.