So long as you keep them electrically isolated during the charging process, there shouldn't be an issue.

It would be simpler to make the battery pack easily removable, and just carry a spare. But if you're up to the challenge of adding the extra wiring, recharge port, etc., then by all means give it a shot.