Previous and Next Portfolio Items: Limit to Category

Hi, I have a site with a dozen or so Portfolio Categories. Each Category has a page where I used Portfolio grid to show all images in that category. What’s going on, though, is the previous and next links are not limited to the category. So, when one portfolio item from Category A is then reassigned to Category B, and browsing through Category B, the items from A get mixed in when browsing via the previous and next arrows.

Is there a way to limit the previous and next arrows to stay only in that category? This would be super, if so.

Thanks for your help.