Adding navigation per category for Portfolio Entires

When turning on blog navigation for Portfolio Entries (blog navigation), the left and right arrows that appear aren’t not cycling through only the entries of a particular category. Rather, they are cycling through ALL the Portfolio Entries. When navigating to a single entry from a category page, or a portfolio grid (set to only show entries of a specific category) how do I limit the navigation to only posts of that category.

Is this a setting that can be achieve in a ALB custom layout?

Thank you in advance for any guidance.