I use a “normal” basic authentication (htaccess) on the wp-admin folder.
I whitelisted the admin-ajax.php and now it works :)

Code:

AuthType Basic
AuthName "Auth Required"
AuthUserFile /path/to/.htpasswd
Require valid-user
<Files admin-ajax.php>
  Order allow,deny
  Allow from all
  Satisfy any
</Files>

Thank you for the food for thought!