We configure haproxy with "verify optional", which makes haproxy request a client certificate, but accept the case where client does not present a certificate, but as described in 1, if client present a certificate and this certificate can not be verified, handshake is aborted. This is not what we want, we want to treat the case of a non verified certificate same as the case of the absence of certificate.
This configures haproxy accordingly, using "crt-ignore-err all" to allow handshake anyway.
Once this was fixed, there was a remaining problem with client_cert_verified acl, haproxy acl are OR, but this rule was supposed to be a AND (client present a certificate AND it is verified), this was rewritten to use inline condition which are AND.