The short answer why it used to work with old versions, is that there was a change to hostapd sometime around version 21. Also that phone OSs are increasing their use of random WiFi MAC addresses for user privacy.
The change to hostapd is that when a MAC filter is active, the AP will no longer answer Probe Requests from a not allowed client MAC. Older versions did not consult the MAC filter, they would answer any Probe Request. Though it isn't mandated by the standard, mobile clients tend to always send a Probe Request and do not further attempt to connect unless they receive a specific Probe Response. This is how they know that the AP is in radio range and also exactly which channel it is operating on.
In a modern OS these Probe Requests are sent from a random MAC, often a new different MAC for every probe-- that may even be the case when the user setting for MAC privacy is turned off, which is of course not recommended. So with the new hostapd, a hidden/filtered AP will never answer random MAC probes, and connection is impossible.
When an AP runs with a hidden SSID, the client must include the SSID in the Probe Request in order for it to be answered. The SSID is transmitted in clear text, and also the answer from the AP contains the SSID in clear text. Any low-level hacker within wifi range can discover your SSID almost instantly. But that's not the worst part. When away from home, with wifi still switched on, your phone will periodically probe for your home network or other known networks. If it is not configured as a hidden SSID, these probes do not contain a SSID, and are from a random MAC address, so they are a low privacy concern.
If you have configured your home network with a hidden SSID, your phone will broadcast that SSID in periodic probe requests if you have left the phone wifi on while away from home. Additionally if you have turned MAC privacy off, the probes may contain your constant factory MAC address (the phone has to work that way to be able to connect to a post-v21 hidden and filtered AP). This is the main reason why hidden SSID is not recommended.