I have a larger home, roughly 2 floors, with an unfinished basement and a large, attached garage. I currently have one functioning wifi router providing internet to many IOT devices, several being cameras. The router is also providing internet to mobile devices and computers. Some of these devices are almost outside the range of the router and frequently drop off the network.
What is the best way to extend the network so as any device I put around or in my home can connect without any issues. I am assuming the best approach would be to create a mesh network, but will the devices connect to the strongest SSID or will they hop back and forth? Also, am I able to configure fast transition for devices that move around the home such as mobile devices, laptops and the likes as long as they are on a separate SSID segregated from the IOT devices?
Mesh as a term is often used incorrectly due to how it is marketed with consumer products. But that said, a mesh network uses a wireless backhaul to connect multiple access points. That is to say that one will be wired into the network, and the others will be wirelessly connected back to that main AP.
If you have the opportunity to run a wire between your main router and secondary APs, that is always the best method in terms of simplicity and performance.
On the other hand, if wiring is not an option, you will want to put out 'repeaters' in some form -- the use of mesh (802.11s) vs WDS vs relayd as methods of repeating the network will entirely depend on the devices you are using (they really all need to be running OpenWrt to use mesh or WDS).
This comes down to properly tuning your APs -- the radio's power levels (almost always reducing the power) and channels (non-overlapping), and the physical position of the APs (to the extent possible) should be optimized to help your client devices make intelligent roaming decisions. Most modern client devices are pretty good at roaming properly when provided with a reasonably tuned and optimized environment.
You're talking about 802.11r -- I would recommend not starting here (and also avoid the k and v standards, too) unless you actually have a demonstrable need to use these after properly tuning the APs around your home. There are many client devices that do not perform properly when these additional standards are employed. Even if you do go this route, you must start with a good foundation of a well optimized radio configuration.
You can setup multiple SSIDs and segregated networks for your untrusted devices (guests/iot/etc). But for this to work, you need to have devices that can support multiple networks. OpenWrt does this easily; most consumer grade equipment doesn't support it with the vendor firmware; business and enterprise kit always does. It's much easier to use multiple networks when you are working with a wired backhaul. Wirelessly linking these multiple networks is a bit more challenging. So again, if you can run wires, that's the place to start.
I suggest 100% wired AP's if possible too. The compromise is mechanical complexity vs configuration complexity and reduced performance....
We can speak generically, but I guess the important bit is how fast does this system need to be in the first place? We talking 50mbit, 200mbit, 500mbit+ internet?
Regarding cabling:
Depending on your building you may have difficulties running physical cabling...
If you can't do it in dedicated conduit.... my opinion is that there's usually options available including but not limited to: under carpet trunking. Over carpet velcro covers, hard covers, square trunking on walls, double sided tape / cable clips, running fibre through preexisting power conduit etc. Depends on aesthetics vs performance. An "unfinished basement" probably means you can run conduit to where it needs to go yes?
More stuff to talk about with an electrician / telecommunications cabler etc in your area if you're not aware of the rules though =P