The obvious best choice for lathe collets would be 5C but there pretty expensive to get fully set up for them and there not designed nor meant to work on any mill for tool holding although there were a very few mills that were made with a 5C spindle taper. And that collet system is still by far the most versatile method for small part holding. There's multiple other collet designs, but today about the cheapest next choice in a home shop would probably be ER collets. ER 25,32,or 40 but there's both smaller and larger series from ER 8 - ER 50. Yeah a whole lot slower to use than even a 5C lever operated set up. But a great deal cheaper since there holding range is just about .040" with the larger series so far fewer collets are needed. The trade off to using them on a mill is moving the tooling as much as 3" below those spindle bearings after measuring my ER-40 chuck. So yes you lose a bit of rigidity and Z axis height. For any R8 mill then likely the ER 25's would be a bit small for both the lathe and mill unless you do a lot of very small work and tool holding. The ER 32's are well suited for the general size of tooling any R8 spindle is rated for. I still chose the larger ER 40 on my R8 mill for it's dual usage and having up to 1" capacity on the lathe. Yes they make MT shanks with about any size of ER collet nose you decide on for a fast easy fit to the lathe. You again lose that spindle through hole. Plain back ER collet chucks https://littlemachineshop.com/products/ ... uctID=2532 as one example and would be my choice to regain that spindle through hole. You'd still need to machine your own back plate for it to fit your lathes spindle mounting. You can even make your own from scratch and save a few bucks.It all depends on what you want to accomplish and what your shops average needs are for which direction you should go and no one can really make that decision but yourself. If you have a fair amount of smaller lathe parts a ER collet set up can do a lot of high precision work that might save dialing it in on a 4 jaw. Some off shore ER collets and there chucks can be a real gamble for what you get though. Youtube has numerous examples of some pretty crappy results. Most of my 3 jaw lathe chucks and even some of my drill chucks can do better than some of what I've seen posted there. If a collet and it's chuck can't perform any better with less run out than the lathe scroll chucks you already have there's not much point in even using them. And high enough run out numbers are just as bad on the mill since it wears out expensive multi flute tooling even quicker.
How tight is tight enough? 25-35 ft. lbs on the drawbar for R8 collets is generally thought of as enough. Use a torque wrench a few times until you get a feel for how much that is. If a tool moves on you, add a bit more. I'd sure agree about keeping any collet clean and oil free. I do add a couple of drops of oil to the drawbar threads every so often though. ER's are a lot different for needing high torque, for the MAXIMUM size each collet series is capable of then the industry recommendations for the 25's is 77 ft. lbs, 32's 100 ft. lbs and the 40's is 130 ft. lbs. As the collet and tooling shank size reduces the required torque drops as well since the cutting loads are also decreased. Those are the numbers for tool holding, using them on a lathe then the torque numbers would be a whole lot lower. For solid carbide tooling on the mill then you'd probably want close to the maximum for torque at most sizes I think just because it's far tougher to get a proper grip on carbide than with HSS tooling. As Russ already mentioned and I've seen the same, any ER collet wrenches the chucks come with are never anywhere close to being long enough to apply those upper level torque numbers. The OEM wrenches seem designed to fit in the box with the chuck and not what's really needed to tighten the nut fully, so you either have to add a snipe to the end of the wrench or buy an after market torque wrench that fits the nut. And according to YouTube many off shore collet chucks don't come with any wrench flats on the chuck body. Make sure if your buying one that it does have them. My mill does have that spindle brake but it doesn't lock in the on position like a real BP so I just use two wrenches.
There's multiple other collet types and tool holding methods with some of it mostly designed for high production cnc use. In a home shop then R8 collets are about the cheapest "IF" you can find decent ones and probably what I'd start with. ER's might be more versatile on both a mill and lathe for round parts and tooling. But if money wasn't tight I'd be set up with Hardinge 5C collets and tooling for all my part holding.
There might be one more option that's worth a look. If I hadn't already bought as much R8 shanked tooling as I did before Tormach came out with there TTS system it's probably what I would have gone with even for my manual mill since it's already designed to work with R8 spindle tapers. It's a fairly decent quick change system that has the design innovation of pulling the arbors up tight against the spindle nose and gaining a bit of rigidity.