Thanks for the ping @Qui-Gon Glenn.
You want the honest to goodness truth, the answer is just a lot of trial-and-error. Before I reached my first stable build release I had to play the games through, shifting order and switching mods out, 4-5 times before I could even complete one. That was back before I was more familiar with what I was doing, but the moral of the story, if you will, is still the same--depending on what you're trying to get to work together, it might just come down to a lot of tinkering.
That said, JC is absolutely correct that incompatibilities are pretty rare. The most common type you'll have are obvious straight file overwrites, which are disproportionately texture or model changes. In those cases it's as simple as installing what you want last. More rarely you have file overwrites that involve more important files, which you can sometimes get away with overwriting if you know that the file from one mod is just minor changes (a rebalance of values in spells.2da, let's say) while another makes additions. Obviously in that case you keep the one that makes the additions, if you're dead-set on both mods, and either manually re-implement the balance changes or just eat the loss. Properly major file overwrites, like two loose appearance.2da files, can still be made to work if you've got the know-how and patience to merge the data from both files. Otherwise it's a matter of deciding which mod you prefer.
TSLPatcher mods are more stable, but whether you'll encounter stability issues as a result of them is often less obvious, as they tend to do more (just by the nature of what mods benefit from using the TSLPatcher) and warnings the patcher spits out can sometimes be intentional, or benign even when unintentional. You don't always know which is which, and even if you go through everything with no errors thrown at all, you can still get a conflict at the end of the day. There's nothing you can do for that, though, except try to structure your install order logically and be smart about what mods you want to integrate. The more you add, and the more that anything you add edits, the more risk you inherently run.
Determining a provisional load order is where you're going to be doing most of your legwork, and it's the step that will most directly impact the stability of your final build. This is probably the one thing I might be able to give some useful advice on. For the builds, the install order favors installing loose-file mods first, then larger TSLPatcher mods descending down to those with the most minor changes. The order of importance thus goes from the most trivial loose-file mods to the most important loose-files, then shifts to the most important TSLPatcher/.exe content down to the most trivial. In addition, mods of similar styles (texture replacers, model edits, mechanics changes, etc.) and those with similar thematic foci (Dantooine, let's say) are also grouped somewhat closely together. I've found that this is helpful in determining where you're likely to have conflicts, because by starting with the smallest mods first you'll have less stuff to go through in your override when/if you have a direct file conflict so you can easily determine what mods you're going to be forced to select between, and by grouping themes you're more likely to run into those potential conflicts back-to-back. From the TSLPatcher side of things, installing the largest mods first lets you know right away whether or not your most important mods are throwing any errors in their install phase, and also allows for mods with more specific changes (which are generally smaller) to overwrite the changes made by your larger, broad-stroke mods.
Your final functional order might not be similar to that, but installing a test run in this way is the best way I've found to quickly diagnose problem areas or potential issues in what you intend to install, which will let you fine-tune what you're using and when you're installing it to get better results. Once you've cleaned up the obvious issues, though, there's nothing for it but to do a run and make sure everything works. Sometimes it won't--most times it doesn't, for me. A little over half the time I have to restart or modify a test partway through due to unforeseen issues that didn't crop up during the install phase. If you're bound and determined to use a bunch of mods, that's just how it goes.