If you give up a food and you feel much better, that should tell you something important. Likewise, if you eat something new and it gives you trouble, that is important information too.
People who "can eat tons of rich food" and remain rail-thin are genetically unusual. We are designed to turn food into fuel and repair material for our bodies. When times were tough, people who couldn't put on weight were in big trouble. Those folks who could put on weight for the lean times survived, while the skinny folks often did not. It's just that nowadays, almost everything we do is automated, so people don't get as much exercise anymore, and it is much easier to gain weight.