Silverstripe for highly transactional app

Would Silverstripe be suitable for a highly transactional app? How many custom modules can you have before it starts to strain?

This question is very broad. In short, you can add endless modules until it starts to strain. If dependency injection is used properly, the amount of modules don’t matter much until they are actively used.
And with that in mind, your limit is the limit that can be executed by the server within a reasonable time.
PHP8 is fast enough to not be a big issue, with Silverstripe specifically, your strain would more likely be in excessive abstraction and loading of templates compared to the actual PHP code.
Consider that Barack Obama’s website for his presidential run was build on Silverstripe, you can imagine that it needs to be able to handle a lot of transactions before things start to crack.

I was asking because of no foreign keys and that all being managed by the ORM. Do people have any issues with orphaned records or data syncing?

Also, was this the website:

Silverstripe ORM uses foreign keys everywhere. I would say quite the opposite: in my queries there are too many foreign keys.

Whenever you feel the ORM is not doing a good job you can resort to raw queries. silverstripe-cms itsself does exactly that to get the last Sort value.

If you feel the ORM is getting in your way, you can also ditch it alltogether and write your own code using only view and controller of Silverstripe. I occasionally fetch data from weird sources and it just has to be formatted as ArrayList of ArrayData to be digestible by the rest of the code.