The Started property tells whether or not there is a physical transaction allocated. A physical transaction must be started just to look at data via a query. This is started automatically for you by IBO whenever necessary. Under certain circumstances IBO will also end the physical transaction even though you may not have requested it to do so. This is a part of the OAT (Oldest Active Transaction) handling which will be covered in detail later on.
The TransactionIsActive property tells whether or not there has been a change posted or if a change is pending for the transaction component. In logical terms a transaction is started once a dataset actually enters a state where a condition of change exists. If that change is cancelled and no other changes have been posted then the logical transaction is no longer active or "started". Once a change is posted in a transaction then it is active until that change is either committed or rolled back.
The InTransaction property tells whether or not you have called the StartTransaction method in order to start an explicit transaction. Starting an explicit transaction will suspend AutoCommit behavior if that has been set to true. AutoCommit behavior resumes after the explicit transaction is ended. This happens when Commit, CommitRetaining, Rollback, RollbackRetaining, Refresh(), Close, etc. has been called causing a commit or rollback to take place. SavePoint does not reset the AutoCommit behavior as it isn't considered an end to the transaction even though it does make all changes to that point permanent.
So, to answer this question, it depends on what transaction aspect you want to know about. If you need to know if there is a physical transaction in progress use the Started property, for a logical transaction check the TransactionIsActive property and for an explicit transaction check the InTransaction property.