another case of undocumented syntax

Yes, absolutely! Just forgot to add it to the list.

Actually I saw the syntax in GCARE too :).

Not sure if I get the meaning of the output - how re-ordering could fail? Probably will just return 1.

It is useless, of course, but it is a relic from another time ... :)

Prior, to make it working, I used:

[z(perm,:),t] = ordschur(z,t,'lhp'); Success = 1;

Actually it might indicate few things (in case of ordqz) - problem is too ill-conditioned and transformed matrices might be too far from generalized Schur or just indicate wrong inputs (mismatching matrix dims or else). In any case toolbox will just return 1 in case of extra output parameters.

I sent to you by email a piece of reference from Matlab's childhood

Thank you for reminding, I almost forget to add this to toolbox - will add right now.

Undocumented syntax has been implemented for both routines, ordqz and ordschur (in trunk).