IMHO, the 3.7 would be the most reliable in our Nitro's. The 3.7 V6 is basically the 4.7 V8, less 2 cylinders. Therefore, it is a truck engine, designed for a Truck. The Nitro is a truck, based on the Liberty. The 3.7 has a chain timing belt, and as far as I can tell, as with the 4.7 in my Jeep Commander, there is no mention of maintenance for the timing belt.
I am not familiar with the Diesel engines, as they are not available in the US. The one available presently in the Nitro as an Export model was previously available in the first generation Jeep Liberty (Cherokee) and was very reliable and a good Italian engine for this truck. But, since diesel engines are not popular in the US, it was discontinued.
The 4.0 is a car engine built for fwd cars, like the minivan and Pacifica. It is a stroked version of the 3.5. Both have a rubber timing belt which has to be changed somewhere around 100,000 miles. Being a car engine is probably why it is not offered in the Liberty - not a rugged enough engine for the Jeep Brand.
The 3.7 is a SOHC engine with 12 valves. The 4.0 is a SOHC engine with 24 valves. Usually, multivalve engines are DOHC, one cam for intakes and one cam for exhaust valves, so the cams on the 4.0 are very busy!
If I had the choice, I'd take the 3.7 with transmission from the 4.0, but this combination is not available. I would like the extra gear to tweak my mileage, but from what I understand, the 4.0 is not rated any better than the 3.7 as far as gas mileage goes.