According to my experience, the single directive you should add to db section to improve performance is:
olcDbCacheSize
OpenLDAP tuning guide covers that in section: 21.4.2.
To enable a 200000 entries cachesize use this ldif:
The enumeration of the whole directory (150000 entries) took 15 seconds without cachesize and less than 6 seconds with it.