Last modified : 19 September, 2017
Today I learned just a wee bit about LDAP. These are my hap-hazard notes more than an article for disseminating information. Whoever said “a little knowledge is a dangerous thing” probably had me in mind.
We have our Kerberos infrastructure set up to learn about identities (and consequently principals) from LDAP. This was all done by Travis Thompson who is awesome.
The LDAP server we chose to use was 389 . Apparently the source is available here although I did not delve deep enough to look at the source. The project was started at UMich, and through some convoluted changing of hands (Netscape, AOL, Sun, Oracle) landed up with RedHat. ActiveDirectory from Microsoft seems another popular choice for LDAP.
The way we have it set up, there is one Master and two Slaves. All writes (modifications) of the database go to the Master. These updates are then pushed to the Slaves by the Master. The slaves are capable of (and preferred for purposes of load balancing) to serve all read requests.
Allen Wittenauer who knows practically everything about Ops and security helped with this analysis along with Zoran Dimitrijevic. The problem in our case happened because the Slave nodes filled up their disks (logrotate and other factors combined….). Consequently, these updates known as “Replications” started to fail. These manifested themselves in Hadoop logs like so :
kinit: Client '<my_principal>' not found in Kerberos database while getting initial credentials
I was able to figure out what was wrong, by running kadmin.local on all three KDC servers (1 Master and 2 Slaves) and checking the existence of <my_principal> using listprincs. Although it would have been nice to get 389-console (which is a JAVA GUI that you are supposed to access via X forwarding, no joke), I was hesitant to install XServer on the LDAP Master. So I tried working through the console.
Here’s some documentation which I found useful :
- 389 Documentation for monitoring replication. The first command basically confirmed our thesis that replication had failed.
- Redhat Documentation that describes what to do in case you need to re-jigger the replication.
Essentially you have to setup ldif files on the Master, telling it to push updates to Slaves. This is done through a “replication agreement”. Although it can be accomplished (much more easily through the GUI), where’s the fun in that? ldapmodify and ldapsearch seem to be the tools of the trade and they have a gazillion options. dirsrv is the daemon that runs and starts a process called slapd. lsof on the slapd process ought to tell you about the log files (although failures in replications were somehow not in any of the access/error logs) . I think I jiggered the replication by blowing away the changesetdb and restarting the dirsrv daemon (although I am not sure if that was it ;-) because technically it worked only after I “unblowed” it back and restarted the daemon a second time.) Let me know if some other way worked for ya. Its safe to say, I’m not proud…..
Please add comments here:
All content on this website is licensed as Creative Commons-Attribution-ShareAlike 4.0 License. Opinions expressed are solely my own.