Summer Student Program ‘16 Work project report

3. Project proposal

4. Contribution

As can be seen on Figure 2, the project has been divided into three main stages:

• 
  The first stage consisted of parsing the AUTHZ file (which is used to manage the SVN repositories) to find out which repositories the E-groups and users have access to and what are their access rights to it;
  
• 
  The second stage consisted of cross referencing the previously extracted information from the SVN with the LDAP information in order to know to which E-group a user belongs to or which users form an E-group;
  
• 
  The third and last stage consisted of developing the web interface to permit the users to query the access rights.
  

1. Stage 1: Parsing

The E-groups forming the list are automatically filled in by their respective members three times a day. However, in my case, if a CERN member generates a query to know to which E-group a user belongs to, the response is retrieved from the LDAP server and not from the AUTHZ file.

In order to parse the AUTHZ file I used two of the many command line options provided by the Subversion User Authorization File Editor (SUAFE). This script, available on GitHub and developed by Shaun Johnson, allowed me to query the AUTHZ file to get the list of E-groups and the list of access rights rules.

Once I have extracted the list of E-groups from the SVN configuration file, I used the ldap3 Python library to query the CERN's LDAP server (reachable at xldap.cern.ch). I coded a Python program that requests the members of each E-groups appearing in the AUTHZ file.

2. Stage 2: Cross-referencing

Figure 2: Overview of the NoSQL database

Secondly, in order to cross-reference the extracted information coming from the SVN and the LDAP, I gather everything into a single NoSQL database using the MongoDB cross-platform. As the extracted data was coming from different sources and with different formats, I used a data pipeline (Logstash) that normalizes it into the JavaScript Object Notation (JSON) format. I made the choice of outputting the information into JSON since MongoDB uses JSON format to store records.

As can be seen on Figure 3, the database called “database” is made of two collections: one for the SUAFE data and the other one for the LDAP data. Each collection is then made of several unique documents, where one document has three distinct fields. However, I found it troublesome to filter the data from its source to JSON because Logstash presents some lack logging information. Anyway, I managed to obtain the desired result by completely personalizing the grok filter provided by Logstash.

3. Stage 3: WEB application development

4. Improvements

5. Evaluation

During the first stage I also had some problems with SUAFE as it was containing bugs. Its developer did not try every possibilities its script is offering, therefore I had to contact him to get the bugs fixed.

The second stage has been particularly challenging as well because I had to learn how to master Logstash. The output in JSON was not in the format I was expected, thus, I needed to filter the input files to eliminate some fields by personalizing the grok filter. This filter is using regular expressions, which took me time to accommodate with.

Finally, I could not be able to properly finish the WEB application. I have some doubt about the architecture of my solution as it might not be RESTful.

6. Conclusion