Ldap Synchronization Connector

I will try to write all the pros & cons of all files type, please update as you see fit !

XML :¶

+ Very easy to read/write for a computer
+ With CDATA we can write multiple lines JS
- Very hard to read/write for a human

YAML :¶

+ Easy to read/write for a human/computer
- Can not write multiple lines JS

Properties :¶

Same as YAML

NeXt/ Open Step configuration files¶

See : http://commons.apache.org/configuration/apidocs/org/apache/commons/configuration/plist/PropertyListConfiguration.html

Same as YAML

Rémy-Christophe Schermesser wrote:

I will try to write all the pros & cons of all files type, please update as you see fit !

Great start. You're right, the first step to this is changing the format.

YAML :¶

+ Easy to read/write for a human/computer - Can not write multiple lines JS

This is not so, you can store multiple lines in YAML, either with a folding syntax (line breaks are replaced by spaces) or as literals (line breaks are preserved). See http://www.yaml.org/spec/1.2/spec.html#id2760844.

So without this, YAML doesn't seem to have any cons :)

Another criterion to think about: can configuration files include/reference other configuration files?

Thanks Jon.

I think another creterion is : can we reference another property ? like the ${my.property} syntax from the .properties files

I update my first comment

Hum ... I can not change my first comment, too bad :(

I will so update the bug notes

What about JSON ? I have heard from others that it is more wide-spread than YAML ?

The problem with JSON, si that a String must be enclosed with quotes.

So I don't think it is a good idea to have this extract chars ine the config (and it is error prone).

I have found a nice Java for YAML : http://code.google.com/p/snakeyaml

It has a nice feature of loading yaml into custom java classes. This way the issue #102 will be also resolved.

Hi,

I have tried to make a tree in yaml for the new configuration. I took all the configuration pages I found, and put an example in my yaml file.

So what do you think of it ?

I changed the some things, to be more flexible and support multi-input and output : check it here : http://pylonshq.com/pasties/9ce57432be91cd02ac332c4d1e2b8e8f

Or here, but without syntax coloration :

connections:    #Defintion of the connections  
    ldap:   #name of this source
        type: ldap  #type of the source
        configuration:   #ldap specific configuration
            derefAliases: never #optionnal, JNDI specific => remove ?
            factory: com.sun.jndi.ldap.LdapCtxFactory #java.naming.factory.initial, optionnal, JNDI specific => remove ?
            version: 3  #optionnal
            referral: ignore         #java.naming.referral, optionnal
            authentification: simple #java.naming.security.authentication, optionnal
            pageSize: 10
            tls: true
        url: ldap://localhost:1390/dc=AD,dc=net
        username: cn=manager,dc=AD,dc=net #java.naming.security.principal
        password: secret    #java.naming.security.credentials

    database:
        type: database
        url: jdbc:hsqldb:file:hsqldb/lsc
        username: elilly
        password:
        configuration:
            driver: org.hsqldb.blabla

    #may be a part of this is the configuration of a CSV output, maybe not                 
    csv: 
        file: /tmp/log.csv
        operations: create, delete
        attrbiutes: cn;dn #was attrs
        separator: ;    #optionnal
        append: true    #optionnal

    ldif:
        file: /tmp/log.csv
        operations: create, delete
        append: false   #optionnal

tasks:  #definition of the tasks              
    task1:  #name of the task
        sources:    #configuration of the sources
            ldap:   #name of the connection
                service: org.lsc.jndi.SimpleJndiSrcService
                base dn: ou=group
                pivot attributes: dn, cn #old pivotAttrs
                fetched attributes: uid, mail, cn   #old attributes
                dn: >
                     'cn=' + srcBean.getAttributeValueById('uid') 
                filters:
                    global: (&(objectClass=inetOrgPerson)(uid=*))   #old filterAll           
                    identity: (&(objectClass=inetOrgPerson)(uid={uid})) #old filterId
        destinations:   #configuration of the destinations 
            csv:
            ldap:
                service: SimpleJndiDstService #we should only specify a class name and LSC will look in the correct packages, or if failed, try to find the classname
        conditions:
            create: 1 > 0
            update: >
                src.getAttr('updateTimeStamp') > dst.getAttr('updateTimeStamp') 
            #we rename srcBean and dstBean by src and dst
            delete: false
            modrdn: false
        action: (force|keep|merge)  #was K, F, M
        attributes:
            cn: #the name of the attribute
                create value: "something" 
                default value: "nothing" 
                force value: "blabla" 

It seems that you paste the configuration YAML file two times in this issue. Am'I right ?

Yes I did, silly me ...

You can see it online http://pylonshq.com/pasties/9ce57432be91cd02ac332c4d1e2b8e8f with some coloration

This YAML format is ok for me.

Overall, this looks good to me. One major query:

• Should we separate sources out from tasks? I mean: the configuration that links a connection (URL, username, password for example) and a set of parameters to a source service (for example: select all inetOrgPerson objects with a filter of _ from connexion _42) could be a in a seperate block, called a source or a source service. Then, we could have several tasks using the same service...

I don't really know at this point what is most adaptable and easy to understand. What do you think?

I also have some queries over details:

1. I assume the connections can have names (like My AD server rather than just ldap)?
2. In the connections, why are some parameters on the first level (url, password) and some under a configuration item?
3. I don't really understand the sources and destinations items in the task definition: do you link a connection from the first part of the file? If so, why not specify something like: connection: My AD Server?
4. basedn, filters, etc are parameters of a [src/dst]Service, so should probably be underneath that item. Also, a connection is probably a parameter to the service too.

We (Jonathan, Sebastien and me) decided for this new configuration format in the FOSDEM, what do you think of it ?

<connections>
    <connection type="ldap">
            <name>myAdServer</name>
            <url>ldap://localhost:1390/dc=AD,dc=net</url>
            <username>cn=manager,dc=AD,dc=net</username>
            <password>secret</password>
            <referral>ignore</referral>
            <derefAliases>never</derefAliases>
            <factory>com.sun.jndi.ldap.LdapCtxFactory</factory>
            <version>3</version>
            <authentification>simple</authentification>
            <pageSize>10</pageSize>
            <tls>true</tls>
     </connection>
    <connection type="database">
        <name>myHSQLDBServer</name>
        <url>jdbc:hsqldb:file:hsqldb/lsc</url>
        <username>elilly</username>
        <password></password>
        <driver>org.hsqldb.blabla</driver>
    </connection>
</connections> 

<audits>   
    <audit type="csv"> 
        <name>csv</name>
        <operations>create, delete</operations>
        <attributes>cn, dn</attributes>
        <separator>;</separator>
        <append>true</append>
        <file>/tmp/log.csv</file>
    </audit>
    <audit type="ldif">
         <name>ldif</name>
        <operations>create, delete</operations>
        <append>false</append>
        <file>/tmp/log.ldif</file>
    </audit>
</audits>                           

<services>
    <service type="SimpleJndiSrcService"> 
        <name>myADAccount</name>
        <connection>myAdServer</connection>
        <baseDn>ou=people</baseDn>
      <pivotAttributes>dn, cn</pivotAttributes>
        <fetchAttributes>uid, mail, cn</fetchAttributes>
        <dn>'cn=' + srcBean.getAttributeValueById('uid')</dn>
      <getAllFilter>(&(objectClass=inetOrgPerson)(uid=*))</getAllFilter>
      <getOneFilter>(&(objectClass=inetOrgPerson)(uid={uid}))</getOneFilter>
    </service>
    <service type="SimpleJndiDstService">
        <name>myDestination</name>
        <connection>myAdServer</connection>
    </service>
</services>   

<tasks>
    <task>
        <name>myTask</name>
        <source>myADAccount</source>
        <destination>myDestination</destination>
        <auditLogs> 
            <audit>csv</audit>
            <audit>ldif</audit>
        </auditLogs>
        <conditions>
            <create>1 > 0</create>
            <update>src.getAttr('updateTimeStamp') > dst.getAttr('updateTimeStamp')</update>
            <delete>false</delete>
            <modrdn>false</modrdn>
        </conditions>
        <syncoptions class="XmlBasedSyncOptions">
            <attribute> 
                <name>cn</name>
                <createValue>toto</createValue>
                <defaultValue>toto</defaultValue>
                <forceValue>toto</forceValue>
                <policy>(force|keep|merge)</policy>
            </atttibute>
        </syncoptions>
    </task>
</tasks>

Hi,

Rémy-Christophe Schermesser wrote:

We (Jonathan, Sebastien and me) decided for this new configuration format in the FOSDEM, what do you think of it ?

I think this is XML? So just a few technical remarks:
1/ There are some missing elements:

<?xml version="1.0" encoding="UTF-8"?>

2/ Be careful to XML syntax, which can cause problems specifically while writing LDAP filters :

>       <getAllFilter>(&(objectClass=inetOrgPerson)(uid=*))</getAllFilter>
>       <getOneFilter>(&(objectClass=inetOrgPerson)(uid={uid}))</getOneFilter>

is incorrect, it should be:

      <getAllFilter>(&amp;(objectClass=inetOrgPerson)(uid=*))</getAllFilter>
      <getOneFilter>(&amp;(objectClass=inetOrgPerson)(uid={uid}))</getOneFilter>

3/ A DTD and/or an XML schema would be fine.

Regards,
Raphaël.

Thanks Raphaël !

For the 1) you are right, I will add them
For the 2) The parser we used seems not to care about & and &, but I will be careful and test it
For the 3), yes we will write a DTD as soon as the the configuration is implemented.

Any thoughts on feature #120?

Jonathan Clarke wrote:

Any thoughts on feature #120?

It should be feasible with xml entities, but I am not sure your parser support them.

Hi,

• the taskName should be highlighted in the configuration and/or the configuration file should be splittable

• javascript should not really be an issue if:
1. it becomes possible to define functions (#182)
2. some helpers are added (ex. #226)
3. support for external javascript files is added (#182 too?)

• the format should not be more verbose than the actual one

Regards,
Raphaël Ouazana.

• Have simple key/value pairs
• Group key/value pairs by section, and sub-section
• Reuse previously defined values (#120)
• Include one config file in another
• Have one section "extend" another section, just changing some values

I like the look of it!

• It is written in Scala. However, it can apparently be used from Java (OK, why not)
• It also includes a logging API, but we can ignore this I think.

See http://www.lag.net/configgy/README.html.

I just love your proposition Jonathan. The syntax is simple, and the API is clean.

And I think that the fact it is in scala is a huge advantage ;)

We should try and do a poc.

Jonathan Clarke wrote:

Just came across a library that aims to solve the configuration file problem... It uses a simple language that allows to:

• Have simple key/value pairs
• Group key/value pairs by section, and sub-section
• Reuse previously defined values (#120)
• Include one config file in another
• Have one section "extend" another section, just changing some values

I like the look of it!

Some drawbacks:

• It is written in Scala. However, it can apparently be used from Java (OK, why not)
• It also includes a logging API, but we can ignore this I think.

See http://www.lag.net/configgy/README.html.