Glue42 Search Service

Overview

The Glue42 Search Service (GSS) allows desktop applications to search arbitrarily among complex items across multiple search providers. The items can be entities, such as "Clients", "Instruments", "Accounts" etc. Each entity is associated with a set of fields, where each field has:

  • name - e.g., firstName, lastName, id etc.;
  • type:
    • scalar types: Boolean, Int, Long, Double, String, DateTime;
    • Composite, which means that the field defines an object (defined again as a sequence of fields);
  • searchType - None, Partial, Exact or Both (supports Exact and Partial matching);
  • isArray - a flag indicating that the field is an array;
  • displayName;
  • description;
  • metadata - an object with arbitrary key/value pairs;

Composite combined with the isArray flag allows for the definition of regular (non-cyclic, directed) object graphs. Both the Composite type and the isArray flag implicitly turn off searching on that field.

Architectural Diagram

GSS

Dedicated JavaScript API

Client applications don't talk to providers directly. Similarly to the Glue42 Notification Service, clients send requests to the GSS Desktop Manager application and receive results over Interop streams. The GSS Desktop Manager is responsible for multiplexing the requests to the providers and demultiplexing the results back to the client applications. It can also optionally implement local caching with configurable expiration.

JavaScript client applications talk to the GSS Desktop Manager using a JavaScript API which internally utilizes Interop streaming. The JavaScript API exposes interfaces for searching (GlueSearchService) and for creating search providers (GlueSearchProvider).

Interop and REST Protocols

GSS providers can use 2 protocols to provide search results - Interop (subscription/push-based) and REST (request-based).

The GSS Desktop Manager configuration is typically held in the Configuration Manager and can be configured with user-specific overrides and modified during runtime. As with the Glue42 Notification Service, local file configuration can also be used for development/test/demo purposes.

Multiple Search Providers

There could be multiple search providers for each entity type - "Clients" can be provided by both Salesforce or Microsoft Dynamics, and by Outlook. Providers are not required to offer the same set of entity fields. This is to say, that the combination of all fields from all providers defines the entity type. However, all providers must use the same field descriptors for a given entity type they support. The only exception is the search type, where providers can specify different search type for any of the fields.

Search Mechanics

A client application issues a search query for an entity type, specifying one or more search fields and their values (selection), and can optionally limit the maximum number of entries that each provider should return, and the set of entity fields (projection) that are required in the result.

The query filter is a set of key/value pairs, where the key is the name of one of the entity type fields, and the value is the value to search for.

Depending on the search type, the relevant providers will perform exact or partial matching (it is an error to pass a field in the search filter, if the field search type is None). Client applications can optionally pass both, a field value and the requested search type. In case the client wants the search to be performed on all searchable fields, or the client knows that providers offer full text search, then the client can specify a field ANY, which will be dispatched to all providers.

Both, the entity type and filter, are required and any filtering is done at the provider side.

Full Text Search

A provider can add a special field, called ANY, to an entity type definition, which allows clients to execute a partial search against all indexed/searchable fields for that entity, or a full text search, if the provider supports that.

Multiple Asynchronous Results from Multiple Providers

When a search is performed, the GSS Desktop Manager looks at the set of fields and finds the appropriate search providers. It then runs parallel queries against these providers. As each provider returns or starts streaming data, the GSS Desktop Manager will start streaming the results back to the client application.

The results from a search query are returned asynchronously and it is expected that different providers will return results with different latencies. Once all the results from all the providers are delivered to the client application, the search query fires a completion callback to indicate that the application should not expect more results. A provider can send multiple results for a single query, if it is multiplexing the requests itself, so the client should expect more than one result from a provider.

Since one query can return multiple results from multiple providers, each result is tagged with the provider name and carries a flag indicating whether this is the last result from this provider (in case the provider batches results or searches across various sources of data internally). Typically, the client code does not need to be aware of the number of providers.

Search Cancellation and Time Out

At any time, a client application can cancel a search request or change the query filter (e.g., the user typing a few more characters).

Also, a client can specify a search timeout - globally or per query. If a search request times out (regardless of whether data was received), the query state will become "timed out" and the completion callback will fire.

Limitations

  • No filtering is performed at the side of the GSS Desktop Manager.
  • No attempt is made to provide joins across entity types in GSS.
  • Transformations and aliasing in a projection are not supported by GSS.
  • No conflation or throttling is performed in the GSS Desktop Manager or by the JavaScript API.

References

GSS API

GSS REST API Swagger Spec

Writing a GSS Search Provider

Overview

The GSS search provider can handle queries for 1 or more entity types.

Creating a Provider

Since the JavaScript GSS library is not part of Glue42, to create a GSS provider, instantiate GlueSearchProvider, passing to it a reference to the Glue42 Interop library:

// creating a provider
const provider = new gss.GlueSearchProvider(glue.interop);

In order for the provider to connect to the GSS Desktop Manager, you need to call the start() method. This returns a Promise, returning the provider, but also takes an optional callback in case you cannot use Promise (in the Glue42 Browser you can use Promise):

function (error: Error, result: GlueSearchProvider);

Example:

provider.start()
    .then((/* already have a ref (provider) */) => {
        // register entity type handler(s)
    })
    .catch((err) => {
        // deal with the error
    });

Without using a Promise:

provider.start((err, result) => {
    if (err) {
        // deal with the error
    }
    else {
        // register entity type handler(s) on result (or provider)
    }
});

Registering Entity Types

In order to handle search requests for an entity type, the provider needs to register a handler for each entity type it supports by calling addEntityType() on the provider instance. The addEntityType() method requires a reference to an entity type object, and a callback function which will be called with the search request when a search is performed.

You can create an entity type using the classes GssEntityType and GssField. However, the library provides an easier way of building an entity type from a JavaScript (or JSON) object - GssEntityType.fromJS(jsObject).

Example:

// define an entity type as a JavaScript (or JSON) object
const partyEntityType = {
    "name": "Party",    // entity type name
    "properties": [     // entity fields
        {
            "name": "partyType",
            "type": "String",
            "searchType": "Exact",
            "displayName": "Party Type",
            "description": "specifies types of parties ('Client', 'Prospect', 'Lead', 'Market Lead', 'Other Contact', 'Authorized Contact') that can be searched...."
        },
        {
            "name": "fullName",
            "type": "String",
            "searchType": "Partial",
            "displayName": "Full Name",
            "description": "Full Name of party."
        },
        {
            "name": "advisor",
            "type": "Composite",
            "properties": [
                {
                    "name": "fullName",
                    "type": "String",
                    "displayName": "Advisor Name"
                },
                {
                    "name": "webId",
                    "type": "String",
                    "displayName": "Advisor WEBID"
                }
            ],
            "description": "Details of Banker/Advisor covering the party."
        }
    ]
};

Here is an example of creating the entity type object from the JSON above:

// create a GssEntityType out of the JavaScript/JSON object
const entityType = gss.GssEntityType.fromJS(partyEntityType);

Once you have an entity type, you can register a search handler for it:

// register an entity type handler for the entity type
// the handler is of type function (query: GssQueryRequest)
provider.addEntityType(partyEntityType, handlePartySearch);

Implementing the Search Handler

The search handler function should accept a single argument of type GssQueryRequest. The request object captures the entity type in the entityType property, and the search query in its query property. Once the provider is ready to push results, it can use the push() method, or if there is an error handling the request, it can use the error() one.

GSS Search Provider Example

Below is an example implementation of a GSS search provider, which performs searches by various identifiers by concurrently calling multiple backend (REST) services:

Defining the entity type (should be in a separate file):

const partyEntityType = {
    "name": "Party",
    "properties": [{
            "name": "partyType",
            "type": "String",
            "searchType": "Exact",
            "displayName": "Party Type",
            "description": "specifies types of parties ('Client', 'Prospect', 'Lead', 'Market Lead', 'Other Contact', 'Authorized Contact') that can be searched. Herut (a GSS provider) will initially support 'Client' and 'Prospect' but eventually will support all party types."
        },
        {
            "name": "portf",
            "type": "String",
            "searchType": "Exact",
            "displayName": "PORTF",
            "description": "Portfolio identifier."
        },
        {
            "name": "netId",
            "type": "String",
            "searchType": "Exact",
            "displayName": "NETID",
            "description": "Legacy client identifier."
        },
        {
            "name": "dynamicsId",
            "type": "String",
            "searchType": "Exact",
            "displayName": "DYNAMICS ID",
            "description": "Auto generated client Dynamics ID."
        },
        {
            "name": "fullName",
            "type": "String",
            "searchType": "Partial",
            "displayName": "Full Name",
            "description": "Full Name of party."
        },
        {
            "name": "salesForceId",
            "type": "String",
            "searchType": "Exact",
            "displayName": "SALESFORCEID",
            "description": "SalesForce identifier."
        },
        {
            "name": "advisor",
            "type": "Composite",
            "properties": [{
                    "name": "fullName",
                    "type": "String",
                    "displayName": "Advisor Name"
                },
                {
                    "name": "webId",
                    "type": "String",
                    "displayName": "Advisor WEBID"
                }
            ],

            "description": "Details of Banker/Advisor covering the party."
        },
        {
            "name": "dmDynamics",
            "type": "String",
            "searchType": "None",
            "displayName": "DM DYNAMICS ID",
            "description": "Auto generated client Dynamics ID for DM."
        },
        {
            "name": "dmPortf",
            "type": "String",
            "searchType": "None",
            "displayName": "DM PORTF",
            "description": "Portfolio identifier for DM."
        },
        {
            "name": "dmName",
            "type": "String",
            "searchType": "None",
            "displayName": "DM",
            "description": "Full Name of DM."
        },
        {
            "name": "dmNetId",
            "type": "String",
            "searchType": "None",
            "displayName": "DM NETID",
            "description": "Legacy client identifier for DM."
        },
        {
            "name": "investor",
            "type": "String",
            "searchType": "None",
            "displayName": "Investor",
            "description": "Full Name of Investor."
        }
    ]
};

Implementation:

// define the entity type
const entityType = gss.GssEntityType.fromJS(partyEntityType);

// holds number of REST API calls in transit
let apiCount = 0;

// holds API call XHR's (which can be aborted)
const apiCalls = [];

// search results can overlap when using different identifiers; this is a "set"
let uniqueClientsPerPortf = {};

// REST API specifics - REST API supports searches by various identifiers
// the only difference being the search field (and its value, populated in the request)
const requestSpecs = [
    {
        searchKey: "DYNAMICS_SEARCH", searchValueField: "dynamics"
    },
    {
        searchKey: "PORTF_SEARCH", searchValueField: "portf"
    },
    {
        searchKey: "NETID_SEARCH", searchValueField: "netId"
    },
    {
        searchKey: "SALESFORCEID_SEARCH", searchValueField: "salesForceId"
    },
    {
        searchKey: "NAME_SEARCH", searchValueField: "firstName"
    },
    {
        searchKey: "NAME_SEARCH", searchValueField: "lastName"
    },
    {
        searchKey: "ACCOUNT_ID_SEARCH", searchValueField: "accountID"
    },
];

// max number of concurrent REST API calls
const apiMaxConcurrent = requestSpecs.length;

// entity type registration
const addPartyEntityType = (provider) => provider.addEntityType(partyEntityType, handlePartySearch);

// resets search on each new search request
const clearAll = () => {
    while (apiCalls.length) {
        xhr = apiCalls.pop();
        xhr.abort();
    }
    apiCount = 0;
    uniqueClientsPerPortf = {};
};

// filtering function discarding duplicates by a certain identifier
const uniquePortf = (client) => !uniqueClientsPerPortf.hasOwnProperty(client.portf);

const getURLParameter = (name) => decodeURIComponent((new RegExp(`[?|&]${name}=([^&;]+?)(&|#|;|$)`).exec(window.location.search) || [undefined, ""])[1].replace(/\+/g, "%20")) || undefined;

// actual GSS search handler
const handlePartySearch = (request) => {
    clearAll();

    // by default, get the search URL from the Glue42 Browser
    let apiUrl = htmlContainer.getContext().url;
    if (typeof apiUrl === "undefined") {
        apiUrl = getURLParameter("url");
    }
    if (typeof apiUrl === "undefined") {
        // make it default to something, if this has not been passed in the context
        apiUrl = "/secure/gcm/gcm-data/rest/client/search";
    }
    console.log(`endpoint URL: ${apiUrl}`);

    // make N parallel search requests for each identifier
    requestSpecs.forEach((spec) => {
        const query = {
            searchKey: spec.searchKey,
            gId: null,
            portf: null,
            dynamics: null,
            salesForceId: null,
            firstName: null,
            lastName: null,
            netId: null,
            accountID: null
        };
        query[spec.searchValueField] = request.query.filter[0].value;
        apiCall(request, apiUrl, query);
    });
};

// ~= _.get(obj, path) (lodash)
const getObjectPath = (obj, path) => {
    if (!obj) {
        throw new Error("obj is required");
    }
    const fields = path ? path.split(".") : [];
    if (fields.length == 0) {
        throw new Error("A path with at least 1 field name is required");
    }
    let fieldName;
    let data = obj;
    for (let i = 0; i < fields.length; ++i) {
        fieldName = fields[i];
        data = data[fieldName];
        if (data === null || data === undefined) {
            break;
        }
    }
    return data;
};

// ~= _.set(obj, path, value) (lodash)
const setObjectPath = (obj, path, value) => {
    if (!obj) {
        throw new Error("obj is required");
    }
    const fields = path ? path.split(".") : []
    if (fields.length == 0) {
        throw new Error("A path with at least 1 field name is required")
    }
    let fieldName;
    let data = obj;
    for (let i = 0; i < fields.length - 1; ++i) {
        fieldName = fields[i];
        if (data[fieldName] === null || data[fieldName] === undefined) {
            data[fieldName] = {};
        }
        data = data[fieldName];
    }
    data[fields[fields.length - 1]] = value;
    return obj;
};

// function, mapping REST to GSS fields
const fieldsMapping = (entity) => {
    // mapping <ext.REST> : <GLUE GSS>
    const mappings = {
        banker: "advisor.fullName",
        dynamics: "dynamicsId",
        clientName: "fullName",
        clientType: "partyType"
    };

    const result = Object.getOwnPropertyNames(entity).reduce((soFar, key) => {
        const sourceValue = getObjectPath(entity, key);
        const mappedField = mappings[key] || key;
        setObjectPath(soFar, mappedField, sourceValue);
        return soFar;
    },
        {});

    return result;
};

const gssPush = (request, data) => request.push(data, apiCount >== apiMaxConcurrent);

// REST call wrapper
const apiCall = (request, apiUrl, apiQuery) => {
    let data;
    const async = true;
    const xhr = new XMLHttpRequest();
    xhr.open("POST", apiUrl, async);
    xhr.setRequestHeader("Content-Type", "application/json; charset=UTF-8");
    xhr.onreadystatechange = () => {
        if (xhr.readyState !== XMLHttpRequest.DONE) {
            return;
        }

        apiCount++;

        if (xhr.status === 200 && xhr.getResponseHeader("Content-Type").includes("json")) {

            console.log("Request succeeded with response", xhr.responseText);

            try {
                data = JSON.parse(xhr.responseText);
                if (!data.hasOwnProperty("data")) {
                    throw new Error("Response is missing data property");
                }
                if (Array.isArray(data.data.constructor)) {
                    throw new Error("Data response is not an array of clients");
                }
                data = data.data.map(fieldsMapping).filter(uniquePortf);
                gssPush(request, data);
            } catch (e) {
                console.error("Response parse failed", e);
                request.error(e);

                // update cache to avoid duplicates
                data.forEach((client) => {
                    uniqueClientsPerPortf[client.portf] = client;
                });
            } 
        } else {
            console.log("Request unsuccessful", xhr.status, xhr.statusText, xhr.responseText);
            gssPush(request, []);
        }

        xhr.send(JSON.stringify(apiQuery));

        apiCalls.push(xhr);
    };
};

// creating the GSS provider
const provider = new gss.GlueSearchProvider(glue.interop, {
    debug: true,
    measureLatency: false
});
provider.start()    // and starting it
    // registering the entity type search handler
    .then(() => addPartyEntityType(provider)) 
    .catch(console.error);

Reference

Reference