Case Settings

Table of Contents

• Case Settings

  • General Settings

  • Display Settings

    • Facet Settings

    • Content Settings

    • Result list Settings

    • Detail View Settings

    • List Style Settings

  • Search Settings

    • Stopword List

    • Synonym List

    • Boosting

      • Boost Query

      • Boost Function

  • API Settings

    • Polling API

      • Setting up your Application

      • Batched result responses

      • Setting up a polling job on the INDICA appliance

    • PowerBI Connection

      • Creating an OAuth client

      • Creating a Personal Access Token

      • Connecting PowerBI to INDICA

      • Creating request URLs

    • Polling API Examples

      • Job GET Endpoint JSON

      • Job Result JSON

      • Document Query JSON

      • Document Query Response JSON

      • Subject Details Request JSON

      • Export Documents JSON

      • Document Export Response JSON

      • GDPR Query JSON

      • GDPR Query Response JSON

      • Batch Response JSON

General Settings

The “General Settings” page allows you to tune some settings like hits per page, enable comments, datechanger, etc.

• Use first folder as Facet or Custodian name?

  This feature is especially useful for eDiscovery investigations. Enabling this option allows you to place data items that belong to a custodian in a separate folder. INDICA will recognise this folder as a custodian and adds filtering capabilities on custodian in the facet bar.

• Enable OCR?

  Toggling this setting will enable/disable the Optical Character Recognition processing. Disabling this setting will result in a higher indexing speed, but it will not fully index the content of scanned documents. The index will be incomplete as it will not contain the content of non-text based data items.

  Tip

  It is possible to process OCR data on a later moment.

• Pre-create document views?

  This option allows you to pre-create document views. This will result in nicer document previews and faster loading times.

• Only index email meta data?

  When this option is enabled, only the metadata of emails is processed. The email content will not be included in the index.

• Use nice document viewer?

  Enabling this option allows you to preview documents in a nicer document previewer.

• Disable this option to remove ACL from shares

  When this option is disabled, the Access Control Lists of shares will not be taken into account when indexing. This means that all INDICA users can see all indexed documents, regardless of Whether or not they can see the documents on the source.

• Enable NLP library?

  When this option is enabled, INDICA will use it’s NLP algorithm to extract NLP keywords from documents.

• How many hits should be shown per page?

  Number of hits per page. Can be changed to “Show None” to allow metadata investigations.

• Enable Downloads?

  Whether or not downloading of documents from the user interface is allowed.

• Enable Comments?

  Whether or not the “Comments” section in the previewer will be shown. This allows users to attach comments to documents.

• Enable Stemming?

  When Stemming is enabled, the indexer will trim down verbs to their stem for indexing.

• Enable datechanger?

  Datechanger enables the user to alter the date of a document. This is especially useful when a document is scanned in from paper. The file creation date will not necessarily reflect the original (paper) document date. The datechanger can change the document date in the index.

Display Settings

Facet Settings

Facet Settings allows you to manage and create filtering capabilities in the INDICA front-end.

This overview shows the list of currently active filters (facets) in the front-end. Don’t see the facet list in your front-end? Click on the “Reset to default” button to seed the current facets. From the overview, it’s possible to:

• Add facets

  Type in a facet key in the top bar (“Enter facet key”), then click the “Add” button. A new facet will be created. The facet keys are based on the fields in the index. For help with selecting the correct field, contact INDICA support.

• Re-order facets

  Re-ordering facets is possible by dragging the icon with the four horizontal lines. Changing the order of the facets will be reflected on the front-end.

• Delete facets

  Click the “X” icon on the right to remove a facet. It can always be re-added if needed.

• Edit facets

  It is possible to change some characteristics of a facet. You can change the name of the facet, if it will be expanded by default, and if the facet should be hidden or not.

• Reset facets to default

  When needed, the facets can be reset to the default setup by clicking the “Reset to default” button.

Changes in the facet settings will be presented to the end user as such:

Content Settings

These content settings are correlated to the viewable content for the specified user group within the case.

• Allow users to see content

  Enabled by default. Disabling this setting will revoke access for the users of the case to preview the content of documents.

• Allow managers to see content

  Enabled by default. Disabling this setting will revoke access for the managers of the case to preview the content of documents.

• Allow admins to see content

  Enabled by default. Disabling this setting will revoke access for the admins of the case to preview the content of documents.

Disabling any of these settings means that these groups will not be able to download and preview documents, but are still able to see meta data. GDPR tokens within the meta data have been stripped and will only show ‘FOUND’ as a value.

Note

Please keep in mind that it’s possible that an AD connection has been set up. AD settings will overwrite these content settings.

Result list Settings

The following option can be changed on this page:

• Allow user to toggle between list and table style

  When enabled, a button will appear on the search page that allows the user to switch between list-style results and table-style results. The table-style results can be used for a custom overview of query results, as the displayed information can be chosen by the user. This feature is very powerful, but less easy to use.

Example of list-style preview:

Example of table-style preview:

Detail View Settings

This panel allows you to change the following options:

• Show Tagging

  This option enables and disables the options to tag documents in the front-end. It is not possible to alter tags when this option is disabled.

• Show data lineage

  This option enables and disables the options to show data lineage button in the front-end. The data lineage feature gives the user the possibility to select text and click on the lineage button. This will fire off a query with the selected text and the results will open automatically in the new “Data Lineage” tab. This feature makes it easier to find the so-called ‘single source of truth’ when dealing with duplicate information.

List Style Settings

These options allow for customizing the information in the list-style results.

The following options can be changed:

• Show date?

  Enable or disable the displaying of the date from result item.

• Show file size?

  Enable or disable the displaying of the file size from result item.

• Show ID?

  Enable or disable the displaying of the document ID from result item.

• Show path?

  Enable or disable the displaying of the file path from result item.

• Show summary?

  Enable or disable the displaying of the document summary from result item.

• Show similar?

  Enable or disable the displaying of the “Similar Documents” button from result item.

• Show duplicates?

  Enable or disable the displaying of the “Duplicates” button from result item.

Search Settings

The search settings allow you to manipulate the search results. This can be done by defining synonyms, stopwords, and editing the boosting settings.

Stopword List

During indexing, it is possible to exclude a list of stopwords from the index. Those words are generally words without informational value, like “a”, “and”, “this”, “the”, etc.

Words can be added by typing them in the text field and then pressing the “Add” button.

INDICA comes with a default lists of stopwords, which can be changed here as well.

Synonym List

It is also possible to define synonyms. This allows you to automatically broaden search results that contain words. Synonyms can be added in the text field and then clicking the “Add” button.

Synonyms need to be added as a comma-separated list, for example: “hello,hi,hey”.

Boosting

Boosting can be done by adding a boost query, or by adding a boost function.

Boost Query

The Boost Query specifies an additional query clause that will be added to the user’s main query to influence the score.

INDICA provides the relevance level of matching documents based on the results found. To boost a query, use the caret, “^”, symbol with a boost factor (a number) at the end of the query you are searching. The higher the boost factor, the more relevant the query will be. Boosting allows you to control the relevance of a document by boosting its query. For example, if you are searching for

jakarta apache

And you want the term “jakarta” to be more relevant, boost it using the ^ symbol along with the boost factor next to the query. You would type:

jakarta^4 apache

This will make documents with the term jakarta appear more relevant. You can also boost Phrase Terms as in the example:

“jakarta apache”^4 “Indica search”

By default, the boost factor for each term or phrase is 1. Although the boost factor must be positive, it can be less than 1 (e.g. 0.2).

Boost Function

A Boost Function can also be added here. This feature is for advanced users. Please refer to the official documentation for help.

API Settings

INDICA supports two types of API’s. The polling API and the high level API.

Polling API

The INDICA Polling API service for external systems is a secure API that enables external systems to query the INDICA index in a limited way. It is designed for asynchronous communication initiated by the INDICA appliance (polling for instructions). The API is only able to transfer meta information, no actual content of document object can ever be transmitted to the external system asking for information.

The external system creates a queue of queries according to its needs in its own external API end point (the command set) to be run against the INDICA index. The INDICA pull API then returns the results back to the external systems designated API endpoint. Query results can return metadata from the index with addition to GDPR issues and for authorized users links to preview in the INDICA system itself.

This API is designed for on-premise INDICA systems and Internet-facing applications.

Setting up your Application

For the API to work, your application requires a very basic job system. A single Basic Auth or preferably OAuth protected endpoint(GET) that returns a query of commands/queries for INDICA to process. And a single protected endpoint(POST) which receives the results from INDICA.

Endpoint paths are configurable and you can create it according to your system limits/needs, but it should reside on two of the endpoints (not more) and explicitly POST endpoint has to match the specified pattern.

Required Endpoints

Type	Path	Ext. App. JSON structure	Description
GET	/api/indica/jobs	Job GET Endpoint JSON	Response body has to match one of the job type structures
POST	/api/indica/jobs/{job_id}	Job Result JSON	These will be result that you will receive from INDICA once specific job type is done.

INDICA also accepts more job types than the ones shown in the examples above. API endpoints stay the same on your system, but required request end response JSON structures change based on every type.

Job Types

Type	Job Queue JSON	Response JSON	Description
document_query	Document Query JSON	Document Query Response JSON	Document query returns hit counts and meta fields you requested
subject_details	Subject Details Request JSON	Document Query Response JSON	Returns the same values as Document Query. The only difference is that the query is generated by INDICA
document_export	Export Documents JSON	Document Export Response JSON	Tag export prepares a package on specified tags and uploads it to a specified vendor with specified security measures
gdpr_query	GDPR Query JSON	GDPR Query Response JSON	Returns GDPR issues on all or specific assets.

Batched result responses

All responses can be batched if the job request includes “batch” : 1 and batch_size fields in the JSON. batch_size has to be an integer of what is the maximum character count in the resulting JSON will be. Batched results have one extra JSON key (“batch_id”) and an extra status - “inprogress”. In which “batch_id” represents the order in a batch and status tells if the batch is still “inprogress” or is “processed”.

Batch Response JSON

Setting up a polling job on the INDICA appliance

For job polling system to work correctly it is required to follow strict guidelines.

Settings configuration currently resides in Settings > API Settings inside the case management.

To create a new polling job, press the “Add new API polling job” button.

Polling jobs will run according to your settings:

• Result data type

  Can be changed based on what data you want to be returned to the external source. Currently, there is “Document query” - which returns document fields that you select further down in “Return fields”, and there is also a choice to return “GDPR issues”. Structure of returned data is described below;

• Run query in user scope

  Option specifies if the query should include only the data which adheres to a specific user’s rights (could be API user with your setup in your AD);

• User

  Setting of a user who’s scope will be used;

• Time between polling

  Is used to specify how often the system will check if there are new jobs in the GET endpoint, and run them. Running jobs too often might degrade the performance of your system;

• Endpoint Authentication type

  Specifies what type of authentication external endpoints use;

• Job Queue GET endpoint

  Specifies endpoint from which jobs list in a JSON format will be retrieved;

• Result POST endpoint

  Specifies endpoint to which the queried Document field(s) or GDPR issues will be sent;

• Require acknowledge

  Specifies if acknowledgment of received jobs should be given. It is strongly recommended to use acknowledgment since it negates duplicate jobs when polling time is shorter, or queries are more difficult. It guarantees that the same job doesn’t get picked up twice or more;

• Job acknowledgment endpoint

  Specifies an endpoint to which acknowledgment will be done.

PowerBI Connection

With INDICA, it is possible to connect PowerBI and use the modeling capabilities of PowerBI to create your own dashboarding.

Setting up an API connection between INDICA and PowerBI requires some setup. The steps needed to take are:

1. Creating an OAuth client

2. Creating a personal access token

3. Connecting PowerBI to INDICA

4. Creating request URLs

In the following sections, the above steps will be described in detail.

Creating an OAuth client

First, navigate to the API Settings page under ‘Manage’ > ‘Settings’ > ‘API Settings’. You will be presented with the following page:

Here, we can create a new OAuth client by clicking the “Create New Client” button. A popup will open:

The following information is required:

• Name

  Choose a descriptive name so you will recognise it later on.

• Redirect URL

  If you don’t know what to fill in as redirect URL, you can use https://127.0.0.1, this will be a placeholder.

After filling in all fields, click “Create” to create the OAuth client.

The settings page should look like this:

Creating a Personal Access Token

The next step is creating a Personal Access Token. This token will be used by PowerBI to let INDICA know that it has access to the data. The access token will give unlimited access to the data within INDICA; be careful with it. Do not share it with untrusted parties.

A new token can be created by clicking the “Create New Token” button. A popup will open:

The following information is required:

• Name

  Choose a descriptive name so you will recognise it later on.

• Scopes

  Scopes define the kind of information is retrievable by the API connection.

The following 5 scopes are available to the user:

• admin.directory.users

  This scope gives the user the ability to: Read and modify users.

• admin.directory.cases

  This scope gives the user the ability to: Read and modify cases.

• api-raw

  This is the most powerful scope, this scope allows the user to send an receive raw SOLR data.

• api-docs

  This scope gives the user access to the documents in the index.

• api-facets

  This scope gives the user access to the facets.

For the PowerBI connection, we will be using the scope “api-raw”. After checking the box for api-raw click “Create” to generate your Personal Access Token.

You will now be presented with your personal access token. Make sure to save this token somewhere safe. It will be needed later on.

Warning

This is the only time you will be presented with this token. When the popup is closed, there is no way to retrieve it. Make sure to store this somewhere safe.

When done, click the “Close” button to close the popup.

The settings page should look like this:

Connecting PowerBI to INDICA

When above steps are completed, the connection between INDICA and PowerBI can be made. Open PowerBI and click the button “Transform data”.

When clicked, the Power Query Editor will open.

Click “New Source” in the menubar on the top (marked by the green arrow). Choose “Web” from the list of available options. A popup will open:

In this popup, choose the “Advanced” option. The popup will now look like this:

The following fields are required:

• URL parts

• HTTP request header parameter

Adding the access token to the request

INDICA will not return any data when the access token is not set correctly. To configure PowerBI to include the access token, type the word Authorization in the field on the left under “HTTP request header parameters”. Then, in the field on the right, type in the word Bearer followed by a space, and then the complete Personal Access Token.

The popup should now look like this:

When everything is filled in, click “OK”.

If a question pops up asking “Select which level to apply these sessings to”, pick the option that ends with /api/solr. Then, click “Connect”. Your connection with PowerBI should now be made.

In some occasions, PowerBI will return an error stating that it is unable to connect:

This happens because INDICA will return data in CSV format, while PowerBI expects JSON. Click “Edit” when the above error is displayed.

Then the following popup will open:

Under “Open file as”, select the option for “Csv Document”. PowerBI will now recognise the output that INDICA generates. Finally, click “OK” to confirm. You should now be presented with a preview of the selected data.

The INDICA and PowerBI connection is now ready for use.

Creating request URLs

The URL can be created by combining serveral parameters:

• The URL or IP of the INDICA appliance (called URL in the example);

• The ID of the case on the INDICA appliance (called GROUP_ID in the example);

• The query you want to use (called QUERY in the example);

• A list of the fields to retrieve (called FIELD_LIST in the example);

• The number of rows that will be retrieved (called NUM_ROWS in the example).

Above parameters may be placed in the following format:

http(s)://URL/api/solr?group_id=GROUP_ID&q=QUERY&fl=FIELD_LIST&wt=csv&rows=NUM_ROWS

Note that depending on the appliance, either http or https needs to be used. A working example of this would be:

http://10.66.34.201/api/solr?group_id=1603627437&q=*:*&fl=tags,date,type&wt=csv&rows=100000

Note

For no query, use *:* as query

When the URL has been prepared, you can copy and paste it in the “URL parts” field in PowerBI.

Polling API Examples

Job GET Endpoint JSON

[
    {
      "job_id": "47ec1fe7db822300e5c4e1bb4b961972",
      "job_type": "document_query",
      "q": "\"David Vikander \"~0",
      "fq": "",
      "return_fields": [
        "file_name",
        "mime_type",
        "size"
      ],
      "status": "processed"
    },
    {
      "job_id": "47ec1fe7db822300e5c4e1bb4b961972",
      "job_type": "document_query",
      "q": "\"Dead Kennedys\"~0 OR \"test\"~0",
      "fq": "",
      "return_fields": [
        "file_name",
        "mime_type"
      ],
      "status": "processed"
    }
]

Job Result JSON

{
    "job_id": "47ec1fe7db822300e5c4e1bb4b961972",
    "job_type": "document_query",
    "result_url": "http://192.168.2.128/search?q=%22David%20Vikander%22~0&page=1&fq={%22GDPR_Combined%22:[%22NAME%22]}&sort=score%20desc&mlt=0",
    "result_collection": {
      "count": 2,
      "entries": [
        {
          "file_name": "John_Doe.doc",
          "mime_type": "doc"
        },
        {
          "file_name": "John Doe.pst",
          "mime_type": "pst"
        }
      ]
    },
    "status": "processed",
    "error_message": "error"
}

Document Query JSON

[
    {
      "job_id": "47ec1fe7db822300e5c4e1bb4b961972",
      "job_type": "document_query",
      "q": "\"David Vikander \"~0",
      "fq": [
        {
          "tags": [
            "2_todelete"
          ]
        }
      ],
      "return_fields": [
        "file_name",
        "mime_type",
        "size"
      ],
      "exporting": true,
      "status": "processed"
    },
    {
      "job_id": "47ec1fe7db822300e5c4e1bb4b961972",
      "job_type": "document_query",
      "q": "\"Dead Kennedys\"~0 OR \"test\"~0",
      "fq": "",
      "return_fields": [
        "file_name",
        "mime_type"
      ],
      "exporting": true,
      "status": "pending"
    }
]

Document Query Response JSON

{
    "job_id": "47ec1fe7db822300e5c4e1bb4b961972",
    "job_type": "document_query",
    "result_url": "http://192.168.2.128/search/SCA0001052?q=a",
    "result_collection_urls": [
      {
        "name": "TestCollections",
        "url": "http://192.168.2.128/search/SCA0001052?module=TestCollections&q=a"
      },
      {
        "name": "Sites",
        "url": "http://192.168.2.128/search/SCA0001052?module=Sites&q=a"
      }
    ],
    "result_collection": {
      "count": 2,
      "entries": [
        {
          "file_name": "John_Doe.doc",
          "mime_type": "doc"
        },
        {
          "file_name": "John Doe.pst",
          "mime_type": "pst"
        }
      ]
    },
    "tags": [
      {
        "title": "To Delete",
        "value": "2_todelete"
      },
      {
        "title": "Unresolved",
        "value": "1_unresolved"
      }
    ],
    "status": "processed",
    "error_message": "error"
}

Subject Details Request JSON

{
    "job_id": "SCA0001017",
    "job_type": "subject_details",
    "subject_details": {
      "name": "John Doe",
      "tel": "88123456789",
      "email": "jonh@example.com",
      "address": "Elm Street 13"
    },
    "status": "pending"
}

Export Documents JSON

[
    {
      "job_id": "47ec1fe7db822300e5c4e1bb4b961972",
      "job_type": "document_export",
      "packaging": 1,
      "export_vendor": "box",
      "query_job_id": "56rsbs0e5css51972",
      "protection": "pass",
      "subject_email": "test@example.com",
      "status": "processed"
    },
    {
      "job_id": "8e6d5fe7db822300e5c4e1bb4b961947",
      "job_type": "document_export",
      "packaging": 1,
      "export_vendor": "box",
      "query_job_id": "56rsbs0e5css51972",
      "protection": "pass",
      "subject_email": "test@example.com",
      "status": "processed"
    }
]

Document Export Response JSON

{
    "job_id": "47ec1fe7db822300e5c4e1bb4b961972",
    "job_type": "document_export",
    "package_url": "https://example.com",
    "status": "processed",
    "error_message": "error"
}

GDPR Query JSON

[
    {
      "job_id": "e6e31e4ddb032300e5c4e1bb4b9619fa",
      "job_type": "gdpr_query",
      "q": "",
      "fq": "",
      "status": "processed"
    }
]

GDPR Query Response JSON

{
    "job_id": 64,
    "job_type": "gdpr_query",
    "result_collection": {
      "count": 1,
      "entries": [
        {
          "asset": {
            "type": "collection",
            "identifiers": {
              "name": "Unstructured Data",
              "ip": "198.168.1.0",
              "path": "/dropbox/JohnDoe/things"
            }
          },
          "issues": [
            {
              "type": "NAME",
              "count": 227
            },
            {
              "type": "EMAIL",
              "count": 53
            },
            {
              "type": "TEL",
              "count": 11
            },
            {
              "type": "CC",
              "count": 4
            }
          ]
        }
      ]
    },
    "status": "processed",
    "error_message": "error"
}

Batch Response JSON

Response 1

[
    {
      "job_id": "47ec1fe7db822300e5c4e1bb4b961972",
      "job_type": "document_query",
      "batch_id": 0,
      "result_url": "indica.lan/query=JonhDoe",
      "result_collection": {
        "count": 2,
        "entries": [
          {
            "file_name": "John_Doe.doc",
            "mime_type": "doc"
          },
          {
            "file_name": "John Doe.pst",
            "mime_type": "pst"
          }
        ]
      },
      "status": "inprogress"
    }
]

Response 2

[
    {
      "job_id": "47ec1fe7db822300e5c4e1bb4b961972",
      "job_type": "document_query",
      "batch_id": 1,
      "result_url": "indica.lan/query=JonhDoe",
      "result_collection": {
        "count": 2,
        "entries": [
          {
            "file_name": "test.pst",
            "mime_type": "email"
          },
          {
            "file_name": "test.txt",
            "mime_type": "text"
          }
        ]
      },
      "status": "processed"
    }
]