What's New​

Version 2.3 includes following new features and changes.

New Explorer Backend​

A lot of effort was put into separating concerns of GBS (UI) and Explorer (formerly Frontend) to increase maintainability and stability. All Explorer specific features were reevaluated and either removed or moved to a new dedicated backend. The following list contains all features that were removed.

• Data Lab provided basic data analysis tools that could be applied to nodes once Property Aggregation types were configured accordingly. It was also possible to define and attach an Interop Template to a Data Lab.
• Entity Configuration was used to define if an entity was used as scope, was a generic entity, could be used in sandboxes or if it should be versioned.
• Group node logic allowed to define a property name in system configuration that held a boolean value to define a node as group node.
  info

  To remove the now obsolete xpl_isGroupNode property from nodes, the graph has to be reprocessed.

• Integration enabled the definition of scripts that could be executed in Explorer.
• Label and Property Filter allowed to define and save filters based on specific properties or labels. While filtering is still possible in Explorer, the ability to store filters was removed.
• Node Settings allowed to configure colors and icons of nodes on a global scale. This feature was removed, but each user is still able to define colors and icons in their personalization settings.
• Property Configuration allowed to define if a property could be changed in a sandbox or not.
• Property Groups were used to define grouping amongst properties over different entities. All properties are now displayed in a "General" group.
• Scope allowed to define global filters based on node data. While this feature was removed, it is still possible to define custom scopes in Explorer.
• Theme allowed to change the active Explorer theme.
• Trafomatrix logic allowed to define a property name in system configuration that held transformation matrices. These were used when providing a geometry to Instant 3DHub.
• Versioning allowed to define entities that should be versioned whenever they were processed.
  caution

  Removing versioning changed the way how uniqueness of nodes is guaranteed. This requires a reprocessing of the graph to ensure data uniqueness and also benefit from indexes.

The following features were moved to the dedicated backend

• EBS endpoints that were specifically developed for Explorer
• Personalization
• Sandbox
• Session

User restriction was removed from Sandbox and Session. This means that all created sandboxes and sessions can be used by any user.

Views and Functions​

Viewpoints were renamed to Views and now also support parameters. Parameters are a powerful feature that enhances query efficiency, security, and readability. They are used to pass values to queries at runtime, allowing for more dynamic and reusable queries.

Functions are analogous to Views, with key distinctions in query execution capabilities and how the Graph Security Layer is applied. Contrary to views, a function query can return any result. A user needs to have granted permissions specified as a Graph Access Rule to be able to execute a defined Cypher Query.

Neo4j AuraDB Support​

This version adds support for Neo4j AuraDB, Neo4j's fully managed cloud database service. In order to use an AuraDB instance either AWS S3 or Azure Blob Storage has to be configured. Check the System Configuration keys starting with graph-processing for more information. While enabling AuraDB support, the underlying processing code was also improved resulting in a better performance on big datasets.

New Way of Defining Data Pump Overrides​

A new way of defining data pump overrides was introduced to make them more resilient and clear. Before, there was only one form to define overrides, whereby adding a specific separator would indicate multiple overrides.

The new approach separates each override into its own configuration which allows for a clear overview of how many overrides are defined and which values are used. Assigning a name to each override makes managing them even easier.

Expression Improvements​

Expression logic was refactored to improve stability, maintainability and performance. For more clarity, when using properties in expressions, a markup-like syntax was introduced. While a property was simply referenced by its name (e.g. name) earlier, it now has to be wrapped with backticks (e.g. `name`).

Improvements​

• [EBS] Update to .NET 8
• [Explorer] Use same format for all dates
• [Explorer] Performance improvements
• [Explorer] Change default right side panel from Artifacts to Properties
• [GBS] Support for predefined [Empty] expression was removed
• [GBS] Performance of "edit relationship" was improved
• [GBS] Streamline pagination behavior in GBS UI
• [GBS] Obsolete task history logic was removed
• [GBS] Rewrite rename logic
• [GBS] Rewrite Keycloak realm creation logic
• [GBS] Add validation when renaming unsaved properties that are used in expressions
• [GBS] Add endpoint to reinforce Content db schema to be up-to-date with target entity definition
• [GBS] Update to .NET 8
• [GBS] Set staging and graph relationships fields to read-only
• [GBS] Add information about "Distinct" to Queue Job context
• [GBS] Consolidate log search endpoints and UI
• [GBS] Add information about "Process Type" to Queue Job context for load plan jobs
• [GBS] Remove action buttons in data importer for business key expression

Bug Fixes​

• [EBS] "Graph Overview" does not match modelling in GBS
• [EBS] Space character in filename leads to crash when downloading file
• [Explorer] When providing nodes, the new session will always use the default session name
• [Explorer] Searching and filtering in dashboard is not working correctly when not on first page
• [Explorer] Frontend is very slow and crashes with multiple tabs
• [GBS] Relationship count between staging layer and graph is different
• [GBS] Fixed an occasion where renaming entities was not possible because of Load Plans
• [GBS] Create workspace: switching between users and groups does not display data correctly
• [GBS] Keycloak groups are not displayed when editing a workspace
• [GBS] Entity configuration is not set correctly when added to processing after load type was selected
• [GBS] Many entities in a load plan lead to overflow of window
• [GBS] When adding multiple properties to an entity they can exceed maximum character limit
• [GBS] Fixed several problems for business key and business key builder
• [GBS] Renaming a property that is used to build millions of relationships leads to error message
• [GBS] Do not allow empty property value conditions in access rules
• [GBS] Delete self relation joining causes problems
• [GBS] Renaming configuration of an entity leads to deselection of the configuration in load plan
• [GBS] Fixed several problems when creating an entity from a container
• [GBS] Clients are not stored via external rules endpoint

API Changes​

Following API related changes are part of this release.

Start of Transition to GBS API v2​

Version 2.3 marks the start of a transition towards a RESTful API in GBS. Views and functions are the first features implemented in GBS API v2. Due to the size of the API the transition will be done step by step over a longer period of time. While new features will only be available in v2, existing functionality will be replicated or newly implemented in v2 but still kept as is in v1. Once the transition is finished, v1 will be deprecated and kept for at least one more version to give everyone enough time to switch to the new API.

Removed Endpoints​

Following endpoints were removed.

• [EBS] /api/data-buckets/*
• [EBS] /api/configuration/*
• [EBS] POST /api/graph/query/nodes
• [EBS] GET /api/graph/query/tree-view
• [EBS] GET /api/graph/navigation/scopes
• [EBS] POST /api/graph/query/related-nodes/successors-same-label
• [EBS] POST /api/graph/query/related-nodes/successors-same-label-count
• [EBS] POST /api/graph/query/related-nodes/per-label moved to Explorer Backend
• [EBS] GET /api/graph/query/related-nodes/primary-same-label
• [EBS] /api/graph/query/viewpoints/* renamed to /api/graph/query/views/*
• [EBS] /api/instant-3d-hub/* moved to Explorer Backend
• [EBS] /api/integrations/*
• [EBS] /api/interop-templates/*
• [EBS] /api/label-filters/*
• [EBS] GET /api/license/licensed-modules
• [EBS] /api/personalization/* moved to Explorer Backend
• [EBS] /api/property-filters/*
• [EBS] /api/sandboxes/* moved to Explorer Backend
• [EBS] /api/sessions/* moved to Explorer Backend
• [EBS] /api/themes/*
• [GBS] GET /api/datainterop/getTemplates
• [GBS] GET /api/entity-configuration/getEntityConfigurations
• [GBS] POST /api/graphprocessing/clearVersioningForEntity
• [GBS] GET /api/graphprocessing/getJsonForEntity
• [GBS] GET /api/logs/getTaskHistory
• [GBS] /api/node/*
• [GBS] /api/properties/*
• [GBS] /api/property-configuration/*
• [GBS] /api/propertygroup/*
• [GBS] GET /api/targetEntities/getEntityProperties
• [GBS] GET /api/targetEntities/getArtifactDefinitionProperties
• [GBS] /api/theme/*
• [GBS] /api/viewpoint/* replaced by new view endpoints /api/v2/views/*
• [GBS] POST /api/workspace/data-buckets

Deprecations​

Following endpoints were deprecated and will be removed in the next version.

• [EBS] POST /api/graph/query/cypher was replaced by functions
• [EBS] GET /api/graph/query/views/{id} was replaced by POST /api/graph/query/views/{id} to support parameters
• [GBS] POST /api/importer/deleteDataImporterExpressionMapping

What's New​

Version 2.2 includes following new features and changes.

Graph Security Layer​

New features to enable additional filtering and automatic Access Rule generation were added to Graph Security Layer.

Custom ExternalGraphSecurity workers allow to additionally filter graph related results after Access Rules are applied. Once registered in GBS, each graph related call to EBS will also call ExternalGraphSecurity workers which may then filter results based on customer specific systems or rules. For more information regarding the implementation of such a worker, refer to the developer documentation.

Since manually creating and maintaining Access Rules can be cumbersome, version 2.2 adds the possibility to automatically generate Access Rules based on external systems. A new Rule Transformation step can be added to Load Plans. This step allows to define a registered RuleTransformation worker that is called when the step is executed. Such a worker can be used to automatically create Access Rules based on customer specific systems. The data a worker needs to create rules (e.g., users, entities, properties, etc.) can be defined in the Rule Transformation step. For more information regarding the implementation of such a worker, refer to the developer documentation.

An active flag was added to workers in GBS which allows to pause the usage of a worker in the system. This comes in handy if for example an ExternalGraphSecurity worker is malfunctioning or not available. By pausing this specific worker the system will still work instead of returning errors because of a faulty worker.

Until now it was assumed that workers only provide GET / and POST / endpoints. Because of this, GBS took the provided worker URL as is to get basic information when adding a new worker to the system. With the new version custom endpoints are supported and GBS ensures to call GET / when registering a new worker no matter which worker URL is provided.

The way property value conditions are displayed in an Access Rule has been improved. Instead of displaying each individual condition in a row, they are now grouped by entity and property.

With version 2.2 EBS' /api/graph/query/cypher endpoint is deprecated. This endpoint allowed to execute any cypher query and directly returned the result from Neo4j which made it impossible to apply Graph Security Layer to any results. The endpoint is superseded by /api/graph/cypher/count and /api/graph/cypher/nodes. These endpoints also allow to execute custom cypher queries but only return a count result respectively nodes contained in a result. Additionally, the latter also applies Graph Security Layer.

Data Pump Overrides​

The way how Data Pump overrides work was changed to be more resilient and performant. Until now all overrides were executed at once. If three overrides were defined, GBS collected data of all overrides before processing them. This could lead to failures while importing because of duplicated entries and impact performance of the system. With version 2.2, overrides are handled individually. Each override will be its own task in Airflow and therefore processed independent of each other.

Additionally, support of placeholders was added to overrides which allows to define dates passed to overrides (see Load Plan documentation) for more information). This can, for instance, be used to pass a date to an endpoint in order to only retrieve data of a certain point in time.

Frontend Graph Layout and Performance Improvements​

Besides overall graph performance improvements, following changes have been made to improve the look and feel when working with the Frontend:

• Replace hierarchical layout algorithm by a more performant one
• Add top down order to by-level layout
• Decrease distance between nodes for by-level layout
• Add lasso selection that allows to freely select multiple nodes
• Add performance indicator for layouts
• Add warning indicator when too many nodes are displayed and display a hint to change the layout
• Add suggestion to change layout in "too many nodes" warning
• Focus and select newly added nodes in sandbox mode
• Add option to toggle automatic centering/fitting of nodes after running layout
• Add min and max zoom limit

Removal of [Empty] Expression​

Data Importer allowed to map an [Empty] expression to any property. This predefined expression was removed since it is also possible to define an empty string with through "Set expression".

Improvements​

• [Frontend] Menu for saving sessions was simplified
• [Frontend] A "quick save" button was added to top menu
• [Frontend] A "delete session" option was added to the top menu
• [Frontend] "Remove selected" was added to the default options of the node menu
• [Frontend] Added error screens for missing nodes in sessions and viewpoints
• [Frontend] Added detailed tooltips for graph controls
• [Frontend] Added "Provide" menu to graph overview
• [Frontend] Added cancel pending request dialog when loading takes too long
• [Frontend] Improved interaction between label and property filters
• [Frontend] Import sandbox functionality was moved from export to top menu
• [Frontend] Webvis library was updated to work with 3DHub 3.7
• [Frontend] Overall performance improvements
• [GBS] Loading data of dashboard tabs is done lazily when clicking on tabs instead of greedily when opening GBS UI
• [GBS] Use same date formats throughout the dashboard
• [GBS] Removed superfluous "Smoke test" button in Data Pump containers
• [GBS] Updated "Clear relationship" icon to make clear that it triggers a destructive behavior
• [GBS] When clicking on "Add related" in graph processing, all related entities are added instead of only related entities in selected workspace

Bug Fixes​

• [Frontend] Node menu is not immediately closed when clicking "expand"
• [Frontend] "Expand" menu might get stuck when opening node menu
• [Frontend] Edges won't reset color after selection in graph overview
• [Frontend] Cytoscape occasionally freezes in Firefox when zooming
• [Frontend] Authentication failed view cannot be rendered because theme is not present
• [Frontend] Filter by label is not working as expected
• [Frontend] Dashboard is crashing when roles are missing
• [Frontend] Property filter is not sorted correctly when toggling labels
• [Frontend] When using a property filter on a deselected entity, other entities cannot be selected
• [Frontend] When saving a renamed session, the old name is displayed shortly before menu is closed
• [GBS] When relationship diagram is moved to the bottom of the page it is rendered on top of the session logs
• [GBS] Changing a property aggregation while entity filter is active breaks filtering
• [GBS] Fixed a few bugs that could appear when creating multiple entities from a container
• [GBS] Dashboard does not consider selected workspaces when opened for the first time
• [GBS] First startup of a new instances fails because not correctly updating internal state
• [GBS] Queue jobs table in dashboard overlaps session logs
• [GBS] In graph processing it is not possible to select a load type if build type was selected before an entity
• [GBS] Entities belonging to a selected workspace are not displayed in property aggregation drop-down
• [GBS] Search index creation fails when empty entity is processed
• [GBS] It is possible to delete files from Data Pump image and save it without any files

Deprecations​

• [EBS] /api/graph/query/cypher has been deprecated and replaced by /api/graph/cypher/count and /api/graph/cypher/nodes endpoints. The deprecated endpoint will be removed in the next release.

What's New​

Version 2.1 includes following new features and changes.

Fulltext Search​

The index that was used for full-text search in older versions was prone to break on very big datasets. In Version 2.1 there is a new way how full-text search indexes are created and queried. Future releases will include further performance improvements of the search.

Performance Improvement of "Merge" Data Import​

The performance of data imports using "Merge" load type was improved drastically. While importing data for an entity with 300.000 nodes and 15 million relationships took 2 hours and 38 minutes with the old logic, the same entity now only takes 6 minutes to import.

GBS Dashboard​

Version 2.1 introduces a new dashboard in the Graph Builder Services. The data previously available on the dashboard was split into different views. Additionally, an overview of the whole system was added to the dashboard.

Telemetry​

OpenTelemetry, Jaeger and Prometheus are added to the stack and integrated in services to trace requests across the stack. This helps to track usage of components and identify bottlenecks in communication between services.

New Way of Storing Relationships​

When importing data, all corresponding relationships are precalculated and stored in the system. The way how these relationships are stored has been changed which results in a performance improvement. In the past it could take a long time to rename a property that is part of a relationship (~4 minutes for 15 million relationships). With the new approach the same rename only takes ~2 minutes and 30 seconds. In the course of these changes obsolete artifact types " PartID" and "RelatedNodePath" were removed.

Improvements​

• [Frontend] Consolidated filtering and removing nodes in sandbox
• [Frontend] Performance improvement when retrieving nodes in a Data Lab
• [Frontend] Improved error messages
• [Frontend] Filter button is permanently visible in top navigation bar
• [Frontend] Cleaned up of left navigation bar
• [Frontend] Added new options for context menu
• [Frontend] Allow easy reset to defaults of different personalization settings
• [Frontend] Result table in add related and search menu allows to select different properties to inspect data
• [Frontend] It is possible to personalize icons for labels
• [Frontend] Consolidated left menu in sandbox mode with menu in exploration mode
• [Frontend] Settings button was added to node context menu to easily access context menu settings
• [Frontend] "Property selection" window was moved from the right menu bar to a dialog opened through setting button in properties panel
• [Frontend] In "Filter by property" the global "Keyword" search was replaced by "Search all properties"
• [Frontend] Filter history now includes what was filtered for
• [Frontend] Active filters are displayed as tags to the filter menu
• [Frontend] Webvis library was updated to work with 3DHub 3.6
• [Frontend] Performance improvements by replacing calls to the now deprecated /api/graph/query/nodes endpoint
• [GBS] List of artifact definitions in target entities now also displays the artifact type [GBS] A warning will be displayed in data importer when selecting a target entity whose connected source entity is missing
• [GBS] Added explanation about order for discrete values
• [GBS] When a property, that is part of a relationship, is deleted, the corresponding relationship is also deleted
• [GBS] When creating an entity from a container, excluded properties are not shown in builders anymore
• [GBS] Separators are added to large numbers
• [GBS] System configuration now requires admin role
• [GBS] In Access Rules, when selecting an already added entity, already added properties will automatically be selected
• [GBS] "DB Query" and "Graph Query" were removed for security reasons
• [GBS] Future dates cannot be selected in Queue Jobs or Logs anymore

Bug Fixes​

• [Frontend] Open new session button in the Data Lab does not open a new tab
• [Frontend] When switching tabs, correlations in Data Lab are scaled down
• [Frontend] After applying a scope, the toggle is malfunctioning
• [Frontend] Discrete values are not correctly handled in Data Lab charts
• [Frontend] Date and range filter are malfunctioning
• [Frontend] Adding a rule group to a scope breaks the layout
• [Frontend] Filter and select is not working for data with whitespaces or hyphens
• [GBS] Opening the data table of an entity breaks layouts in other GBS views
• [GBS] Properties used in Access Rules cannot be deleted
• [GBS] Masked values in System Configuration cannot be edited
• [GBS] Clicking "Create new entities" in Data Pump container twice, shows empty list
• [GBS] Renaming an entity with properties ending on _ID, created superfluous properties
• [GBS] Fixed multiple problems when creating entities from container
• [GBS] Renaming a property that is used in multiple relationships causes an error
• [GBS] Creating a relationship without selecting a property but typing its name leads to broken relationship

Deprecations​

• [EBS] /api/graph/query/nodes endpoint will be replaced by specific endpoints in future versions and should not be used anymore. For retrieving artifacts /api/artifacts/per-nodeid endpoint was added. Related nodes can be retrieved via already existing endpoints.

Deployment Changes​

• OpenTelemetry, Jaeger and Prometheus containers were added to enable tracing and metrics of communication between services in the stack.
• All containers in the stack now run in non-root mode

Software Updates​

3^rd party software was updated to the following versions:

Name	Version
Neo4j	5.5
Apache Airflow	2.6
Keycloak	21
OpenTelemetry	0.76
Jaeger	1.42
Prometheus	2.42

What's New​

Version 2.0 includes following new features and changes.

Features​

• Worker Framework: The worker framework is the basis for using workers in the EDCH stack. Workers are isolated applications that are loosely integrated with GBS.
• Expressions: Expressions allow the dynamic definition of property values. The required Expression Worker utilizes Python based Pandas to evaluate the expressions. Other use cases are the business key and artifacts builders which allow to define keys and artifacts based on expressions.
• Graph Security Layer: The Graph Security Layer allows defining fine-grained rules for data visibility when retrieved via EBS endpoints. The rules are defined in GBS UI and are loaded by EBS. When EBS receives a graph related request it loads the corresponding rules and delegates filtering to a Graph Security specific worker.
• GBS UI: GBS UI got some small updates to be more consistent throughout all pages for example by always having button descriptions and icons. Information texts have been moved to dedicated icons to save space for displaying data.
• Import/Export Entities (Beta): A possibility to import and export target entities was added. It is possible to select which entities and dependent configurations should be exported. Exported entities can then be imported in any EDCH instance.
• Performance & Stability: A lot of changes have been made to increase performance and stability of the whole stack.

Frontend​

• Redesign: The most prominent change in the Frontend is the complete UI redesign to have a modern look & feel. It features a new left menu structure and interaction as well as improved menus for searching, adding and filtering nodes. The graph interface was improved and offers additional layout options. The new dashboard allows to easily jump into different modes and open data you recently worked on.
• Label View Layout: Performance improvements and automatic detection of sensitivity for each label view. The latter automatically sets the sensitivity to the nearest where related nodes can be found.
• Personalisation: More customization settings for graph layout were added. It is also possible to set personalized colors for labels.
• Data Lab: Data Lab was redesigned and a new charts library was integrated to further improve UI and interaction possibilities. It is possible to have multiple properties within one correlation and multiple property correlations next to each other. Important correlations can also be saved to the pinboard for later use or comparison.
• Performance & Stability: Overall performance and stability has been improved.

Breaking Changes​

Version 2.0 includes following breaking changes.

General​

• Naming conventions of Keycloak roles have changed. Therefore the license has to be re-uploaded once version 2.0 is deployed. By doing so, Keycloak roles will be recreated with correct naming.

GBS​

• Required roles on endpoints have been reworked and changed. Refer to GBS API's Swagger documentation for more information.

EBS​

• Membership related endpoints, which allowed to get and verify access tokens, where removed for security reasons. Applications requiring an access token should get one directly via Keycloak.
• Support for parameters was removed from /api/graph/query/cypher endpoint for security reasons. Since it is not possible to apply the Graph Security Layer to this endpoint it is now additionally protected by xpl-admin-execute role.
• Required roles on endpoints have been reworked and changed. Refer to EBS API's Swagger documentation for more information.

Deployment Changes​

Starting with 2.0 it is highly recommended to deploy EDCH on a Kubernetes environment. This is mandatory for automatic scaling of workers. The stack still works with a docker compose deployment but performance will suffer.

Software Updates​

3^rd party software was updated to the following versions. A mandatory update indicates that EDCH is not compatible with an older version anymore.

Name	Version	Mandatory
PostgreSQL	15.x
Neo4j	5.x
RabbitMQ	3.11.x
Apache Airflow	2.5.x	✔️
Keycloak	20.x	✔️