Latent Inspector: An Interactive Tool for Probing Neural Network Behaviors Through Arbitrary Latent Activation

1 Introduction

The steadily increasing amount of available data and the accompanying rise in model complexity makes it ever more challenging to understand and interpret the behaviors of machine learning models. Apart from the necessary design and training of machine learning models, verification and validation should not be neglected in the development workflow to ensure the trained models behave as the architects have intended. The intrinsic model property to behave like a black box, where input data is supplied and a specific output is given as result, requires proper methods to open the model in a way that a user can thoroughly probe the inner details. [Mohseni et al., 2021; Samek et al., 2017]

While there are other tools revealing the explainability and transparency such as CAM [Jung and Oh, 2021] or Tensorboard [Abadi et al., 2015], there lacks a tool to trigger posttraining models with arbitrary input or latent activation values, especially those outside the scope of the training data. In real-world applications, behaviors of those triggers can be important as real-world data can be unexpected, with faulty data sources, or with external factors that might interfere with the data integrity of the computing devices [Ziegler, 1996].

We introduce Latent Inspector, an interactive tool to circumvent the black box approach by disassembling the model into its fundamental parts without impairing functionality. Figure 1 shows the general pipeline for developing and analyzing Neural Networks in comparison to our adapted Latent Inspector pipeline. Instead of retrieving the necessary information in form of logging during the training process, our approach starts post-training as an inference intermediary. The tool dynamically adapts and actively works with the trained model by extracting, modifying, and processing the model's layer-based architecture and the available data. We provide several concepts to visualize the high-dimensional data processing in human-perceptible, two-dimensional plots. Furthermore, the user has the option to directly interact with the model through the tool, which enables extended functionalities like out-of-bounds inspection, fine-tuning layer weights, and data generation.

2 Functionality and Use Case

To offer potential users the possibility to probe their trained model in the Latent Inspector, we designed the tool as an active framework with a focus on generality and versatility. Together with a straightforward user interface and a plain file structure, most of the steps that are typically necessary to examine a custom model in this scope were transferred from the user to automatic and dynamic adapting functions inside the tool. In the following, implemented functionalities and potential use cases are introduced together with figure 2.

2.1 Functionality

The user interface follows a common and intuitive design layout containing fundamental controls in the sidebar and a grid system in the main window for interactive visualizations. After a short and simple setup process in which the user adds the necessary files to the designated folder in form of a new project, the Latent Inspector can directly be launched due to its ability of dynamically adapting to the provided files.

Dissecting the neural network model and accessing the architecture is a key element of our tool to realize the idea of inspecting every latent layer individually. The tool creates a list of the present layers from which the user may arbitrarily select one for inspection. An iterative process for retrieving layer properties allows the creation of sub-models based on the user selection. As described in figure 2, the model is split into two parts, one containing the preceding layers of the selection and the other containing the remaining part of the model. To maintain the sub-models fundamental operation, each layer is cloned from the original network to preserve the available parameters.

The user-provided data set is propagated through the first sub-network to obtain the input shape of the inspected latent layer. Usually, this preliminary output is further propagated to receive the final prediction, whereas, for the Latent Inspector, this output is utilized for visualizing the outcome as the latent space in a scatter plot on the left side of the main screen. If the user provides additional ground truth data within the project, each dot of the plot representing a data point is additionally colorized to highlight data distribution. The tool provides common unsupervised dimension reduction techniques like PCA, UMAP, or TSNE from the sklearn library to decrease the high dimensional output to displayable two dimensions [Pedregosa et al., 2011]. Due to our universal approach, each dimension reduction technology aims to provide the best data representation by self-optimizing its parameters, considering the accompanying loss of information when transforming into lower dimensions.

To realize our novel feature to interactively work with the network, the latent space plot is designed to handle user click events. If the selected dimension reduction algorithm supports an inverse transformation, for instance, PCA and UMAP, the coordinates of the created data point are reversely scaled up into the inspected layer dimension and propagated through the second sub-network containing the remaining layers. The outcome of this process constitutes the model prediction of the user click event, which can be visualized through several predefined or customized components.

2.2 Use Case

A major difference between Latent Inspector and existing latent space visualizers is that our tool hosts the trained model and activates it with user-selected embedding values. Thus our tool is an active framework, which others like Tensorboard using only log file visualizations cannot achieve [Abadi et al., 2015]. The user can trigger the network's output from any value at any latent layer through the interactive UI, resulting in the ability for out-of-bounds inspection to verify unknown or even impossible data points of real-world applications. Furthermore, we provide the option to save the propagated data points to files for the creation of new data sets or the refinement of existing ones.

To constitute the full potential of Latent Inspector, we demonstrate the functionality of data generation with user interactions in the case of classification based on an autoencoder feature extractor from the MNIST dataset [Bank et al., 2020; Deng, 2012]. As emphasized with the green color in figure 2, our tool provides an extended feature to train a Logistic Regression classifier based on the inspected layer. The user needs to provide enough training and validation data beforehand to ensure proper classification. Again, our tool automatically adapts to the provided dataset with the help of the sklearn library, allowing either binary or multi-class classification [Pedregosa et al., 2011]. For the MNIST use case scenario, the user input of the click event is propagated through the autoencoder to reconstruct the image of a handwritten number as well as through the trained classifier to label the autoencoder's output. Further, the user may manually modify the classifier label output via a text box if the prediction is incorrect, or to add new labels like for instance "not a number". The final result is stored in the tools project folder as a labeled dataset.

Additionally to the MNIST project, we added the PAMAP

[Reiss and Stricker, 2012] and MoCaPose [Zhou et al., 2023] projects as examples to validate our generalized and versatile approach. Apart from the introduced features, Latent Inspector is designed to be expanded with more functionalities due to its modular architecture.

3 System Structure

To achieve the above-mentioned functionalities and to keep Latent Inspector open for additional features, we decided to build the system with a common front-end and back-end architecture. This separation circumvents the problem of crossorigin resource sharing (CORS) policies when accessing the file system directly through the front end, and further creates the possibility of running the back end on outsourced systems to match possible performance requirements.

The back end is based on Python and uses the Flask API framework to communicate with the front end through the IP network [Grinberg, 2018]. A new project is added to the platform's file system as a new folder, containing data sets, the model to inspect, and additional outsourced functions to allow custom data processing and visualizations. For the moment, Latent Inspector accepts file formats of .json, .npy, .mat, and .csv for the data sets and .h5 files for the network model. Every arising processing is done in the back end to minimize the front-ends performance requirement.

The front end uses Node.js along with Typescript as a basis and the PlotlyJS library for our main components to visualize data through interactive plots [Inc., 2015]. All together form a responsive and fluently scaling web application.

As shown in figure 3, Latent Explorer is divided into three parts following the idea of the above-mentioned functionalities: Starting with the setup process, the back end retrieves and forwards project information from the file system to the front end. After the user selection of the project and layer to inspect, the preparation process starts by splitting the model, propagating the dataset through the first sub-model, and reducing the dimensions with one of the provided techniques for visualization. Lastly, after the initial visualization is done, the user can interact with the model by applying inverse dimension reduction and propagating the user input through the remaining model layers.

All parts in the system have to follow one another, however, it is possible to jump back or iterate through parts of the system again if the user triggers another click event or modifies the current selections. Some minor features of Latent Inspector that do not affect the basic workflow are removed in figure 3 to resolve complexity.

The whole project is designed to be bundled into executable files. For the front end, we are using Electron as a wrapper around the web application to generate a crossplatform desktop application, and for the back end, we are using the python-based pyinstaller library. All relevant libraries to run the system are packed inside the executable to enable running on platforms without an integrated development environment.

4 Limitations and Outlook

4.1 Limitations

For the moment, Latent Inspector supports models with regression, classification, or combinational outputs. Since the tool needs to split the model at the inspected layer, it is only applicable for models with a causal, linear relationship of sequential layers. Models with branched connections like for instance the U-Net architecture with multiple direct input-tooutput layer connections are rejected by the tool. However, models with localized branched connections such as ResNet may still be used since Latent Inspector can unite it as one layer without accessing the depth of the branches.

From the technical side, we require the model to be defined through the Keras Framework and saved in a .h5 file with all its trained weights [Chollet and others, 2015]. Furthermore, the loss of information when applying dimensional reduction techniques should be considered at any time. For PCA, the accuracy is shown in the UI and can drop rapidly if strongly high dimensional data sets are utilized. Speaking about the dataset, there is a noticeable loss in the Latent Inspector's front-end performance when aiming to plot more than 10,000 data points due to the high plotting precision that is currently applied.

4.2 Outlook

In the future, we will continue our generalized and versatile approach to support more machine learning architectures and development frameworks. Together with our modular system architecture, we plan to extend our tool with additional features like for instance interactive loss manipulation by dragging data points in the latent space [Wei et al., 2022].