Entropy

Research

12 pages, 7353 KiB

Open AccessArticle

Description Generation Using Variational Auto-Encoders for Precursor microRNA

by Marko Petković and Vlado Menkovski

Entropy 2024, 26(11), 921; https://doi.org/10.3390/e26110921 - 30 Oct 2024

Viewed by 481

Micro RNAs (miRNA) are a type of non-coding RNA involved in gene regulation and can be associated with diseases such as cancer, cardiovascular, and neurological diseases. As such, identifying the entire genome of miRNA can be of great relevance. Since experimental methods for [...] Read more.

Micro RNAs (miRNA) are a type of non-coding RNA involved in gene regulation and can be associated with diseases such as cancer, cardiovascular, and neurological diseases. As such, identifying the entire genome of miRNA can be of great relevance. Since experimental methods for novel precursor miRNA (pre-miRNA) detection are complex and expensive, computational detection using Machine Learning (ML) could be useful. Existing ML methods are often complex black boxes that do not create an interpretable structural description of pre-miRNA. In this paper, we propose a novel framework that makes use of generative modeling through Variational Auto-Encoders to uncover the generative factors of pre-miRNA. After training the VAE, the pre-miRNA description is developed using a decision tree on the lower dimensional latent space. Applying the framework to miRNA classification, we obtain a high reconstruction and classification performance while also developing an accurate miRNA description. Full article

(This article belongs to the Special Issue Deep Generative Modeling: Theory and Applications)

► Show Figures

Figure 1

26 pages, 17748 KiB

Open AccessArticle

Dimma: Semi-Supervised Low-Light Image Enhancement with Adaptive Dimming

by Wojciech Kozłowski, Michał Szachniewicz, Michał Stypułkowski and Maciej Zięba

Entropy 2024, 26(9), 726; https://doi.org/10.3390/e26090726 - 26 Aug 2024

Viewed by 659

Abstract

Enhancing low-light images with natural colors poses a challenge due to camera processing variations and limited access to ground-truth lighting conditions. To address this, we propose Dimma, a semi-supervised approach that aligns with any camera using a small set of image pairs captured [...] Read more.

Enhancing low-light images with natural colors poses a challenge due to camera processing variations and limited access to ground-truth lighting conditions. To address this, we propose Dimma, a semi-supervised approach that aligns with any camera using a small set of image pairs captured under extreme lighting conditions. Our method employs a convolutional mixture density network to replicate camera-specific noise present in dark images. We enhance results further by introducing a conditional UNet architecture based on user-provided lightness values. Trained on just a few real image pairs, Dimma achieves competitive results compared to fully supervised state-of-the-art methods trained on large datasets. Full article

(This article belongs to the Special Issue Deep Generative Modeling: Theory and Applications)

► Show Figures

Figure 1

16 pages, 479 KiB

Open AccessArticle

NodeFlow: Towards End-to-End Flexible Probabilistic Regression on Tabular Data

by Patryk Wielopolski, Oleksii Furman and Maciej Zięba

Entropy 2024, 26(7), 593; https://doi.org/10.3390/e26070593 - 11 Jul 2024

Viewed by 919

Abstract

We introduce NodeFlow, a flexible framework for probabilistic regression on tabular data that combines Neural Oblivious Decision Ensembles (NODEs) and Conditional Continuous Normalizing Flows (CNFs). It offers improved modeling capabilities for arbitrary probabilistic distributions, addressing the limitations of traditional parametric approaches. In NodeFlow, [...] Read more.

We introduce NodeFlow, a flexible framework for probabilistic regression on tabular data that combines Neural Oblivious Decision Ensembles (NODEs) and Conditional Continuous Normalizing Flows (CNFs). It offers improved modeling capabilities for arbitrary probabilistic distributions, addressing the limitations of traditional parametric approaches. In NodeFlow, the NODE captures complex relationships in tabular data through a tree-like structure, while the conditional CNF utilizes the NODE’s output space as a conditioning factor. The training process of NodeFlow employs standard gradient-based learning, facilitating the end-to-end optimization of the NODEs and CNF-based density estimation. This approach ensures outstanding performance, ease of implementation, and scalability, making NodeFlow an appealing choice for practitioners and researchers. Comprehensive assessments on benchmark datasets underscore NodeFlow’s efficacy, revealing its achievement of state-of-the-art outcomes in multivariate probabilistic regression setup and its strong performance in univariate regression tasks. Furthermore, ablation studies are conducted to justify the design choices of NodeFlow. In conclusion, NodeFlow’s end-to-end training process and strong performance make it a compelling solution for practitioners and researchers. Additionally, it opens new avenues for research and application in the field of probabilistic regression on tabular data. Full article

(This article belongs to the Special Issue Deep Generative Modeling: Theory and Applications)

► Show Figures

Figure 1

18 pages, 3854 KiB

Open AccessArticle

Diffusion-Based Causal Representation Learning

by Amir Mohammad Karimi Mamaghan, Andrea Dittadi, Stefan Bauer, Karl Henrik Johansson and Francesco Quinzan

Entropy 2024, 26(7), 556; https://doi.org/10.3390/e26070556 - 28 Jun 2024

Viewed by 1242

Abstract

Causal reasoning can be considered a cornerstone of intelligent systems. Having access to an underlying causal graph comes with the promise of cause–effect estimation and the identification of efficient and safe interventions. However, learning causal representations remains a major challenge, due to the [...] Read more.

Causal reasoning can be considered a cornerstone of intelligent systems. Having access to an underlying causal graph comes with the promise of cause–effect estimation and the identification of efficient and safe interventions. However, learning causal representations remains a major challenge, due to the complexity of many real-world systems. Previous works on causal representation learning have mostly focused on Variational Auto-Encoders (VAEs). These methods only provide representations from a point estimate, and they are less effective at handling high dimensions. To overcome these problems, we propose a Diffusion-based Causal Representation Learning (DCRL) framework which uses diffusion-based representations for causal discovery in the latent space. DCRL provides access to both single-dimensional and infinite-dimensional latent codes, which encode different levels of information. In a first proof of principle, we investigate the use of DCRL for causal representation learning in a weakly supervised setting. We further demonstrate experimentally that this approach performs comparably well in identifying the latent causal structure and causal variables. Full article

(This article belongs to the Special Issue Deep Generative Modeling: Theory and Applications)

► Show Figures

Figure 1

15 pages, 1153 KiB

Open AccessArticle

In Search of Dispersed Memories: Generative Diffusion Models Are Associative Memory Networks

by Luca Ambrogioni

Entropy 2024, 26(5), 381; https://doi.org/10.3390/e26050381 - 29 Apr 2024

Cited by 11 | Viewed by 3422

Abstract

Uncovering the mechanisms behind long-term memory is one of the most fascinating open problems in neuroscience and artificial intelligence. Artificial associative memory networks have been used to formalize important aspects of biological memory. Generative diffusion models are a type of generative machine learning [...] Read more.

Uncovering the mechanisms behind long-term memory is one of the most fascinating open problems in neuroscience and artificial intelligence. Artificial associative memory networks have been used to formalize important aspects of biological memory. Generative diffusion models are a type of generative machine learning techniques that have shown great performance in many tasks. Similar to associative memory systems, these networks define a dynamical system that converges to a set of target states. In this work, we show that generative diffusion models can be interpreted as energy-based models and that, when trained on discrete patterns, their energy function is (asymptotically) identical to that of modern Hopfield networks. This equivalence allows us to interpret the supervised training of diffusion models as a synaptic learning process that encodes the associative dynamics of a modern Hopfield network in the weight structure of a deep neural network. Leveraging this connection, we formulate a generalized framework for understanding the formation of long-term memory, where creative generation and memory recall can be seen as parts of a unified continuum. Full article

(This article belongs to the Special Issue Deep Generative Modeling: Theory and Applications)

► Show Figures

Figure 1

38 pages, 19811 KiB

Open AccessArticle

Multi-Modal Latent Diffusion

by Mustapha Bounoua, Giulio Franzese and Pietro Michiardi

Entropy 2024, 26(4), 320; https://doi.org/10.3390/e26040320 - 5 Apr 2024

Cited by 5 | Viewed by 1874

Abstract

Multimodal datasets are ubiquitous in modern applications, and multimodal Variational Autoencoders are a popular family of models that aim to learn a joint representation of different modalities. However, existing approaches suffer from a coherence–quality tradeoff in which models with good generation quality lack [...] Read more.

Multimodal datasets are ubiquitous in modern applications, and multimodal Variational Autoencoders are a popular family of models that aim to learn a joint representation of different modalities. However, existing approaches suffer from a coherence–quality tradeoff in which models with good generation quality lack generative coherence across modalities and vice versa. In this paper, we discuss the limitations underlying the unsatisfactory performance of existing methods in order to motivate the need for a different approach. We propose a novel method that uses a set of independently trained and unimodal deterministic autoencoders. Individual latent variables are concatenated into a common latent space, which is then fed to a masked diffusion model to enable generative modeling. We introduce a new multi-time training method to learn the conditional score network for multimodal diffusion. Our methodology substantially outperforms competitors in both generation quality and coherence, as shown through an extensive experimental campaign. Full article

(This article belongs to the Special Issue Deep Generative Modeling: Theory and Applications)

► Show Figures

Figure 1

24 pages, 1977 KiB

Open AccessArticle

Semi-Supervised Variational Autoencoders for Out-of-Distribution Generation

by Frantzeska Lavda and Alexandros Kalousis

Entropy 2023, 25(12), 1659; https://doi.org/10.3390/e25121659 - 14 Dec 2023

Viewed by 1335

Abstract

Humans are able to quickly adapt to new situations, learn effectively with limited data, and create unique combinations of basic concepts. In contrast, generalizing out-of-distribution (OOD) data and achieving combinatorial generalizations are fundamental challenges for machine learning models. Moreover, obtaining high-quality labeled examples [...] Read more.

Humans are able to quickly adapt to new situations, learn effectively with limited data, and create unique combinations of basic concepts. In contrast, generalizing out-of-distribution (OOD) data and achieving combinatorial generalizations are fundamental challenges for machine learning models. Moreover, obtaining high-quality labeled examples can be very time-consuming and expensive, particularly when specialized skills are required for labeling. To address these issues, we propose BtVAE, a method that utilizes conditional VAE models to achieve combinatorial generalization in certain scenarios and consequently to generate out-of-distribution (OOD) data in a semi-supervised manner. Unlike previous approaches that use new factors of variation during testing, our method uses only existing attributes from the training data but in ways that were not seen during training (e.g., small objects of a specific shape during training and large objects of the same shape during testing). Full article

(This article belongs to the Special Issue Deep Generative Modeling: Theory and Applications)

► Show Figures

Figure 1

22 pages, 18479 KiB

Open AccessArticle

Diffusion Probabilistic Modeling for Video Generation

by Ruihan Yang, Prakhar Srivastava and Stephan Mandt

Entropy 2023, 25(10), 1469; https://doi.org/10.3390/e25101469 - 20 Oct 2023

Cited by 125 | Viewed by 5538

Abstract

Denoising diffusion probabilistic models are a promising new class of generative models that mark a milestone in high-quality image generation. This paper showcases their ability to sequentially generate video, surpassing prior methods in perceptual and probabilistic forecasting metrics. We propose an autoregressive, end-to-end [...] Read more.

Denoising diffusion probabilistic models are a promising new class of generative models that mark a milestone in high-quality image generation. This paper showcases their ability to sequentially generate video, surpassing prior methods in perceptual and probabilistic forecasting metrics. We propose an autoregressive, end-to-end optimized video diffusion model inspired by recent advances in neural video compression. The model successively generates future frames by correcting a deterministic next-frame prediction using a stochastic residual generated by an inverse diffusion process. We compare this approach against six baselines on four datasets involving natural and simulation-based videos. We find significant improvements in terms of perceptual quality and probabilistic frame forecasting ability for all datasets. Full article

(This article belongs to the Special Issue Deep Generative Modeling: Theory and Applications)

► Show Figures

Figure 1

18 pages, 4606 KiB

Open AccessArticle

Learning Energy-Based Models in High-Dimensional Spaces with Multiscale Denoising-Score Matching

by Zengyi Li, Yubei Chen and Friedrich T. Sommer

Entropy 2023, 25(10), 1367; https://doi.org/10.3390/e25101367 - 22 Sep 2023

Cited by 1 | Viewed by 2184

Abstract

Energy-based models (EBMs) assign an unnormalized log probability to data samples. This functionality has a variety of applications, such as sample synthesis, data denoising, sample restoration, outlier detection, Bayesian reasoning and many more. But, the training of EBMs using standard maximum likelihood is [...] Read more.

Energy-based models (EBMs) assign an unnormalized log probability to data samples. This functionality has a variety of applications, such as sample synthesis, data denoising, sample restoration, outlier detection, Bayesian reasoning and many more. But, the training of EBMs using standard maximum likelihood is extremely slow because it requires sampling from the model distribution. Score matching potentially alleviates this problem. In particular, denoising-score matching has been successfully used to train EBMs. Using noisy data samples with one fixed noise level, these models learn fast and yield good results in data denoising. However, demonstrations of such models in the high-quality sample synthesis of high-dimensional data were lacking. Recently, a paper showed that a generative model trained by denoising-score matching accomplishes excellent sample synthesis when trained with data samples corrupted with multiple levels of noise. Here we provide an analysis and empirical evidence showing that training with multiple noise levels is necessary when the data dimension is high. Leveraging this insight, we propose a novel EBM trained with multiscale denoising-score matching. Our model exhibits a data-generation performance comparable to state-of-the-art techniques such as GANs and sets a new baseline for EBMs. The proposed model also provides density information and performs well on an image-inpainting task. Full article

(This article belongs to the Special Issue Deep Generative Modeling: Theory and Applications)

► Show Figures

Figure 1

30 pages, 6120 KiB

Open AccessArticle

How Much Is Enough? A Study on Diffusion Times in Score-Based Generative Models

by Giulio Franzese, Simone Rossi, Lixuan Yang, Alessandro Finamore, Dario Rossi, Maurizio Filippone and Pietro Michiardi

Entropy 2023, 25(4), 633; https://doi.org/10.3390/e25040633 - 7 Apr 2023

Cited by 16 | Viewed by 4444

Abstract

Score-based diffusion models are a class of generative models whose dynamics is described by stochastic differential equations that map noise into data. While recent works have started to lay down a theoretical foundation for these models, a detailed understanding of the role of [...] Read more.

Score-based diffusion models are a class of generative models whose dynamics is described by stochastic differential equations that map noise into data. While recent works have started to lay down a theoretical foundation for these models, a detailed understanding of the role of the diffusion time T is still lacking. Current best practice advocates for a large T to ensure that the forward dynamics brings the diffusion sufficiently close to a known and simple noise distribution; however, a smaller value of T should be preferred for a better approximation of the score-matching objective and higher computational efficiency. Starting from a variational interpretation of diffusion models, in this work we quantify this trade-off and suggest a new method to improve quality and efficiency of both training and sampling, by adopting smaller diffusion times. Indeed, we show how an auxiliary model can be used to bridge the gap between the ideal and the simulated forward dynamics, followed by a standard reverse diffusion process. Empirical results support our analysis; for image data, our method is competitive with regard to the state of the art, according to standard sample quality metrics and log-likelihood. Full article

(This article belongs to the Special Issue Deep Generative Modeling: Theory and Applications)

► Show Figures

Figure 1

Journal Menu

Journal Browser

Deep Generative Modeling: Theory and Applications

Share This Special Issue

Special Issue Editor

Special Issue Information

Keywords

Benefits of Publishing in a Special Issue

Published Papers (10 papers)

Research

Further Information

Guidelines

MDPI Initiatives

Follow MDPI