Kubeflow

Kubeflow 1.10 Release Announcement

2025-03-26T00:00:00-05:00

Kubeflow 1.10.0 delivers essential updates that enhance the flexibility, efficiency, and scalability of machine learning workflows. The new features span across several components, improving both user experience and system performance.

Highlight features

Trainer 2.0
New UI for Model Registry
Spark Operator as a core Kubeflow component
Kubernetes and container security (CISO compatibility)
Hyperparameter Optimization for LLMs Fine-Tuning
Loop parallelism in Pipelines
New parameter distributions for Katib
Deeper Model Registry integrations with KServe
New Python SDK, OCI storage, and model caching for KServe
New security contexts and rootless Istio-CNI integrations for Spark Operator

Kubeflow Platform (Manifests & Security)

The Kubeflow Platform Working Group focuses on simplifying Kubeflow installation, operations, and security. See details below.

Manifests:

Spark Operator 2.1.0 included in Kubeflow platform, although not installed yet by default
Documentation updates that make it easier to install, extend and upgrade Kubeflow
For more details and future plans please consult the 1.10.0 and 1.10.1/1.11.0 milestones

Notebooks	Dashboard	Pipelines	Katib	Trainer	KServe	Model Registry	Spark
1.10	1.10	2.4.1	0.18	1.9	0.14	0.2.15	2.1.0

Kubernetes	Kind	Kustomize	Cert Manager	Knative	Istio	Dex	Oauth2-proxy
1.31-1.33	0.26	5.4.3	1.16.1	1.16	1.24	2.41	7.7

Security:

CVE reductions - regular scanning with trivy
Kubernetes and container security best practices:
- Rootless containers / PodSecurityStandards restricted for: Istio-CNI, Knative, Dex, Oauth2-proxy, Spark
- 50 % done: KFP, Notebooks / Workspaces, Katib, Trainer, Kserve, …
- Istio-CNI as default for rootless Kubeflow postponed to 1.10.1
OIDC-authservice has been replaced by oauth2-proxy
Oauth2-proxy and Dex documentation for external OIDC authentication (Keycloak, and OIDC providers such as Azure, Google etc.)

Trivy CVE scans March 25 2025:

Working Group	Images	Critical CVE	High CVE	Medium CVE	Low CVE
Katib	17	11	101	417	734
Pipelines	15	57	490	4030	1922
Workbenches(Notebooks)	12	12	59	179	224
Kserve	16	21	305	6803	1588
Manifests	14	8	4	94	52
Trainer	1	0	0	1	0
Model Registry	6	1	13	153	188
Spark	1	5	37	1640	141
All Images	81	115	1009	13275	4804

Pipelines

Support for Placeholders in Resource Limits

Kubeflow Pipelines 2.4.1 introduces support for placeholders in resource limits, enhancing flexibility in pipeline execution.This update allows users to define dynamic resource limits using parameterized values, enabling more adaptable and reusable pipeline definitions.

Support for Loop Parallelism

Kubeflow Pipelines 2.4.1 introduces a new Parallelism Limit for ParallelFor tasks, giving users the ability to run massively parallel inference pipelines, with more control over parallel execution in their workflows. This feature allows users to specify the maximum number of parallel iterations, preventing resource overutilization and improving system stability. When running large pipelines with GPUs, proper use of this feature could save your team thousands of dollars in compute expenses.

Implement SubDAG Output Resolution

Kubeflow 1.10 ensures that pipelines using nested DAGs work correctly and reliably when treated as components. Outputs from deeply nested DAGs will now resolve properly, avoiding broken dependencies.

Model Registry

Model Registry introduces a new user interface and enhanced model management capabilities.

Model Registry UI

The new Kubeflow Model Registry UI provides a user-friendly web interface for managing machine learning models within the Kubeflow platform. It centralizes model metadata, version tracking, and artifact management, streamlining MLOps workflows.

Key features include:

Easy model registration with custom metadata
Comprehensive model management with filtering and sorting
Archiving capabilities
Version control
Metadata editing

The UI interacts with the Model Registry’s REST API, making it accessible to users of all technical backgrounds and enhancing collaboration across data science, ML engineering, and MLOps teams.

To get started with the Model Registry UI, which is currently in Alpha, you can follow the instructions here.

The Kubeflow Model Registry UI Team would like to conduct user research to identify possible enhancements we can contribute in future iterations of the Kubeflow Model Registry UI. If you are interested in participating in this study, please fill out this survey.

Custom Storage Initializer

The Model Registry Custom Storage Initializer (CSI) is a custom implementation of the KServe ClusterStorageContainer. This feature allows users to utilize Model Registry metadata to download and deploy models efficiently. With the newest release of the Model Registry, it is now possible to install and use the Custom Storage Initializer (CSI).

You can find detailed installation instructions and a small example in the “Getting Started” section of the Model Registry component on the Kubeflow website.

For additional information and future developments towards better integration with KServe, you can refer to the slides here.

Training Operator (Trainer) & Katib

Kubeflow 1.10 enhances the Training Operator and Katib, providing new tools and APIs for hyperparameter optimization, particularly for large language models.

Moreover, the Kubeflow Training Operator now supports JAX for distributed training, enabling users to leverage JAX’s capabilities for efficient and scalable model training.

Finally, if you want to get involved with Trainer V2, take a look at this KEP and issue.

Hyperparameter Optimization API for LLMs

Katib introduces a new high-level API for hyperparameter tuning, streamlining LLMOps workflows in Kubernetes. This API integrates Katib and the Training Operator to automate hyperparameter optimization, reducing manual effort for data scientists fine-tuning large language models. For more information, refer to the feature release blog post.

Support for Various Parameter Distributions

Katib now adds support for multiple probability distributions. Previously limited to uniform distributions, Katib now supports log-uniform, normal, and log-normal distributions, providing data scientists with greater flexibility in tuning hyperparameters. This is particularly useful for parameters like learning rates, which benefit from log-uniform sampling, or values expected to vary around a mean, suited for normal distributions.

Push-Based Metrics Collection

Katib now allows users to push metrics to Katib DB directly. The new push-based design provides administrative and performanace improvements to the existing pull based design. For further details, please refer to the Push-Based Metrics Collection blog post.

Dashboard & Notebooks

Kubeflow 1.10 improves the observability and usability of Notebooks, while providing updated default images.

Prometheus Metrics for Notebooks

Both the Notebooks component and CRUD backends now feature Prometheus metrics. Notebooks expose custom metrics using the prom-client library, and CRUD backends utilize the prometheus_flask_exporter library. This ensures consistent metrics integration across all backend services.

More Descriptive Error Messages

Error messages for notebook creation failures due to resource constraints are now more descriptive. Users can quickly identify issues such as insufficient resources.

Spark Operator

The Spark Operator, now integrated as a core Kubeflow component, includes several key enhancements focusing on architecture, security, and performance:

Rebuilt with Controller Runtime (v2.0.0): Modernized core architecture using controller-runtime, aligning with Kubernetes controller patterns for improved structure, extensibility, and testability.
YuniKorn Gang Scheduling Support (v2.0.0): Enables efficient scheduling of Spark driver & executor pods as a group, ideal for large-scale data pipelines with resource guarantees.
Enhanced Security Contexts & SeccompProfile Support (v2.1.1): Adds support for seccompProfile: RuntimeDefault & readOnlyRootFilesystem, aligning with Kubernetes Pod Security Standards and minimizing security risk.

KServe

KServe v0.14.1 introduces several essential features that enhance its capabilities for deploying and managing machine learning models.

New Python SDK

The release includes a new Python SDK with both REST and GRPC inference clients, offering asynchronous support and the ability to handle tensor data in binary format.

OCI Storage for Models

OCI storage for models has also been promoted to a stable feature, with improvements to stability by configuring OCI models as init containers.

Model Cache Feature

Additionally, the introduction of the Model Cache feature leverages local node storage to reduce model load times, especially for large models, enhancing scalability.

Hugging Face Integration

KServe v0.14.1 further expands integration with Hugging Face, enabling direct model deployment from the Hugbing Face hub via a new hf:// URI schema.

What comes next?

If you want to take a peek into the Kubeflow 1.11 roadmap planning and contribute with your ideas, see Notebooks, Manifests & Security, Pipelines, Model Registry, Katib, Training Operator.

How to get started with 1.10

Visit the Kubeflow 1.10 release page or head over to the Getting Started and Support pages.

Join the Community

We would like to thank everyone for the contribution to Kubeflow 1.10, especially Ricardo Martinelli De Oliveira for his work as the v1.10 Release Manager, all the release team and the working group leads, who relentlessly dedicate their time to this great project.

Release team members : Ricardo Martinelli De Oliveira, Dimitris Poulopoulos, Matteo Mortari, Julius von Kohout Valentina Rodriguez Sosa, Helber Belmiro, Vraj Bhatt, Diego Lovison, Dagvanorov Lkhagvajav, Sailesh Duddupudi, Manos Vlassis, Tarek Abouzeid, Milos Grubjesic

Working Group leads : Andrey Velichkevich, Julius von Kohout, Mathew Wicks, …

Kubeflow Steering Committee : Andrey Velichkevich, Julius von Kohout, Yuan Tang, Johnu George, Francisco Javier Araceo

Participating Distributions : Charmed Kubeflow (Canonical), Nutanix, OpenShift AI (RedHat), QBO

You can find more details about Kubeflow distributions here.

Want to help?

The Kubeflow community Working Groups hold open meetings and are always looking for more volunteers and users to unlock the potential of machine learning. If you’re interested in becoming a Kubeflow contributor, please feel free to check out the resources below. We look forward to working with you!

Visit our Kubeflow website or Kubeflow GitHub Page.
Join the Kubeflow Slack channel.
Join the kubeflow-discuss mailing list.
Attend our weekly community meeting.

🚀 Announcing the Kubeflow Spark Operator Benchmarking Results

2025-03-15T00:00:00-05:00

Kubernetes has become the go-to platform for running large-scale Apache Spark workloads. But as workloads scale, how do you ensure your Spark jobs run efficiently without hitting bottlenecks? Managing thousands of concurrent Spark jobs can introduce severe performance challenges—from CPU saturation in the Spark Operator to Kubernetes API slowdowns and job scheduling inefficiencies.

To address these challenges, we are excited to introduce the Kubeflow Spark Operator Benchmarking Results and Toolkit—a comprehensive framework to analyze performance, pinpoint bottlenecks, and optimize your Spark on Kubernetes deployments.

🔍 What’s Included?

This benchmarking effort provides three key outcomes to help you take full control of your Spark on Kubernetes deployment:

✅ Benchmarking Results – A detailed evaluation of performance insights and tuning recommendations for large-scale Spark workloads.
🛠 Benchmarking Test Toolkit – A fully reproducible test suite to help users evaluate their own Spark Operator performance and validate improvements.
📊 Open-Sourced Grafana Dashboard – A battle-tested visualization tool designed specifically to track large-scale Spark Operator deployments, providing real-time monitoring of job processing efficiency, API latencies, and system health.

❌ The Challenges: Why Benchmarking Matters

Running thousands of Spark jobs on Kubernetes at scale uncovers several performance roadblocks that can cripple efficiency if left unresolved:

🚦 Spark Operator Becomes CPU-Bound: When handling thousands of Spark jobs, the controller pod maxes out CPU resources, limiting job submission rates.
🐢 High API Server Latency: As workloads scale, Kubernetes API responsiveness degrades—job status updates slow down, affecting observability and scheduling efficiency.
🕒 Webhook Overhead Slows Job Starts: Using webhooks adds ~60 seconds of extra latency per job, reducing throughput in high-concurrency environments.
💥 Namespace Overload Causes Failures: Running 6,000+ SparkApplications in a single namespace resulted in pod failures due to excessive environment variables and service object overload.

💡 So, how do you fix these issues and optimize your Spark Operator deployment?
That’s where our benchmarking results and toolkit come in.

🛠 Tuning Best Practices for Spark Operator

Based on our benchmarking findings, we provide clear, actionable recommendations for improving Spark Operator performance at scale.

If you’re running thousands of concurrent Spark jobs, here’s what you need to do:

Deploy Multiple Spark Operator Instances

💡 Why? A single Spark Operator instance struggles to keep up with high job submission rates.
✅ Solution: When a single Spark Operator instance struggles with high job submission rates, leading to CPU saturation and slower job launches, deploying multiple instances can help. Distribute the workload by assigning different namespaces to each instance. For example, one instance can manage `20 namespaces while another handles a separate set of 20 namespaces. This prevents bottlenecks and ensures efficient Spark job execution.

Disable Webhooks for Faster Job Starts

💡 Why? Webhooks introduce ~60 seconds of delay per job due to validation and mutation overhead, reducing throughput in large workloads. ✅ Solution: Instead of using webhooks for volume mounts, node selectors, or taints, define Spark Pod Templates directly within the Spark job definition—no additional files are needed. Disable webhooks by setting webhook.enable=false in the Helm chart.

Increase Controller Workers

💡 Why? By default, the operator runs with 10 controller workers, but our benchmarks showed increasing this to 20 or 30 workers improved job throughput.
✅ Solution: Set controller.workers=20 if your Operator pod runs on a 36-core CPU or higher to enable faster parallel job execution. For larger workloads (e.g., 72+ cores), increase to 40+ workers for better parallel job execution.

Enable a Batch Scheduler (Volcano / YuniKorn)

💡 Why? Kubernetes’ default scheduler isn’t optimized for batch workloads, leading to inefficient job placements.
✅ Solution: Enable Volcano or YuniKorn (batchScheduler.enable=true) to optimize job scheduling. These schedulers provide gang scheduling, queue management, and multi-tenant resource sharing. Benchmarks show that Apache YuniKorn schedules jobs faster than the default Kubernetes scheduler.

Optimize API Server Scaling

💡 Why? API server latency spikes to 600ms+ under heavy load, affecting Spark job responsiveness.
✅ Solution: Scale API server replicas, allocate more CPU and memory, and optimize event handling. Ensure your Kubernetes API server and etcd auto-scale to handle bursty workloads efficiently. Monitor kube-apiserver metrics and scale etcd accordingly. If running thousands of Spark pods, consider manually increasing control plane node sizes.

Distribute Spark Jobs Across Multiple Namespaces

💡 Why? Running too many jobs in a single namespace causes environment variable overflows, leading to pod failures.
✅ Solution: When too many pods are placed in a single namespace, operations like listing or modifying resources can generate large API server responses, increasing latency. For example, retrieving all pods may result in a substantial size in response, consuming significant server resources. Additionally, etcd, Kubernetes’ key-value store, can become a bottleneck when handling frequent updates from a high number of pods in one namespace. Heavy read and write operations can strain etcd, causing increased latencies and potential timeouts. To improve performance and stability, it is recommended to distribute workloads across multiple namespaces.

Monitor & Tune Using the Open-Source Grafana Dashboard

💡 Why? Observability is key to identifying performance bottlenecks.
✅ Solution: Use our Spark Operator Scale Test Dashboard to track job submission rates, API latencies, and CPU utilization in real time.

📖 Learn More & Get Started

The Kubeflow Spark Operator Benchmarking Results and Toolkit provide an in-depth performance playbook for running Spark at scale on Kubernetes. Whether you’re troubleshooting an existing deployment or planning for future growth, this toolkit arms you with data-driven insights and best practices for success.

🚀 Ready to optimize your Spark workloads? Dive into the full results and toolkit below:
📖 Kubeflow Spark Operator Benchmarks

Optimizing RAG Pipelines with Katib: Hyperparameter Tuning for Better Retrieval & Generation

2025-02-21T00:00:00-06:00

Introduction

As artificial intelligence and machine learning models become more sophisticated, optimising their performance remains a critical challenge. Kubeflow provides a robust component, Katib, designed for hyperparameter optimization and neural architecture search. As a part of the Kubeflow ecosystem, Katib enables scalable, automated tuning of underlying machine learning models, reducing the manual effort required for parameter selection while improving model performance across diverse ML workflows.

With Retrieval-Augmented Generation (RAG) becoming an increasingly popular approach for improving search and retrieval quality, optimizing its parameters is essential to achieving high-quality results. RAG pipelines involve multiple hyperparameters that influence retrieval accuracy, hallucination reduction, and language generation quality. In this blog, we will explore how Katib can be leveraged to fine-tune a RAG pipeline, ensuring optimal performance by systematically adjusting key hyperparameters.

Let’s Get Started!

STEP 1: Setup

Since compute resources are scarcer than a perfectly labeled dataset :), we’ll use a lightweight Kind cluster (Kubernetes in Docker) cluster to run this example locally. Rest assured, this setup can seamlessly scale to larger clusters by increasing the dataset size and the number of hyperparameters to tune.

To get started, we’ll first install the Katib control plane in our cluster by following the steps outlined in the documentation.

STEP 2: Implementing RAG pipeline

In this implementation, we use a retriever model, which encodes queries and documents into vector representations to find the most relevant matches, to fetch relevant documents based on a query and a generator model to produce coherent text responses.

Implementation Details:

Retriever: Sentence Transformer & FAISS (Facebook AI Similarity Search) Index
- A SentenceTransformer model (paraphrase-MiniLM-L6-v2) encodes predefined documents into vector representations.
- FAISS is used to index these document embeddings and perform efficient similarity searches to retrieve the most relevant documents.
Generator: Pre-trained GPT-2 Model
- A Hugging Face GPT-2 text generation pipeline (which can be replaced with any other model) is used to generate responses based on the retrieved documents. I chose GPT-2 for this example as it is lightweight enough to run on my local machine while still generating coherent responses.
Query Processing & Response Generation
- When a query is submitted, the retriever encodes it and searches the FAISS index for the top-k most similar documents.
- These retrieved documents are concatenated to form the input context, which is then passed to the GPT-2 model to generate a response.
Evaluation: BLEU (Bilingual Evaluation Understudy) Score Calculation
- To assess the quality of generated responses, we use the BLEU score, a popular metric for evaluating text generation.
- The evaluate function takes a query, retrieves documents, generates a response, and compares it against a ground-truth reference to compute a BLEU score with smoothing functions from the nltk library.

To run Katib, we will use the Katib SDK, which provides a programmatic interface for defining and running hyperparameter tuning experiments in Kubeflow.

Katib requires an objective function, which:

Defines what we want to optimize (e.g., BLEU score for text generation quality).
Executes the RAG pipeline with different hyperparameter values.
Returns an evaluation metric so Katib can compare different hyperparameter configurations.

def objective(parameters):
    # Import dependencies inside the function (required for Katib)
    import numpy as np
    import faiss
    from sentence_transformers import SentenceTransformer
    from transformers import pipeline
    from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction
    
    # Function to fetch documents (Modify as needed)
    def fetch_documents():
        """Returns a predefined list of documents or loads them from a file."""
        return [
            ...
        ]
        # OR, to load from a file:
        # with open("/path/to/documents.json", "r") as f:
        #     return json.load(f)

    # Define the RAG pipeline within the function
    def rag_pipeline_execute(query, top_k, temperature):
        """Retrieves relevant documents and generates a response using GPT-2."""

        # Initialize retriever
        retriever_model = SentenceTransformer("paraphrase-MiniLM-L6-v2")

        # Sample documents
        documents = fetch_documents()

        # Encode documents
        doc_embeddings = retriever_model.encode(documents)
        index = faiss.IndexFlatL2(doc_embeddings.shape[1])
        index.add(np.array(doc_embeddings))

        # Encode query and retrieve top-k documents
        query_embedding = retriever_model.encode([query])
        distances, indices = index.search(query_embedding, top_k)
        retrieved_docs = [documents[i] for i in indices[0]]

        # Generate response using GPT-2
        generator = pipeline("text-generation", model="gpt2", tokenizer="gpt2")
        context = " ".join(retrieved_docs)
        generated = generator(context, max_length=50, temperature=temperature, num_return_sequences=1)

        return generated[0]["generated_text"]

    # TODO: Provide queries and ground truth directly here or load them dynamically from a file/external volume.
    query = ""  # Example: "Tell me about the Eiffel Tower."
    ground_truth = ""  # Example: "The Eiffel Tower is a famous landmark in Paris."

    # Extract hyperparameters
    top_k = int(parameters["top_k"])
    temperature = float(parameters["temperature"])

    # Generate response
    response = rag_pipeline_execute(query, top_k, temperature)

    # Compute BLEU score
    reference = [ground_truth.split()]  # Tokenized reference
    candidate = response.split()  # Tokenized candidate response
    smoothie = SmoothingFunction().method1
    bleu_score = sentence_bleu(reference, candidate, smoothing_function=smoothie)

    # Print BLEU score in Katib-compatible format
    print(f"BLEU={bleu_score}")

Note: Make sure to return the result in the format of <parameter>=<value> for Katib’s metrics collector to be able to utilize it. More ways to configure the output are available in Katib Metrics Collector guide.

STEP 3: Run a Katib Experiment

Once our pipeline is encapsulated within the objective function, we can configure Katib to optimize the BLEU score by tuning the hyperparameters:

top_k: The number of documents retrieved (eg. between 10 and 20).
temperature: The randomness of text generation (eg. between 0.5 and 1.0).

Define hyperparameter search space

parameters = {
    "top_k": katib.search.int(min=10, max=20),
    "temperature": katib.search.double(min=0.5, max=1.0, step=0.1)
}

Let’s submit the experiment! We’ll use the tune API that will run multiple trials to find the optimal top_k and temperature values for our RAG pipeline.

katib_client = katib.KatibClient(namespace="kubeflow")

name = "rag-tuning-experiment"
katib_client.tune(
    name=name,
    objective=objective,
    parameters=parameters,
    algorithm_name="grid",  # Grid search for hyperparameter tuning
    objective_metric_name="BLEU",
    objective_type="maximize",
    objective_goal=0.8,
    max_trial_count=10,  # Run up to 10 trials
    parallel_trial_count=2,  # Run 2 trials in parallel
    resources_per_trial={"cpu": "1", "memory": "2Gi"},
    base_image="python:3.10-slim",
    packages_to_install=[
        "transformers==4.36.0",
        "sentence-transformers==2.2.2",
        "faiss-cpu==1.7.4",
        "numpy==1.23.5",
        "huggingface_hub==0.20.0",
        "nltk==3.9.1"
    ]
)

Once the experiment is submitted, we can see output indicating that Katib has started the trials:

Experiment Trials status: 0 Trials, 0 Pending Trials, 0 Running Trials, 0 Succeeded Trials, 0 Failed Trials, 0 EarlyStopped Trials, 0 MetricsUnavailable Trials
Current Optimal Trial:
 {'best_trial_name': None,
 'observation': {'metrics': None},
 'parameter_assignments': None}
Experiment conditions:
 [{'last_transition_time': datetime.datetime(2025, 3, 13, 19, 40, 32, tzinfo=tzutc()),
 'last_update_time': datetime.datetime(2025, 3, 13, 19, 40, 32, tzinfo=tzutc()),
 'message': 'Experiment is created',
 'reason': 'ExperimentCreated',
 'status': 'True',
 'type': 'Created'}]
Waiting for Experiment: kubeflow/rag-tuning-experiment to reach Succeeded condition

.....

Experiment Trials status: 9 Trials, 0 Pending Trials, 2 Running Trials, 7 Succeeded Trials, 0 Failed Trials, 0 EarlyStopped Trials, 0 MetricsUnavailable Trials
Current Optimal Trial:
 {'best_trial_name': 'rag-tuning-experiment-66tmh9g7',
 'observation': {'metrics': [{'latest': '0.047040418725887996',
                              'max': '0.047040418725887996',
                              'min': '0.047040418725887996',
                              'name': 'BLEU'}]},
 'parameter_assignments': [{'name': 'top_k', 'value': '10'},
                           {'name': 'temperature', 'value': '0.6'}]}
Experiment conditions:
 [{'last_transition_time': datetime.datetime(2025, 3, 13, 19, 40, 32, tzinfo=tzutc()),
 'last_update_time': datetime.datetime(2025, 3, 13, 19, 40, 32, tzinfo=tzutc()),
 'message': 'Experiment is created',
 'reason': 'ExperimentCreated',
 'status': 'True',
 'type': 'Created'}, {'last_transition_time': datetime.datetime(2025, 3, 13, 19, 40, 52, tzinfo=tzutc()),
 'last_update_time': datetime.datetime(2025, 3, 13, 19, 40, 52, tzinfo=tzutc()),
 'message': 'Experiment is running',
 'reason': 'ExperimentRunning',
 'status': 'True',
 'type': 'Running'}]
Waiting for Experiment: kubeflow/rag-tuning-experiment to reach Succeeded condition

We can also see the experiments and trials being run to search for the optimized parameter:

kubectl get experiments.kubeflow.org -n kubeflow
NAME                    TYPE      STATUS   AGE
rag-tuning-experiment   Running   True     10m

kubectl get trials --all-namespaces
NAMESPACE   NAME                             TYPE      STATUS   AGE
kubeflow    rag-tuning-experiment-7wskq9b9   Running   True     10m
kubeflow    rag-tuning-experiment-cll6bt4z   Running   True     10m
kubeflow    rag-tuning-experiment-hzxrzq2t   Running   True     10m

The list of completed trials and their results will be shown in the UI like below. Steps to access Katib UI are available in the documentation:

Conclusion

In this experiment, we leveraged Kubeflow Katib to optimize a Retrieval-Augmented Generation (RAG) pipeline, systematically tuning key hyperparameters like top_k and temperature to enhance retrieval precision and generative response quality.

For anyone working with RAG systems or hyperparameter optimization, Katib is a powerful tool—enabling scalable, efficient, and intelligent tuning of machine learning models! We hope this tutorial helps you streamline hyperparameter tuning and unlock new efficiencies in your ML workflows!

Synthetic Data Generation with Kubeflow Pipelines

2025-02-16T00:00:00-06:00

Synthetic Data Generation - Why and How?

When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it’s worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.

To address these challenges, synthetic data provides an alternative. It mimics real data’s statistical properties while preserving privacy and accessibility. Synthetic data generators (synthesizers) are models trained on real data to generate new datasets that follow the same statistical distributions and relationships but do not contain real records. This allows for accelerated development, improved data availability, and enhanced privacy.

Depending on the technique used, synthetic data not only mirrors statistical base properties of real data but also preserves correlations between features. These synthesizers — such as those based on Gaussian Copulas, Generative Adversarial Networks (GANs), and Variational Autoencoders (VAEs) — enable the creation of high-fidelity synthetic datasets. See more description of these techniques below.

Key Benefits of Using Synthetic Data

While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for creating (synthetic) data:

Enhanced Privacy and Security
Mimics real datasets without containing sensitive or personally identifiable information, mitigating privacy risks and ensuring compliance with regulations like GDPR.
Improved Data Availability
Enables testing and training of models without requiring extensive real-world data collection.
Innovation and Experimentation
Allows safe experimentation with new algorithms and models without exposing sensitive data, fostering rapid prototyping in a secure environment.
Ethical and Responsible AI Development
Ensures training data is free from biases present in real-world datasets, promoting fair and unbiased AI systems.
Accelerated Testing and Deployment
Supports testing of new products, services, and systems in a controlled yet realistic setting, ensuring they are robust, scalable, and ready for real-world use.
Cost Efficiency
Reduces expenses related to data collection, storage, and compliance by eliminating the need for large-scale real-world data acquisition.
Regulatory Compliance Simplification
Helps organizations navigate complex data regulations by offering a compliant alternative to real-world datasets, easing cross-border data transfers.
Balanced and Augmented Datasets
Supplements real-world data by balancing underrepresented classes, improving model performance, and reducing biases in AI training.
Resilience Against Data Scarcity
Enables AI development in domains where real-world data is limited, expensive, or difficult to obtain—such as healthcare and cybersecurity—by generating high-quality alternative datasets.

To realize these benefits, we need effective tools for generating synthetic data. Different frameworks exist for this purpose, ranging from cloud-based platforms to open-source solutions. In this post, we focus on open-source synthetic data generation frameworks that provide control, flexibility, and on-premise deployment options.

Frameworks for Creating Synthetic Data

This post focuses exclusively on open source frameworks. Some data cannot be sent to the cloud, so some cloud-based synthetic data generation solutions are not always a good fit. For data already in cloud, we can use other cloud-based frameworks to generate synthetic data.

Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.

So, what framework did we (initially) choose? Currently, we are using the open source version of SDV, an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of Foundation models is certainly an area of interest.

One of the most widely used open-source libraries for synthetic data generation is Synthetic Data Vault (SDV). It provides multiple synthesizers, each tailored for different types of data and statistical properties.

The Synthetic Data Vault (SDV)

When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc. - see below), it trains a model based on the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between different features in the dataset. The synthesizer doesn’t memorize individual records from the dataset. Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data.

Below are the (free) synthesizers provided by SDV that we evaluated on each use case. Each synthesizer does this differently:

GaussianCopulaSynthesizer: Uses statistical copula functions to model relationships between features, ensuring accurate marginal distributions.
CTGANSynthesizer: Uses Generative Adversarial Networks (GANs) to learn complex data distributions, particularly effective for categorical and mixed-type data.
TVAESynthesizer: Leverages Variational Autoencoders (VAEs) to capture latent representations, useful for continuous and structured data.
CopulaGANSynthesizer: Combines Copula-based statistical modeling with GANs to generate data with complex dependencies.
PARSynthesizer: Uses autoregressive models to generate sequential data while preserving temporal dependencies.

There are more synthesizers, also from SDV, but not all are open source. We used the first four, when evaluating optimal synthesizer for our different use cases.

Generators - generating new data - on demand

Synthesizers are statistical and (more often) AI models trained to mimic the real data. Once developed, the resulting models are used to create as much synthetic data as you find useful for your use case. Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.

Evaluation Criteria for Synthetic Data

But, how good is synthetic data, how do we evaluate it?

There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthetic data generation technique (synthesizer) is best for our specific dataset and use case.

We need to ensure a good balance between:

Usability – How useful is the synthetic data for the intended use case?
Fidelity – How well does the synthetic data preserve statistical properties of the real data?
Privacy – Does the generated data ensure an acceptable level of privacy for the given use case?

For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below to assess usability.

Comments on privacy and privacy preserving techniques

Ensuring privacy in synthetic data is a non-trivial problem, even if there are techniques to ensure levels of privacy, it remains an active area of research.

Privacy problems, in synthetic data?

While synthetic data enhances privacy by removing personally identifiable information, it is not inherently risk-free. Some key challenges include:

Overfitting and Memorization: If a synthesizer is overfitted, it may generate synthetic records that closely resemble real data, leading to privacy leakage.
Anomaly Exposure: Unique individuals or rare events in the dataset (e.g., a very wealthy individual or a rare disease) may be unintentionally replicated in synthetic data, creating a risk of re-identification.
Re-identification Attacks: Even if synthetic data is statistically different from real data, attackers may use background knowledge to infer sensitive details about individuals.

One additional problem here is that it might be the anomalies we really are looking for. Currently we are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.

Our On-Premise Analytics Platform: ARCUS

ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases. The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.

Needed environment to create synthetic data

For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).

Kubeflow pipelines
GPU capabilities (for performance and efficiency)
Development (IDE) environment (for framework building and running)
Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets

Parallelism needed

In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the ‘winner’.

Note: In earlier version of Kubeflow we noticed that the parallelism wasn’t acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.

Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.

Exploring the Creation and Usefulness of Synthetic Data

This is what we want to do: we have a use case, the supporting data, and developed ML model.

How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
How well do the ML models based on synthetic data keep up with ML models based on real data?

Validation of synthetic data techniques

Create the synthetic data and save the best synthetic data generator. In this step similarity measures are created by the out of the box SDV framework
Create the ML model (in our case classifier model) both on real data and the using the synthetic data. Compare the performance of both models against the same real data testset.

From above, we have an example where the final synthesizer is collected and saved. This step is used in the example below, exporting the resulting synthetic data generator to cloud.

Using Synthetic Data Generators to Enable Multiple Environments without Data Transfer

Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.

Problem statement:

Our data cannot be moved from on-premise to cloud.
We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)

Solution:

Create synthetic data for our on-premise environment use-cases, and - as a side product we save away the synthetic data generator (the pickled model used to create synthetic data).
Copy the synthetic data generator to cloud
Use the synthetic data generator in the cloud, creating synthetic data for training of an ML model
Copy the ML model on-premise, and use it for new incoming data
Evaluate: Compare the on-premise AI model with the model created in the cloud - against the same test data

Division of work, what is done on-premise with Kubeflow, and what is done in cloud (AWS SageMaker)?

On-premise

See the above Validation of synthetic data techniques.

Develop the model on real data – for the comparison later with the cloud model.
Create synthetic generators, evaluate the generators, and export the best generator to AWS.

Cloud

Use the imported synthetic generator (from on-premise)
Create synthetic data using the synthetic data generator
Develop the model and determine which synthetic generator is the best
Increase the amount of synthetic data, to see if the increase of synthetic data improves model performance (not for sure it will, see below comment)
Export model to on-premise

In some more detail below.

On-premise

Compare real data model against synthetic data model – using real test data.

In the current examples we see near equivalent performance of the ML models (a few percentage points lower for models created using synthetic data). We experimented with increasing the size of the synthetic dataset, with minor improvements. Augmenting the training data is expected (not tested here) to have more effects when using deep learning algorithms.

Summary

Clearly, the above workflows would be very cumbersome to build and maintain without Kubeflow. Our solution is entirely open source, Kubernetes based, and uses Kubeflow and SDV to give us the scalability, robustness, and detailed control that is required.

The area of synthetic data generation is moving fast with the overall AI field. Reports from IBM and others, of the increased usage of synthetic data for e.g. LLM training is frequent but the application areas are much greater. We also expect more capable synthesizers and, hopefully, privacy preserving techniques to keep up with the innovation in this area. Our original main motivator was speed up in innovation and experimentation, and overall - speed to market. Often a key pain for our teams.

Looking ahead, we are exploring the development of a synthesizer catalog — ideally integrated into our overall data catalog — to enable users to rapidly experiment with ideas and get started more efficiently.

Kubeflow and Me: A Story Started with Push-based Metrics Collection

2024-09-28T00:00:00-05:00

This summer, I gained a precious opportunity to participate in the Google Summer of Code(GSoC), in which I would contribute to Katib and fulfill a project named “Push-based Metrics Collection in Katib” within 12 weeks. Firstly, I got to know about GSoC and Kubeflow with the recommendation from the former active maintainer Ce Gao(gaocegege)’s personal blog. And I was deeply impressed by the idea of cloud native AI toolkits, I decided to dive into this area and learn some skills to enhance my career and future. In the blog, I’ll provide my personal insight into Katib, for those who are interested in cloud native, AI, and hyperparameters tuning.

Problem

The project aims to provide a Python SDK API interface for users to push metrics to Katib DB directly.

The current implementation of Metrics Collector is pull-based, raising design problems such as determining the frequency at which we scrape the metrics, performance issues like the overhead caused by too many sidecar containers, and restrictions on developing environments that must support sidecar containers and admission webhooks. And also, for data scientists, they need to pay attention to the format of metrics printed in the training scripts, which is error prone and may be hard to recognize.

Solution

We decided to implement a new API for Katib Python SDK to offer users a push-based way to store metrics directly into the Kaitb DB and resolve those issues raised by pull-based metrics collection.

In the new design, users just need to set metrics_collector_config={"kind": "Push"} in the tune() function and call the report_metrics() API in their objective function to push metrics to Katib DB directly. There are no sidecar containers and restricted metric log formats any more. After that, Trial Controller will continuously collect metrics from Katib DB and update the status of Trial, which is the same as pull-based metrics collection.

If you are interested in it, please refer to this doc and example for more details.

My Contributions during the GSoC

I raised numerous PRs for the Katib and Training-Operator project. Some of them are related to my GSoC project, and others may contribute to the completeness of UTs (Unit Tests), simplicity of dependency management, and the compatibility of the UI component.

For reference, the coding period can be rougly divided into 3 stages:

Convert the proposal to a KEP and discuss the architecture, API design, etc. (~4 weeks) with the mentors
Develop a push-based metrics collection interface according to the KEP. (~8 weeks)
Write some examples and documentation & Present my work to the Kubeflow Community.

Also, I raised some issues not only to describe the problems and bugs I met during the coding period, but also to suggest the future enhancement direction for Katib and the Training-Operator.

There is a Github Issue tracks the progress of developing push-based metrics collection for katib during the GSoC coding phase. If you are interested in my work or Katib, please can check this issue for more details.

Lessons Learned

Think Twice, Code Once: Andrey taught me that we should think of the API specification and all the related details before coding. This can significantly reduce the workload of the coding period and avoid big refactor of the project. Meanwhile, my understanding of Katib got clear gradually during the over-and-over rounds of re-think and re-design of the architecture.
Dive into the Source Code: Engineering projects nowadays are extremely complex and need much effort to understand them. The best way to get familiar with the project is to dive into the source code and run several examples.
Communication: Communication is the most important thing when collaborating with others. Expressing your idea precisely and making others understand you easily are significant skills not only in the open source community but also in various scenarios such as at a company and in group work.

In the End

Special Thanks:

To my mentors @andreyvelich @johnugeorge @tenzen-y, especially to Andrey. Your great knowledge about the code base and the industry impressed me a lot. Thanks for your timely response to my PRs and for always attending the weekly meetings to solve my pending problems, from which I benefited a lot. What’s more, I can well remember that, in that night, you explained the usage of Kubeflow in the industry to me with greate patience, and encouraged me not to doubt about myself, just do it and explore more, contribute more. You ignite the flame of my desire to contribute to cloud native AI.
To @gaocegege. You recommend me to the Kubeflow Community. Thanks for your patient answers for my endless silly questions.
To Google. Thanks for offering such a precious opportunity for me to begin my journey in the open source world!

I hold a firm belief that every small step counts, and everybody in the community is unique and of great significance. There is no doubt that our joint efforts will surely contribute to the flourishing of our Kubeflow Community, make it the world-best community managing AI lifecycle on Kubernetes, and attract much more attention from the industry. Then, more and more new comers will pour in and work along with us.

Again, I’ll continue to contribute to Kubeflow.

Links

For more details about Kubeflow and the upcoming GSoC’25 event, please check:

LLM Hyperparameter Optimization API: My Google Summer of Code Journey with Kubeflow

2024-09-19T00:00:00-05:00

This summer, I had the opportunity to participate in the Google Summer of Code (GSoC) program, where I contributed to Kubeflow, an open-source machine learning toolkit. My project focused on developing a high-level API for optimizing hyperparameters in Large Language Models (LLMs) within Katib, Kubeflow’s automated hyperparameter tuning system. I’d like to share insights from this experience with others interested in Kubeflow, GSoC, or optimizing LLMs.

Motivation

The rapid advancements and rising popularity of LLMs, such as GPT and BERT, have created a growing demand for efficient LLMOps in Kubernetes. To address this, we have developed a train API within the Training Python SDK, simplifying the process of fine-tuning LLMs using distributed PyTorchJob workers. However, hyperparameter optimization remains a crucial yet labor-intensive task for enhancing model performance.

Goal

Hyperparameter optimization is essential but time-consuming, especially for LLMs with billions of parameters. This API simplifies the process by handling Kubernetes infrastructure, allowing data scientists to focus on model performance rather than system configuration.

With this API, users can import pretrained models and datasets from Hugging Face and Amazon S3, define parameters including the hyperparameter search space, optimization objective, and resource configuration. The API then automates the creation of Experiment, which contains multiple Trials with different hyperparameter settings using PyTorch distributed training. It then collects and analyzes the metrics from each Trial to identify the optimal hyperparameter configuration.

For detailed instruction on using the API, please refer to this guide

My Contributions to the GSoC Project

My work on the project can be broadly divided into four stages:

Stage 1: Designing the API, drafting the project proposal, and refining it into a Kubeflow Enhancement Proposal (KEP).
Stage 2: Developing and implementing the high-level API.
Stage 3: Implementing unit tests and end-to-end tests for the API.
Stage 4: Creating documentation and presenting the work to the Kubeflow community.

In addition, I addressed several critical bugs in previous Katib and Training Operator releases and contributed new features, such as writing end-to-end tests for the train API.

For those interested, here is a detailed summary of all the pull requests I submitted during this process.

Lessons Learned

This is my first experience contributing to an open source project, and I gained extensive technical knowledge throughout this project, including Docker, Kubernetes, and Kubeflow itself. Before developing and implementing the API, I invested significant time onboarding and familiarizing myself with Kubeflow. The official documentation and GitHub repository were invaluable resources during this process.

Beyond these technical skills, I also learned several key lessons that extend into broader personal and professional growth.

Think from the User’s Perspective

One key lesson was the importance of considering the user’s needs. Discussing API design with my mentors taught me to focus on what functionalities users need and how they prefer to use them. Listening to users’ feedback is crucial for effective product design.

Don’t Fear Bugs

I used to feel overwhelmed by bugs and unsure how to tackle them. When a bug caused a container failure during a Katib trial, my mentor guided me through the debugging process, teaching me how to systematically trace and understand the issue. The key is to approach debugging methodically and think through each step of the problem.

Communication is Important

Communication is important in collaboration, especially in open source projects. There are various ways of communicating in open-source projects, such as GitHub issues or PRs, Slack, and community meetings. And I’m grateful to my mentor for discussing my challenges during our weekly meetings and providing invaluable guidance.

Every Contribution Counts

Initially, I thought contributing to open source was complex. I learned that every contribution, no matter how small, is valuable and appreciated. For example, contributing to documentation is crucial, especially for newcomers.

In The End

I am deeply grateful to everyone who supported me throughout this project. Your suggestions, advice, and encouragement were invaluable in helping me complete my work.

I especially want to extend my heartfelt thanks to my mentor, Andrey Velichkevich. His deep knowledge of both the project and the industry, combined with his willingness to help, has been incredibly inspiring. I greatly appreciate the time and effort he dedicated to guiding me, from the high-level design of the API to the finer details like code formatting. I have learned so much from his mentorship.

Looking ahead, I am excited to continue contributing to Kubeflow. I also look forward to helping future contributors by improving documentation and sharing my experiences with newcomers in the community.

If you’re interested in open-source and want to be part of Kubeflow, GSoC 2025 applications are now open! Check out the details here—we’d love to have you join us!

Kubeflow 1.9: New Tools for Model Management and Training Optimization

2024-07-22T00:00:00-05:00

Kubeflow 1.9 significantly simplifies the development, tuning and management of secure machine learning models and LLMs. Highlights include:

Model Registry: Centralized management for ML models, versions, and artifacts.
Fine-Tune APIs for LLMs: Simplifies fine-tuning of LLMs with custom datasets.
Pipelines: Consolidation of Tekton and Argo Workflows backends for improved flexibility.
Security Enhancements: Network policies, Oauth2-proxy, and CVE scanning.
Integration Upgrades: Improved integrations with Ray, Seldon, BentoML, and KServe for LLM GPU optimizations.
Installation and Documentation: Streamlined installation, updated platform dependencies, and enhanced documentation.

These updates aim to simplify workflows, improve integration dependencies, and provide Kubernetes-native operational efficiencies for enterprise scale, security, and isolation.

Model Registry

A model registry provides a central catalog for ML model developers to index and manage models, versions, and ML artifacts metadata. It fills a gap between model experimentation and production activities. It provides a central interface for all stakeholders in the ML lifecycle to collaborate on ML models. Model registry has been asked by the community for a long time and we are delighted to introduce it to the Kubeflow ecosystem.

This initial release includes REST APIs and a Python SDK to track model artifacts and model metadata with a standardized format that can be reused across Kubeflow components, such as to deploy Inference Servers. You can get started by following the Model Registry tutorial on the Kubeflow website, or see a short demo video of the Model Registry in action.

We are just getting started. This is an Alpha version and we look forward to feedback. The model registry working group meets biweekly: you can provide feedback by joining the meeting or directly on the repository.

Fine-Tune APIs for LLMs

In the rapidly evolving ML/AI landscape, the ability to fine-tune pre-trained models represents a significant leap towards achieving custom solutions with less effort and time. Fine-tuning with custom datasets allows practitioners to adapt large language models (LLMs) to their specific needs.

However, fine-tuning tasks often require extensive manual intervention, including the configuration of training environments and the distribution of data across nodes. The new Fine-Tune API aims to simplify this process, offering an easy-to-use Python interface that abstracts away the complexity involved in setting up and executing fine-tuning tasks on distributed systems.

By providing this API, Training Operator not only simplifies the user experience for ML practitioners but also leverages its existing infrastructure for distributed training. You can take advantage of Kubernetes’ ability to dynamically schedule GPU, thus saving on compute resources and cost. Training Operator also gives you fault tolerant guarantees, to save your training procedures from cluster node failures.

Pipelines

v1 Feature Parity

We made significant progress towards KFPv1 feature parity by adding more Kubernetes resources to the Pipelines code with the new kfp-kubernetes 1.2.0 Python package. We encourage every KFP user to test the new V2 functionality and plan your migration from V1 to V2. We still have some outstanding features that need to be ported over to V2, please help us to identify what’s missing by openining a new issue in the KFP repository.

Argo Workflows and Tekton Backends Consolidation

The Pipelines Tekton backend has been merged into the main Kubeflow Pipelines repository. You can now choose what workflow engine to use from the same Pipelines version. This proves the extensibility and flexibility of the KFP v2 architecture, which encourages other contributors to bring support for other workflow engines.

Both Argo Workflows and Tekton provide unique advantages. Argo Workflows is known for its simplicity and ease of use, making it a popular choice for many users. Tekton offers extensive customization options with its pipeline definitions and reusable components, which can be advantageous for integrating into various CI/CD systems. Depending on your specific requirements and preferences, you can leverage the strengths of either Argo Workflows or Tekton to optimize your machine learning workflows.

In this blog post, you can find more details about the benefits of running KFP with either Tekton or Argo Workflows.

Argo Workflows Upgrade

Kubeflow Pipelines’s Argo Workflows backend is upgraded to 3.4.16. This upgrade moves the supported version closer to the latest upstream version and brings lots of CVE resolutions. The previous minor version was no longer being patched by the Argo community, so lots of security issues had accumulated over time.

Katib

Kubeflow 1.9 ships with Katib 0.17, which brings official support for ARM64, getting us one step closer to full ARM64 coverage.

For Data Scientists who submit training jobs with the Python SDK, you can now set the algorithm settings and environment variables from the tune method. Previously, you had to rely directly on Kubernetes CRD submission for these. You can also take advantage of the latest features from TensorFlow 2.16 and PyTorch 2.2. The team also worked to resolve environmental conflicts that prevented the Katib Python SDK to be installed alongside the Kubeflow Python SDK.

There are tons of additional improvements and bug fixes. Check out the full changelog here.

Central Dashboard

This release bring several improvements to the Kubeflow Central Dashboard, including:

Styling improvements to the sidebar, including grouping all Kubeflow Pipelines links to reduce clutter
Significant improvements to the “manage contributors” page, including the ability to manage contributors for all profiles that you are the owner of, and see which profiles you have access to, even when you are not the owner
Allow external services to parse the current profile (namespace) by sending the namespace selector value to non-iframed applications
Significant updates to dependencies to reduce CVEs

Notebooks

With this release, we provide significant updates to all example notebook images including PyTorch 2.3.0, Tensorflow 2.15.1 and many other library updates. While you can continue to use the old images, we recommend updating to use the greatest and latest ML libraries.

Additionally, notebooks images now run with a non-root SecurityContext, allowing for an improved security.

Take a look at the changelog for a full list of bug fixes and improvements.

While this release was light on new Notebooks features, the Working Group is hard at work on an exciting new project: we are actively developing Notebooks V2, with contributions from various companies, in the new repository. Take a look here and join our Working Group meetings to get involved!

Kubeflow Platform (Security and Manifests)

Security

Network Policies

Network policies are enabled for the Kubeflow core services as a second layer of defense before Istio authorization policies. This gives administrators a better network overview and segmentation while also enforcing common enterprise security guidelines. You can read more about the current implementation and architecture here.

Authentication

Oauth2-proxy replaces oidc-authservice, which brings improved token-based authentication. Machine Learning engineers can now use tokens instead of insecure passwords for CI/CD automation of Kubeflow deployment and maintenance (e.g. using GitHub actions). You can read more about the current implementation and architecture here.

CVE Scanning

With this release we are introducing automated CVE scanning with Trivy on the manifests master branch. We appreciate contributions to reduce the number of CVEs, the Security Working Group needs help to build a more secure platform. You can find more details about our security scanning process and disclosure policy here. Here are is a summary from June 25th:

You can find a detailed Security WG roadmap here.

Manifests

Installation and documentation improvements

The documentation has been improved and now contains guidelines for upgrading and extending the Kubeflow Platform for administrators. New users can now install kubeflow on their laptop in just a few minutes

Platform dependencies updates:

Component	Kubernetes	Kustomize	Istio	Dex	Cert-Manager	Knative
KF 1.9 Version	1.27 - 1.29	5.2.1+	1.22.1	2.39.1	1.14.5	1.12.4

NOTE Kubernetes 1.30+ is also expected to work, but was not officially tested.

NOTE Kustomize 5.2.1+ support with way less warnings. So platform engineers have a modern and supported installation tool chain.

Integration with third-party ML tools

We have updated the third-party components in /contrib to provide integration with the broader ML ecosystem.

BentoML 1.2.28 and 1.1.21
Seldon 1.18.1
Ray 2.23 and Kuberay 1.1.1

Find a detailed Manifests WG roadmap here.

KServe

We upgraded to KServe 0.13. This release includes:

Enhanced Hugging Face Runtime Support: Hugging Face models are supported out-of-the-box, implementing a KServe Hugging Face Serving Runtime. Currently supported tasks include sequence classification, token classification, fill mask, text generation, and text to text generation.
vLLM Support: Dedicated runtime support for vLLM is now included, streamlining the deployment process for LLMs.
OpenAI Schema Integration: KServe now supports endpoints for generative transformer models, following the OpenAI protocol. This enables KServe to be used directly with OpenAI’s client libraries or third-party tools like LangChain and LlamaIndex.

Documentation

MLOps is a complex subject and users have asked for clear, up-to-date and comprehensive documentation. We are happy to announce that we started a restructuring process to better align the various components’ docs to have a similar structure. We are revamping our docs to better align with user expectations and how you expect technical docs to be organized. We will continue to improve the quality and completeness of our docs, by adding new user guides, tutorials, and reference architecture topics.

We are looking for new members who can help us craft complete and high quality documentation. Please get involved by opening and reviewing PRs in the Kubeflow website.

Honorable Mentions

Google Spark Operator migration to Kubeflow

We’re excited to announce the migration of Google’s Spark Operator to the Kubeflow Spark Operator, marking the launch of a significant addition to the Kubeflow ecosystem. The Kubeflow Spark Operator simplifies the deployment and management of Apache Spark applications on Kubernetes. This announcement isn’t just about a new piece of technology, it’s about building a stronger, open-governed, and more collaborative community around Spark on Kubernetes.

Kubeflow Spark Operator is not yet officially included in the Kubeflow release, but you can install it by following the instructions here.

Read more about Kubeflow Spark operator in the announcement blog post.

Google Summer of Code

This year, Kubeflow was excited to participate in Google Summer of Code (GSoC), attracting a wave of enthusiastic students! Over 250 students joined our Slack channel, eager to learn about contributing to the Kubeflow community and crafting impactful proposals.

We were also fortunate to have a dedicated group of 20 mentors ready to guide these talented individuals. From a pool of nearly 70 proposals, we have selected 10 proposals and were awarded with 8 outstanding students by Google. They are now actively contributing to various Kubeflow features, making a real difference in various Kubeflow components.

We’ll be following their progress and sharing their accomplishments through a series of blog posts in the future, so stay tuned! A big thank you to all the mentors and students who are making Kubeflow’s GSoC 2024 a huge success!

Thanks to all the studets: Adem Baccara, Biswajit Pattnaik, Hansini Karunarathne, Hezhi Xie, Sandipan Panda, Shao Wang, Shashank Mittal, SIVASUBRAMANIAM L. Visit this page for more details about each project and respective mentors.

What’s next

The community continues to see growth, especially with the ever growing interest from the CloudNative community in AI topics. We have recently elected the first Kubeflow Steering Committee. This is the first step towards a more mature governance structure and a democratic and open community.

If you want to take a peek into the Kubeflow 1.10 roadmap planning and contribute with your ideas, see Notebooks, Manifests & Security, Pipelines, Model Registry, Katib, Training Operator.

How to get started with 1.9

Visit the Kubeflow 1.9 release page or head over to the Getting Started section to learn more about installation, architecture and quick start examples.

Join the Community

We would like to thank everyone for their contribution to Kubeflow 1.9, especially Ricardo Martinelli De Oliveira for his work as the v1.9 Release Manager, all the release team and the working group leads, who relentlessly dedicate their time to this great project.

Release team members: Ricardo Martinelli De Oliveira, Stefano Fioravanzo, Helber Belmiro, Diego Lovison, Ajay Nagar, Mathew Wicks, Steven Irvin, Milos Grubjesic, Andrew Scribner, Julius von Kohout.

Working Group leads: Andrey Velichkevich, Ce Gao, Chaoran Yu, Chen Sun, Christian Kadner, Ilias Katsakioris, James Liu, James Wu, Johnu George, Julius von Kohout, Kimonas Sotirchos, Mathew Wicks, Matteo Mortari, Ramesh Reddy, Stefano Fioravanzo, Tommy Li, Vara Bonthu, Yannis Zarkadas, Yuan Tang, Yuki Iwai.

Kubeflow Steering Committee: Andrey Velichkevich, Johnu George, Josh Bottum, James Wu, Yuan Tang.

Participating Distributions: Charmed Kubeflow (Canonical), IBM IKS, Nutanix, OpenShift AI (RedHat), Oracle Cloud Infrastructure, DeployKF, VMWare, QBO. You can find more details about Kubeflow distributions here.

Want to help?

Visit our Kubeflow website or Kubeflow GitHub Page
Join the Kubeflow Slack channel
Join the kubeflow-discuss mailing list
Attend our weekly community meeting

Announcing the Kubeflow Spark Operator: Building a Stronger Spark on Kubernetes Community

2024-04-15T00:00:00-05:00

The Journey to Kubeflow Spark Operator

The journey of the Kubeflow Spark Operator began with Google Cloud Platform’s Spark on Kubernetes Operator (https://cloud.google.com/blog/products/data-analytics/data-analytics-meet-containers-kubernetes-operator-for-apache-spark-now-in-beta). With over 2.3k stars and 1.3k forks on GitHub, this project laid the foundation for a robust Spark on Kubernetes experience, enabling users to deploy Spark workloads seamlessly across Kubernetes clusters.

Growth and innovation require not just code but also community. Acknowledging the resource and time limitations faced by Google Cloud’s original maintainers, Kubeflow has taken up the mantle.This transition is not merely administrative but a strategic move towards fostering a vibrant, diverse, and more actively engaged community.

Why Kubeflow?

Enhanced Community Engagement: Transitioning to Kubeflow opens the door to a broader developer base, encouraging contributions and collaboration. Since Kubeflow is a CNCF incubating project this transition will help consolidate Cloud Native and Spark communities to work more closely to build robust infrastructure to run Spark applications on Kubernetes.
Stronger Governance: Kubeflow’s governance model provides a structured environment for decision-making and project management, ensuring sustainable growth for the Spark Operator.
Unified Ecosystem: By bringing the Spark Operator under the Kubeflow umbrella, we’re not just merging projects; we’re building a cohesive ecosystem that enhances the Spark on Kubernetes experience.
Integration with AI/ML: Kubeflow provides several components to address many stages of the AI/ML lifecycle. The Spark distributed data processing capabilities are a natural expansion, allowing the Spark community to closely collaborate and better integrate within the end-to-end ML lifecycle.

What’s Next?

We are dedicated to not just maintaining but enhancing the Kubeflow Spark Operator for the long term. Here’s what you can look forward to:

Upcoming roadmap: As part of the first release, we aim to update the documentation with references to Kubeflow, address GitHub workflow issues, and update the container registry with Kubeflow, along with any other critical issues.
Ongoing Support and Enhancements: At the time of migration to the Kubeflow repository, the repository comprised 450+ issues and 60+ pull requests. We kindly request contributors to rebase their code and update the PR with a comment indicating its continued relevance. As for open issues, they will be considered for resolution as the broader community and contributors engage in upcoming releases.The operator will continue to evolve, incorporating new features and improvements to stay at the forefront of Kubernetes deployments.
Rich Community Resources: From detailed documentation to hands-on tutorials, we’re crafting resources to help you succeed with the Spark Operator. We are planning to host regular Spark Operator calls to discuss users issues, questions, and future roadmaps.
Open Doors for Contributions: This is a call to arms for developers, writers, and enthusiasts! Your contributions are the lifeblood of this project, and there’s a place for everyone to make a mark.
Kubeflow Working Group Data: To consolidate efforts around new data tools in the Kubeflow ecosystem such as Spark Operator and Model Registry the new Working Group Data will be formalized soon. Feel free to review this PR to get involved and provide your feedback on the charter.

Join the Movement

The Kubeflow Spark Operator is more than just software. It’s a community endeavor. Here’s how you can be a part of this journey:

Dive In: Visit our GitHub repository to start your journey with the Kubeflow Spark Operator.
Contribute: Every code snippet, documentation update, and piece of feedback counts. Find out how you can contribute on GitHub.
Be Part of the Community: Join the CNCF Slack Workspace and then join the conversation in the #kubeflow-spark-operator channel. Whether you’re seeking advice, sharing insights, or just listening in, your presence enriches us. Follow this guide to learn more about Kubeflow community.
Kubeflow Spark Operator Community Call: We’re excited to announce Spark Operator Community Monthly Meetings for Open Source Contributors starting May 17th, 2024 (10-11 AM PST). These meetings, held every third Friday, are your chance to discuss project updates, share ideas, and collaborate with the community. You can find the Zoom call details and meeting notes in this Google Doc.

In the spirit of collaboration fostered on platforms like Slack, and with the generous support of the Google Cloud team, we’re set to sail into a promising future. The Kubeflow Spark Operator isn’t just a tool, it’s our collective step towards harnessing the true potential of Spark on Kubernetes. Together, let’s shape the future of cloud-native big data processing.

Reference Issues

Kubeflow Project Steering Committee Announced

2024-01-31T00:00:00-06:00

We’re thrilled to announce the results of the 2023 Kubeflow Steering Committee (KSC) election.

The Kubeflow community has shown a strong commitment to the project’s future by casting their votes for the new leadership. Please welcome Yuan (Terry) Tang (Red Hat), Andrey Velichkevich (Apple), and Johnu George (Nutanix) who will be joining Josh Bottom (Consultant) and James Wu (Google) as KSC members. The three new members will serve a two-year term ending in 2025, beginning immediately.

The election saw a turnout of 72.88%, with 43 out of 59 eligible voters participating. The three nominees were chosen from a pool of candidates, which also included Kimonas Sotirchos, Julius von Kohout, and Mathew Wicks.

This election represents a significant step forward for the Kubeflow project. We extend our deepest gratitude to the interim KSC members and the election officials for their service and to all those who were nominated and participated in this election. We eagerly anticipate the contributions of our new leadership and what we can accomplish as a project moving forward.

Quotes:

“Excited to start my two-year term on the Kubeflow Steering Committee on behalf of Red Hat! Thanks to everyone who’s supported me along the way, and congratulations to all the new committee members. Johnu, Andrey, and I have collaborated on various Kubeflow subprojects for many years. I look forward to working with them and the rest of the committee more closely. Kubeflow is an umbrella of projects that provide an excellent foundation for AI/ML applications in the cloud-native world. Its contributor velocity, community momentum, and industry adoption have proliferated. Together with the Kubeflow community, ecosystem projects, supporting organizations, and partners, I am confident that we will steer the project towards a successful CNCF journey and continue thriving.” - Yuan (Terry) Tang (Red Hat)

“I am thrilled to join the Kubeflow Steering Committee for the two-year term. Being an active member of this community for almost six years, it was great to see how the project evolves towards open governance and widespread user adoption. Thanks to everyone for your support, collaboration, and contributions throughout these years. Kubeflow stands as the foundational framework to run AI/ML workloads on Kubernetes. It bridges ML, Big Data, and Cloud Native ecosystems to facilitate a new generation of AI applications. Kubeflow components empower users at every stage of ML lifecycle from model development and training to fine-tuning and deployment. I am excited about the future of CNCF and Kubeflow together, to build an open CloudNative AI/ML platform accessible to everyone.” - Andrey Velichkevich (Apple)

“I am really excited to join the first Kubeflow Steering Committee formed post CNCF incubation. Kubeflow is one of the most popular enterprise-ready MLOps platforms used in production at various companies. It has become the de facto platform to run complex ML pipelines, managing model lifecycles on a large scale. Having been part of Kubeflow leadership since its inception, it is delightful to see its growing ecosystem proving its relevance in the current times. Thanks to the entire Kubeflow user and developer community, who have immensely contributed to its success over the years. I am excited to drive the future of this vibrant community and look forward to the next phase of the Kubeflow journey in the CNCF ecosystem.” -Johnu George (Nutanix)

Additional Information:

Get Involved

Getting involved in the Kubeflow Community offers numerous opportunities for learning, networking, and contributing to open-source development. Participating in the Kubeflow community you’ll be joining a vibrant ecosystem of developers, data scientists, AI enthusiasts and others who are pushing the boundaries of machine learning operations with Kubeflow. By actively participating and sharing your ideas, you can make a meaningful impact and be part of this dynamic community. Check out the weekly community call, get involved in discussions on the [mailing list]

Kubeflow Community Holds First Election for Kubeflow Steering Committee

2023-12-12T00:00:00-06:00

The Kubeflow Community, known for its dedication to making machine learning workflows on Kubernetes simple, portable, and scalable, has recently announced a significant milestone in its journey. For the first time, they are holding elections for the Kubeflow Steering Committee (KSC)!

The KSC will be crucial in guiding the project’s direction, ensuring it continues to meet the needs of its growing user base. Candidates for this committee are drawn from the community’s diverse members, embodying the spirit of open-source collaboration that Kubeflow cherishes. This election marks a new chapter in Kubeflow’s history, reflecting the community’s commitment to a democratic project governance model. It serves as a testament to the community’s growth, maturity, and dedication to inclusivity and shared decision-making.

The election process for the KSC is structured to ensure that every phase is transparent and fair. As per the announced timeline, we are currently in the nomination phase, where eligible community members are encouraged to nominate themselves or others for a position on the Committee. This stage will determine the group of candidates who will move forward to the testimonial phase and then the voting phase. It’s an exciting time for the community, as members step up to the challenge and show their readiness to drive the project’s future direction.

Elections like this in the open source ecosystem highlight the collaborative spirit that drives these initiatives. They provide the opportunity for anyone, regardless of their role in the project, to step up and help guide its future. Let’s all join in congratulating the Kubeflow Community on this milestone and look forward to the outcomes of this exciting election!

Get Involved:

We are an open and welcoming community of software developers, data scientists, AI enthusiasts, organizations and more! Getting involved in the Kubeflow Community is an exciting journey that offers numerous opportunities for learning, networking, and contributing to open-source development. By actively participating and sharing your ideas, you can make a meaningful impact and be part of this dynamic community. Check out the weekly community call, get involved in discussions on the mailing list or chat with others on the Slack Workspace!

Important Dates:

Exceptions Phase: 4 December 2023 at 0900 Pacific Time (Starts) - 10 December 2023 at 12:00pm Pacific Time (Ends)
Nomination Phase: 11 December 2023 at 0900 Pacific Time (Starts) - 24 December 2023 at 12:00pm Pacific Time (Ends)
Testimonial Phase: 25 December 2023 at 0900 Pacific Time (Starts) - 7 January 2024 at 12:00pm Pacific Time (Ends)
Voting Phase: 8 January 2024 at 0900 Pacific Time (Starts) - 29 January 2024 at 12:00pm Pacific Time (Ends)
Announcement of Election Results: 30 January 2024