TAS wins VMware Explore 2023 Las Vegas Hackathon

I haven't slept much in the past 24 hours, but I want to document my thoughts after the VMware Explore Las Vegas Hackathon! 😎

I am honored to announce that the Tanzu Application Service (TAS) team took 1st place in the hackathon with our concept of expanding the existing Tanzu Application Service, Cloud Foundry, and BOSH ecosystem with Gen AI/LLM-as-a-Service technologies.

First and foremost, I want to thank Vladimir Velikov, Cami Hough, Franky Barragan, and Brian Chang for helping to plan and run the hackathon!! You all made a great hackathon event possible, and it was great to see the VMware community hack away at many avenues of the VMware ecosystem!

Also, a big thank you to all the judges; it was great to talk with all of you and hear your feedback and ideas on how we can work with the rest of the VMware ecosystem.

In particular, it was great to talk to the "Wolf of VMware," Chris Wolf and Alan Renouf, to hear their take on our project as they lead the cutting-edge work at VMware AI Labs. I was very excited to see the announcement of VMware Private AI today at Explore, and I think our project aligns with that philosophy.

What a privilege it was to work with Long Nguyen, Amelia Downs, and Jonathon Regher to showcase Tanzu Application Service and Cloud Foundry BOSH running applications that use OpenAI-compatible and LLM-as-a-Service technologies.

In particular, Long Nguyen, our project's lead engineer, developed and pioneered this work with his contributions to BOSH, custom stemcell work, and various updates to the Cloud Foundry ecosystem. He was the one who got BOSH to deploy and manage VMs with GPUs – which was the critical first step to open the door for so many more things in the GenAI/LLM space. 😎

I also want to send some special thanks to David Stevenson, who helped mentor our team and assisted with providing access to his awesome TAS-hosted environment for our demo.

Disclaimer: The work documented here was an experimental proof of concept. It did not represent a product commitment, and the architecture was subject to change.

What was our problem statement?

As the slide says, our team was focused on enabling the Tanzu Application Service Customer base and the Cloud Foundry ecosystem that runs millions of application instances or containers globally with GenAI and LLMs.

If you have worked with Cloud Foundry for some time, you know that GPU-backed AI workloads were not traditionally found inside the application runtime. TAS expects applications to decouple state, and packaging large GPU drivers directly into an application container is not an attractive operating model.

(IMHO- This is a type of workload that's best suited for a virtual machine, especially one managed by BOSH.)

At a high level, our team extended the BOSH ecosystem to deploy GPU-equipped virtual machines that hosted large language model inference beside Cloud Foundry. That provided the model endpoint used by ChatGPT-style applications running on TAS.

Going one level deeper, our team produced a custom BOSH release and stemcell with the following pieces:

• Packaged FastChat's API server with Llama 2 models into a BOSH release.
• Deployed it via BOSH side-by-side with our TAS deployment.
• Built a TAS app that uses our new prototype, "Tanzu AI Service," via hardcoded credentials.

One key benefit of this proposed architecture is enabling Tanzu Application Service customers to run multi-cloud private Gen AI & LLM services. Yes, that's right, the customer or end user of the TAS deployment could run this entirely in the walls of their own data center without exposing their sensitive IP and data to a third-party cloud or API.

Let's see it in action:

The concept of our demo was pretty simple.

• Ask the ChatGPT-style application a question outside the base model's knowledge (in this case, Llama 2).
• Upload a text file containing the answer. The application chunks and embeds that content, then stores the vectors in a pgvector-compatible Postgres database for retrieval. This augments the prompt; it does not retrain the LLM.
• Run a TAS application that calls the prototype Tanzu AI Service through an OpenAI-compatible endpoint. The proof of concept used credentials that emulated a future service binding.

After we return from Explore, we will sit down and properly record our demo, but I can share a few screens/steps of what was demonstrated at the Hackathon.

Ask the app running on TAS a question outside the model's existing knowledge.

"Upload the domain-specific information as a test file– AKA some information that can provide the answer to your question."

Watch the GPU workload increase as the new information is embedded and queried.

The nvtop graphs show GPU utilization on the BOSH deployment while the application performs model inference and processes the uploaded context.

Ask the same question again – (we are looking for team 8):

There it is: a functional ChatGPT-style application on TAS, calling a prototype LLM-as-a-Service deployment managed by BOSH and retrieving domain context from embeddings stored in Postgres.

So, How does this work?

The diagram shows the prototype's overall architecture: an OpenAI-compatible model endpoint beside TAS, plus Postgres with pgvector for storing and retrieving document embeddings. The vector database supplies relevant context at request time; it is not training the underlying model.

From a future perspective, we could potentially envision a new tile or service to be loaded into the Tanzu Application Service or Cloud Foundry marketplace.

We want your feedback!

Thanks again to everyone who attended the hackathon this year! I would love to hear if you are potentially interested in this use case so we can help drive this prototype forward into an actual product.

At the time, two days of VMware Explore sessions remained. The companion guide below preserves the TAS session list from the event.

A Guide for Tanzu Application Service at VMware Explore 2023 Las Vegas