LLM Inference on Web

https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference/web_js

https://github.com/jasonmayes/Web-AI-Spotify-DJ

I was watching TikTok and chilling in my favorite shop per usual and that's when it showed me a one peculiar demo.

Turns out people at Google has already implementing a support to run LLM on device in browser. In other words your web browser can open a .html page or execute an inference call to any LLM.

Actually it is much more than that, your browser via WebAssembly can use many more capabilities such as operating on video and audio files via ffmpeg that is widely used on server and desktop applications, sqlite which is a lightweight, cutting edge sql database. It doesn't stop there a popular "graphics" library Raylib can be compiled for OpenGL for desktop platform and for WebGL in WASM.

https://github.com/raysan5/raylib/wiki/Working-for-Web-(HTML5)

This makes the browser the easiest platform of distribution for a wide range of software. That can later be integrated more tightly to the platform by stepping down each level: from browser platform to crossplatform applications and further to native platform to make better use of hardware by making steps to run closer to it.

A lost art of platform optimization and embedded software development per se.

With a vibecoding evolution, it becomes mostly irrelevant on what language are you writing since the core linguistics of it comes down to Calculus and Church-Turing thesis.

All total functions are computable functions

Evolution of Autonomous AI Agents

Let's take a step back and shift our focus towards AI creator economy.

Last year a crypto AI agents received a lot of adoption on the launches of their tokens, a few ones of the top of my head:

Truth terminal: https://truthterminal.wiki/docs/home, https://x.com/truth_terminal
Aixbt: https://x.com/aixbt_agent
- Finance Wizard: https://x.com/financewizardio
- Gork https://x.com/gork

All of them, even Gork showed a practical case of on chain monetization of AI tokens.

Yet they are very far from wide adoption due to novelty and general ignorance.

The thing is, their sheer scale of speed and computational power already surpassing common market labor, essentially squeezing out the rest of us.

There is no point doing better anymore, only in making a better AI that will do it for you.

Still, what I am saying sounds quite abstract and far fetched to the point that it is just being brushed off - well that not gonna apply to me or my job.

Except there is already an ecosystem being established, Agent Commerce Protocol: https://app.virtuals.io/research/agent-commerce-protocol

In other words you can make a contract for a token exchange to buy a derivative of inference token... I hope you are catching my drift here.

In other words you buy a prompt, this prompt can act as a command to generate a content or an asnwer back. You already doing it via ChatGPT or Cursor.

Open AI went further to provide a dialog experience with a "black circle", a few iterations later this "black circle" will be transformed to a user friendly head.

On the other side, there is a whole industry of VTubing, a Japanese influence on running a virtual youtube online entertainers.

This agent (Luna) https://app.virtuals.io/virtuals/68 can provide a good representation of current capabilities at this moment. The online stream part looks a bit rudimentary yet, but not so long the quality of the video and interaction will be on par and eventually surpass the level of individual person.

So where this "friendly head" would exist if not on the stream of a social media platform.

While Twitter has already got occupied by Agents, Facebook platform seems to be unaffected, you think? Or maybe you don't just seem them in plain sight...

As for Instagram platform it looks like the general catalyst would be the ability to produce a high grade image that would drive AI adoption, on TikTok I don't know anymore what is AI generated and what is the recorded video.

Regardless of a platform, all of them, in my opinion favors a stream, engagement that is automated. And running 24/7 stream on a platform is not that trivial.

Music streaming Agent

That Spotify DJ demo inspired me to make my own spin of it too. This is my results of research and my trials of running a stream.

Since the Web is the easiest method of delivery and prototype already exist, this was my starting point.

Having a browser version it is possible to adapt to a stream setup or standalone application. However you wont be able to run Spotify under them.

This is where the things start to fall apart, WebGL engine does not have a support on OBS browser source for text to speech model, so it does have to be generated server side, web llm inference should be supported but I haven't confirmed it yet.

I'll make another article on streaming setups if there will be any interest.

Summary

The browser platform comes as a variation for different places and puts its own constraints on software solutions. None of it contributes to viability of calling llm inside of a browser in favor of backend.

So the streaming solution has to be heavily customized. Since js can also be run on the server with Node and Bun it is relatively easy to shuffle around business logic and abstract the interfaces from which it is called from. Once again proving the point of how malleable the "soft" in software.

On a low spec hardware scenario though, it is unbearably laggy and expensive to call an inference compared to calling Open AI endpoint for a few cents. $5 is cheaper than the effort and hardware needed to set up a llm, if hardware allows though, simple ollama would work for streaming or local setup.

In summary, streaming and standalone desktop app, despite the same environment, takes different approaches than web browser, yet the browser is the most accessable and licensed. Below is the link to a demo and a source code.

https://jellyninja.gumroad.com/l/siiom

0 comments

Lead through any storm with the Shackleton decision manual

New templates announcement

How Leaf Panda Keeps Your Roses Safe with Multi-Sig Magic

The Master Key App

The Playbook Revolution: How to Close More Deals Without the Guesswork

See all posts from Vlad Rimsha