For the inference part, you can dockerise your model and use https://banana.dev for serverless GPU. They have examples on github on how to deploy and I’ve done it last year and was pretty straightforward. - Source: Hacker News / about 1 month ago

I want to first check the user's ID and only if the user has an active subscription then the request will be forwarded to my API on banana.dev else the request will be blocked at the middleware itself. Should I use Express JS for the middleware i.e. Authentication and forwarding requests? Is there any other better way to improve my project structure? Currently it looks like:. Source: 6 months ago

Hey! Would love to have you try https://banana.dev (bias: I'm one of the founders). We run A100s for you and scale 0->1->n->0 on demand, so you only pay for what you use. I'm at erik@banana.dev if you want any help with it :). - Source: Hacker News / about 1 year ago

CAN you do this in AWS? Of course, do they have a service that does exactly what this banana.dev does? Probably not. Source: about 1 year ago

I've been using banana.dev for easily running my ML models on GPU in a serverless manner, and interacting with them as an API. Although the principle of the service is sound, it is currently too buggy to take into production (very long cold boots, errorring requests, always hitting capacity). Source: about 1 year ago

This is a good place to mention https://nuitka.net/ which aims to compile python programs into standalone binaries. - Source: Hacker News / about 1 month ago

For Python, you could make a proper deployment binary using Nuitka (in standalone mode – avoid onefile mode for this). I'm not pretending it's as easy as building a Go executable: you may have to do some manual hacking for more unusual unusual packages, and I don't think you can cross compile. I think a key element you're getting at is that Go executables have very few dependencies on OS packages, but with Python... - Source: Hacker News / about 2 months ago

There is already an AOT compiler for Python: Nuitka[0]. But I don't think it's much faster. And then there is mypyc[1] which uses mypy's static type annotations but is only slightly faster. And various other compilers like Numba and Cython that work with specialized dialects of Python to achieve better results, but then it's not quite Python anymore. [0] https://nuitka.net/ [1] - Source: Hacker News / 5 months ago

Nuitka deals pretty well with those in general: https://nuitka.net/. - Source: Hacker News / 10 months ago

Have a look at Nuitka, which is a "real" Python compiler into C. It uses CPython in its backend and should be completely compatible to "regular" Python. The compiled code can, but does not have to improve performance. It's probably worth looking into. Source: about 1 year ago