Oops, realized I didn’t answer your question about actually crawling, dig into common crawl documentation, they provide a bunch of technical data and stats that show you the scale…2-4billion pages per month
And note CC just does a sample of the pages it finds. So the more monthly dumps don’t contain all of the data afaik
And the number above are for one of the monthly dumps
https://commoncrawl.github.io/cc-crawl-statistics/plots/crawlermetrics
The index (i.e just the list of urls) of common crawl is 250gb. The actual crawl is 500tb ish. Then pull out just the text you get down to 30tb ish. And then you need that all to be in a fts index… (something like opensearch/elasticsearch) idk how big the cluster you would need for that is… maybe 2tb per data node max. So 15 pretty beefy vms (8cpu w/64gb memory) with 2tb nvme for data and a few more as controller nodes
Common crawl numbers pulled from this paper
https://aclanthology.org/anthology-files/anthology-files/pdf/L/L18/L18-1286.pdf
Cluster sizing numbers pulled from tea leaf divination
Are you the llm though…this is a mangled rephrasing of the punchline… was your prompt something like rephrase a common from a similar thread
Better Nate than lever?
Is this an llm bugging out by rephrasing?
Nice :)
Nice, ok hmm… sounds like you are using the router for internal DNS, did you set all the services to point at caddy?
Eg you could try a quick hack in hosts (assuming the caddy ip is 192.168.10.10)
192.168.10.10 grafana.domain.example
192.168.10.10 prometheus.domain.example
192.168.10.10 jellyfin.domain.example
And then navigate to one of the services over https
Main thing is all your internal DNS records need to point at caddy, and then it forwards to the real container IP
Not strictly required but could be worthwhile to get a dedicated internal dns server up… I use technitium, or there’s pi hole or a few other options …
I’m trying to highlight the absurdity of tieing buggyness to sloppyness, they are two distinct axis, not a spectrum. There are plenty of problems with LLMs but bugginess is not one of them. Idk if you’ve been watching the news lately, bug squashing in old code is up big time, can you guess why? And I wonder have you ever worked on a real codebase? Because the only perfect code is the code you wrote 5minites ago… go look at Linux kernel code… arguably the most used but of software out there, find me anything perfect in there… like what is going on here in this artisanal, hand rolled, organic code?! You’re arguing devs need more time to work on security, well guess what’s giving them more time to work on security…
Ps. Sorry to the Bluetooth dev, didn’t mean for you to catch strays, I love your code anyway!
If you are doing a dns challenge then You need to give caddy access to your domain to perform the dns challenge. Depending on domain provider that is easy or hard. It will briefly change some records that tells let’s encrypt you own the domain and then issue the cert
Or you can do http challenge, this requires you to manually configure your domain to point to a publicly accessible caddy endpoint which is probably not what you want, maybe there is a way to make it work through a cloud flare tunnel
Who is your domain provider?
Its so annoying that all code has been perfect and bug free until these silly LLM people came and made buggy code
Envoy is what i use, or squid is an option
They can be used for training…
That’s all software, I don’t think anyone is doing back prop in hardware. Your are right about cuda, but z.ai just demonstrated that moat is evaporating quick
DANE provides user friction…but as OP mentioned TLS doesn’t work
Very old article, they are absolutely just using llms nowdays
Edit: this is the paper https://arxiv.org/pdf/1611.04558 the article is about
Maybe not apt…but sudo dnf install nix nix-daemon
Edit: which doesn’t answer your question about free runners… but can you trust them anyway? They are running a container afaik
Not too difficult to fix this in post.
Do database roles even exist anymore? It’s all been “left shifted” to devs these days
Hey now, don’t shoot the messenger, I was just grounding the conversation in the reality of how I’ve seen doctors typically work (in Australia).
But …your doc has an In person scribe? That sounds pretty great! It is another arrow in the side of the for profit medical industry that not everyone has access to this level of service.
Don’t think they’re call centers, docs just record in a little recorder and send it to transcription services, it’s not live or anything. They don’t get more context than the recording itself…
A computer algorithm can absolutely be non deterministic, pick your language and run random.rand()
There is a whole class of such algorithms https://en.wikipedia.org/wiki/Nondeterministic_Turing_machine
But I don’t see how it makes a difference, even deterministic algorithms can produce infinite complexity, eg linear neural networks are proven to be able to compute any computable function, or the Mandelbrot set which is produced from a single recurrent statement but encodes a space of infinite complexity
Maybe your point is to say that a brain is performing a non computable function…but since your brain is computing that function…is that a counter example?