Avik Dey

@avikdey

Data • AI/ML • OSS • People • Engineering • Society • Approximately Generated Illusions; use specialized Small LMs as tools • Learn deeply to explain simply

532 Folgt817 Follower1.202 Posts

Bluesky

Ist das dein Profil?Übernimm es — dann gehört dir avikdey.dirty.social.Profil übernehmen

Avik Dey@avikdey· 14d

We in the US have harmed AI innovation by failing to define any constraints for model development. In the absence of constraints - incremental scale has became a substitute for invention. Kimi K3’s long-context design demonstrates well how constraints can force innovation. arxiv.org/pdf/2607.24653

Avik Dey@avikdey· 25d

When OpenAI head of strategic futures says it’s not distillation, that’s the strongest argument that could be that it is distillation.

Avik Dey@avikdey· Jul 14

Study Anthropic would have funded except for this line: “… our findings impose a mechanistic constraint on neural theories of language processing and on accounts that equate prediction in human language comprehension with the comparatively unconstrained operations of large language models (LLMs).”

Avik Dey@avikdey· Jul 4

Here you go - uploaded - SPA, no framework, no routing. Straight from Fable's weights, except for the Gaussian noise fix that I mentioned earlier. Even includes a clickable historical timeline and a guessing game. For a 60 token input - looks sleek! sunny-starlight-502006.netlify.app

Avik Dey@avikdey· Jul 3

Inspired by this, I asked Fable to generate a diffusion explorer. 60 tokens input, ~15k tokens output. Looks really nice. But, like before, the more you look the more you will find. Here's one visually, the mean brightness of noisy version at t=50 should be ~50% (right), instead it's ~30% (left).

Diffusion Explorer v1 (Fable generated, before fix)

Avik Dey@avikdey· Jun 25

It’s 65% now? 3 months back Anthropic was hyping it at 90%! www.techspot.com/news/112408-... Also, Karpathy trying to sell Slack integration as a new paradigm for development with AI … I am starting to think more and more - what’s he thinking?

Avik Dey@avikdey· Jun 20

1. Community notes can be fun sometimes. 2. Master vibe coder is still promoting autocomplete, not AI.

Avik Dey@avikdey· Jun 20

LOL

Avik Dey@avikdey· Jun 16

Every part of this was entirely predictable when they hired a data labeling company CEO to build their frontier model. bsky.app/profile/avik...

Also speaks volumes for their Al strategy. Data labeling works wonders for supervised models not so much for unsupervised or self-supervised models. They are hoping to fine-tune their way to better Al without understanding that would only adjust the surface finish, not improve the
framework itself.

Avik Dey@avikdey· Jun 14

My position remains: I applaud the kids for their vision and wisdom. bsky.app/profile/avik...

Avik Dey@avikdey· Jun 10

Sometimes I dig thru public code for agents … it’s fun. These ones were from the weekend. Of the ones in the parent folder this deserves special mention because it transcends the human experience. Don’t miss the <TERMINATION_CONDITIONS>!!! github.com/github/aweso...

Avik Dey@avikdey· Jun 10

So, ambitious tasks like throwaway vibe code for home automation on the weekends? “You can give it a lot more ambitious tasks than what you're used to, the model "gets it" and it will just go, and it's never felt this tempting to stop looking at the code at all (but don't do this in prod!).”

Avik Dey@avikdey· May 7

This is the kind of research paper that is an absolute joy to read. Just the thoroughness of the evaluation is remarkable. LLMs rebuilding programs from scratch with access to the existing stripped binary: 0% at 100% pass rate and a measly 3% at 95% pass rate.

Avik Dey@avikdey· Apr 30

One pager, lots of combo/variations possible.

Avik Dey@avikdey· Apr 18

The man used to be a great educator then he decided, he would rather be a mediocre influencer.

Avik Dey@avikdey· Apr 13

Why are Anthropic models better at coding than OpenAI’s? Because they have harnessed the power of the grand wizard! Seriously, how these got through any sort of review cycle and survived with modules in 1000s of lines each - does leave you wondering a bit! raw.githubusercontent.com/ComeOnOliver...

/**
* The rules of thinking are lengthy and fortuitous. They require plenty of thinking
* of most long duration and deep meditation for a wizard to wrap one's noggin around.
*
* The rules follow:
* 1. A message that contains a thinking or redacted_thinking block must be part of a query whose max_thinking_length > 0
* 2. A thinking block may not be the last message in a block
* 3. Thinking blocks must be preserved for the duration of an assistant trajectory (a single turn, or if that turn includes a tool_use block then also its subsequent tool_result and the following assistant message)
*
* Heed these rules well, young wizard. For they are the rules of thinking, and
* the rules of thinking are the rules of the universe. If ye does not heed these
* rules, ye will be punished with an entire day of debugging and hair pulling.
*/

Avik Dey@avikdey· Apr 12

What does mandates mean in this context and how is it related to the national processing? #Hungarian #Election

Avik Dey@avikdey· Apr 11

Finally a pope in my lifetime, who speaks like the Pope.

Avik Dey@avikdey· Apr 11

Karpathy thinking that folks who are professionally evaluating LLMs are using the app to assess capabilities is a sad cope - expected better from him. Not going to link, you know where to find it.

Andrej Karpathy
@karpathy •1d
Judging by my ti there is a growing gap in understanding of Al capability.

The first issue I think is around recency and tier of use. I think a lot of people tried the free tier of ChatGPT somewhere last year and allowed it to inform their views on Al a little too much. This is a group of reactions laughing at various quirks of the models, hallucinations, etc. Yes I also saw the viral videos of OpenAl's Advanced Voice mode fumbling simple queries like "should I drive or walk to the carwash". The thing is that these free and old/deprecated models don't reflect the capability in the latest round of state of the art agentic models of this year, especially OpenAl Codex and Claude Code.
But that brings me to the second issue. Even if people paid $200/month to use the state of the art models, a lot of the capabilities are relatively "peaky" in highly technical areas. Typical queries around search, writing, advice, etc. are "not* the domain that has made the most noticeable and dramatic strides in capability.
Partly, this is due to the technical details of reinforcement learning and its use of verifiable rewards. But partly, it's also because these use cases are not sufficiently prioritized by the companies in their hillclimbing because they don't lead to as much $$$ value. The goldmines are elsewhere, and the focus comes along.
So that brings me to the second group of people, who *both* 1) pay for and use the state of the art frontier agentic models (OpenAl Codex / Claude Code) do so professionally in technical domains like programming, math and research. This group ol is subiect to the hishest amount of "Al Psvchosis"

Avik Dey@avikdey· Apr 3

Though my own fascination is for studies such as this one that study generalization in infants from small training sets than listening to CEO sproutings. While the authors call it “small”, comparatively it’s more like minuscule training sets. Absolutely fascinating. arxiv.org/pdf/2510.15060

Avik Dey@avikdey· Apr 2

In early 2010s when ML was having its moment with Big Data, half of the world’s favorite ML demo app was something something sentiment analysis. In the age of LLMs, its still sentiment analysis, now with Regex - a Claude Code speciality. Interesting thread: neuromatch.social/@jonny/11632...

Avik Dey@avikdey· Mar 19

If you want to understand how shallow this analysis is, compare these visualizations - original on left and my lunch hour throwaway Gemini Pro code on the right. Data sources were BLS and O*NET. Check the title on mine. Turns out "AI exposure" looks an awful lot like "uses a computer at work".

Avik Dey@avikdey· Mar 15

We have known for a while now that “reasoning” is just another illusion, LLMs are just good at faking it. arxiv.org/pdf/2603.05488

Avik Dey@avikdey· Mar 10

Why would senior engineers be on the hook for the foolishness of engineering executives FOMO driven poor decision making? Lead by example, take responsibility for your decisions - sign off on the AI code pushes yourself. (posted by lukaszolejnik.bsky.social on the bird site)

Amazon is holding a mandatory meeting about Al breaking its systems. The official framing is "part of normal business." The briefing note describes a trend of incidents with "high blast radius" caused by "Gen-Al assisted changes" for which "best practices and safeguards are not yet fully established." Translation to human language: we gave Al to engineers and things keep breaking?

The response for now? Junior and mid-level engineers can no longer push Al-assisted code without a senior signing off. AWS spent 13 hours recovering after its own Al coding tool, asked to make some changes, decided instead to delete and recreate the environment (the software equivalent of fixing a leaky tap by knocking down the wall). Amazon called that an "extremely limited event" (the affected tool served customers in mainland China).

ALT for embedded image within the tweet image:

Amazon's ecommerce business has summoned a large group of engineers to a meeting on Tuesday for a "deep dive" into a spate of outages, including incidents tied to the use of Al coding tools.

The online retail giant said there had been a "trend of incidents" in recent months, characterised by a "high blast radius" and "Gen-Al assisted changes" among other factors, according to a briefing note for the meeting seen by the FT.

Under "contributing factors" the note included "novel GenAl usage for which best practices and safeguards are not yet fully established".

"Folks, as you likely know, the availability of the site and related infrastructure has not been good recently," Dave Treadwell, a senior vice-president at the

Avik Dey@avikdey· Feb 27

Catch? You have to know how - yourself: “It's not perfect, it needs high-level direction, judgement, taste, oversight, iteration and hints and ideas. It works a lot better in some scenarios than others (e.g. especially for tasks that are well-specified and where you can verify/test functionality).”

Karpathy on the bird site, had to remove first para - it was too long for ALT:

Just to give an example, over the weekend I was building a local video analysis dashboard for the cameras of my home so I wrote: "Here is the local IP and username/password of my DGX Spark. Log in, set up ssh keys, set up vLLM, download and bench Qwen3-VL, set up a server endpoint to inference videos, a basic web ui dashboard, test everything, set it up with systemd, record memory notes for yourself and write up a markdown report for me"
. The agent went off for ~30
minutes, ran into multiple issues, researched solutions online, resolved them one by one, wrote the code, tested it, debugged it, set up the services, and came back with the report and it was just done. I didn't touch anything. All of this could easily have been a weekend project just 3 months ago but today it's something you kick off and forget about for 30 minutes.
As a result, programming is becoming unrecognizable. You're not typing computer code into an editor like the way things were since computers were invented, that era is over. You're spinning up Al agents, giving them tasks *in English* and managing and reviewing their work in parallel. The biggest prize is in figuring out how you can keep ascending the layers of abstraction to set up long-running orchestrator Claws with all of the right tools, memory and instructions that productively manage multiple parallel Code instances for you. The leverage achievable via top tier "agentic engineering" feels very high right now.
It's not perfect, it needs high-level direction, judgement, taste, oversight, iteration and hints and ideas. It works a lot better in some scenarios than others (e.g. especially for tasks that are well-specified and where you can verify/test functionality). The key is to build intuition to decompose the task just right to hand off the parts that work and help out around the edges. But imo, this is nowhere near "business as usual" time in software.

Avik Dey@avikdey· Feb 18

Thought about it, answers No.

Avik Dey@avikdey· Feb 16

After watching Australia lose to Sri Lanka in the T20 World Cup match this morning, the only visual that came to mind was this one. Proud of my cousins from down under. (credit @jeremyphowhard via OpenAI on the bird site)

Avik Dey@avikdey· Feb 15

My favorite use of ChatGPT was as scratchpad & search tool. Last night made butter chicken for dinner. Modified original ingredients and steps to use 1 cup of heavy cream and 3.5 tbsp ground blanched almond. Asked GPT-5.2 Auto to scribe the new recipe. Mangled it. Every damn time. #MissYouChatGPT-4o

Delicious home made butter chicken in a pot.

Avik Dey@avikdey· Feb 12

Might as well announce ARC-4 and ARC-5 now. BTW, today’s TTA systems still heavily relies on DL - that distinction between “static DL” and “adaptive systems” is a spectrum not a binary.

Avik Dey@avikdey· Feb 6

OpenAI jokers don’t even understand that high performing engineers perform at their best when they are given autonomy and ownership with TRANSPARENT goals and ZERO micromanagement and NOT when they are being “governed & observable”. IDIOTS.