FFmpeg Assembly Language Lessons
Prior discussion 2025-02-22, 222 comments: https://news.ycombinator.com/item?id=43140614
Shame this doesn't start with a quick introduction to running the examples with an actual assembler like NASM.
What is the actual process of identifying hotspots caused suboptimal compiler generated assembly?
Would it ever make sense to write handwritten compiler intermediate representation like LLVM IR instead of architecture-specific assembly?
So the main issues here are not what people think they are. They generally aren't "suboptimal assembly", at least not what you can reasonably expect out of a C compiler.
The factors are something like:
- specialization: there's already a decent plain-C implementation of the loop, asm/SIMD versions are added on for specific hardware platforms. And different platforms have different SIMD features, so it's hard to generalize them.
- predictability: users have different compiler versions, so even if there is a good one out there not everyone is going to use it.
- optimization difficulties: C's memory model specifically makes optimization difficult here because video is `char *` and `char *` aliases everything. Also, the two kinds of features compilers add for this (intrinsics and autovectorization) can fight each other and make things worse than nothing.
- taste: you could imagine a better portable language for writing SIMD in, but C isn't it. And on Intel C with intrinsics definitely isn't it, because their stuff was invented by Microsoft, who were famous for having absolutely no aesthetic taste in anything. The assembly is /more/ readable than C would be because it'd all be function calls with names like `_mm_movemask_epi8`.
Normally you spin up a tool like vtune or uprof to analyze your benchmark hotspots at the ISA level. No idea about tools like that for ARM.
> Would it ever make sense to write handwritten compiler intermediate representation like LLVM IR instead of architecture-specific assembly?
IME, not really. I've done a fair bit of hand-written assembly and it exclusively comes up when dealing with architecture-specific problems - for everything else you can just write C (unless you hit one of the edge cases where C semantics don't allow you to express something in C, but those are rare).
For example: C and C++ compilers are really, really good at writing optimized code in general. Where they tend to be worse are things like vectorized code which requires you to redesign algorithms such that they can use fast vector instructions, and even then, you'll have to resort to compiler intrinsics to use the instructions at all, and even then, compiler intrinsics can lead to some bad codegen. So your code winds up being non-portable, looks like assembly, and has some overhead just because of what the compiler emits (and can't optimize). So you wind up just writing it in asm anyway, and get smarter about things the compiler worries about like register allocation and out-of-order instructions.
But the real problem once you get into this domain is that you simply cannot tell at a glance whether hand written assembly is "better" (insert your metric for "better here) than what the compiler emits. You must measure and benchmark, and those benchmarks have to be meaningful.
> Normally you spin up a tool like vtune or uprof to analyze your benchmark hotspots at the ISA level. No idea about tools like that for ARM.
perf is included with the Linux kernel, and works with a fair amount of architectures (including Arm).
perf doesn't give you instruction level profiling, does it? I thought the traces were mostly at the symbol level
Hit enter on the symbol, and you get instruction-level profiles. Or use perf annotate explicitly. (The profiles are inherently instruction-level, but the default perf report view aggregates them into function-level for ease of viewing.)
> Would it ever make sense to write handwritten compiler intermediate representation like LLVM IR instead of architecture-specific assembly?
Not really. There are a couple of reasons to reach for handwritten assembly, and in every case, IR is just not the right choice:
If your goal is to ensure vector code, your first choice is to try slapping explicit vectorize-me pragmas onto the loop. If that fails, your next effort is either to use generic or arch-specific vector intrinsics (or jump to something like ISPC, a language for writing SIMT-like vector code). You don't really gain anything in this use case from jumping to IR, since the intrinsics will satisfy your code.
If your goal is to work around compiler suboptimality in register allocation or instruction selection... well, trying to write it in IR gives the compiler a very high likelihood of simply recanonicalizing the exact sequence you wrote to the same sequence the original code would have produced for no actual difference in code. Compiler IR doesn't add anything to the code; it just creates an extra layer that uses an unstable and harder-to-use interface for writing code. To produce the best handwritten version of assembly in these cases, you have to go straight to writing the assembly you wanted anyways.
Loop vectorization doesn't work for ffmpeg's needs because the kernels are too small and specialized. It works better for scientific/numeric computing.
You could invent a DSL for writing the kernels in… but they did, it's x86inc.asm. I agree ispc is close to something that could work.
I was expecting to read pearls of wisdom gleaned from all the hard work done on the project, but I’m not really getting how this relates to ffmpeg.
The few chapters I saw seemed to be pretty generic intro to assembly language type stuff.
Why not include the required or targeted math lessons needed for the FFmpeg Assembly Lessons in the GitHub repository? It'd be easier for people to get started if everything was in one place :)
NTA but if the assumption is that the reader has only a basic understanding of C programming and wants to contribute to a video codec there is a lot of ground that needs to be covered just to get to how the cooley/tukey algorithm works and even that's just the basic fundamentals.
I read the repo more as "go through this if you want to have a greater understanding of how things work on a lower level inside your computer". In other words, presumably it's not only intended for people who want to contribute to a video codec/other parts of ffmpeg. But I'm also NTA, so could be wrong.
Love it. Thanks for taking the time to write this. Hope it will encourage more folks to contribute.
How do they make these assembly instructions portable across different cpus?
I think there's a generic C fallback, which can also serve as a baseline. But for the big (targeted) architectures, there one handwritten assembly version per arch.
Yup.
On startup, it runs cpuid and assigns each operation the most optimal function pointer for that architecture.
In addition to things like ‘supports avx’ or ‘supports sse4’ some operations even have more explicit checks like ‘is a fifth generation celeron’. The level of optimization in that case was optimizing around the cache architecture on the cpu iirc.
Source: I did some dirty things with chromes native client and ffmpeg 10 years ago.
They don't. It's just x86-64.
The lessons yes, but the repo contains assembly for the 5-6 architectures in wide use in consumer hardware today. Separate files of course. https://github.com/FFmpeg/FFmpeg/tree/master/libavcodec
Yeah, sure. I was specifically referring to the tutorials. Ffmpeg needs to run everywhere, although I believe they are more concerned about data center hardware than consumer hardware. So probably also stuff like power pc.
More interesting than I thought it could be. A domain specific tutorial is so much better.
There is serious abuse of nasm macro-preprocessor. Going to be tough to move away to another assembler.
Why move away?
Where? There's very little code in those lessons
The lessons reference `cglobal` in `x86inc.asm`:
https://github.com/FFmpeg/FFmpeg/blob/master/libavutil/x86/x...
I feel like I just got a 3 page intro to autism.
It's glorious.
I can’t imagine the scale that FFMPEG operates at. A small improvement has to be thousands and thousands of hours of compute saved. Insanely useful project.
Their commitment to performance is a beautiful thing.
Imagine all projects were similarly committed.
There's tons of backlash here as if people think better performance requires writing in assembly.
But to anyone complaining, I want to know, when was the last you pulled out a profiler? When was the last time you saw anyone use a profiler?
People asking for performance aren't pissed you didn't write Microsoft Word in assembly we're pissed it takes 10 seconds to open a fucking text editor.
I literally timed it on my M2 Air. 8s to open and another 1s to get a blank document. Meanwhile it took (neo)vim 0.1s and it's so fast I can't click my stopwatch fast enough to properly time it. And I'm not going to bother checking because the race isn't even close.
I'm (we're) not pissed that the code isn't optional, I'm pissed because it's slower than dialup. So take that Knuth quote you love about optimization and do what he actually suggested. Grab a fucking profiler, it is more important than your Big O
like Slack or Jira... lol.
That would be an enormous waste of time. 99.9% of software doesn't have to be anywhere near optimal. It just has to not be wasteful.
Sadly lots of software is blatantly wasteful. But it doesn't take fancy assembly micro optimization to fix it, the problem is typically much higher level than that. It's more like serialized network requests, unnecessarily high time complexities, just lots of unnecessary work and unnecessary waiting.
Once you have that stuff solved you can start looking at lower level optimization, but by that point most apps are already nice and snappy so there's no reason to optimize further.
Sorry, I would word it differently. 99.9% software should be decently performant. Yes, don't need 'fancy assembly micro optimization'. That said, today some large portion of software is written by folks who absolutely doesn't care about performance - just duct-taping some sh*t to somehow make it work and call it a day.
Seems to me like we're in agreement.
Yeah no, I'd like non-performance critical programs to focus on other things than performance thank you
Hard disagree. I'd like word processors to not need ten seconds just to start up. I'd like chat clients not to use _seconds_ to echo my message back to me. I'd like news pages that don't empty my mobile data cap just by existing. All of these are “non-performance critical”, but I'd _love_ for them to focus on performance.
So you’re a PM for a word processor. You have a giant backlog.
Users want to load and edit PDFs. Finnish has been rendering right to left for months, but the easy fix will break Hebrew. The engineers say a new rendering engine is critical or these things will just get worse. Sales team says they’re blocked on a significant contract because the version tracking system allows unaudited “clear history” operations. Reddit is going berserk because the icon you used (and paid for!) for the new “illuminated text mode” turns out to be stolen from a Lithuanian sports team.
Knowing that most of your users only start the app when their OS forces a reboot… just how much priority does startup time get?
This is an incredibly convoluted hypothetical trying to negate the idea that users notice and/or appreciate how quickly their applications start. Usually as a PM you are managing multiple engineers, one of which I would assume is capable of debugging and eventually implementing a fix for faster start times. Even if they can't fix it immediately due to whatever contrived reason you've supposed, at least they will know where and how to fix it when time does come. In fact, I would argue pretending there is no issue because of your mountain of other problems is the worst possible scenario to be in.
When I was in school I had a laundry app (forced to use) that took 8 seconds to load, mostly while it scanned the network for the machines. It also had the rooms out of order in the room listing and no caching so every time you wanted to check the status (assuming it even worked) it took no less than a minute. It usually took less time to physically check, which also had a 100% accuracy.
Fuck this "we don't need to optimize" bullshit. Fuck this "minimum viable product" bullshit. It's just a race to the bottom. No one paper cut is the cause of death, but all of them are when you have a thousand.
> I'd like news pages that don't empty my mobile data cap just by existing.
To be fair, this is because they mostly care about serving ads. Without the ads, the pages are often fine.
Many things are slow because few programmers (or managers) care. Because they'll argue about "value" but all those notions of value are made up anyways.
People argue "sure, it's not optimal, but it's good enough". But that compounds. A little slower each time. A little slower each application. You test on your VM only running your program.
But all of this forgets what makes software so powerful AND profitable: scale. Since we always need to talk monetary value, let's do that. Shaving off a second isn't much if it's one person or one time but even with a thousand users that's over 15 minutes, per usage. I mean we're talking about a world where American Airlines talks about saving $40k/yr by removing an olive and we don't want to provide that same, or more(!), value to our customers? Let's say your employee costs $100k/yr and they use that program once a day. That's 260 seconds or just under 5 minutes. Nothing, right? A measly $4. But say you have a million users. Now that's $4 million!
Now, play a fun game with me. Just go about your day as normal but pay attention to all those little speedbumps. Count them as $1m/s and let me know what you got. We're being pretty conservative here as your employee costs a lot more than their salary (2-3x) and we're ignoring slowdown being disruptive and breaking flow. But I'm willing to bet in a typical day you'll get on the order of hundreds of millions ($100m is <2 minutes).
We solve big problems by breaking them into a bunch of smaller problems, so don't forget that those small problems add up. It's true even if you don't know what big problem you're solving.
I have uBO, they're still obscenely large.
untrue. what bloats the modern web is the widespread AND suboptimal use of web frameworks. otherwise, making adblockers would dramatically speed up the loading of every website that uses ads, while it is true to some extent, is not the entire picture. anyways, i'm not saying that these libraries are always slow, but the users aren't aware of the performance characteristics and perf habits they should use while making use of such libraries. do you have any idea how many tens of layers of abstractions a "website" takes to reach your screen?
untrue. what makes bloats the modern web is competing incentives and businesses choosing what they think is going to make them the most money.
> None of these are “non-performance critical”, but I'd _love_ for them to focus on performance
Then you agree with the poster. Performance critical software should focus on performance.
This mentality brings you a loading screen when you start the calculator on windows.
What? Calculator starts up faster than I can figure out on where and on which screen it decided to open
Surely all programs are performance critical. Any program we think isn't is just a program where the performance met the criteria already.
Safety critical systems say hello.
> Safety critical systems
Any concrete examples where we can see the code?
sqlite is probably our best example. The project touts use within Airbus A350 and DO-178B certification.
Indeed. All else remaining the same, a faster program is generally more desirable than a slower program, but we don't live in generalities where all else remains the same and we simply need to choose fast over slow. Fast often costs more to produce.
Programming is a small piece of a larger context. What makes a program "good" is not a property of the program itself, but measured by external ends and constraints. This is true of all technology. Some of these constraints are resources, and one of these resources is time. In fact, the very same limitation on time that motivates the prioritization of development effort toward some features other than performance is the very same limitation that motivates the desire for performance in the first place.
Performance must be understood globally. Let's say we need a result in three days, and it takes two days to write a program that takes one day to get the result, but a week to write a program that takes a second to produce a result, then obviously, it is better to write the program the first way. In a week's time, your fast program will no longer be needed! The value of the result will have expired.
This is effectively a matter of opportunity cost.
There's nothing more permanent than a temporary fix that works.
Seems so easy! You only need the entire world even tangentially related to video to rely solely on your project for a task and you too can have all the developers you need to work on performance!
ffmpeg has competition. For the longest time it wasn't the best audio encoder for any codec[0], and it wasn't the fastest H.264 decoder when everyone wanted that because a closed-source codec named CoreAVC was better[1].
ffmpeg was however, always the best open-source project, basically because it had all the smart developers who were capable of collaborating on anything. Its competition either wasn't smart enough and got lost in useless architecture-astronauting[2], or were too contrarian and refused to believe their encoder quality could get better because they designed it based on artificial PSNR benchmarks instead of actually watching the output.
[0] For complicated reasons I don't fully understand myself, audio encoders don't get quality improvements by sharing code or developers the way decoders do. Basically because they use something called "psychoacoustic models" which are always designed for the specific codec instead of generalized. It might just be that noone's invented a way to do it yet.
[1] I eventually fixed this by writing a new multithreading system, but it took me ~2 years of working off summer of code grants, because this was before there was much commercial interest in it.
[2] This seems to happen whenever I see anyone try to write anything in C++. They just spend all day figuring out how to connect things to other things and never write the part that does anything?
I seem to recall that they lamented on twitter the low amount of (monetary or code) contribution they got, despite how heavily they are used.
No one is forcing them to produce code for free. There is something toxic about giving things away for free with the ulterior motive of getting money for it.
You know friend, if open source actually worked like that I wouldn’t be so allergic to releasing projects. But it doesn’t - a large swath of the economy depends on unpaid labour being treated poorly by people who won’t or can’t contribute.
It'd be nice, though, to have a proper API (in the traditional sense, not SaaS) instead of having to figure out these command lines in what's practically its own programming language....
FFMpeg does have an API. It ships a few libraries (libavcodec, libavformat, and others) which expose a C api that is used in the ffmpeg command line tool.
They publish doxygen generated documentation for the APIs, available here: https://ffmpeg.org/doxygen/trunk/
Don't know how I overlooked that, thanks. Maybe because the one Python wrapper I know about is generating command lines and making subprocess calls.
They're relatively low level APIs. Great if you're a C developer, but for most things you'd do in python just calling the command line probably does make more sense.
It could even make sense in C. In some circumstances, I wouldn’t feel bad for cutting that corner.
For future reference, if you want proper python bindings for ffmpeg* you should use pyav.
* To be more precise, these are bindings for the libav* libraries that underlie ffmpeg
If you are processing user data, the subprocess approach makes it easier to handle bogus or corrupt data. If something is off, you can just kill the subprocess. If something is wrong with the linked C api, it can be harder to handle predictably.
Also because you can apply stricter sandboxing/jail/containerization to the process.
I get why the CLI is so complicated, but I will say AI has been great at figuring out what I need to run given an English language input. It's been one of the highest value uses of AI for me.
hell yeah, same here. i made a little python GUI app to edit videos