.NET Performance Optimization for High-Traffic Web Apps: A Backend Engineering Checklist

This is what makes backend performance so aggravating, because everything can work perfectly. The app got deployed a month ago, nobody has changed anything about it, everything works, and suddenly it doesn’t. And there are a thousand different reasons why.

Maybe a request had a timeout connecting over the internet for some reason. Maybe the database was running a backup at that very moment and was slow. Maybe some third party API that the app depends on broke. Cloud computing makes it easy to deploy apps, but it also introduces lots of points of failure, all these various small apps connecting to all these various small services, and any of them could go down at any time.

None of it will show up in a Lighthouse report. You can tune the front end to be perfect, but the page will still wait for the backend to take 800ms to respond.

Why Is the App Slow Only Sometimes?

Usually because something else is running on a schedule and nobody is looking right at it when it happens.

A team had an issue with a page that would sometimes half work. Half of it would load and the other half wouldn’t. When users reported it, the developer checked and everything seemed fine. The developer checked again the next day and it was fine. The day after, it was fine. There was no reason it sometimes worked and sometimes didn’t, and it couldn’t be reproduced.

Deployed software requires at least some sort of monitoring. Instead of relying on “This developer tested it and it was OK” or “That developer tested it and it was OK,” it’s nice to have something test it for you, every minute, possibly from multiple locations across the country, or the world.

The team decided to add a synthetic monitor. It simply took the web page URL and tried to load it every minute. They checked back an hour later, looking at the graph, wondering whether it was working, or how slow it was. And then they saw the pattern. Every 15 minutes, it went slow for two to three minutes, and otherwise, it worked normally.

Now the question was, what, in the world, was running on a schedule every 15 minutes, right on the 15s?

They went into the cloud logs and the database logs and found it. There was a scheduled job that attempted to delete old logging data out of the database that nobody ever really cared about anyway, and the table had gotten large enough that the delete operations would time out. Without the observability, that’s actually quite hard to determine. With it, the investigation took one afternoon.

Another team hit a similar issue in a slightly different way. They had some reminder emails that they needed to go out on schedule since scheduling mattered for their users. The emails would come out at 12:16 or 12:31 instead of 12:15 and 12:30, and no one knew why. Their database would take a little backup every 15 minutes, and during the backup the database would be locked. It would only take 30 seconds or maybe a minute. That was enough.

How Do You Find the Bottleneck?

Ask one question: Where does everything stop? Performance problems are typically easy to solve once you ask this. Work passes through a system like water. At some point there is a dam. Find the point where flow stops, and start fixing there.

The Data Access Layer

In .NET applications, the bottleneck is often data access. Entity Framework Core allows developers to easily write the N+1 query and easily overlook it: a page displaying 50 records fires off 50 additional queries for related data, and works perfectly against a development database with 12 rows.

Poorly indexed tables can do the same thing. It worked great when you first created the schema, but then the data grew, and no code change shows up in a diff to explain why the page that worked last month doesn’t work today. Furthermore, read-only API calls that incur the overhead of change tracking can go here; that’s what AsNoTracking() is for.

Blocking Calls in Async Code

The more insidious version is a design choice that is great for one thing but terrible for another. A system may read great but be disastrous for concurrent writes. Blocking calls such as .Result and .Wait() in asynchronous code is a classic .NET issue because it takes threads hostage, and the thread pool starves.

But the thing is, if you’re not seeing heavy load, the problem will never manifest. It will start showing up once the load comes, and then it looks just like a slow database and people end up tuning the wrong things. Also this is why the test must be done before the load increases, not after. Maybe these technologies were right for the initial load levels, and the app has simply outgrown them over time.

Where the Cache Lives

Caching is the cheapest win on the board. Someone says “we should use more caching”, it gets enabled, they write a couple lines of code and the app serves a lot more traffic. Where the cache lives is a judgment call. You shouldn’t be trying to solve million-user issues from day one, and a single instance app might be able to live off IMemoryCache for years. The goal is to architect the app so that if you need more than 1 of everything in the future, then more than 1 database, more than 1 redis cache, more than 1 queue – you have a path to getting there. Cross the bridge when you get to it: when you reach two servers, then your two in-memory caches will be out of sync and it’s time to consider a distributed cache.

Why Isn’t an Uptime Check Enough?

Because your app might be up, technically, but the backend sync job hasn’t worked in three days. It’s like, “Our website is up, our app is available.” But the calendar sync on the backend isn’t working. This is why an uptime SLA by itself isn’t good enough, and why teams establish SLOs for the various parts of a system that perform various functions. How frequently does the sync run? Does it function or not? You have to keep tabs on every moving component because, sooner or later, something is going to break.

There will be a lot of noise coming out of monitoring. Some sort of random database timeout, some sort of this or that. Apps will throw errors at random; it’s just part of running them. If no one is watching the errors, then no one is going to be able to sift through them and find out which ones are important and which ones aren’t, and the important ones won’t end up in the next sprint to be fixed.

As far as response time goes, the Server-Timing API is an easy first step. The server adds its own values to the response, namely database duration, cache duration, time spent waiting on some third-party API, and they appear in browser dev tools alongside the frontend waterfall. Combine this with an APM tool and “the site feels slow” turns into “this endpoint spends 600ms on a single query.”

What If the Team Can’t Get to the Work?What If the Team Can’t Get to the Work?

When a team has something like this, they sometimes have to make choices and prioritize. There is some work that needs to get done that a team just isn’t great at. And that’s fine! Every developer is really good at some things and bad at other things.

If the thing they aren’t great at is the team’s weak spot, then someone has to realize that and say, hey, this isn’t something we should be doing right now, we’re not good at that thing. We need someone who is really good at that thing.

Anybody who has heard of the term opportunity cost knows that it refers to the value of the next best alternative. So what is the value of what the team didn’t choose to do? If the team is racking up a ton of opportunity cost because it is trying to do a lot of work that isn’t its strong suit, the engineers could be shipping the roadmap rather than reading query plans. At that point, companies bring in dedicated C# developers to run the backend performance work as its own scoped project or to hang out with the team until the backlog clears.

A Backend Performance Checklist

Every item in this list is about the same question: where does everything stop? Follow the water through the system and check the usual dams.

• Find out what’s actually running every 15 minutes on the servers. Something is.
• List pages and dashboard endpoints are the most common spots for N+1 queries. Audit them first. Add AsNoTracking() to any read-only queries.
• Make sure the indexes cover what the application is currently querying, not what was requested at launch.
• Read-heavy tables should be cached.
• Search the codebase for .Result and .Wait().
• Stress test before the marketing team does it for you.
• Inspect each part in isolation. A sync process can fail silently for weeks and still report 100% availability.

The Short Version

Even the best designed and most thoughtfully crafted software is going to break, and a lot of the causes will be outside of anything a team can control. The important thing is to have some level of basic monitoring, at least some basic monitoring, so you have a basis for understanding if the core functionality of your product is working or not, and you know who to call when it isn’t.

Compared to achieving that level of visibility, the fix is nearly always the easy part. What you really need is someone’s time. Because one of the things that operations teams do when running a software platform is make sure that the software platform is running. And somebody needs to take ownership of that.

Frequently Asked Questions (FAQs) Related to .NET Performance Optimization

Q1. Does frontend optimization still matter if the backend is slow?

Of course it matters, every ms after the initial response is better. But there’s a limit. If the server takes 800ms to respond, then that 800ms is already gone, you can’t trim the bundle enough to save those 800ms.

Q2. Is Redis always necessary?

Nope, and if you try to fix all possible scaling issues on day 1, that’s a trap. It’s entirely normal for a single instance to survive on in-memory caching for years. Pick solutions that allow you to scale later, and don’t worry about those problems until your second instance goes live.

Q3. How much of this can automated tools catch?How much of this can automated tools catch?

A modern platform makes you feel like a kid in a candy store: queuing, caching, all these different database technologies, all available in a few clicks. The hard part has become knowing which ones to use and how to use them.

APM tools are good at finding slow endpoints and N+1s. Judging which kind of caching strategy fits what you’re trying to access is still up to the backend dev.

Q4. Where should a team start if nothing is monitored today?Where should a team start if nothing is monitored today?

A single synthetic test on the page you care about, at 1-minute intervals, and someone who logs the errors. That’s all it takes to go from “sometimes it’s slow” to a graph with a trend line.

[/vc_column_text][/vc_column][/vc_row]