If the start date and end date are both week days, then the total number of week days in between is simply:

(total difference in days) – (total difference in weeks) * 2

DateDiff(dd, @start, @end) – DateDiff(ww, @start, @end)*2

… since the DateDiff() function with weeks returns the number of week "boundaries" that are crossed; i.e., the number of weekends.
If you have a table of holidays, then you can simply subtract them out as well:

DateDiff(dd, @start, @end) –
DateDiff(ww, @start, @end)*2 –
(select count(*) from holidays where holiday_date between @start and @end)

Now, what if the start day or the end day is on a weekend? In that case, you need to define what to do in those situations in your requirements.
For example, if the start date is Sunday, Nov 20th, and the end day is Monday, Nov 21st — how many week days are between those dates? There’s no universal correct answer; it could be 0, or 1, or perhaps even "undefined" (null) depending on your needs.