Coburn pounced before half-time when he headed in at the back post after Tristan Crama had hooked a ball back into the danger zone and that was enough to seal all three points.
And on a hard night for John Eustace's side, they have been left with one fit senior goalkeeper after Vickers appeared to pull a muscle in the final stages of the game with Jacob Widell Zetterstrom already on the treatment table.
Derby could have taken the lead when a corner from Joe Ward picked out Patrick Agyemang whose towering header beat Anthony Patterson, but not Zak Sturge who managed to clear off the line.
Milwall's response was for Zak Sturge to see a header clawed away by Josh Vickers before Alex Neil's side rattled the post, although it would not have counted due to a foul from Josh Coburn.
Joe Ward had a long shot from distance that fizzed past the upright but just before the break, Millwall took the lead.
Tristan Crama managed to turn a ball into the danger area at the back post and there to meet it was Coburn who headed into the corner of the net.
Rhian Brewster tried to find Ben Brereton Diaz at the back post just before the break, but the latter just couldn't reach his cross.
Eustace sent on Bobby Clark at half-time for Ozoh and the former teed up an effort for Lewis Travis after the break. but the skipper's effort flew past the upright.
But the best chance Derby created came when a clever ball over the top from Szmodics allowed Agyemang to use his pace during which he fended off a challenge Crama.
That took him into the area, but with just the goalkeeper to beat, he dragged his shot wide of the post. Szmodics then dragged a shot past the upright as Derby pushed for an equaliser.
And they were so close to making it 1-1 when Carlton Morris headed inches wide of the post. There was still time for Agyemang to almost snatch a point but his shot cannoned off the upright.
Derby almost had an equaliser at the end as Agyemang hit the post, but it was a much-improved second-half performance. Big chances wasted mind. Ratings to follow.
It's almost 1-1 but Morris heads inches wide of the post.
Szmodics drags a shot wide past the upright. Another good opportunity goes begging.
Szmodics plays a ball over the top and I said earlier that I fancied Derby to get some change out of Milwall's high line.
Crama lets the ball bounce, Agyemang then does brilliantly to fend him off, turn and make space for the shot.
He's left with just the keeper to beat, but drags it wide.
Cross from Neghli comes into the near post where Ivanovic is there to meet it but his header flies straight into the arms of Vickers.
BBD and Brewster are off and Banel and Morris are on.
Clark cuts inside from the left and feeds Travis, but his shot is the wrong side of the post.
Millwall fans think they have won a penalty, but they haven't. It was a foul the other way and Derby had the free-kick.
Coburn just starting to show his influence.
And Bobby Clark is on for Ozoh.
Millwall have the lead as Crama turns a ball into the back post and there is Josh Coburn to head in at the back post.
Derby just not been at it to be fair in an attacking sense.
And it's not a classic by any stretch.
Derby have been unable to get anything going in the final third and at the other end, Vickers has only been tested once.
Matt Clarke has been superb so far by the way.
Now Joe Ward is down with an injury.
Ward has an ambitious strike from distance that whistles past the upright. Derby just enjoying a spell in possession at the moment.
But at least Agyemang is moving a bit better which is a relief.
Millwall are just applying some pressure now and Vickers has just had to parry Sturge's header away. From the resulting corner.
Alex Neil's side then hit the post from the following corner, but it wouldn't have counted due to a foul from Coburn.
There's a big concern here for Patrick Agyemang who just seemed to hurt himself when going for that header just a few moments ago.
Carlton Morris was stripped and ready to come on, but the American went over for further treatment on the sidelines and now looks okay to continue.
How long for though is anybody's guess.
Now it's Derby's time to apply some pressure. This time from a corner after Sanderson hooks a ball into the box and Brereton Diaz's header takes a deflection.
The corner is delivered by Ward into the box and Agyemang 's header is cleared off the line by Sturge.
And it comes from the hosts Mitchell fires a ball down the line which Ward should take care of but Sturge bundles him aside and his cross to the near post is cleared behind by Clarke.
And not a lot has happened to be fair as of yet. Both teams feeling each other out. Millwall are defending with a high line though.
I just wonder whether the pace of Agyemang could exploit that.
Derby get us started and the Millwall fans break into voice.
However, an offside flag goes up immediately.
The teams are out and lots of empty seats here, unless people are still stuck in the pub!
I just wondered whether that change has been made because of Millwall's physicality in the middle of the pitch.
They are a physically imposing team and Ozoh is more than capable of handling that.
One change for Derby which is David Ozoh in for Bobby Clark.
I am expecting a full-blooded Championship game. I was disappointed in the atmosphere at the New Den last season, but expect it to be a lot more intense tonight.
Derby have goals in them. I reckon it will be a 2-1 Derby win.
Derby head coach John Eustace does not expect any of the squad carrying knocks to be available for the trip to the New Den.
Eustace did say, however, there are no fresh concerns ahead of the game against Alex Neil's side as Derby look to keep pace with the top six.
But Callum Elder, Lars-Jorgen Salvesen, Liam Thompson, Sondre Langas, Jacob Widell Zetterstrom, Max Johnston and Danny Batth look set to be absent.
The press room just beginning to fill up here and the Moose is here from TalkSport.
Perry Groves, the former Arsenal player, is here with him too.
And welcome from the New Den, where Derby County are seeking back-to-back wins in the Championship.
It's probably one of the toughest assignments of the season given how well they've done this season and Alex Neil has proven again that he's a very good manager at this level.
I expect Derby's squad to be the same as Saturday as no injured players are expected to return as Eustace said yesterday it was very much status quo.
Anyway, team news will be with you at 18:30.