Fundamentally, if the fileserver can serve the file at decent speed then the network chip isn't a bottleneck.
Looks like most of the traffic is looped around within the server box, so you are at the mercy of virtual machine scheduling and virtualisation IO overhead on the network packets.
There may be some optimisation you have missed (I don't use Hyper-V or even Windows so don't know specifics, sorry) but I would google for efficient network adaptors and make sure any client OS drivers are loaded as they reduce the overheads.
Expect the lazy way out would be a bigger CPU in the server
