When you remove a driver from device manager, and reboot your PC it just comes back again via plug and play. To permently remove a driver you need to delete it form c:\windows\inf. To find the driver just search for it by the same name as what it appears in device manager:
Here I'm finding which files belong to a HP NC7781 network card:
find /c "NC7781" c:\windows\inf\*.inf | find ":" | find /v ": 0"
Once found delete all files that associated with the INF. There is usually always a PNF file that is associated with the INF which needs to be deleted too.
If you want to keep your server/pc clean, you can delete the driver files as well by going to device manager, going to the device and bringing up the properties.