Before the yield point, the deformation is elastic, and then it is plastic. But what happens to the elastic 'part' of strain after the load is removed? Is it retracted fully or partly, and how does this relate to the extent of plastic deformation?
When the load is removed, the elastic strain is recovered completely (by definition); the plastic strain is not recovered at all (by definition):
If the load is reapplied, the curve picks up where it left off (i.e., the system moves back up the unloading line and resumes plastic deformation). This point is explained here.
In ductile materials there is a well defined straight area at the beginning of the stress/strain curve which is purely elastic, the slope of this part of the curve is Young's modulus. If you remove the load the sample will completely relax to its original length and can give back all the work put into it.
If you keep loading the specimen beyond this elastic region and then remove the load it will not fully recover to its original length, but it will shrink with the same slope as Young's modulus. The area under this loading and unloading is the work gone into the system and never recovered.
This could be a huge amount of energy in structural frames and connections. It is employed to provide the last precious extra minutes of safety in case of a big earthquake, by absorbing the energy of earthquake.