A photoelastic experiment was conducted to analyze the evolution characteristics of force chain network and granule displacement in pentagons under a complete loading and unloading cycle. The results demonstrate that both contact force within the strong force chain and granule displacement show a visible “hysteresis” phenomenon during unloading; the area where granule displacement response is positive during unloading is larger than the area during loading. The displacement recovery ratio ηd((1<ηd<1) was defined to describe the movement locus of granules. Then the relationship between ηd and five types of movement locus were built. ηd increases along the depth. Evolution characteristics of both force chain and granule displacement show a critical level of superload. The residual contact force in granular material is at the state of stable equilibrium when superload is higher than the critical level while reaches the limit equilibrium state when unloading to the critical level. Further unloading will result in instant release of residual contact force as well as the redistribution of granules.