Don't take me for granite!
He was tired of everyone mistaking him for granite."
Because he takes it for granite.
He took his wife for granite.
Not much. One likes getting stones, the other likes getting stoned.
Because he thought people were taking him for granite.