Don't take me for granite!
His boss always took him for granite.
Nobody takes him for granite.
Stone deaf...
He took his wife for granite.
Because he thought people were taking him for granite.