When a local politician from Tokyo’s Ebisu district last week condemned media coverage of lesbian, gay, bisexual, and transgender (LGBT) rights issues and called gay people “abnormal” on Twitter, it came as a reminder of times past. It was just four years ago when Tokyo’s governor publicly called gay people “deficient.”

But a lot has changed in Japan since 2011, including the recognition of same-sex relationships in parts of Tokyo; IBM Japan and other major companies extending some benefits to employees’ same-sex partners; and Osaka’s Yodogawa ward, in 2013, and Okinawa’s capital, Naha City, in July 2015, declaring themselves “LGBT friendly” municipalities.

Around the world, progress by LGBT people has often provoked a backlash, such as the Ebisu representative’s homophobic remarks. And while the politician’s colleagues reacted swiftly to criticize him, even calling for his resignation and prompting him to apologize, such statements cut deep, and contribute to a sense that LGBT people in Japan are under siege – particularly those most vulnerable to hateful comments from authorities.