[KCD2] Got accused of murder after fighting bandits

Got attacked by bandits south of Troskowitz when doing the Canker quest. Killed them, went back to Troskowitz the next day and got accused of killing and robbing a bunch of wayfarers with a fine of 4k. Went away when I reloaded the save before fighting them.

So do I just not do the Canker quest? Is it bugged forever or something? Am I at risk of getting accused of murder every time I fight off bandits?