AITA My BF could have killed my cat

A couple weeks ago my (26) boyfriend (29) found a plant outside by a dumpster and grabbed it, when we brought it home we figured out that it was a Desert Rose and that it is toxic to cats so we decided to keep it out on our balcony so our cat can’t reach it. Yesterday there were some storms in Dallas and some really bad winds so he brought it inside and left it in our living room. When I saw that my cat kept going near it and at one point looked like she was chew on something/ kinda licking her lips after going near it, I decided to put it in the closet and keep the door closed. Later that night I told my boyfriend he needed to put the plant back outside because I didn’t want to risk accidentally leaving the door open and having my cat get to the plant. He said he wants to wait for the wind to die down and I got upset because that plant is not more important than my cats well being. With that being said he said he would put it outside tomorrow and will keep the closet door closed and won’t forget because “the plant is the first thing you see when you walk into the closet” and that he wouldn’t forget to close the door. Lo and behold he forgot to close the closet door this morning. He wakes up earlier than me for work but works from home, that door could have been open for hours. We have spent all day fighting and he keeps saying it was an accident and that my cat is okay but I think he’s just genuinely inconsiderate and selfish at this point. I packed up to take my cat and myself to my mom’s and he has been screaming that it’s an accident and I’m blowing it out of proportion. Am I the asshole?