Remove ListView item by clicking button on the item

I'm having a bit of a struggle trying to remove the correct item from my ListView. I have a ListView where the items have their own buttons, and when you click on a button on a specific item, then it should remove this specific item. I can sometimes make it work when only removing one item, but when I try to remove the next item, some funky stuff can happen (e.g. removing more than one item or removing the wrong item). I think I know the source of this trouble, but I don't know how else to do it.
I set up the item layout in the getView method in my BaseAdapter:

    public override View GetView(int position, View convertView, ViewGroup parent)
    {
        View view = convertView;
        IInvitation invite = this [position];

        if (view == null) // no view to re-use, create new
            view = context.LayoutInflater.Inflate(Resource.Layout.InvitationLayoutItem, null);

        view.FindViewById<TextView>(Resource.Id.headingInvite).Text = invite.GroupName; //InviteItem.Heading
        view.FindViewById<TextView>(Resource.Id.subheadingInvite).Text = invite.InviterName;

        ImageButton declineButton = view.FindViewById<ImageButton>(Resource.Id.imageButtonDecline);
        declineButton.Click += (object sender, EventArgs e) => 
            {
                invites.Remove (invite);
                Console.WriteLine ("You declined the invite for " + invite.GroupName + " from " + invite.InviterName);
                this.NotifyDataSetChanged ();
            };

        return view;
    }

I believe the problem starts at the declineButton.Click line. Also the invites.Remove method is probably not the right way to remove the correct item, but I've also tried removing the correct item based on the position, but that creates problems when my ListView is very long.

So how do I create an EventHandler correctly for buttons in a ListView item, so I can remove the correct item from my List and view? And is it going to work on a ListView with many items?