Is it OK to reference non PCLs?

OK So I'm still evaluating, using Xamarin Studio for my Android code.

I have had some success with referencing libraries from my existing code base, they seem to run fine.

I ran into some problems about specific system DLLs not being referenced but I managed to factor most of them out.

However this one I can't factor out: Error CS0012: The type 'System.Xml.XmlNode' is defined in an assembly that is not referenced. You must add a reference to assembly 'System.Xml, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'

I got the same error for System.Xml, Version=4.0.0.0 when my DLLs were targeting .NET 4.5.1

I have added a reference to System.Xml, but it seems to be Version 2.0.5 (xamarin-android)

Am I doing the wrong thing by referencing non PCL libraries?

I would really like this to work, as I don't want to have to move all of my core libraries over to PCLs and thus be tied to Xamarin for eternity for my non mobile projects.