character: special to use for parenthesized terms - i.e. random effects terms with unspecified structure
1. atom (not a call or an expression): NULL
2. special, i.e. foo(...) where "foo" is in specials: return term
3. parenthesized term: if the head of the head is | (i.e.
it is of the form (xx|gg), then convert it to the default
special type; we won't allow pathological cases like
((xx|gg)) ... [can we detect them?]