Note that this interface is going to be changed in next release. Some of internally used C routines are now exported. This interface should be considered experimental. List of exported C routines and their signatures are provided below in the usage section.
# SEXP subsetDT(SEXP x, SEXP rows, SEXP cols);
# p_dtCsubsetDT = R_GetCCallable("data.table", "CsubsetDT");
For details how to use those see Writing R Extensions manual Linking to native routines in other packages section.
https://cran.r-project.org/doc/manuals/r-release/R-exts.html