Learn R Programming
⚠️There's a newer version (4.1.0) of this package.Take me there.

quanteda v0.99 Important Changes

Many important changes to the package have been underway, including API changes, as we approach a stable API, "1.0" release in October 2017. Version 0.99 represents the last version that will contain many of the deprecated object types and methods that date several releases.

v0.99 also implements many enhancements and performance improvements over previous releases. See NEWS.md for details, and Quanteda Structure and Design for a description of the package's underlying logic and design philosophy.

About the package

An R package for managing and analyzing text, created by Kenneth Benoit in collaboration with a team of core contributors: Kohei Watanabe, Paul Nulty, Adam Obeng, Haiyan Wang, Ben Lauderdale, and Will Lowe. Supported by the European Research Council grant ERC-2011-StG 283794-QUANTESS.

For more details, see the package website.

How to cite the package:

To cite package 'quanteda' in publications please use the
following:

  Benoit, Kenneth et. al. ().  "quanteda: Quantitative Analysis of
  Textual Data".  R package version: 0.99.12.  http://quanteda.io.

A BibTeX entry for LaTeX users is

  @Manual{,
    title = {quanteda: Quantitative Analysis of Textual Data},
    author = {Kenneth Benoit and Kohei Watanabe and Paul Nulty and Adam Obeng and Haiyan Wang and Benjamin Lauderdale and Will Lowe},
    note = {R package version 0.99.12},
    url = {http://quanteda.io},
  }

Leave feedback

If you like quanteda, please consider leaving feedback or a testimonial here.

Features

Powerful text analytics

Generalized, flexible corpus management. quanteda provides a comprehensive workflow and ecosystem for the management, processing, and analysis of texts. Documents and associated document- and collection-level metadata are easily loaded and stored as a corpus object, although most of quanteda's operations work on simple character objects as well. A corpus is designed to efficiently store all of the texts in a collection, as well as meta-data for documents and for the collection as a whole. This makes it easy to perform natural language processing on the texts in a corpus simply and quickly, such as tokenizing, stemming, or forming ngrams. quanteda's functions for tokenizing texts and forming multiple tokenized documents into a document-feature matrix are both extremely fast and extremely simple to use. quanteda can segment texts easily by words, paragraphs, sentences, or even user-supplied delimiters and tags.

Works nicely with UTF-8. Built on the text processing functions in the stringi package, which is in turn built on C++ implementation of the ICU libraries for Unicode text handling, quanteda pays special attention to fast and correct implementation of Unicode and the handling of text in any character set, following conversion internally to UTF-8.

Built for efficiency and speed. All of the functions in quanteda are built for maximum performance and scale while still being as R-based as possible. The package makes use of three efficient architectural elements: the stringi package for text processing, the Matrix package for sparse matrix objects, and the data.table package for indexing large documents efficiently. If you can fit it into memory, quanteda will handle it quickly. (And eventually, we will make it possible to process objects even larger than available memory.)

Super-fast conversion of texts into a document-feature matrix. quanteda is principally designed to allow users a fast and convenient method to go from a corpus of texts to a selected matrix of documents by features, after defining and selecting the documents and features. The package makes it easy to redefine documents, for instance by splitting them into sentences or paragraphs, or by tags, as well as to group them into larger documents by document variables, or to subset them based on logical conditions or combinations of document variables. A special variation of the "dfm", a feature co-occurrence matrix, is also implemented, for direct use with embedding and representational models such as text2vec.

Extensive feature selection capabilities. The package also implements common NLP feature selection functions, such as removing stopwords and stemming in numerous languages, selecting words found in dictionaries, treating words as equivalent based on a user-defined "thesaurus", and trimming and weighting features based on document frequency, feature frequency, and related measures such as tf-idf.

Qualitative exploratory tools. Easily search and save keywords in context, for instance, or identify keywords. Like all of quanteda's pattern matching functions, users have the option of simple "glob" expressions, regular expressions, or fixed pattern matches.

Dictionary-based analysis. quanteda allows fast and flexible implementation of dictionary methods, including the import and conversion of foreign dictionary formats such as those from Provalis's WordStat, the Linguistic Inquiry and Word Count (LIWC), Lexicoder, Yoshioder, and YAML.

Text analytic methods. Once constructed, a dfm can be easily analyzed using either quanteda's built-in tools for scaling document positions (for the "wordfish" and "Wordscores" models, direct use with the ca package for correspondence analysis), predictive models using Naive Bayes multinomial and Bernoulli classifiers, computing distance or similarity matrixes of features or documents, or computing readability or lexical diversity indexes.

In addition, quanteda a document-feature matrix is easily used with or converted for a number of other text analytic tools, such as:

  • topic models (including converters for direct use with the topicmodels, LDA, and stm packages);

  • machine learning through a variety of other packages that take matrix or matrix-like inputs.

Planned features. Coming soon to quanteda are:

  • Bootstrapping methods for texts that makes it easy to resample texts from pre-defined units, to facilitate computation of confidence intervals on textual statistics using techniques of non-parametric bootstrapping, but applied to the original texts as data.

  • Additional predictive and analytic methods by expanding the textstat_ and textmodel_ functions. Current textmodel types include correspondence analysis, "Wordscores", "Wordfish", and Naive Bayes; current textstat statistics are readability, lexical diversity, similarity, and distance.

  • Expanded settings for all objects, that will propogate through downstream objects.

  • Object histories, that will propogate through downstream objects, to enhance analytic reproducibility and transparency.

How to Install

  1. From CRAN: Use your GUI's R package installer, or execute:

    install.packages("quanteda")

  2. From GitHub, using:

    # devtools packaged required to install quanteda from Github 
devtools::install_github("kbenoit/quanteda")

    Because this compiles some C++ source code, you will need a compiler installed. If you are using a Windows platform, this means you will need also to install the Rtools software available from CRAN. If you are using OS X, you will need to to install XCode, available for free from the App Store, or if you prefer a lighter footprint set of tools, just the Xcode command line tools, using the command xcode-select --install from the Terminal.

  3. Additional recommended packages:

    The following packages work well with or extend quanteda and we recommend that you also install them:

    • readtext: An easy way to read text data into R, from almost any input format.

    • spacyr: NLP using the spaCy library, including part-of-speech tagging, entity recognition, and dependency parsing.

    • quantedaData: Additional textual data for use with quanteda.

      devtools::install_github("kbenoit/quantedaData")

    • LIWCalike: An R implementation of the Linguistic Inquiry and Word Count approach to text analysis.

      devtools::install_github("kbenoit/LIWCalike")

Getting Started

See the package website, which includes the Getting Started Vignette.

Demonstration

library(quanteda)
## quanteda version 0.99.12
## Using 7 of 8 threads for parallel computing
## 
## Attaching package: 'quanteda'
## The following object is masked from 'package:utils':
## 
##     View

# create a corpus from the immigration texts from UK party platforms
uk2010immigCorpus <- 
    corpus(data_char_ukimmig2010,
           docvars = data.frame(party = names(data_char_ukimmig2010)),
           metacorpus = list(notes = "Immigration-related sections of 2010 UK party manifestos"))
uk2010immigCorpus
## Corpus consisting of 9 documents and 1 docvar.
summary(uk2010immigCorpus)
## Corpus consisting of 9 documents:
## 
##          Text Types Tokens Sentences        party
##           BNP  1125   3280        88          BNP
##     Coalition   142    260         4    Coalition
##  Conservative   251    499        15 Conservative
##        Greens   322    679        21       Greens
##        Labour   298    683        29       Labour
##        LibDem   251    483        14       LibDem
##            PC    77    114         5           PC
##           SNP    88    134         4          SNP
##          UKIP   346    723        27         UKIP
## 
## Source:  /Users/kbenoit/Dropbox (Personal)/GitHub/quanteda/* on x86_64 by kbenoit
## Created: Fri Oct  6 09:35:32 2017
## Notes:   Immigration-related sections of 2010 UK party manifestos

# key words in context for "deport", 3 words of context
kwic(uk2010immigCorpus, "deport", 3)
##                                                                     
##   [BNP, 157]        The BNP will | deport | all foreigners convicted
##  [BNP, 1946]                . 2. | Deport | all illegal immigrants  
##  [BNP, 1952] immigrants We shall | deport | all illegal immigrants  
##  [BNP, 2585]  Criminals We shall | deport | all criminal entrants

# create a dfm, removing stopwords
mydfm <- dfm(uk2010immigCorpus, remove = stopwords("english"), remove_punct = TRUE)
mydfm
## Document-feature matrix of: 9 documents, 1,547 features (83.8% sparse).

topfeatures(mydfm, 20)  # 20 top words
## immigration     british      people      asylum     britain          uk 
##          66          37          35          29          28          27 
##      system  population     country         new  immigrants      ensure 
##          27          21          20          19          17          17 
##       shall citizenship      social    national         bnp     illegal 
##          17          16          14          14          13          13 
##        work     percent 
##          13          12

# plot a word cloud
set.seed(100)
textplot_wordcloud(mydfm, min.freq = 6, random.order = FALSE,
                   rot.per = .25, 
                   colors = RColorBrewer::brewer.pal(8,"Dark2"))

Contributing

Contributions in the form of feedback, comments, code, and bug reports are most welcome. How to contribute:

Copy Link

Version

Install

install.packages('quanteda')

Monthly Downloads

23,966

Version

0.99.12

License

GPL-3

Maintainer

Kenneth Benoit

Last Published

October 6th, 2017

Functions in quanteda (0.99.12)

as.corpus

coerce a compressed corpus to a standard corpus
as.corpus.corpuszip

coerce a compressed corpus to a standard corpus
as.list.dist

coerce a dist object into a list
as.list.dist_selection

coerce a dist_selection object into a list
as.matrix.dfm

coerce a dfm to a matrix or data.frame
as.matrix.dist_selection

coerce a dist_selection object to a matrix
View

View methods for quanteda
applyDictionary

apply a dictionary or thesaurus to an object
as.dictionary

coercion and checking functions for dictionary objects
as.dist.dist

coerce a dist into a dist
bootstrap_dfm

bootstrap a dfm
cbind.dfm

Combine dfm objects by Rows or Columns
corpus_sample

randomly sample documents from a corpus
corpus_segment

segment texts on a pattern match
corpus_trimsentences

remove sentences based on their token lengths or a pattern match
create

utility function to create a object with new set of attributes
as.matrix.simil

Coerce a simil object into a matrix
as.tokens

coercion, checking, and combining functions for tokens objects
convert

convert a dfm to a non-quanteda format
corpus-class

base method extensions for corpus objects
changeunits

deprecated name for corpus_reshape
char_tolower

convert the case of character objects
corpus_subset

extract a subset of a corpus
data_char_sampletext

a paragraph of text for testing various text-based functions
data_char_ukimmig2010

immigration-related sections of 2010 UK party manifestos
dfm_group

combine documents in a dfm by a grouping variable
dfm_lookup

apply a dictionary to a dfm
docnames

get or set document names
docvars

get or set for document-level variables
featnames

get the feature labels from a dfm
features

deprecated function for featnames
is_regex

internal function for select_types() to check if a string is a regular expression
joinTokens

join tokens function
ngrams

deprecated function name for forming ngrams and skipgrams
nscrabble

count the Scrabble letter values of text
coef.textmodel

extract text model coefficients
collocations

deprecated function names for textstat_collocations
corpus

construct a corpus object
corpus_reshape

recast the document units of a corpus
corpus_trim

remove sentences based on their token lengths or a pattern match
data_corpus_inaugural

US presidential inaugural address texts
data_corpus_irishbudget2010

Irish budget speeches from 2010
deprecated-textstat

deprecated textstat names
dfm-class

Virtual class "dfm" for a document-feature matrix
dictionary

create a dictionary
docfreq

compute the (weighted) document frequency of a feature
dfm-internal

internal functions for dfm objects
dfm

create a document-feature matrix
dfm_weight

weight the feature frequencies in a dfm
data_dfm_lbgexample

dfm from data in Table 1 of Laver, Benoit, and Garry (2003)
data_dictionary_LSD2015

Lexicoder Sentiment Dictionary (2015)
dfm2lsa

convert a dfm to an lsa "textmatrix"
dfm_compress

recombine a dfm or fcm by combining identical dimension elements
dfm_sort

sort a dfm by frequency of one or more margins
escape_regex

internal function for select_types() to escape regular expressions
fcm-class

Virtual class "fcm" for a feature co-occurrence matrix
keyness

compute keyness (internal functions)
kwic

locate keywords-in-context
dictionary2-class

print a dictionary object
head.dfm

return the first or last part of a dfm
is.dfm

coercion and checking functions for dfm objects
metacorpus

get or set corpus metadata
metadoc

get or set document-level meta-data
ntoken

count the number of tokens or types
pattern

pattern for feature, token and keyword matching
quanteda_options

get or set package options for quanteda
read_dict_liwc

Import a LIWC-formatted dictionary
sample

randomly sample documents or features
dfm_subset

extract a subset of a dfm
groups

grouping variable(s) for various functions
head.corpus

return the first or last part of a corpus
list2dictionary

internal function to convert a list to a dictionary
removeFeatures

remove features from an object
remove_attributes

utility function to remove all attributes
remove_empty_keys

utility function to remove empty keys
as.yaml

convert quanteda dictionary objects to the YAML format
attributes<-

function extending base::attributes()
compress

compress a dfm by combining similarly named dimensions
replace_dictionary_values

internal function to replace dictionary values
syllables

deprecated name for nsyllable
textfile

old function to read texts from files
textmodel_NB

Naive Bayes classifier for texts
pattern2id

convert various input as pattern to a vector used in tokens_select, tokens_compound and kwic.
phrase

declare a compound character to be a sequence of separate pattern matches
reexports

Objects exported from other packages
convert-wrappers

convenience wrappers for dfm convert
scrabble

deprecated name for nscrabble
textmodel-internal

internal functions for textmodel objects
textmodel

fit a text model
textplot_keyness

plot word keyness
textplot_scale1d

plot a fitted scaling model
regex2fixed

convert regex and glob patterns to type IDs or fixed patterns
similarity

compute similarities between documents and/or features
slots<-

function to assign multiple slots to a S4 object
sparsity

compute the sparsity of a document-feature matrix
stopwords

access built-in stopwords
textplot_wordcloud

plot features as a wordcloud
textplot_xray

plot the dispersion of key word(s)
merge_dictionary_values

internal function to mege values of duplicated keys
nsentence

count the number of sentences
nsyllable

count syllables in a text
print.phrases

print a phrase object
textmodel_ca

correspondence analysis of a document-feature matrix
texts

get or assign corpus texts
textstat_collocations

identify and score multi-word expressions
quanteda-package

An R package for the quantitative analysis of textual data
segment

segment: deprecated function
selectFeatures

select features from an object
sort.dfm

sort a dfm by one or more margins
spacyr-methods

extensions of methods defined in the quanteda package
textmodel_fitted-class

the fitted textmodel classes
textmodel_wordfish

wordfish text model
tfidf

compute tf-idf weights from a dfm
toLower

Convert texts to lower (or upper) case
tokens_segment

segment tokens object by patterns
tokens_select

select or remove tokens from a tokens object
valuetype

pattern matching using valuetype
weight

weight or smooth a dfm
tokens_hash

Function to hash list-of-character tokens
tokens_lookup

apply a dictionary to a tokens object
tokens_tolower

convert the case of tokens
tokens_wordstem

stem the terms in an object
data-deprecated

datasets with deprecated or defunct names
data-internal

internal data sets
dfm_sample

randomly sample documents or features from a dfm
dfm_select

select features from a dfm or fcm
dfm_tolower

convert the case of the features of a dfm and combine
dfm_trim

trim a dfm using frequency threshold-based feature selection
textstat_lexdiv

calculate lexical diversity
textstat_readability

calculate readability
tokens_ngrams

create ngrams and skipgrams from tokens
tokens_recompile

recompile a hashed tokens object
fcm

create a feature co-occurrence matrix
fcm_sort

sort an fcm in alphabetical order of the features
ndoc

count the number of documents or features
nest_dicitonary

utility function to generate a nested list
textstat_frequency

tabulate feature frequencies
textstat_keyness

calculate keyness statistics
tokenize

tokenize a set of texts
tokens

tokenize a set of texts
wordstem

stem words
plot-deprecated

deprecated plotting functions
predict.textmodel_NB_fitted

prediction method for Naive Bayes classifier objects
print.dfm

print a dfm object
print.dist_selection

print a dist_selection object
selectFeaturesOLD

old version of selectFeatures.tokenizedTexts
settings

Get or set the corpus settings
subset.corpus

deprecated name for corpus_subset
summary.corpus

summarize a corpus
textmodel_wordscores

Wordscores text model
textmodel_wordshoal

wordshoal text model
textstat_dist

Similarity and distance computation between documents or features
tf

compute (weighted) term frequency from a dfm
tokens_compound

convert token sequences into compound tokens
tokens_group

recombine documents tokens by groups
topfeatures

identify the most frequent features in a dfm
trim

deprecated name for dfm_trim