When you use style = 'bootstrap' in datatable(), you can specify the CSS classes for Bootstrap tables. You can find the possible classes on this page. We show the appearance of a few classes here:

iris2 = head(iris, 20)
options(DT.options = list(pageLength = 5))
# default Bootstrap style in DT
datatable(iris2, style = 'bootstrap')
# with cell borders
datatable(iris2, style = 'bootstrap', class = 'table-bordered')
# with cell borders, and make the table compact
datatable(iris2, style = 'bootstrap', class = 'table-bordered table-condensed')

Note the appearance depends on which Bootstrap theme you use. We are using the Flatly theme on this website, and the tables may look different when you use a different theme.