Usually, developers only work on Jsoup with default options, unaware that it provides various useful options. This recipe will acquaint you with some common-use options.
userAgent
: It is very important to always specify userAgent
when sending HTTP requests. What if the web page displays some information differently on different browsers? The result of parsing might be different.Document doc = Jsoup.connect(url).userAgent("Mozilla/5.0 (Windows NT 6.1)").get();
Especially when using Jsoup in Android, you must always specify a user agent; otherwise, it won't work properly.
Document doc = Jsoup.connect(url).ignoreContentType(true).get();
By default, Jsoup only allows working with HTML and XML content type and throws exceptions for others. So, you will need to specify this properly in order to work with other content types, such as RSS, Atom, and so on.
Document doc = Jsoup.connect(url).timeout(5000).get();
The default timeout for Jsoup is 3000 milliseconds (three seconds). Zero indicates an infinite timeout.
Document doc = Jsoup.connect(url).data("author", "Pete Houston").get();
In dynamic web, you need to specify a parameter to make a request; the data()
method works for this purpose.
Please refer to the following link for more information:
http://jsoup.org/apidocs/org/jsoup/Connection.html#data(java.lang.String, java.lang.String)
post
.Document doc = Jsoup.connect(url).data("author", "Pete Houston").post();
Document
class.This option works through the Document.OutputSettings
class.
Please refer to the following link for more information:
http://jsoup.org/apidocs/org/jsoup/nodes/Document.OutputSettings.html
This class outputs HTML text in a neat format with the following options:
For example, display the HTML output with; charset as utf-8
and the indentation amount as four spaces, enable the HTML outline
mode, and enable pretty printing:
Document.OutputSettings settings = new Document.OutputSettings(); settings.charset("utf-8").indentAmount(4).outline(true).prettyPrint(true); Document doc = …// create DOM object somewhere.doc.outputSettings(settings); System.out.println(doc.html());
After setting the output format to Document
, the content of Document
is processed into the according format; call the Document.html()
method for output result.
Jsoup provides two parser types: HTML parser and XML parser.
By default, it uses HTML parser. However, if you are going to parse XML such as RSS or Atom, you should change the parser type to XML parser or it will not work properly.
Document doc = Jsoup.connect(url).parser(Parser.xmlParser()).get();
The previously mentioned options in Jsoup are important ones that the developers should know and make use of.
However, there are several more that you can try:
DataUtil
: This provides methods to load a file, or stream and transform into a Document
object. To know more about this option, go to the following location:StringUtil
: This provides methods to handle strings; for example, to search in array, join string array, or test string. To know more about this option, go to the following location:Validate
: This provides methods to test objects, such as test empty, test null, and so on. To know more about this option, go to the following location:Entities
: This provides methods to test or get HTML entities. To know more about this option, go to the following location:Parser
: This provides methods to parse HTML into Document
. To know more about this option, go to the following location:3.17.204.7