Problem

You need to read from a URL, e.g., to connect to a RESTful web service or to download a web page or other resource over http/https.

Solution

Use the standard Java 11 HttpClient or the URLConnection class.

This technique applies anytime you need to read from a URL, not just a RESTful web service.

Discussion

Prior to Java 11, you had to either use the URLConnection class or download and use the older Apache HTTP Client Library . With Java 11, there is a fairly easy-to-use and flexible API in standard Java. This also supports HTTP/2.0; which the Apache HttpClient and the URLConnection do not, as of this writing.

As our simple example, we’ll use Google’s “suggestion” service, that is, what you see when you type the first few characters of a search into the Google web search engine.

This Google service supports various output formats. The base URL is just the following: append the word you want suggestions on.

https://suggestqueries.google.com/complete/search?client=firefox&q=

The client=firefox tells it we want a simple JSON format; with client=chrome it contains more fields.

To use the Java HTTPClient API, you need a HttpClient object which you get using the Builder pattern, then create a Request object.

        // This object would be kept for the life of an application
        HttpClient client = HttpClient.newBuilder()
            .followRedirects(Redirect.NORMAL)
            .version(Version.HTTP_1_1)
            .build();

        // Build the HttpRequest object to "GET" the urlString
        HttpRequest req =
            HttpRequest.newBuilder(URI.create(urlString +
                URLEncoder.encode(keyword)))
            .header("User-Agent", "Dept of Silly Walks")
            .GET()
            .build();

The HttpRequest object can be sent using the client to get a HttpResponse object, from which you can get the status and/or the body. Sending can be done either synchronously (if you need the results right away) or asynchronously (if you can usefully do something else in the meantime).

        // Send the request - synchronously
        HttpResponse<String> resp =
            client.send(req, BodyHandlers.ofString());

        // Collect the results
        if (resp.statusCode() == 200) {
            String response = resp.body();
            System.out.println(response);
        } else {
            System.out.printf("ERROR: Status %d on request %s
",
                resp.statusCode(), urlString);
        }

        // Send the request - asynchronously
        client.sendAsync(req, BodyHandlers.ofString())
            .thenApply(HttpResponse::body)
            .thenAccept(System.out::println)
            .join();

Here is the output; the line has been broken at commas to make it fit on the page:

$ java HttpClientDemo.java
["darwin",["darwin thompson","darwin","darwin awards","darwinism",
 "darwin australia","darwin thompson fantasy","darwin barney",
 "darwin theory","darwinai","darwin dormitorio"]]

Should you not wish to use the HttpClient library, you could use the legacy code in java.net, since all we usually need here is the ability to open and read from a URL. Here is the code using a URLConnection:

public class RestClientURLDemo {
    public static void main(String[] args) throws Exception {
        URLConnection conn = new URL(
            HttpClientDemo.urlString + HttpClientDemo.keyword)
            .openConnection();
        try (BufferedReader is =
            new BufferedReader(new InputStreamReader(conn.getInputStream()))) {

            String line;
            while ((line = is.readLine()) != null) {
                System.out.println(line);
            }
        }
    }
}

The output should be identical to what the HttpClient version produced.

See Also

Don’t confuse this HttpClient with the older Apache HttpClient Library, still available at https://hc.apache.org/httpcomponents-client-ga/index.html.

You can find more information on REST services (including implementing the server-side components for them) in Bill Burke’s RESTful Java with JAX-RS 2.0, 2nd Edition (O’Reilly).

Problem

You need to contact a server using TCP/IP.

Solution

Just create a java.net.Socket, passing the hostname and port number into the constructor.

Discussion

There isn’t much to this in Java. When creating a socket, you pass in the hostname and the port number. The java.net.Socket constructor does the gethostbyname() and the socket() system call, sets up the server’s sockaddr_in structure, and executes the connect() call. All you have to do is catch the errors, which are subclassed from the familiar IOException. Example 12-2 sets up a Java network client, but doesn’t actually do any I/O yet. It uses try-with-resources to ensure that the socket is closed automatically when we are done with it.

import java.net.Socket;

/* Client with NO error handling */
public class ConnectSimple {

    public static void main(String[] argv) throws Exception {

        try (Socket sock = new Socket("localhost", 8080)) {

            /* If we get here, we can read and write on the socket "sock" */
            System.out.println(" *** Connected OK ***");

            /* Do some I/O here... */

        }
    }
}

This version does no real error reporting, but a version called ConnectFriendly does; we’ll see this version in Recipe 12.4.

See Also

Java supports other ways of using network applications. You can also open a URL and read from it (see Recipe 12.8). You can write code so that it will run from a URL, when opened in a web browser, or from an application.

Problem

You want to look up a host’s address name or number or get the address at the other end of a network connection.

Solution

Get an InetAddress object.

Discussion

The InetAddress object represents the Internet address of a given computer or host. It has no public constructors; you obtain an InetAddress by calling the static getByName() method, passing in either a hostname like darwinsys.com or a network address as a string, like 1.23.45.67. All the “lookup” methods in this class can throw the checked UnknownHostException (a subclass of java.io.IOException), which must be caught or declared on the calling method’s header. None of these methods actually contact the remote host, so they do not throw the other exceptions related to network connections.

The method getHostAddress() gives you the numeric IP address (as a string) corresponding to the InetAddress. The inverse is getHostName(), which reports the name of the InetAddress. This can be used to print the address of a host given its name, or vice versa:

public class InetAddrDemo {
    public static void main(String[] args) throws IOException {
        String hostName = "darwinsys.com";
        String ipNumber = "8.8.8.8"; // currently a well-known Google DNS server

        // Show getting the InetAddress (looking up a host) by host name
        System.out.println(hostName + "'s address is " +
            InetAddress.getByName(hostName).getHostAddress());

        // Look up a host by address
        System.out.println(ipNumber + "'s name is " +
            InetAddress.getByName(ipNumber).getHostName());

        // Look up my localhost addresss
        final InetAddress localHost = InetAddress.getLocalHost();
        System.out.println("My localhost address is " + localHost);

        // Show getting the InetAddress from an open Socket
        String someServerName = "google.com";
        // assuming there's a web server on the named server:
        try (Socket theSocket = new Socket(someServerName, 80)) {
            InetAddress remote = theSocket.getInetAddress();
            System.out.printf("The InetAddress for %s is %s%n",
                someServerName, remote);
        }
    }
}

You can also get an InetAddress from a Socket by calling its getInetAddress() method. You can construct a Socket using an InetAddress instead of a hostname string. So, to connect to port number myPortNumber on the same host as an existing socket, you’d use:

InetAddress remote = theSocket.getInetAddress( );
Socket anotherSocket = new Socket(remote, myPortNumber);

Finally, to look up all the addresses associated with a host—a server may be on more than one network—use the static method getAllByName(host), which returns an array of InetAddress objects, one for each IP address associated with the given name.

A static method getLocalHost() returns an InetAddress equivalent to “localhost” or 127.0.0.1. This can be used to connect to a server program running on the same machine as the client.

If you are using IPv6, you can use Inet6Address instead.

See Also

See NetworkInterface in Recipe 13.2, which lets you find out more about the networking of the machine you are running on.

There is not yet a way to look up services—i.e., to find out that the HTTP service is on port 80. Full implementations of TCP/IP have always included an additional set of resolvers; in C, the call getservbyname("http", "tcp"); would look up the given service¹ and return a servent (service entry) structure whose s_port member would contain the value 80. The numbers of established services do not change, but when services are new or installed in nonroutine ways, it is convenient to be able to change the service number for all programs on a machine or network (regardless of programming language) just by changing the services definitions. Java should provide this capability in a future release.

Problem

You want more detailed reporting than just IOException if something goes wrong.

Solution

Catch a greater variety of exception classes. SocketException has several subclasses; the most notable are ConnectException and NoRouteToHostException. The names are self-explanatory: the first means that the connection was refused by the machine at the other end (the server machine), and the second completely explains the failure. Example 12-3 is an excerpt from the Connect program, enhanced to handle these conditions.

public class ConnectFriendly {
    public static void main(String[] argv) {
        String server_name = argv.length == 1 ? argv[0] : "localhost";
        int tcp_port = 80;
        try (Socket sock = new Socket(server_name, tcp_port)) {

            /* If we get here, we can read and write on the socket. */
            System.out.println(" *** Connected to " + server_name  + " ***");

            /* Do some I/O here... */

        } catch (UnknownHostException e) {
            System.err.println(server_name + " Unknown host");
            return;
        } catch (NoRouteToHostException e) {
            System.err.println(server_name + " Unreachable" );
            return;
        } catch (ConnectException e) {
            System.err.println(server_name + " connect refused");
            return;
        } catch (java.io.IOException e) {
            System.err.println(server_name + ' ' + e.getMessage());
            return;
        }
    }
}

Problem

Having connected, you wish to transfer textual data.

Solution

Construct a BufferedReader or PrintWriter from the socket’s getInputStream() or getOutputStream().

Discussion

The Socket class has methods that allow you to get an InputStream or OutputStream to read from or write to the socket. It has no method to fetch a Reader or Writer, partly because some network services are limited to ASCII, but mainly because the Socket class was decided on before there were Reader and Writer classes. You can always create a Reader from an InputStream or a Writer from an OutputStream using the conversion classes. The paradigm for the two most common forms is:

BufferedReader is = new BufferedReader(
    new InputStreamReader(sock.getInputStream( )));
PrintWriter os = new PrintWriter(sock.getOutputStream( ), true);

Example 12-4 reads a line of text from the “daytime” service, which is offered by full-fledged TCP/IP suites (such as those included with most Unixes). You don’t have to send anything to the Daytime server; you simply connect and read one line. The server writes one line containing the date and time and then closes the connection.

Running it looks like this. I started by getting the current date and time on the local host, then ran the DaytimeText program to see the date and time on the server (machine darian is one of my Unix servers):

C:javasrc
etwork>date 
Current date is Sun 01-23-2000
Enter new date (mm-dd-yy):
C:javasrc
etwork>time
Current time is  1:13:18.70p
Enter new time:
C:javasrc
etwork>java network.DaytimeText darian
Time on darian is Sun Jan 23 13:14:34 2000

The code is in class DaytimeText, shown in Example 12-4.

public class DaytimeText {
    public static final short TIME_PORT = 13;

    public static void main(String[] argv) {
        String server_name = argv.length == 1 ? argv[0] : "localhost";

        try (Socket sock = new Socket(server_name,TIME_PORT);
            BufferedReader is = new BufferedReader(new
                InputStreamReader(sock.getInputStream()));) {
            String remoteTime = is.readLine();
            System.out.println("Time on " + server_name + " is " + remoteTime);
        } catch (IOException e) {
            System.err.println(e);
        }
    }
}

The second example, shown in Example 12-5, shows both reading and writing on the same socket. The Echo server simply echoes back whatever lines of text you send it. It’s not a very clever server, but it is a useful one. It helps in network testing and also in testing clients of this type!

The converse() method holds a short conversation with the Echo server on the named host; if no host is named, it tries to contact localhost, a universal alias² for “the machine the program is running on.”

public class EchoClientOneLine {
    /** What we send across the net */
    String mesg = "Hello across the net";

    public static void main(String[] argv) {
        if (argv.length == 0)
            new EchoClientOneLine().converse("localhost");
        else
            new EchoClientOneLine().converse(argv[0]);
    }

    /** Hold one conversation across the net */
    protected void converse(String hostName) {
        try (Socket sock = new Socket(hostName, 7);) { // echo server.
            BufferedReader is = new BufferedReader(new
                InputStreamReader(sock.getInputStream()));
            PrintWriter os = new PrintWriter(sock.getOutputStream(), true);
            // Do the CRLF ourself since println appends only a 
 on
            // platforms where that is the native line ending.
            os.print(mesg + "
"); os.flush();
            String reply = is.readLine();
            System.out.println("Sent "" + mesg  + """);
            System.out.println("Got  "" + reply + """);
        } catch (IOException e) {
            System.err.println(e);
        }
    }
}

It might be a good exercise to isolate the reading and writing code from this method into a NetWriter class, possibly subclassing PrintWriter and adding the and the flushing.

Problem

Having connected, you wish to transfer binary data, either raw binary data or serialized Java objects.

Solution

For plain binary date, construct a DataInputStream or DataOutputStream from the socket’s getInputStream() or getOutputStream(). For serialized Java object data, construct an ObjectInputStream or ObjectOutputStream.

Discussion

The simplest paradigm for reading/writing on a socket is:

DataInputStream is = new DataInputStream(sock.getInputStream());
DataOutputStream is = new DataOutputStream(sock.getOutputStream( ));

If the volume of data might be large, insert a buffered stream for efficiency. The paradigm is:

DataInputStream is = new DataInputStream(
    new BufferedInputStream(sock.getInputStream( )));
DataOutputStream is = new DataOutputStream(
    new BufferedOutputStream(sock.getOutputStream( )));

The program example in Example 12-6 uses another standard service that gives out the time as a binary integer representing the number of seconds since 1900. Because the Java Date class base is 1970, we convert the time base by subtracting the difference between 1970 and 1900. When I used this exercise in a course, most of the students wanted to add this time difference, reasoning that 1970 is later. But if you think clearly, you’ll see that there are fewer seconds between 1999 and 1970 than there are between 1999 and 1900, so subtraction gives the correct number of seconds. And because the Date constructor needs milliseconds, we multiply the number of seconds by 1,000.

The time difference is the number of years multiplied by 365, plus the number of leap days between the two dates (in the years 1904, 1908, . . . , 1968)—i.e., 19 days.

The integer that we read from the server is a C-language unsigned int. But Java doesn’t provide an unsigned integer type; normally when you need an unsigned number, you use the next-larger integer type, which would be long. But Java also doesn’t give us a method to read an unsigned integer from a data stream. The DataInputStream method readInt() reads Java-style signed integers. There are readUnsignedByte() methods and readUnsignedShort() methods, but no readUnsignedInt() method. Accordingly, we synthesize the ability to read an unsigned int (which must be stored in a long, or else you’d lose the signed bit and be back where you started from) by reading unsigned bytes and reassembling them using Java’s bit-shifting operators:

At the end of the code, we use the new date/time API (see Chapter 6) to construct and print a LocalDateTime object to show the current date and time on the local (client) machine:

$ date
Thu Dec 26 09:48:36 EST 2019
java network.RDateClient aragorn
Remote time is 3786360519
BASE_DIFF is 2208988800
Time diff == 1577371719
Time on aragorn is 2019-12-26T09:48:39
Local date/time = 2019-12-26T09:48:41.208180
$

The name aragorn is the hostname of one of my OpenBSD Unix computers. Looking at the output, you can see that the server agrees within a second or two. That confirms the date calculation code in Example 12-6. This protocol is commonly known as rdate, so the client code is called RDateClient.

public class RDateClient {
    /** The TCP port for the binary time service. */
    public static final short TIME_PORT = 37;
    /** Seconds between 1970, the time base for dates and times
     * Factors in leap years (up to 2100), hours, minutes, and seconds.
     * Subtract 1 day for 1900, add in 1/2 day for 1969/1970.
     */
    protected static final long BASE_DAYS =
        (long)((1970-1900)*365 + (1970-1900-1)/4);

    /* Seconds since 1970 */
    public static final long BASE_DIFF = (BASE_DAYS * 24 * 60 * 60);

    public static void main(String[] argv) {
        String hostName;
        if (argv.length == 0)
            hostName = "localhost";
        else
            hostName = argv[0];

        try (Socket sock = new Socket(hostName,TIME_PORT);) {
            DataInputStream is = new DataInputStream(new
                BufferedInputStream(sock.getInputStream()));
            // Read 4 bytes from the network, unsigned.
            // Do it yourself; there is no readUnsignedInt().
            // Long is 8 bytes on Java, but we are using the
            // existing time protocol, which uses 4-byte ints.
            long remoteTime = (
                ((long)(is.readUnsignedByte()) << 24) |
                ((long)(is.readUnsignedByte()) << 16) |
                ((long)(is.readUnsignedByte()) <<  8) |
                ((long)(is.readUnsignedByte()) <<  0));
            System.out.println("Remote time is " + remoteTime);
            System.out.println("BASE_DIFF is " + BASE_DIFF);
            System.out.println("Time diff == " + (remoteTime - BASE_DIFF));
            Instant time = Instant.ofEpochSecond(remoteTime - BASE_DIFF);
            LocalDateTime d = LocalDateTime.ofInstant(time, ZoneId.systemDefault());
            System.out.println("Time on " + hostName + " is " + d.toString());
            System.out.println("Local date/time = " + LocalDateTime.now());
        } catch (IOException e) {
            System.err.println(e);
        }
    }
}

Object serialization is the ability to convert in-memory objects to an external form that can be sent serially (a byte at a time). To read or write Java objects via serialization, you need only construct an ObjectInputStream or ObjectOutputStream from an InputStream or OutputStream; in this case, the socket’s getInputStream() or getOutputStream().

This program (and its server) provide a service that isn’t a standard part of the TCP/IP stack; it’s a service I made up as a demo. The server for this service is introduced in Recipe 13.3. The client code in Example 12-7 is quite similar to the DaytimeBinary program in the previous recipe, but the server sends us a LocalDateTime object already constructed. Example 12-7 shows the portion of the client code that differs from Example 12-6.

        try (Socket sock = new Socket(hostName, TIME_PORT);) {
            ObjectInputStream is = new ObjectInputStream(new
                BufferedInputStream(sock.getInputStream()));

            // Read and validate the Object
            Object o = is.readObject();
            if (o == null) {
                System.err.println("Read null from server!");
            } else if ((o instanceof LocalDateTime)) {

                // Valid, so cast to LocalDateTime, and print
                LocalDateTime d = (LocalDateTime) o;
                System.out.println("Time on " + hostName + " is " + d);
            } else {
                throw new IllegalArgumentException(
                    String.format("Wanted LocalDateTime, got %s, a %s",
                        o, o.getClass()));
            }

I ask the operating system for the date and time, and then run the program, which prints the date and time on a remote machine:

$ date
Thu Dec 26 09:29:02 EST 2019
C:javasrc
etwork>java network.DaytimeObject aragorn
Time on aragorn is 2019-12-26T09:29:05.227397
C:javasrc
etwork>

Again, the results agree within a few seconds.

Problem

You need to use a datagram connection (UDP) instead of a stream connection (TCP).

Solution

Use DatagramSocket and DatagramPacket.

Discussion

Datagram network traffic is a kindred spirit to the underlying packet-based Ethernet and IP (Internet protocol) layers. Unlike a stream-based connection such as TCP, datagram transports like UDP transmit each “packet,” or chunk of data, as a single “entity” with no necessary relation to any other.³ A common analogy is that TCP is like talking on the telephone, whereas UDP is like sending postcards or maybe fax messages.

The differences show up most in error handling. Packets can, like postcards, go astray. When was the last time the postman rang your bell to tell you that the post office had lost one of several postcards it was supposed to deliver to you? That’s not going to happen, because the post office doesn’t keep track of postcards. On the other hand, when you’re talking on the phone and there’s a noise burst—like somebody yelling in the room, or even a bad connection—you notice the failure in real time, and you can ask the person at the other end to repeat what they just said.

With a stream-based connection like a TCP socket, the network transport layer handles errors for you: it asks the other end to retransmit. With a datagram transport such as UDP, you have to handle retransmission yourself. It’s kind of like numbering the postcards you send so that you can go back and resend any that don’t arrive—a good excuse to return to your vacation spot, perhaps.

Another difference is that datagram transmission preserves message boundaries. That is, if you write 20 bytes and then write 10 bytes when using TCP, the program reading from the other end will not know if you wrote one chunk of 30 bytes, two chunks of 15, or even 30 individual characters. With a DatagramSocket, you construct a DatagramPacket object for each buffer, and its contents are sent as a single entity over the network; its contents will not be mixed together with the contents of any other buffer. The DatagramPacket object has methods like getLength(), setPort(), and so on.

So why would we even use UDP? UDP has a lot less overhead than TCP, which can be particularly valuable when sending huge amounts of data over a reliable local network or a few hops on the Internet. Over long-haul networks, TCP is probably preferred because TCP handles retransmission of lost packets for you. And obviously, if preserving record boundaries makes your life easier, that may be a reason for considering UDP. UDP is also the way to perform Multicast (broadcast to many receivers simultaneously), though Multicast is out of scope for this discussion.

Example 12-8 is a short program that connects via UDP to the Daytime date and time server used in Recipe 12.5. Because UDP has no real notion of “connection,” the client typically initiates the “conversation,” which sometimes means sending an empty packet; the UDP server uses the address information it gets from that to return its response.

public class DaytimeUDP {
    /** The UDP port number */
    public final static int DAYTIME_PORT = 13;

    /** A buffer plenty big enough for the date string */
    protected final static int PACKET_SIZE = 100;

    /** The main program that drives this network client.
     * @param argv[0] hostname, running daytime/udp server
     */
    public static void main(String[] argv) throws IOException {
        if (argv.length < 1) {
            System.err.println("usage: java DayTimeUDP host");
            System.exit(1);
        }
        String host = argv[0];
        InetAddress servAddr = InetAddress.getByName(host);
        DatagramSocket sock = new DatagramSocket();
        //sock.connect(servAddr, DAYTIME_PORT);
        byte[] buffer = new byte[PACKET_SIZE];

        // The udp packet we will send and receive
        DatagramPacket packet = new DatagramPacket(
            buffer, PACKET_SIZE, servAddr, DAYTIME_PORT);

        /* Send empty max-length (-1 for null byte) packet to server */
        packet.setLength(PACKET_SIZE-1);
        sock.send(packet);
        System.out.println("Sent request");

        // Receive a packet and print it.
        sock.receive(packet);
        System.out.println("Got packet of size " + packet.getLength());
        System.out.print("Date on " + host + " is " +
            new String(buffer, 0, packet.getLength()));

        sock.close();
    }
}

I’ll run it to my Unix box just to be sure that it works:

$
$ java network.DaytimeUDP aragorn
Sent request
Got packet of size 26
Date on aragorn is Sat Feb  8 20:22:12 2014
$

Problem

Having heard these terms, you want to know the difference between a URI, URL, and URN.

Solution

Read on. Or see the javadoc for java.net.uri.

Discussion

A URL is the traditional name for a network address consisting of a scheme (like “http”) and an address (site name) and resource or pathname. But there are three distinct terms in all:

• URI (Uniform Resource Identifier)

• URL (Uniform Resource Locator)

• URN (Uniform Resource Name)

A discussion near the end of the Java documentation for the new class explains the relationship among URI, URL, and URN. URIs form the set of all identifiers: URLs and URNs are subsets.

URIs are the most general; a URI is parsed for basic syntax without regard to the scheme, if any, that it specifies, and it need not refer to a particular server. A URL includes a hostname, scheme, and other components; the string is parsed according to rules for its scheme. When you construct a URL, an InputStream is created automatically. URNs name resources but do not explain how to locate them; typical examples of URNs that you will have seen include mailto: and news: references.

The main operations provided by the URI class are normalization (removing extraneous path segments including “..”) and relativization (this should be called “making relative,” but somebody wanted a single word to make a method name). A URI object does not have any methods for opening the URI; for that, you would normally use a string representation of the URI to construct a URL object, like so:

URL x = new URL(theURI.toString( ));

The program in Example 12-9 shows examples of normalizating, making relative, and constructing a URL from a URI.

public class URIDemo {
    public static void main(String[] args)
    throws URISyntaxException, MalformedURLException {

        URI u = new URI("https://darwinsys.com/java/../openbsd/../index.jsp");
        System.out.println("Raw: " + u);
        URI normalized = u.normalize();
        System.out.println("Normalized: " + normalized);
        final URI BASE = new URI("https://darwinsys.com");
        System.out.println("Relativized to " + BASE + ": " + BASE.relativize(u));

        // A URL is a type of URI
        URL url = new URL(normalized.toString());
        System.out.println("URL: " + url);

        // Demo of non-URL but valid URI
        URI uri = new URI("bean:WonderBean");
        System.out.println(uri);
    }
}

See Also

There are many better structured ways to write a chat client, including WebSockets, RMI, and JMS. RMI is Java’s RPC interface, and is included both in Java SE and in Java EE; it is not described in this edition of this book, but you can find the RMI chapter from previous editions on the author’s website. The other technologies are part of the Java Enterprise so, again, we refer you to Arun Gupta’s Java EE 7 Essentials (O’Reilly).

If your communication goes over the public internet, you do need to encrypt your socket connection, so check out Sun’s JSSE (Java Secure Socket Extension). If you took my earlier advice and used the standard HTTP protocol, you can encrypt the conversation just by changing the URL to https.

For a good overview of network programming from the C programmer’s point of view, see the late W. Richard Stevens’ Unix Network Programming. Despite the book’s name, it’s really about socket and TCP/IP/UDP programming and covers all parts of the (Unix) networking API and protocols such as TFTP in amazing detail.