Efficiently Fetching and Processing Web (HTTP) Data in Java: A Comprehensive Guide

How to Send HTTP Request and Capture Response in Java? – url.openStream()

There are more than 8 ways you could send request to any URL using HTTP GET or HTTP POST call in Java.

With the latest Java APIs and every release had new improved way to make URL calls.

Life of developer definitely improved a lot over last few years with frequent release of Java JDK.

In this tutorial we will go over How to send HTTP Request and Capture complete Response (Payload) in Java using url.openStream() API.

Here’s what each part of the code does:

Import Statements:
- import java.io.BufferedReader;: Import the BufferedReader class, which is used to efficiently read text from a character input stream.
- import java.io.InputStreamReader;: Import the InputStreamReader class, which converts bytes from an input stream into characters.
- import java.net.URL;: Import the URL class, which represents a Uniform Resource Locator and is used to open connections to web resources.
Class Declaration:
- public class CrunchifyFetchURLData { ... }: Define a public class named CrunchifyFetchURLData.
Main Method:
- public static void main(String[] args) { ... }: The entry point of the program. This is where the code execution starts.
Try-Catch Block:
- try { ... } catch (Exception ex) { ... }: The try block contains the main code logic, and any exceptions that occur are caught and handled in the catch block.
Fetching and Printing Web Data:
- URL url = new URL(“https://ift.tt/8rLDVNU Create a URL object representing the web page you want to fetch.
- try (BufferedReader br = new BufferedReader(new InputStreamReader(url.openStream()))) { … }: Open a connection to the specified URL and create a BufferedReader to read its content. The try-with-resources statement ensures that the resources are properly closed after use.
- String strTemp;: Declare a string variable to store each line of the web page content.
- while ((strTemp = br.readLine()) != null) { ... }: Read each line of the web page content using the BufferedReader and print it. The loop continues until there are no more lines to read (readLine() returns null).
Exception Handling:
- ex.printStackTrace();: If an exception occurs during the process, it will be caught in the catch block, and the stack trace will be printed to the console. This helps in diagnosing issues during development.

Java Code:

package crunchify.com.tutorials;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.URL;

/**
 * @author crunchify.com
 * Efficiently Fetching and Processing Web (HTTP) Data in Java: A Comprehensive Guide
 */

public class CrunchifyFetchURLData {

        public static void main(String[] args) {
                try {
                        // Create a URL object representing the web page to fetch
                        URL url = new URL("https://crunchify.com/");

                        // Open a connection to the URL and create a BufferedReader to read its contents
                        try (BufferedReader br = new BufferedReader(new InputStreamReader(url.openStream()))) {
                                String strTemp;
                                // Read and print each line of the web page content
                                while ((strTemp = br.readLine()) != null) {
                                        System.out.println(strTemp);
                                }
                        }
                } catch (Exception ex) {
                        // Handle any exceptions that might occur
                        ex.printStackTrace();
                }
        }
}

IntelliJ IDEA result:

<!DOCTYPE html>
<html lang="en-US">
<head >
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name='robots' content='index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1' />

        <title>Crunchify • Largest free Technical and Blogging resource site for Beginner. We help clients transform their great ideas into reality!</title>
        <meta name="description" content="Largest free Technical, Blogging resource site for Beginners. We help clients transform their great ideas into reality!" />
        <link rel="canonical" href="https://crunchify.com/" />
        <link rel="next" href="https://crunchify.com/page/2/" />
        <meta property="og:locale" content="en_US" />
        <meta property="og:type" content="website" />
        <meta property="og:title" content="Crunchify" />
        <meta property="og:description" content="Largest free Technical, Blogging resource site for Beginners. We help clients transform their great ideas into reality!" />
        <meta property="og:url" content="https://crunchify.com/" />
        <meta property="og:site_name" content="Crunchify" />
        <meta property="og:image" content="https://crunchify.com/wp-content/uploads/2018/12/crunchify.com-Cover.png" />
        <meta property="og:image:width" content="2560" />
        <meta property="og:image:height" content="1363" />
        <meta property="og:image:type" content="image/png" />
        <meta name="twitter:card" content="summary_large_image" />
        <meta name="twitter:site" content="@Crunchify" />
        <script type="application/ld+json" class="yoast-schema-graph">{"@context":"https://schema.org","@graph":[{"@type":"CollectionPage","@id":"https://crunchify.com/","url":"https://crunchify.com/","name":"Crunchify • Largest free Technical and Blogging resource site for Beginner. We help clients transform their great ideas into reality!","isPartOf":{"@id":"https://crunchify.com/#website"},"about":{"@id":"https://crunchify.com/#organization"},"description":"Largest free Technical, Blogging resource site for Beginners. We help clients transform their great ideas into reality!","breadcrumb":{"@id":"https://crunchify.com/#breadcrumb"},"inLanguage":"en-US"},{"@type":"BreadcrumbList","@id":"https://crunchify.com/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Crunchify"}]},{"@type":"WebSite","@id":"https://crunchify.com/#website","url":"https://crunchify.com/","name":"Crunchify","description":"Largest free Technical and Blogging resource site for Beginner. We help clients transform their great ideas into reality!","publisher":{"@id":"https://crunchify.com/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://crunchify.com/search/?q={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https://crunchify.com/#organization","name":"Crunchify, LLC","url":"https://crunchify.com/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https://crunchify.com/#/schema/logo/image/","url":"https://crunchify.com/wp-content/uploads/2023/01/crunchify.png","contentUrl":"https://crunchify.com/wp-content/uploads/2023/01/crunchify.png","width":1024,"height":1024,"caption":"Crunchify, LLC"},"image":{"@id":"https://crunchify.com/#/schema/logo/image/"},"sameAs":["https://www.facebook.com/Crunchify","https://twitter.com/Crunchify","https://www.pinterest.com/Crunchify","https://www.youtube.com/@_crunchify"]}]}</script>
        <meta name="google-site-verification" content="vJfmWV91XhXdCun07nj_e5xclx8xDhdaILEtNA5RFCk" />

<link rel='dns-prefetch' href='//ajax.googleapis.com' />
<link rel='dns-prefetch' href='//cdnjs.cloudflare.com' />
.......
.......

and so on... long payload

Fetching and processing web data in Java can be done efficiently and reliably using built-in classes and libraries.

By optimizing and expanding upon the provided code, you’ve learned how to implement best practices, handle exceptions, and enhance user experience. This knowledge forms the foundation for building more advanced web data retrieval applications tailored to your specific needs.

The post Efficiently Fetching and Processing Web (HTTP) Data in Java: A Comprehensive Guide appeared first on Crunchify.